Skip to content
Article Nielsen Norman Group Oct 2026

NNGroup: Synthetic users vs. human respondents — same conclusion, less variation

Rachel Banawa at Nielsen Norman Group published this study on October 16, 2026, comparing AI-generated synthetic respondents to real survey participants on the same open-ended question. The study asked both groups which professional field best describes turning user insights into product recommendations — a question relevant to research, design, and product roles. The setup makes the comparison meaningful because it involves a real content question rather than a constructed benchmark, and the human sample is large enough (604 tech and design professionals) to establish what genuine variation looks like.

The synthetic group consisted of 120 respondents generated via the Synthetic Users platform. The broad conclusion matched: both groups named UX as the dominant field. But the comparison at the level of variation is where the findings become useful. Human respondents produced more than 50 distinct labels, including unusual, specialized, and job-title-specific terms that reflected real occupational diversity. The synthetic respondents collapsed onto approximately 12 labels, dropping all of the less common terms entirely. Banawa calls this effect “mode collapse” — the AI gravitates toward the most statistically representative response and underrepresents the long tail.

The demographic skew compounds the problem. Only 20% of the synthetic respondents were designated as UX professionals, yet 84% of their responses named UX as the relevant field. Among the 604 human respondents — 67% of whom actually were UX professionals — the figure was 70%. The AI model was more UX-centric than the human UX professionals it was supposed to simulate.

Role effects persisted in synthetic data but in a muted form. Synthetic PMs attributed the work to UX at a 93% rate; human PMs did so at 44%. This means synthetic data can show that subgroup differences exist, but not how large those differences actually are in practice. The article also notes a methodological complication: the Synthetic Users platform conducts simulated interviews rather than direct surveys, which means respondents were answering a longer conversational exchange rather than a single question — a format difference that affects how results can be compared.

The practical guidance from the study is specific. Synthetic users can be useful for generating hypotheses, stress-testing whether a broad pattern holds, or exploring which directions are worth investigating with real participants. They are not reliable for measuring subgroup differences, capturing minority or specialized perspectives, or producing the kind of natural variation that identifies edge cases in a design. Teams using synthetic respondents to decide what to build or what to skip are working with data that is systematically biased toward whatever the model considers normal.