Arguing against AI-first research — Smashing Magazine analysis

What the article covers

Vitaly Friedman addresses the growing pressure on UX teams to replace real user research with AI-driven “synthetic testing.” The article provides specific counter-arguments researchers can use when stakeholders propose substituting AI-generated customer responses for actual user studies.

Context

Published during a period of aggressive marketing by synthetic user testing tools, the article responds to a real organizational dynamic: executives who see AI research as fast, cheap, and easily repeatable, and do not understand why those qualities make it dangerous rather than advantageous for product decisions.

Key takeaway

The core argument is that AI research creates an illusion of customer experience rather than documenting actual experience. LLMs are trained to produce the most plausible output based on patterns, which means they reflect averages and expectations rather than the specific, surprising behaviors that make real user research valuable. Friedman recommends triangulating rather than validating: start with human research using real customers, then layer AI analysis to check for gaps, rather than starting with AI-generated hypotheses that create confirmation bias. The article also links to a curated collection of critical resources from researchers including Greg Nudelman, Stephanie Walter, Nikki Anderson, and Debbie Levitt.

Who should read this

Researchers who need ready-made arguments for stakeholder conversations about why synthetic testing should not replace user research, and research leaders who want to establish organizational positions on AI’s role in research practice.