Last month I almost shipped a survey with no humans in it. I wanted feedback on an onboarding flow for a side project, and recruiting real respondents felt like a lot of ceremony for a Tuesday night. So I did the thing half the industry is quietly doing anyway: I asked a model to role-play 40 users and fill the survey in for me. The responses looked great. Suspiciously great. Thoughtful, well written, and in complete agreement with everything I already believed.
I didn’t use it, mostly out of guilt. Then two papers landed on arXiv this week that put numbers on that guilt, and the numbers are worse than I expected.
Short version for the impatient: LLM synthetic data is useful for structure and volume, and close to useless for the one thing you actually run a survey for. If you want the details, keep reading.
Five models, zero surprises
The first paper has my favorite title of the year: “Stochastic Parrots or Singing in Harmony?”. The authors ran a real survey of 420 Silicon Valley coders and developers, then had five leading models generate synthetic respondents for the same instrument: ChatGPT Thinking 5 Pro, Claude Sonnet 4.5 with CoWork, Gemini Advanced 2.5 Pro, Incredible 1.0, and DeepSeek 3.2. Then they compared the fake answers to the real ones.
The synthetic results were plausible. Distributions looked sane, and the models actually replicated the conventional findings better than the authors assumed they would going in. If you skimmed the aggregate charts, you’d sign off on the fake data.
Then the part that got me. None of the five models captured the counterintuitive findings, meaning the answers that made the human survey worth running in the first place. And when the models deviated from the real data, they deviated together. Their errors clustered on the same side. The actual humans ended up as the statistical outlier in their own study.
Sit with that for a second. If models failed randomly, you could average five of them and wash the noise out. But they fail identically. Five models isn’t five independent witnesses. It’s one witness wearing five hats.
Peer review fails the same way
The second paper, “How Closely Do LLM Reviews Align with Human Peer Review?”, asks the same question in a different room. The authors took 300 ICLR 2026 submissions, split evenly between oral acceptances, posters, and rejections, stripped the decision information, and had GPT-5.4, Gemini 3.1 Pro Preview, and Claude Opus 4.6 review every paper under identical instructions.
All three models separated accepted papers from rejected ones. Good. None of them could reproduce the oral versus poster distinction, which is exactly the judgment call that requires taste. Each provider also had its own scoring personality; Gemini graded systematically higher than the other two, like that one reviewer who thinks a 7 is an insult.
If you’re using an LLM as a judge in your eval pipeline, this result should make you slightly nervous. Accept-versus-reject is the easy half of the job, and the models handle it. The half you probably built the eval for, ranking good outputs against slightly better ones, is the half where all three providers went blind at the same time. I’ve watched a judge model hand out 8s to two completions where one was clearly worth shipping and the other clearly wasn’t. I used to treat that as a prompting problem on my end. This paper suggests it’s structural, and that swapping judge providers won’t fix it either, since they share the same blind spot.
Different domain, same failure shape. The coarse gradient survives. The expert distinction evaporates. A model can tell you a paper is probably fine. It can’t tell you a paper is special, because “special” is by definition a deviation from what usually gets written, and deviation from the usual is the exact thing these systems are built to sand off.
Why they all miss the same things
There’s nothing mystical going on. These models are trained on overlapping slices of the same internet, and generation pulls toward the statistically likely middle of that distribution. Ask one to simulate a developer answering your survey and you get the internet’s median opinion about what developers think, delivered fluently and with total confidence.
Real survey data earns its money in the opposite place. You pay for 420 humans because eight of them will say something that contradicts the median, and that contradiction is the finding. Everything else in the report is packaging.
This also explains why my fake onboarding survey agreed with me so completely. My priors came from the same internet the model was trained on. We’d read the same blog posts. Of course we agreed. I had built the world’s most expensive mirror, and it cost me twelve cents.
I ran into a cousin of this failure once before, when adding more few-shot examples made my outputs worse: feed a model more of one distribution and it commits harder to the middle of it. Regression to the mean is the operating principle of these systems, and no prompt gets you around it.
Where I still use LLM synthetic data
Here’s the part where I don’t tell you to swear it off, because I use synthetic data constantly. The trick is that none of my uses involve claims about human beings.
Test fixtures are the obvious one. When I’m building a support triage pipeline, I need hundreds of realistic tickets and I don’t have a support inbox to raid. So:
personas = ["first-time user", "agency admin", "developer on a deadline"]
issues = ["billing confusion", "a webhook that stopped firing", "an SSO login loop"]
prompts = [
f"Write a short, mildly annoyed support ticket from a {p} about {i}."
for p in personas
for i in issues
]
# pipe these through a cheap model, feed the outputs to the triage queue
That’s a legitimate use, because the tickets only need to be plausible. Nobody is going to make a product decision based on what the fake customer wants.
Load test corpora fall in the same bucket. So does seeding an eval set, as long as a human reviews and corrects the labels before anything depends on them. Synthetic examples are cheap scaffolding for evals in the same way they are for fine-tuning datasets, and the safety condition is identical: a person owns the final labels.
What’s not fine is asking the fake customers whether they’d pay for the feature. The moment synthetic data is answering a question about what people think or would do, both papers point at the same outcome: you get a fluent restatement of conventional wisdom, and you get it from every model on the market, so cross-checking providers buys you nothing.
The rule I’ve settled on: synthetic data can populate a system. It can’t inform a decision about humans.
A sanity check you can run this week
If you’ve ever run a real survey or a round of real user interviews, you’re sitting on a free calibration exercise.
Take the old instrument. Prompt a model to simulate your respondent pool, matching whatever demographics you recorded. Generate the same number of synthetic responses. Then diff the two datasets, and look specifically at the most surprising thing your real research found. Did the model reproduce it?
I did this with a survey from an old client project, wearing my consultant hat (that’s most of what I do, between blog posts). The model nailed every boring aggregate and whiffed on the single insight that had changed the roadmap. Which, after reading these two papers, is exactly what I should have predicted.
The whole exercise took me half a day, needs no budget, and leaves you with something better than a rule of thumb: a concrete, product-specific demo of what synthetic respondents miss. Run it before someone on your team suggests replacing the next user study with an API call. Someone will.
