Okay, this is going to sound dumb, but I spent most of a Tuesday last month arguing with a client about whether a model was “good enough” for their support triage, and we were both right. He was pasting tickets into the ChatGPT app and getting mediocre results. I was running the same model through the API in a harness with a tidy system prompt and getting clean labels. Same model name on the invoice. Different product.
I filed that under “prompting is hard” and moved on. Then this week a paper from Stanford landed that says, with numbers, that the gap I was hand-waving about is real, measurable, and in some cases as large as a whole model generation. So I’m writing this partly to say I told you so, and partly to admit I’d been quoting API benchmark scores to clients as if they described the chat window they’d actually be using.
What the paper actually measured
The paper is API Benchmark Scores Do Not Reliably Transfer to Chatbot Interfaces by Jennifer Wang, Joachim Baumann, Daniel E. Ho and Sanmi Koyejo. The setup is simple enough that I’m annoyed nobody did it earlier. They took ChatGPT, Claude and Gemini, seven systems in total, and ran nine benchmarks through two doors: the developer API, and the consumer chat interface that most humans actually use. The benchmarks cover general capability, social bias and sycophancy, so it isn’t only “can it do maths”.
The headline number: API evaluations scored 3.4 percentage points higher in accuracy on average than the interface versions of the same models. Consistency was worse too. Test-retest agreement, meaning “ask the same thing twice, get the same answer”, was 2.1 points higher through the API.
The line that made me put my coffee down is about ChatGPT specifically. The difference between API access and interface access was bigger than the difference between GPT 5.3 and GPT 5.4 measured through the API alone. Switching from the API to the app cost more than a version downgrade would have. If you’ve ever wondered why the model that topped a leaderboard feels underwhelming when you use it in the app, this is a candidate explanation with data behind it.
Why the same model behaves differently
I don’t have the vendors’ internals, and neither do the authors, so this section is inference. But the mechanics are not mysterious. A chat interface is the model plus a system prompt you didn’t write, plus tool routing you can’t see, plus safety layers, plus a sampling temperature somebody in product picked, plus whatever “memory” feature is quietly injecting context about you. Every one of those is a variable that a benchmark harness holds constant and the app does not.
The authors tested whether you could close the gap from the API side by fiddling with the knobs you do control: system prompts, sampling parameters and reasoning settings. Their finding, in the full paper, is that these shift behaviour in some cases but “do not reliably eliminate the gap”. That matches my experience. I’ve tried to reproduce app behaviour through the API for debugging and given up more than once, because the app’s system prompt is a moving target that changes without a changelog.
There’s a name for this in the paper: a context-validity gap. Measurements taken in one context (API) don’t generalise to another (deployed interface). It’s the same problem as testing a car engine on a dyno and assuming that tells you how it drives in traffic. Useful, but not the same thing.
The uncomfortable part for people who quote benchmarks
I’ve done this. A client asks “is model X good enough”, I go look at whatever leaderboard is current, HELM or one of the others, and I read a number back to them. That number was almost certainly produced through the API, with a fixed prompt, at a fixed temperature, in a harness that retries on malformed output. The client is going to use the app on their phone.
I wrote about a related version of this problem a while back in my post on ARC-AGI-3 and the harness gap, where one model got two different scores depending on the scaffolding around it. That was about agent harnesses. This paper is the consumer-facing version of the same lesson, and I think it matters more, because far more people use the app than build harnesses.
The bias and sycophancy results deserve a separate mention. If the API and the app differ on social bias benchmarks, then a vendor’s published safety evaluation, which almost certainly ran through the API, doesn’t necessarily describe what a teenager gets in the app. Policy people should read that section carefully. I’m not a policy person, so I’ll leave it there, but it’s the part of the paper I expect to get cited most.
What this means for how I evaluate models now
I’m changing a few things in how I run evaluations for client work, and none of them are clever.
First, I evaluate through the surface the client will use. If they’re going to use the ChatGPT app, then my eval runs against the ChatGPT app, or at least a sample of it does. That’s slower and uglier than an API loop. It’s also the only number that’s honest. For teams that build their own products on the API, the API number is the right one, and this paper is less of a problem for them.
Second, I stopped trusting single runs. The paper’s test-retest finding is a reminder that a model can pass your eval on Monday and fail it on Tuesday, and the interface versions are worse at this than the API versions. I now run the same set three times minimum and report the spread, not the best run. I covered why that spread matters for structured extraction in my post on LLM-as-a-judge, and the lesson is the same here.
Third, I write down which door I used. It sounds trivial. But every eval report I’ve written before this month says “GPT 5.4” without saying “via API, temperature 0, system prompt attached”. Going forward, the access surface goes in the header, next to the model version.
Where I think the paper could be wrong, or at least incomplete
I’m not going to pretend this is the final word. Nine benchmarks across seven systems is a decent sample, but interfaces change weekly and the paper is a snapshot. It’s possible the gap has already narrowed for some vendors, or widened for others, since the runs were done. The 3.4 point average also hides a lot of variance; some system and benchmark pairs will have a much bigger gap and some will have none.
I’d also push back on anyone who reads this as “the API is better”. It’s better at benchmarks, which is what benchmarks were designed for. The app might be better at the thing a normal person wants, which is a helpful answer with guardrails and no prompt engineering. Those are different goals. The paper measures one of them. My client with the support tickets didn’t care about accuracy on a bias benchmark, he cared about whether the app would sort his inbox, and for that the honest answer was “run it in the app and see”.
One more thing I’m unsure about: whether the vendors will respond by publishing interface-based evals, or by quietly tuning the apps to match API behaviour. The first would be good for everyone. The second would make the gap smaller without making it more transparent, and I suspect that’s the path of least resistance.
What to do this week
Take one eval you already run, whatever it is, and run it once through the chat app instead of the API. Twenty prompts is enough. Copy and paste if you have to. Compare the numbers. If they match, great, you’ve learned your workload doesn’t have this problem. If they don’t, you’ve just found out something about your product that a leaderboard was never going to tell you, and you found it before your users did.
If you want a second pair of eyes on an evaluation setup, that’s part of what I do for clients; my portfolio has examples of the kind of work I take on. But the twenty-prompt test costs nothing and takes an hour, so do that first.
