Short version for the impatient: your agent eval suite is probably testing the wrong time scale. Mine was. If you want the two papers that finally made this click for me, read on.
I got burned by this in June. A client agent, order triage, nothing exotic, sailed through our eval set. Above ninety percent in every task category. Three weeks into production it had quietly invented a “needs review” holding state and started parking tickets there, and nobody caught it for four days because every individual decision still looked defensible when you pulled it up on its own. The evals weren’t wrong. Each step was fine. The trajectory was garbage.
Then two benchmark papers crossed my feed in the same week and put a name on the thing I’d been chewing on. Both measure what happens when a model has to stay coherent over a long horizon instead of nailing one turn. Both found that the long game looks nothing like the leaderboard.
The benchmark said yes, week three said no
Almost every eval you can run today has the same shape: one input, one output, one score. Doesn’t matter if it’s a hand-rolled pytest file, promptfoo, LangSmith, or a spreadsheet with a thumbs-up column. You feed the model a case, you grade the answer, you average the grades.
That shape is fine for a chatbot that answers a question and moves on. It’s the wrong shape for an agent, because an agent’s output at step 40 depends on the state it built up across steps 1 through 39. Errors don’t just happen, they compound. A slightly weird decision at step 3 becomes the context for step 4, and by step 60 the agent is operating in a world of its own making.
My triage agent’s invented holding state was exactly this. No single decision was a failure. The failure was a slow lean, the way a shopping cart with one bad wheel drifts left. You can’t see a lean in a single frame.
What happens when an LLM runs a company for a year
The first paper is a benchmark that hands an LLM a simulated retail company and makes it live with its choices. AI Playing Business Games runs five models (the free web versions of Gemini, ChatGPT, Meta AI, Mistral, and Grok) through a month-by-month management simulation: pricing, purchasing, marketing, the works. Every month’s decision lands on top of the last one.
The authors built it because there’s a shortage of benchmarks for long-term coherence, and the few that exist keep turning up the same uncomfortable result. They point to Vending-Bench, where agents ran a simulated vending machine business and models with strong single-task scores produced wildly different outcomes from run to run. Same model, same setup, one run ends profitable and another spirals into a doom loop of misremembered inventory. The average hides it. The variance is the story.
That variance is what my eval suite never measured. I ran each test case once. Why would you run it ten times? The answer, which I now find obvious in the embarrassing way all hindsight is obvious: because for a long-running agent, run-to-run spread is the metric. A model that’s brilliant nine times and deranged once will pass almost any sampled eval you write, then be deranged in production on a Tuesday.
One lucky forecast tells you less than you’d hope
The second paper is more fun. The AI World Cup benchmark had ten LLM assistants forecast the entire 2026 FIFA World Cup before kickoff. Same data snapshot, same prompt, same JSON schema, one shot each: group results, the full knockout bracket, the champion. Then reality played out all 104 matches and the forecasts got scored.
GPT-5.5 Thinking won with 744 points and was the only model that picked Spain, who beat Argentina 1-0 in the final. Nice headline. The finding I actually care about is buried in the correlations: total score correlated with knockout-stage points at r = 0.986, and with group-stage match points at r = 0.055. Getting the early, easy, high-volume predictions right contributed almost nothing. The whole ranking came down to a handful of late branch points.
Agents work the same way. A trajectory is a compound prediction, and a few branch decisions dominate the outcome. Your per-step accuracy can be 95% while the 5% sits exactly on the branches that matter. When I audited my triage agent afterward, the “needs review” habit traced back to two early decisions in the first week. Everything after was locally reasonable and globally wrong.
Why your llm evaluation tools don’t catch this
I don’t think the tools are bad. I use them daily. They’re just built around three assumptions that stop holding the moment your model runs longer than one turn.
First, they assume samples are independent. Score 200 cases, average them, done. Agents violate this by design, since the whole point is carrying state forward.
Second, they report means. Long-horizon performance is high variance, and both papers above found the spread between runs of the same model can be wider than the spread between models. A mean with no distribution around it is close to noise.
Third, they grade outputs, not states. Drift lives in the state: the memory that’s slowly filling with junk, the label taxonomy the agent is quietly extending. I wrote about a cousin of this problem in my post on tool schemas as an attack surface, where everyone audits the prompt and nobody audits the machinery around it. Same blind spot, different corner.
How I test agents now
This is the part I can actually hand you. Since June, no agent I build goes out without what I’ve started calling trajectory evals. The recipe is short.
Run whole scenarios, not steps. Build a small simulator of the environment, even a crude one, and let the agent run for the full horizon you expect in production. A day of simulated tickets, a month of simulated inventory.
Run each scenario at least ten times and look at the spread before the mean:
results = [run_trajectory(agent, scenario, seed=i) for i in range(10)]
scores = [r.final_score for r in results]
print(f"mean: {statistics.mean(scores):.2f}")
print(f"stdev: {statistics.stdev(scores):.2f}")
print(f"worst: {min(scores):.2f}")
# ship gate: the worst run has to be acceptable,
# and the spread has to be boring
assert min(scores) > FLOOR
assert statistics.stdev(scores) < MAX_SPREAD
The gate on min(scores) matters more than the one on the mean. Production doesn’t experience your average run. Somebody always gets the worst one.
Then checkpoint the state, not just the output. Every N steps, assert on the agent’s working state: how many items are in each status, how big the memory has grown, whether any categories exist that you didn’t define. That last check is the one that would have caught my invented holding state in an afternoon instead of three weeks.
None of this is exotic. It’s maybe 150 lines of harness around the eval stack you already have, and it’s now a standard part of the pre-launch checklist in my consulting work. The papers gave me the vocabulary, but the practice is just: test the time scale you’re actually shipping.
Something to run this week
Take one agent you have in production or close to it. Pick one realistic scenario, run it end to end ten times, and diff the ten final states against each other. Not the outputs. The states.
If all ten look alike, good, you’ve earned some sleep. If they don’t, you’ve just found out what your eval suite has been hiding, and you found it on a weekday afternoon instead of in week three. That trade is the entire point.
