Short version for the impatient: GPT-6 Astra scored 62.7% on ARC-AGI-3 and also 99.9% on ARC-AGI-3, on the same day, on the same test set, and both numbers are real. The difference is the harness. If you only read one headline this week, make it that one, because it’s the most useful thing anyone has said about benchmarks in a long time.
I spent Thursday evening reading the ARC Prize write-up on Astra instead of the thing I was supposed to be doing, which was fixing a flaky test in a client’s checkout flow. The test is still flaky. But I came away with a clearer picture of why the agent evals I run for my own projects keep giving me numbers I don’t trust, and it’s the same reason Astra has two scores.
This isn’t a “is it AGI” post. ARC Prize themselves say it isn’t, and I have no interest in relitigating that. It’s a post about what the 37-point gap between those two scores means for anyone who ships agents, which at this point is most of us.
What ARC-AGI-3 actually tests
Quick background, because ARC-AGI-3 is only six months old and the earlier versions were different animals. ARC-AGI-1 and 2 were grid puzzles: here’s a few input-output pairs, infer the rule, apply it. Static. One shot.
ARC-AGI-3 launched in March 2026 as an interactive benchmark. Each task is a small turn-based game with no instructions, no stated goal, and no rules explained. The agent has to poke at the environment, work out what the mechanics are, figure out what “winning” even means, and then do it, across levels that get harder. ARC Prize built hundreds of these and calibrated difficulty with about 500 members of the public, who solved all of them. At launch, every frontier model scored under 1%.
The thing I find interesting about the design is that it scores action efficiency on top of completion. Finishing a level in 400 moves when the median human took 40 counts against you. That’s a much closer match to how I’d judge an agent doing real work. An agent that eventually fixes the bug after 300 tool calls is not the same as one that fixes it in twelve.
The two harnesses, in plain terms
Here’s the part that matters. ARC Prize ran Astra two ways.
The Standard harness is their provider-neutral interface. Every model gets the same thing: the game state, the action space, and the ability to carry forward whatever notes it chooses to write for itself between turns. What it keeps is up to the model. What it doesn’t write down, it loses.
The Provider Adapter harness lets the model use the context-management features its provider built for it. For Astra, that means the opaque reasoning state (the stuff OpenAI doesn’t show you) persists between requests, and long conversations get compacted rather than truncated. The model gets to reuse its own prior thinking instead of reconstructing it from notes.
Standard: 62.7% at max reasoning effort, for about $26,000. Provider Adapter: 99.9% at high effort, for about $19,000. Faster too. ARC Prize reports the adapter runs were roughly 3.66 times quicker by elapsed time and used 49% fewer tokens across the 167 game-and-reasoning-level pairs both harnesses solved.
Read that again. Cheaper, faster, and 37 points better. Same model weights. The only thing that changed is whether the model was allowed to remember how it had been thinking.
Why the gap is the story, not the 99.9
I’ve seen a few takes this week along the lines of “the 99.9 is fake, the real score is 62.7”. I don’t think that’s right, and ARC Prize don’t frame it that way either. They’re going to report both on the leaderboard, labelled, going forward. The Standard number answers “how do models compare on an identical minimal interface”. The adapter number answers “how well does this model do when it’s used the way its provider intends”.
Both are legitimate questions. But if you’re building something, the second one is the one you’re living with. Nobody ships an agent on a deliberately handicapped interface to keep the comparison fair.
What the gap tells you is how much of an agent’s performance lives outside the weights. In Astra’s case, more than a third of the score. That lines up uncomfortably well with something I noticed on my own projects and wrote up in my post on agent memory being a dose, not a switch: the same model, given a slightly different memory policy, behaved like a different product. I’d assumed that was me being sloppy with prompts. Turns out the best-resourced eval team in the field sees the same thing at a scale of 37 points.
The Standard-harness results across reasoning levels are also worth a look. Astra at “none” (no reasoning) scored 35.2%. At “low” it scored 17.5%. Lower. Some reasoning did worse than no reasoning on this task under the Standard harness, and I don’t have a tidy explanation for that. ARC Prize don’t offer one either. Under the adapter harness, every level from none to max lands between 96.7% and 99.9%. The harness raised the ceiling and flattened the whole curve at the same time.
What Astra was actually doing in there
The bit of the write-up I keep coming back to isn’t the numbers. It’s the replays.
Under the Standard harness, Astra invented its own shorthand for tracking game state. Not English notes. Things like L8: hub q2 (8↓). Lengths: 14=1, 9=1 to record a level, a rotation index, and mechanism lengths, and extend8 to3; retract10 to2; shorten8 to1 as an ordered plan. ARC Prize call it an on-the-fly algebraic notation. It’s a model compressing its own world model into the smallest thing that survives the note-passing boundary, because the harness forces it to.
That’s the same thing a person does when the whiteboard is too small. It’s also, I think, why the Standard score is as high as it is. The model built a workaround for the missing memory. The adapter harness just removed the need for the workaround.
In a third setup, the PRO-LONG harness where the model gets a code sandbox, Astra wrote itself small libraries per game: maze_solver.py, patrol_solver.py, a sync_state.py to check predictions against what it observed. ARC Prize are careful to say that’s model-plus-tools and not comparable to the human baseline. Fair. But it’s also exactly what you’d want an agent to do when handed a codebase it’s never seen.
The part that should worry you about your own evals
Here’s my uncomfortable conclusion. If a 37-point swing can come from the harness alone on a benchmark this carefully built, then the numbers coming out of my own agent evals, run on a harness I threw together in an afternoon, are measuring my harness at least as much as they’re measuring the model.
I made this point from a different angle in the post about LLM evaluation tools missing the long game. Most eval tooling scores single turns. Agents don’t live in single turns. The thing that made Astra go from 62.7 to 99.9 is a multi-turn concern: what persists, what gets compacted, what the model is allowed to reuse. If your eval harness resets context per task and your production agent doesn’t, you’re not testing what you ship.
The fix isn’t complicated, it’s just work. Run your evals through the exact context-management path your production agent uses. Same compaction, same memory, same tool wiring. If the provider offers a native session or reasoning-state feature, test with it on and off, and record both, the way ARC Prize now does. If the two numbers are close, great, your harness isn’t the variable. If they’re 30 points apart, you’ve just learned where your engineering time should go, and it isn’t prompt tweaking.
Simon Willison’s note on the Astra launch makes the related point that Astra’s headline benchmark wins arrived alongside Artificial Analysis still ranking it below Fable 5.1 on their intelligence index. Different harnesses, different questions, different winners. That’s not a contradiction. That’s what measuring agents looks like now.
What I’m doing about it this week
Two things, and you can steal both.
First, I’m adding a “harness” column to my eval results table. Every run records which context policy it used. I’ve been treating that as an implementation detail. It’s a variable, and it’s a bigger one than the model version in some cases.
Second, I’m re-running my agent evals with provider-native session state turned on, for the two providers I use that support it, and comparing against my hand-rolled note-passing. I have a suspicion my notes approach is the Standard harness of my own little world and I’ve been paying a tax I didn’t know about.
If you want to argue with any of this, the replays are public and you can play the games yourself. I lasted about four minutes on the first one before I understood the rules, which puts me somewhere behind Astra and, I suspect, most of the 500 test participants. This is the kind of evaluation work I do on the agent projects I take on; there’s more about that on my site.
