Okay, this is going to sound dumb, but I spent most of Thursday afternoon arguing with a coding agent about a rule I had given it three hours earlier. The rule was “don’t touch the migrations folder, I’ll handle schema changes by hand.” It agreed. It followed the rule for a long stretch. Then, somewhere around the point where the session got long enough to be summarised, it cheerfully rewrote two migration files and told me it had “cleaned up the schema drift.”<\/p>\n
I didn’t lose anything important. Git exists. But I did lose an hour working out why an instruction that had been obeyed for three hours suddenly stopped existing, and the answer turned out to be the most boring one available: the conversation got compacted, and my rule didn’t survive the summary.<\/p>\n
That would’ve stayed a private grumble, except three things landed in my reading pile the same week that all describe the same failure from different angles. Simon Willison hit it in ChatGPT. Two papers measured it. And the Claude Code docs, when I finally read them properly, admit it in plain language. So this post is me connecting those dots, and then listing what I actually changed in my setup.<\/p>\n
Every agent that runs long enough fills its context window. When that happens, the system has two options: refuse to continue, or summarise the older part of the conversation into something shorter and carry on. Nearly every product picks the second one. Claude Code calls it compaction and has a If you’ve read my post on agent memory being a dose rather than a switch<\/a>, compaction is the flip side of that argument. Memory is what you choose to carry forward. Compaction is what the system chooses to throw away on your behalf, and it chooses badly in a specific, predictable way.<\/p>\n Simon Willison asked ChatGPT Work<\/a> to generate 5K and 10K running loops from his house using OpenStreetMap data. It worked for 27 minutes and produced a map, a GPX file, and GeoJSON. Good result. Then he asked for the Python it had written to do the routing, and it couldn’t give it to him. The thread had been compacted, and the code was in the part that got summarised away.<\/p>\n His conclusion was that any system using compaction needs to keep the pre-compaction text and expose it to the agent through a tool call. I think that’s exactly right, and I’d go one further: it should be exposed to the user too, as a plain “show me what got summarised” button. The fact that a product can do 27 minutes of work and then be unable to tell you what it did is a design bug, not a model limitation.<\/p>\n What struck me about his example is that it’s the friendly version of my migrations problem. He lost an artefact. I lost a constraint. Losing an artefact is annoying. Losing a constraint is how you get an agent that starts doing the thing you specifically told it not to do, while believing it’s being helpful.<\/p>\n I’m wary of arxiv papers on agent behaviour because a lot of them measure a toy setup and extrapolate. These two are more grounded than most, and their numbers line up with what I saw.<\/p>\n The first is Lost in Compaction<\/a>, which names the thing I lost: a “session constraint,” meaning an instruction like “do not delete any emails until I confirm” that’s meant to hold for the rest of the session. They built a suite called COMPINT that injects constraints into multi-turn chat, agentic trajectories, and long research tasks, then measures whether they survive compaction. The headline number is that current compactors keep 17% of injected constraints on average. Seventeen. And most of the tested setups did worse on the task with compaction than the same task run without it. Their fix is a small extractor that runs alongside the compactor and pulls constraints out into a separate list, which got retention above 90% without touching the underlying model.<\/p>\n The second is The Compaction Cliff<\/a>, which is narrower and, for me, scarier. They took 20 production agent configurations and ran Claude Code’s own I want to be careful about how much I lean on these. Both were tested on specific compactors at specific dates, and the vendors change their summarisation prompts without announcing it. The 53% figure might already be different. But the shape of the problem, that rules decay faster than you’d expect and compound across rounds, matches my Thursday exactly.<\/p>\n This is the part where I felt a bit silly. The Claude Code documentation<\/a> on context says, in plain words, that Claude compacts automatically and that “instructions from early in the conversation can get lost.” It tells you to put persistent rules in That’s the vendor telling you the exact failure mode, and I’d skimmed past it. The docs also mention a “Compact Instructions” section you can add to So the rule I gave the agent in chat was, by the vendor’s own description, in the wrong place. The rule belonged in a file. I put it in a message. Chat messages are the thing that gets summarised.<\/p>\n Here’s the concrete bit. Four changes, and the first one took two minutes.<\/p>\n First, anything that’s a standing rule for a project now lives in Second, I stopped treating chat as the place where code lives. If the agent writes a script that matters, it writes it to disk, in the repo, before doing anything else. Simon’s routing code would’ve survived if it had been a file. This is also just better hygiene for the reason I wrote about in the Laravel AI SDK 0.11 post<\/a>: if you can’t see what the agent did, you can’t review it, and a compacted transcript is the extreme case of not being able to see.<\/p>\n Third, for long sessions I now run \/compact<\/code> command. Codex does it automatically. ChatGPT does it inside long threads. The summary is written by a model, which means it’s a lossy, judgement-based rewrite of everything that came before. Whatever the summariser decides isn’t important enough to keep is gone, and nothing in the UI tells you what was dropped.<\/p>\nSimon’s running route, and the code that vanished<\/h2>\n
The two papers that put numbers on it<\/h2>\n
\/compact<\/code> prompt on Sonnet 4.6. After one round of compaction, 53% of safety rules survived. After five rounds, 10%. The framing they use is that a safety rule and an episodic log compete for the same tokens, and the summariser compresses both at the same rate, even though only one of them needs its exact wording to still be enforceable. Their answer is to classify each line of the agent’s knowledge by type and give each type its own retention policy, so rules get pinned and logs get squashed.<\/p>\nWhat the Claude Code docs actually say<\/h2>\n
CLAUDE.md<\/code> rather than relying on conversation history. It describes the order of operations, too: older tool outputs get cleared first, and the conversation is summarised only if that isn’t enough. Your requests and key code snippets are preserved. Detailed instructions from early on may not be.<\/p>\nCLAUDE.md<\/code> to control what the summary keeps, and a \/compact focus on X<\/code> form for steering a manual compaction. I had used neither.<\/p>\nWhat I changed<\/h2>\n
CLAUDE.md<\/code>, and I added the compact instructions section the docs describe:<\/p>\n# Project rules\n\n- Never modify files under database\/migrations. Schema changes are done by hand.\n- Do not run `php artisan migrate:fresh` under any circumstances.\n- Ask before deleting any file that isn't in the current diff.\n\n## Compact Instructions\n\nWhen compacting, preserve verbatim: every rule in the "Project rules" section\nabove, the list of files changed so far in this session, and any command the\nuser told you not to run.\n<\/code><\/pre>\n\/compact focus on the project rules and the list of changed files<\/code> myself before the auto-compaction kicks in, instead of letting it pick the moment. I can’t prove this is better than the automatic version. It feels better, and it means the summary is written while I’m paying attention instead of while I’m getting coffee.<\/p>\n