AI Agent Memory Is a Dose, Not a Switch

Short version for the impatient: giving your agent more memory can make it worse, and the point where that happens depends on which model you’re running. If you want the reasoning, keep going.

I’ve built two agent systems this year where “add memory” was on the roadmap as if it were a checkbox. Turn it on, agent gets smarter, ship. Both times the results were muddier than that, and both times I assumed I’d implemented something wrong. Turns out that’s a documented pattern, and someone finally ran the numbers on it properly.

IBM Research published a study this week testing agent memory across eight models, and the framing in it has already changed how I’d scope this work. Their line: memory is a dose you calibrate, not a feature you switch on.

What “memory” means here, because the word is doing too much work

This is the part that trips up most conversations. When people say ai agent memory they might mean any of four things: raw conversation transcripts replayed into context, a vector store of past documents, a summary buffer, or distilled behavioural guidelines.

The IBM work is about the last one. Their tool, ALTK-Evolve, has the agent attempt tasks, then mines its own trajectories, successful and failed, for reusable guidelines: strategies that worked, mistakes to avoid, edge cases it hit. Those guidelines go back into context at inference time. No weight updates, no human labelling, no fine-tuning run.

I like this framing because it’s the cheap version. Nobody needs a GPU cluster to write lessons into a text file and prepend them to a prompt. That’s a weekend, not a quarter.

The finding that surprised me

They evaluated on AppWorld, a benchmark of 585 multi-step tasks across nine simulated apps, and compared three setups: no memory, the full guideline set injected on every reasoning step, and curated retrieval, which is a fixed high-confidence core plus a few task-relevant guidelines fetched per task.

Three patterns fell out, and they don’t track parameter count the way you’d assume.

Strong models with room to improve wanted everything. DeepSeek-V3.2 gained 9.5 percentage points on task completion with the full guideline set. Claude Opus 4.6, already at 90.5%, still gained 4.1 points.

Weaker models drowned in it. gpt-oss-120b gained 16.1 points with curated retrieval, but less than that with the full set, while burning about 50% more tokens to do worse. More memory, worse outcome, higher bill.

And one model gained nothing at all. GLM-5 sat at 87.5% with memory and without it, to the decimal. The authors call this the saturated pattern and are careful to say the label describes what they saw rather than a proven cause. It could be a ceiling on those tasks, or guidelines that didn’t address its actual failures, or the model just not applying the guidance. They don’t know. I appreciate that they said so instead of inventing a story.

The full writeup is on Hugging Face, with the technical report on arXiv if you want the ablations.

The cost table is the part to screenshot

Everyone assumes better means more expensive. Here it doesn’t, and this is the number I’d put in front of a client.

For gpt-oss-120b, curated retrieval was both the most accurate configuration and nearly the cheapest: +16.1 points of task completion for +5% tokens. The full set cost ten times the token overhead and performed worse.

The overhead comes from re-sending guidelines on every reasoning step, not from the agent taking longer routes. DeepSeek ran roughly the same number of steps either way, around eighteen or nineteen. So it’s input inflation, which means prompt caching is the lever that matters in production. Keep the static guideline prefix stable across steps and it stays cacheable. Shuffle it, or interleave it with per-task content, and you’ve thrown that away without noticing.

I’ve watched a bill triple from exactly this mistake: dynamic content injected above the stable prefix, cache invalidated on every call. Cache-aware prompt layout isn’t an optimisation you do later. It’s the difference between memory being affordable and memory being the line item someone asks about.

One caveat I want to flag on my own reading of this, because it’s easy to walk away with the wrong rule of thumb. “Weak models want less, strong models want more” is a tidy summary and the authors explicitly refuse to make it. They point out that parameter count isn’t what determines the pattern; benchmark headroom, context window size, architecture, guideline quality, and the task distribution all seem to matter, and they haven’t separated them yet. They also flag context window size as a plausible factor they haven’t run controlled experiments on. So the takeaway isn’t a lookup table. It’s that the dose is a variable you have to measure per model, and previously most of us weren’t even treating it as a variable.

Two metrics, and the second one is the honest one

They report task completion (TGC), the share of tasks fully completed, and scenario completion (SGC), which bundles several variants of the same task and only counts it if the agent gets every variant right.

SGC moved more than TGC almost everywhere. DeepSeek gained 9.5 on TGC and 16.1 on SGC. GPT-5.5 and Opus, both near the ceiling on task completion, still gained 7.2 and 7.1 points of scenario completion.

That gap is the interesting bit. It says memory helps most with reliability, not raw capability. An agent that solves a task four times out of five looks fine on the headline metric and is unusable in production, because the fifth run is a support ticket. If you’re evaluating your own agent on a single pass-rate number, you’re measuring the thing that matters least.

What I’d change in my own setup

Concretely, three things I’m doing differently after reading this.

I’m going to stop treating memory as binary in scoping conversations. “Does it have memory” is the wrong question. “How much, delivered how, and does this model benefit” is three questions and all of them have measurable answers.

I’m going to test both delivery strategies on whatever model the project actually runs, not the model the benchmark used. The paper’s own caveat is that these results come from one benchmark, and AppWorld is not your app. What transfers is the method, not the numbers.

And I’m going to audit prompt layout for cacheability before adding a single guideline. That’s a half-day of work that determines whether the rest is affordable.

The retrieval side has a known weakness too, which the authors flag: their selection ranks guidelines by cosine similarity, and they’ve shown that doesn’t reliably predict which guidelines actually help a given task. A selector trained on outcomes is the obvious next step and it doesn’t exist yet. So if your retrieval feels like it’s pulling the wrong lessons, that’s not you being bad at embeddings.

The decision this actually sits next to

Every time a client asks about improving agent accuracy, the conversation collapses into fine-tuning versus retrieval within about four minutes. I wrote up the question I ask before that argument starts, and distilled guidelines are a third option that rarely gets raised: no training run, no vector infrastructure, portable across models, and cheap enough to test in an afternoon.

It won’t fix a model that fundamentally can’t do your task. Nothing prompt-shaped will. But for the very common case of a capable model that keeps making the same three mistakes, writing those mistakes down and handing them back is embarrassingly effective for the effort involved.

Try this before your next agent sprint

Take your last fifty agent runs. Read the twenty that failed. Write, by hand, the ten rules that would have prevented them. Put those ten rules at the top of your system prompt, above anything that changes per request, and re-run the same fifty tasks.

That’s a manual version of the whole loop and it takes an afternoon. If it moves your numbers, you’ve justified building the automated version. If it doesn’t, you’ve saved yourself a sprint, and you’ve learned that your problem is somewhere other than memory. Both outcomes are worth the afternoon. This is roughly the diagnostic I run at the start of agent work I take on, and the failures are almost never where the client expects.