I spent most of May blaming my prompts for failures my prompts had nothing to do with. An agent I built for a client kept falling apart around step twelve of a twenty-step job. It would re-call tools it had already called, forget constraints I’d stated at the start of the run, and once it confidently declared a task finished that it had never started. I rewrote the system prompt nine times. I added capital letters. I added threats. Nothing held.
The actual problem: every tool call was dumping 40KB of raw JSON into the conversation, and by step twelve the model was swimming in it. The prompt was fine. The context was a junk drawer.
That month is why I stopped treating “context engineering vs prompt engineering” as a naming debate. They’re different jobs. Prompt engineering is deciding what you say to the model. Context engineering is deciding what the model gets to see at all. Once you’re building agents, the second job is where the failures live.
The difference, in one working definition
Prompt engineering optimizes a single instruction. Phrasing, examples, output format, the order you present constraints in. It’s the right tool when you have one shot at one task: a classifier, a summarizer, a one-off extraction. You tweak, you eyeball the output, you ship.
Context engineering manages the whole window across a long run. Which tool outputs stay verbatim, which get compressed, what history gets dropped, what sits in a file and only comes back when the agent asks for it. Anthropic’s engineering team frames it well in their post on effective context engineering: context is a finite resource with diminishing returns, and the job is finding the smallest set of high-signal tokens that make the right behavior likely.
I’d put it more bluntly. Your agent usually doesn’t fail because the model is dumb. It fails because you fed it 60,000 tokens it didn’t need and the three sentences it did need got buried somewhere around turn four.
If you’re doing single-shot completions, you can close this tab. Prompt engineering covers you. The moment you add tools and loops, keep reading.
Where my agents actually died: tool responses
Here’s the lazy default I was running, and I suspect most people are running, because it’s what every tutorial shows:
result = api_client.search_orders(query)
messages.append({
"role": "tool",
"content": json.dumps(result) # every field, every record, every time
})
Enterprise APIs return everything. Internal IDs, audit timestamps, nested metadata, pagination cursors, localization variants. The model needs almost none of it. But it all lands in the window, and it lands again on the next call, and the call after that.
Here’s what that handler looks like now:
result = api_client.search_orders(query)
slim = [
{"id": o["id"], "status": o["status"], "total": o["total"]}
for o in result["orders"][:20]
]
messages.append({
"role": "tool",
"content": json.dumps({
"count": result["total_count"],
"orders": slim,
"note": "call get_order(id) if you need full details"
})
})
Three fields instead of forty. A count so the model knows what it’s not seeing. A pointer telling it how to get the rest. That last line matters more than it looks: the agent can still reach the full record, it just has to want it. Detail became opt-in instead of ambient.
The other failure this fixed was stale state. My agent kept acting on order data from fifteen steps earlier, because the old snapshot was still sitting in context looking exactly as authoritative as the fresh one. Once full records stopped accumulating in the window, the agent re-fetched when it needed current state. Not because it got smarter. Because the stale copy wasn’t there to mislead it.
The paper that put numbers on what I saw
I thought this was just my problem until I read Less Context, Better Agents, a paper from last week on long-horizon tool-using agents in enterprise workflows. The authors studied exactly this failure mode: verbose tool responses causing context overflow, stale-state errors, and inflated inference cost in automated enterprise tasks. Their fix is the same shape as mine, engineered context handling for tool output rather than raw pass-through, and they report better task completion at lower cost.
One paper, one setting, so hold it loosely. But it matched my production logs closely enough that I stopped feeling clever about my fix and started feeling late to it.
The cost angle deserves a number of its own. Tokens you put in the window get charged on every subsequent call. A 40KB tool response on step three isn’t one payment, it’s a subscription you keep paying through step twenty. Trimming tool output was the single biggest cost reduction I made on that client project, and it wasn’t close.
What my setup looks like now
Three habits, in the order I’d adopt them again.
First, compact finished work. When a sub-task completes, I replace its tool-call transcript with a two-line summary of what happened and what was decided. The agent doesn’t need the play-by-play of a search it ran twenty minutes ago. It needs the conclusion.
Second, offload to files. Big artifacts, like a fetched document or a long test log, go to a scratch file. Context gets the path and a one-line description. The Manus team’s post on context engineering calls the file system the ultimate context: unlimited, persistent, and readable on demand. They also make a point I hadn’t considered about keeping context append-only to preserve KV-cache hits, which is the kind of detail you only learn from people running agents at scale.
Third, retrieve just in time. I used to preload everything the agent might need. Now it starts light and pulls references when the task actually calls for them. Slower on paper. Faster in practice, because the model stops attending to material that never becomes relevant.
I keep a fourth habit for the output side: when context quality drops, hallucinations climb. The model fills gaps with plausible inventions, and a polluted window has a lot of gaps. I wrote up how I catch hallucinations separately, and most of those catches trace back to context problems upstream.
Prompt engineering still matters, just later
None of this makes prompts irrelevant. It changes when they’re worth tuning. A prompt tweak inside a polluted context window is unmeasurable. You change a sentence, the agent behaves differently, and you can’t tell whether the edit caused it or some interaction with 50K tokens of accumulated noise did. Clean context first. Then prompt changes become legible again, and small ones start producing repeatable effects.
This is the same lesson I keep relearning in agentic coding, where the feedback loop beats a smarter model: the environment around the model decides more than the model does. Most of the agent work I take on now starts with an audit of what’s in the window, not a rewrite of the instructions.
Something to run this week
Log your agent’s full message array right before its next failure, then count tokens by category: system prompt, tool definitions, tool responses, conversation history. In every agent I’ve audited, tool responses dominate, and they’re the cheapest thing to fix. Slim your noisiest tool’s response to the fields the model actually uses, add a “call X for details” pointer, and re-run the task that was failing.
Mine went from dying at step twelve to finishing all twenty. Your number will differ, but I’d bet money on the category. It’s almost never the prompt.
