Your Agent Is Not Dumb. Its Context Is Just Wrong.
Teams routinely hit the same wall: an AI agent that dazzles in demos falls apart in production. The instinctive response is to swap models or write a better prompt. In our experience, and in most post-mortems we read, the real culprit is elsewhere — the context the model sees at each step of the loop. The model can only reason over what is in front of it. If that window is polluted, stale, or structured the wrong way, no amount of model quality will save the run.
A useful mental model: an agent's context is nothing more exotic than a growing list of messages — system prompt, user goal, the agent's own actions, tool results. Everything about "memory" and "attention" reduces to engineering decisions about that list. Once you see it this way, a set of practical rules emerges.
Rule 1: Keep the Prefix Stable
Modern inference stacks cache the computation for the tokens the model has already seen. That cache is prefix-based: it stays valid only while the beginning of the conversation stays byte-for-byte identical. Two consequences follow:
- Put static material — role instructions, tool definitions, domain rules — at the very start, and never edit it mid-session.
- Append dynamic material at the end. Anything that mutates the middle — a refreshed timestamp, a reordered list, a rewritten summary — silently invalidates the cache for everything after it.
Teams that instrument this report dramatic swings: something as innocent as injecting the current time into the middle of the prompt can turn a cheap, fast session into one that re-pays the full input cost on every turn. Cache-awareness is not a micro-optimization. It is an architectural constraint that should shape how you structure the entire conversation.
Rule 2: Give Every Kind of Information a Lifecycle
Not all context ages the same. A production-grade agent treats context like a storage system with tiered retention, not a single bottomless bucket:
- Tool outputs get a budget. Large results (file dumps, API responses, search hits) go to disk; the model sees a short preview plus a reference it can expand on demand.
- Noise gets deleted, not summarized. Summarizing irrelevant content is a waste of tokens twice over — once to summarize, forever to carry.
- Progress gets archived per turn. Structured round-by-round notes — closer to a git log than a squash — preserve the logic of the session while the raw transcript shrinks.
- Full compression is the last resort. And it deserves a circuit breaker: in real traffic, a noticeable fraction of sessions get stuck in repeated compression failures. Detect the loop and stop paying for it.
Rule 3: Isolation Beats Compression
Compression is a subtraction performed after information has already entered the context. The stronger move is to keep bulky intermediates out of the main window entirely. Delegate wide-scope work — "go search the codebase", "read these forty pages" — to a sub-agent with its own context, and let only the distilled answer flow back to the orchestrator.
This mirrors a discipline every engineering organization already knows: important decisions and their rationale belong in durable documents, not in chat history. Agents benefit from the same habit — writing progress notes to files as they go, so the session can survive compaction, and the next session can pick up the trail.
Rule 4: Compress by Task, Not by Template
The same transcript deserves different compression depending on the job. Retrieval tasks need breadth (keep the list of candidates). Analysis tasks need depth (keep the evidence chain). Creative tasks need inspiration triggers (keep the oddities). A compression strategy that ignores task type will delete exactly the wrong things — usually early architectural decisions, the reasoning behind constraints, and the paths that failed. Those are the most expensive items to lose.
Key Takeaways
- Treat context as a message list you architect, not a chat history that happens.
- Static content first, byte-stable prefix, dynamic content appended at the end.
- Layer information retention: budgets for tool output, deletion for noise, archival summaries for progress, full compression only with a circuit breaker.
- Prefer sub-agent isolation over clever compression for bulky intermediate work.
- Have agents write progress to durable documents — sessions are ephemeral, documents are not.
Context engineering is where most of the leverage in agent building lives today. It is cheaper than fine-tuning, safer than prompt-injection whack-a-mole, and it compounds: every improvement to the context discipline lifts every task the agent performs.
