Compression Is Not a Length Problem
When teams first hit context limits, they reach for summarization: shrink the transcript, keep the conversation going. Then quality drops mysteriously, and the summary gets blamed. The real issue: context compression interacts with three different systems at once — token cost, reasoning quality, and the KV cache that makes long sessions affordable. Optimize one without the others and the session gets worse in a dimension you weren't watching.
Why Deletion Hurts More Than You'd Expect
In-context learning behaves more like retrieval than reasoning: the model answers largely by pulling from what is literally present in the window. Delete a detail and the model doesn't "know" it is gone — it confabulates from what remains. This is why lossy compression fails asymmetrically: what gets lost first is exactly what matters most — early architectural decisions, the reasoning behind constraints, and the paths already tried and failed.
A useful discipline before compressing anything: classify the transcript by task type. Retrieval tasks need breadth (candidate lists survive, prose can go). Analysis tasks need depth (the evidence chain survives, redundant confirmations can go). Creative tasks need their oddities kept — the unusual detail is the prompt for the good idea.
The Production Layer Cake
Systems that run long sessions at scale don't pick one strategy — they stack them, cheapest first:
- Tool-output budgets. Big results go to disk at write time; the context keeps a short preview and a reference. The replacement decision is frozen once made, so the cache stays stable.
- Noise deletion. Low-value content (endless search-result pages where three lines mattered) is removed outright. Summarizing noise doubles the waste.
- API-side editing. Some inference APIs let you drop specified messages from the served prefix while your local history stays intact. Zero local complexity — but everything after the removal point rebuilds the cache once, so use it when an overflow was coming anyway.
- Archival summaries. Per-turn structured notes, kept round-by-round like a git log rather than squashed into one blob. The logic of the session survives; the raw tokens go.
- Full LLM-driven compression. The last resort — and it needs a circuit breaker. Real traffic shows a stubborn fraction of sessions stuck in repeated compression failures, burning money on every retry. Count consecutive failures and stop.
| Layer | What it does | Cache impact | When to use |
|---|---|---|---|
| 1. Tool-output budgets | Big results go to disk at write time; the window keeps a preview + reference | None — frozen once made | Always, by default |
| 2. Noise deletion | Low-value content removed outright (summarizing noise doubles the waste) | Low — cheap regions only | Endless search-result dumps |
| 3. API-side editing | Specified messages dropped from the served prefix, local history intact | One rebuild after removal point | When an overflow was coming anyway |
| 4. Archival summaries | Per-turn structured notes kept like a git log, raw tokens go | Rebuild at turn boundary | Closed topics in long sessions |
| 5. Full LLM compression | Model summarizes the transcript — last resort with a circuit breaker | Full rebuild; failure loop risk | Only after 1–4 failed |
The Cache Conflict — and Its Resolution
Here's the trap: caches key on the conversation prefix. Compression edits the middle. Every edit invalidates the cache for everything after it, and a long session suddenly pays full input price again. Aggressive compression can raise total cost while lowering token counts.
The reconciliation is layering plus timing: compression that happens at natural boundaries (task completion, topic change) amortizes the rebuild across the next long stretch; the early layers (budgets, noise deletion) touch content whose cache region was cheap anyway. Compression should feel like garbage collection in a managed runtime — scheduled, batched, rarely in the hot path.
Isolation Beats Compression
Before compressing, ask why the window is full. If the answer is "the agent read forty pages to find one number", the fix is not compression — it is isolation. Delegate wide-scope, bulky-intermediate work to a sub-agent whose context absorbs the noise; only the distilled answer enters the main window. And on the human side, the same rule: important decisions and their rationale belong in durable documents the agent maintains, not in an ever-growing chat transcript.
A Worked Example: Budgets, Boundaries, and the Math
Numbers make the trade-offs concrete. Consider a code-migration agent with a 180K-token session: 120K of repository context, 40K of tool outputs, 20K of conversation. At $3/M input tokens with Anthropic-style prompt caching (cached reads at 0.1×, writes at 1.25×), a steady-state turn costs roughly $0.12. Run naive summarization mid-session and the cache is fully invalidated — that turn pays the full uncached price, about $0.54, and every later turn until the cache re-warms costs several times the steady state. Compress at the task boundary instead and the rebuild amortizes across the next fifty turns: about $0.13 per turn all-in. Same content, same model — a 4× swing in input cost purely from where the compression happens. The exact multiplier varies by vendor (OpenAI's automatic caching yields a smaller ~2× swing, Google's ~3×), but the shape of the curve is the same: compression timing dominates compression ratio.
The implementation sequence that works in practice:
- Instrument first. Log per-turn token counts split by source (system prompt, repo context, tool outputs, conversation) and the cache-hit ratio. You cannot budget what you cannot see.
- Set tool-output budgets at the tool layer, not the prompt layer. A search tool returns top-3 previews plus a file reference; the agent reads the full file only when the preview earns it. This deletes noise before it ever enters the window.
- Define task boundaries explicitly. A boundary is where state is checkpointed to durable storage and compression is allowed. Between boundaries, the window is append-only.
- Compress in tiers at boundaries. Drop raw tool payloads (already on disk), then collapse superseded intermediate reasoning, then summarize closed topics — cheapest and safest first, never touching the architectural decisions layer.
- Verify with a canary. After each compression, ask the agent to recall three facts from the compressed region. Two failures in a row means the tier cut too deep — roll back to the checkpoint.
Key Takeaways
- Compression serves three masters: cost, quality, and cache. Tune them together.
- The model retrieves from what is present — deleted facts are confabulated, not remembered as missing.
- Stack strategies cheapest-first: budgets, deletion, API editing, archival summaries, full compression.
- Compress at task boundaries; every mid-prefix edit rebuilds the cache downstream.
- Full compression needs a circuit breaker — some sessions never recover.
- If the window is full of bulky intermediates, isolate the work instead of compressing the transcript.
