The Invoice Shock Pattern
A team ships an LLM feature. The first month's invoice is a rounding error. Three months later, usage grew, prompts grew with it, an agentic loop made every user action 8–15 model calls, and the invoice quietly crossed the salary of the engineer maintaining the feature. Nobody decided to spend that much — the cost accrued through unmeasured decisions: chunk sizes, retries, prompt boilerplate, and a model tier chosen once and never revisited.
LLM cost optimization is not about choosing a cheaper model. It is a measurement problem first: until cost is attributed to specific pipeline stages and query classes, every optimization is a guess. Here is the sequence we use to bring agent and RAG workloads under cost control.
Step 1: Attribute Cost per Span, Not per Invoice
The invoice tells you the total; it cannot tell you what to fix. Instrument the pipeline with OpenTelemetry traces carrying token counts and model tier per span. Within a day you will have the breakdown that drives all further decisions: retrieval vs generation vs reranking vs tool-output summarization, per query class. In most RAG audits we run, two stages account for the majority of spend — and neither is the one the team assumed.
| Stage | Typical share of spend | Usual offender |
|---|---|---|
| Generation (final answer) | 40–60% | Verbose system prompts, oversized output limits |
| RAG context (input tokens) | 25–40% | top-k without relevance filtering — dead chunks paid per query |
| Reranking | 5–15% | Reranking candidates that keyword rules would have dropped |
| Agent loop overhead | Highly variable | Retries, redundant re-planning steps, full-history re-sends |
Step 2: Attack Input Tokens First
Input tokens are the quiet majority in RAG systems, and they compound: every one of your daily queries pays the same context tax. The highest-leverage moves, in order of typical ROI:
- Relevance-filtered retrieval. Sending top-k chunks "for safety" means paying for dead context on every query and degrading answer quality at the same time. Set recall budgets per query class and drop chunks below the relevance floor.
- Prompt boilerplate extraction. Instructions repeated verbatim in every prompt belong in the system message with caching (Anthropic prompt caching, OpenAI automatic prefix caching) — cached input tokens cost a fraction of uncached.
- History compaction in agent loops. Re-sending the full conversation every turn is quadratic growth. Summarize stale turns; keep the recent window verbatim.
- Structured, minimal output instructions. "Answer in at most three sentences, JSON schema X" beats both rambling output and post-hoc trimming.
Step 3: Route by Query Class, Not by Habit
Most workloads have a natural tiering: a large share of queries are easy, a small share are hard. Paying frontier-model prices for "summarize this ticket" is the most common waste we find.
- Cascade routing. Try the small model first; escalate to the frontier model only when a confidence check (self-consistency, schema validity, eval-scored sample) fails. For typical support workloads this cuts generation spend substantially at near-identical measured quality — because the eval suite, not vibes, defines "identical".
- Semantic caching. Repeated near-identical queries (password resets, status checks) hit cache instead of the model. Cache hits must be validated against the eval set: a cached answer that is wrong for a slightly different query is worse than an expensive correct one.
- Batch what is not latency-bound. Nightly summarization, embedding backfills, report generation — batch APIs price at a discount, and no user is waiting.
Step 4: Make Cost a Regression Metric
Cost optimizations decay silently. A prompt tweak here, a "temporary" frontier-model migration there — and three months later the spend is back. The fix is the same one that protects quality: add cost per task to the regression suite. Every change re-runs the eval set and reports both numbers — quality pass rate and cost per completed task. A change that improves quality at 2x cost, or cuts cost while failing cases, gets caught in review, not on the invoice.
This is the same discipline as our approach to load testing LLM systems, where the invoice is a load-test metric: retries and verbose failures burn budget exactly like they burn capacity.
Key Takeaways
- Attribute cost per pipeline span and per query class before optimizing anything — the invoice cannot tell you what to fix.
- Input tokens are the quiet majority in RAG: relevance-filtered retrieval, prompt caching, and history compaction are the first levers.
- Route by query class: cascades from small to frontier models, validated by the eval suite, cut spend without a quality trade-off.
- Make cost per task a regression metric alongside quality — cost drift is quality-drift's silent twin.
We build this cost-attribution and routing layer as part of our AI agent engineering practice — and the observability stack behind it comes from our performance engineering toolkit. If your LLM feature's economics have outgrown its value, that combined audit is the fastest way to fix it.
