Your API Can Handle 5,000 RPS. Your LLM Can't Handle 30.
Teams that already run load tests eventually turn to their shiny new LLM feature and ask the
habitual question: "how many requests per second can it take?" The load test fires 200
concurrent completions, the provider starts returning 429 Too Many Requests, the
p95 latency lands at 45 seconds, and the invoice for the afternoon is four figures. The test
"failed" — but it failed because the question was wrong, not because the system is weak.
Load testing LLM applications is a different discipline from load testing conventional APIs. The difference is not that LLMs are slow. It is that the three assumptions classic load testing is built on — constant response cost, monotonic capacity, and stable quality under load — do not hold. This piece is the methodology we use when load testing agent and RAG systems for clients.
Three Assumptions That Break
1. Response cost is constant. A CRUD endpoint costs roughly the same whether it returns one row or fifty. An LLM completion can produce 10 tokens or 2,000 — a 200x difference in generation time and cost. Under concurrent load, output length also drifts: degraded context handling, prompt-response feedback loops, and retry storms change what the model produces. Your load profile is not "requests per second"; it is "requests × token distribution per second."
2. Capacity is monotonic. Conventional systems degrade gracefully: more load, longer queues, higher latency, eventually errors. LLM endpoints have provider-side rate limits (RPM and TPM tiers) that convert excess load into immediate hard failures — not queues. Between "fine" and "429 for everyone" there is often no middle ground, and the boundary moves as the provider rebalances capacity.
3. Quality is constant under load. The nastiest one. Timeouts, truncated contexts after aggressive retrieval under memory pressure, and silent retry logic can degrade answer quality without any error being thrown. A load test that only measures latency can report green while the system returns garbage.
What to Model Instead: Tokens, Not Clicks
The unit of load for an LLM system is the token flow, in both directions:
- Input tokens per minute — hits the provider's TPM limit; driven by prompt and context size, especially RAG chunks.
- Output tokens per minute — also counts against TPM; driven by generation length, which you must sample from a realistic distribution, not fix.
- Requests per minute — the RPM limit; usually hit first by agentic loops, where one user action triggers 5–20 API calls.
- Concurrent generations — the provider's soft capacity; self-hosted models fail here first (VRAM exhaustion shows as OOM or queueing).
Build the profile from production telemetry: the distribution of input sizes, the distribution of output lengths (sample from it per request — do not replay a constant), and the call multiplier of agentic workflows. Percentiles of output length matter more than the mean, because p95 generation time sets your user-perceived latency.
Metrics That Matter for LLM Systems
| Metric | Why it matters | Typical target |
|---|---|---|
| Time to first token (TTFT) | Perceived responsiveness in streaming UIs; users tolerate slow completion but not silence | < 1–2 s |
| Tokens per second (streaming) | Reading speed of the stream; below ~10 tok/s a chat UI feels broken | > 20 tok/s |
| End-to-end p95 completion time | The SLA your users actually experience, including retrieval and tool calls | Per product SLA |
| 429/5xx rate under stage load | Provider limits convert overload into hard failures; retry storms amplify them | 0 at 100% profile |
| Quality sample pass rate | Catches silent degradation — answers that are wrong, truncated, or empty | ≈ unloaded baseline |
| Cost per completed task | Retries and verbose failures burn budget; the invoice is a load-test metric | Stable across stages |
The quality sample is the step almost everyone skips, and it is cheap: from each load-test stage, take a random sample of responses and run them through your evaluation suite (deterministic checks first, LLM-as-judge for the rest). If pass rate drops while latency holds, the system is quietly degrading — usually via truncated retrieval contexts or broken tool outputs under concurrency.
A Stage Sequence That Works
- Baseline single-stream. One user, production-realistic prompts. This is the reference for TTFT, tok/s, output-length distribution, and quality pass rate.
- 100% of the modeled profile. Concurrent streams sampling the real distributions, agentic call multipliers included. SLOs must hold here with zero 429s.
- 120–150% stages. Find where the first SLO breaks. For provider-backed systems, expect the failure mode to be rate-limit cliffs; for self-hosted, queueing and then OOM. The boundary is your MaxPerf.
- Failure-mode drills. Deliberately exhaust the provider limit and watch your retry logic: exponential backoff with jitter, or a thundering herd that turns one limit event into a ten-minute outage? Also test provider failover if you have a fallback model.
- Quality sampling per stage. Compare against the baseline pass rate. Any drop is a finding, even if all latency SLOs are green.
Common Failure Modes We See in Engagements
- The retry storm. Client libraries default to aggressive retries. Under a rate limit, every client retries N times, multiplying effective load exactly when the provider can least afford it. Cap concurrency, jitter backoff, and shed load client-side.
- RAG context inflation. Retrieval that returns top-k chunks regardless of relevance quietly inflates input tokens; under load, TPM limits arrive early and per-request cost doubles. Budget retrieval per query class.
- Agentic multipliers. One user request becomes 15 LLM calls and 40 tool calls. Load models that treat "user" and "LLM request" as 1:1 underestimate load by an order of magnitude. Instrument the multiplier; it is the single most load-relevant number in an agent.
- Silent quality decay. Everything green, answers worse. Found only if quality sampling runs per stage — see above.
Where the Discipline Fits
Load testing LLM applications is where our two service lines meet: the performance engineering methodology — profiles, stages, SLO gates, capacity verdicts — adapted for token flow, and the agent engineering practice that supplies the eval suite used for quality sampling. If you are about to put an LLM feature under real traffic and want a capacity number you can take to planning — not a bill shock story — that combination is exactly what we deliver.
