From Benchmark Tourism to a Regression Suite
Public leaderboards tell you which model is generally impressive. They cannot tell you whether your agent got better after Tuesday's prompt change. Production agent teams converge on the same realization: the eval infrastructure that matters is internal, small, and boring — a regression suite built from your own traffic, run automatically on every change.
Here is the full pipeline, from raw failures to a system that improves itself.
Step 1: Harvest Tasks From Reality
The eval set must come from production, not from the team's imagination. Three sources, in descending value:
- Escalations. Every session a human had to rescue is a task the agent failed — with the trace attached. These are golden: real distribution, real messiness, known failure mode.
- Anomalies. Budget blowouts, loop detections, tool-error spikes — each yields a task plus the diagnostic question ("does the loop still form?").
- Sampled routine traffic. A small random sample keeps the suite honest about the common case, not just the pathological one.
Aim for 50-200 tasks to start. Below that, signal drowns in noise; above that, maintenance cost bites. Quality of tasks beats quantity every time.
Step 2: Grade Mechanically Wherever Possible
The grading pyramid, most reliable at the base:
- Programmatic checks — unit-test-style assertions: file exists, JSON parses, number matches, query returned rows. Zero ambiguity, zero cost per run. Design tasks so more of them can be checked this way.
- Rule-based rubrics — required sections present, citation links resolve, tone keywords absent. Still deterministic, slightly fuzzier.
- LLM-as-a-judge — for genuinely fuzzy qualities (helpfulness of an explanation). Necessary, but the judge is itself a model with biases: spot-check it against human labels regularly, watch for position bias and verbosity bias, and never let a single judge grade its own family's outputs uncalibrated.
Step 3: Read the Numbers Like a Statistician
Two mistakes make eval dashboards lie:
- Small-sample theater. 30 tasks, 70% vs 73% success — that difference is noise. Before celebrating a "3% improvement", compute the confidence interval; with task-level binomial variance, n=30 can barely distinguish 60% from 80%. Either grow the set or only trust large deltas.
- Averaging over a bimodal distribution. If half the tasks score 100% and half score 0%, the mean describes nothing. Slice results by task category and difficulty; a regression confined to one slice is invisible in the aggregate.
Step 4: From Score to Action
A score without attribution is trivia. When the suite catches a regression or a persistent failure cluster, the diagnostic chain is: which slice → which step (retrieval? tool? context assembly? reasoning?) → what did the trace show? The trace (full execution tree) is what turns "the agent got worse" into "retrieval stopped returning the pricing chunk after the chunking change". This is why evals and observability must share identifiers — the eval case should link directly to the production trace it came from.
Step 5: Simulation, the Bridge to Training
Once the suite is trusted, it becomes an environment: the same tasks, run continuously against candidate versions — new prompts, new models, new harness code — before they touch users. And if you ever graduate to reinforcement learning, the simulation you built for evaluation becomes the RL environment. Evaluation infrastructure is not a testing cost center; it is the training ground for every future capability.
A Worked Example: The First 90 Days
For teams starting from zero, here is what the first three months look like when done properly — no benchmark tourism, straight to a regression suite:
- Weeks 1–2: mine the escalations. Pull every support escalation and internal complaint from the last quarter. Extract the 10–20 recurring failure patterns. Each becomes a named eval case with the original trace attached. Cost so far: one engineer, no model calls.
- Weeks 3–4: grade mechanically. For each case, find the cheapest check that catches the failure: exact string for IDs, JSON schema for format, unit tests for computations, an LLM judge only for tone and coverage — and validate the judge against ~100 human-labeled examples before trusting it.
- Month 2: wire it into CI. The suite runs on every prompt, model, or tool change. Gate: no release if any named case regresses. Add sampling — 1% of production traffic replayed nightly — to catch drift the named cases miss.
- Month 3: close the loop. New escalations become new cases within a week (the pipeline from Step 1). The dashboard shows score by case, trend by release, and — most important — which product decisions the suite has blocked or approved.
The end state: when a vendor ships a new model version, the team knows within a day whether to upgrade — with numbers per failure mode, not vibes. That decision speed is the actual product the eval suite delivers.
Key Takeaways
- Build the eval set from escalations, anomalies, and sampled traffic — reality, not imagination.
- Grade with the most mechanical check that works; treat the LLM judge as a model that needs calibration.
- Respect statistics: small n lies, and bimodal means mislead.
- Every eval case links to the production trace it came from — attribution over scores.
- The trusted internal suite is the bridge to simulation and, later, to training environments.
