The Demo That Lies
Every team building agents eventually presents one: the agent handles five tricky requests in a row, the room nods, the project gets green-lit. Three months later the same system is drowning in production escalations. Nobody lied in the demo — but nobody measured the right thing, either.
There are two fundamentally different questions you can ask about an agentic system:
- Can it succeed at all? If we try enough times, does a correct solution come out?
- Does it succeed every time? If a real user runs it once, will it work?
The first is an exploration metric. The second is a reliability metric. They are related, but they are not interchangeable — and the gap between them is where failed agent projects live.
Capability Ceiling vs Business Reliability
Call the first family Pass@k: the probability of at least one success in k attempts. Call the second Pass^k: the probability that all k attempts succeed. The math is unforgiving. With a per-attempt success rate of 60%:
- Pass@5 is already around 99% — the system looks flawless if retries are free.
- Pass^5 is around 8% — a five-step pipeline succeeds end-to-end less than once in ten runs.
Same system. Same 60%. One number sells the project, the other predicts its fate. Which one you should optimize depends entirely on your task's retry economics:
- Retries are cheap and private (drafting an email, brainstorming names, generating code variations for a developer to pick from) — optimize Pass@k and let the user or the harness pick the best result.
- Every attempt is visible, costly, or irreversible (customer-facing support, payments, long-running pipelines, actions with side effects) — optimize Pass^k, because the user experiences the single roll of the dice, not your leaderboard.
Fix the Measurement Before the Model
Once you know which number matters, a practical evaluation loop looks like this:
- Define tasks from real traffic. Not toy prompts — the actual requests your users made last month, including the weird ones.
- Grade mechanically where possible. Unit tests for code, schema validation for structured output, deterministic checkers for facts. Save LLM-as-a-judge for the fuzzy remainder, and spot-check the judge itself against human labels.
- Measure per-attempt success first. That single number powers both families of metrics, so it is the cheapest thing to track and the most informative.
- Attribute failures before tuning anything. Is the model reasoning badly, or is the tool returning garbage, or is the context missing a fact it needs? The fix differs wildly: better harness, better tools, better context — or only then, a different model.
- Re-run the suite on every change. Prompt tweak, model upgrade, tool refactor — each is a regression risk that only an eval suite can catch.
What This Means for the Roadmap
The eval numbers also tell you where to invest. Low per-attempt success on well-specified tasks usually means the harness or context is weak — fix that before shopping for models. Unstable output format points at fine-tuning with demonstration data. A measurable reward signal with headroom for improvement is the precondition for reinforcement-learning-style optimization. Teams that skip the diagnosis tend to run expensive training loops on problems that a one-line prompt change would have solved.
Key Takeaways
- Pass@k measures capability ceiling; Pass^k measures reliability. They diverge fast — a 60% step is 99% Pass@5 and 8% Pass^5.
- Choose the metric from retry economics: cheap invisible retries favor Pass@k, user-facing single-shot flows demand Pass^k.
- Build the eval set from real traffic and grade mechanically wherever possible.
- Attribute failures to harness, tools, or context before blaming the model.
- An agent project without a regression eval suite is a demo, not a product.
