Back to Blog
AI/ML

Agent Observability: Traces, Cost Attribution, and Catching Loops in Real Time

October 5, 20264 min read · 775 words
AI AgentsObservabilityLLM

The $2.10 Loop

A real monitoring dashboard pattern from agent products in production: today's spend looks normal, until the anomaly list surfaces task #892 — an agent that called the same search tool fourteen times in a loop and burned a few dollars on a question worth cents. Multiply by thousands of sessions, and "the AI bill" becomes a line item nobody can explain.

Agent observability is the discipline that makes that bill explainable — and, more importantly, makes quality regressions visible before customers file them. It rests on three primitives: traces, metrics, and evals wired to the same identifiers.

Task #892 session · task · span IDs $2.10 total LLM call planning — $0.42 LLM call answer — $0.31 LLM call retry — $0.38 search_tool 240 ms · $0.02 search_tool ×14 loop alert
One trace of task #892: every LLM and tool call carries its own cost, and the loop detector flags the search tool repeated fourteen times — the $2.10 pattern, caught in real time.

Traces: The Execution Tree Is the Source of Truth

A session is not a string of text. It is a tree: one task, containing LLM calls, each with model, latency, token counts, and cost; tool calls nested under them, each with arguments, results, and duration. When something goes wrong — a wrong answer, a budget blowout, a loop — the reconstruction question is always the same: what exactly did the agent see and do? Only a full trace answers it.

Practical requirements for traces:

  • Completeness. Every LLM call and every tool call with inputs and outputs. A missing tool result in the trace is a debugging dead end — and, as failure post-mortems show, its loss in the live trajectory can outright cause infinite loops.
  • Correlation IDs. One ID threads session, task, and each sub-call, so cost and failures roll up cleanly.
  • Cost per span. Tokens per call, price per token, computed at write time. Aggregate cost questions are unanswerable retroactively if per-call costs were never recorded.

Metrics: The Few That Matter

Dashboards drown or mislead; a handful of series carries most of the signal:

  • Task success rate — measured by an automatic grader where possible, user signals where not. The headline number.
  • Steps per task — a silent regression indicator. When the median step count creeps up after a prompt change, something started wandering.
  • Tool error rate — per tool. Spikes localize failures to a specific integration instead of "the agent got worse".
  • Cost per task — and per outcome, not just per token. Cheap tokens that waste steps are expensive.
  • Loop detectors — explicit alerts on repeated identical tool calls. The #892 pattern is detectable in real time, not just in the post-mortem.

Wire Observability to Evals

Observability and evaluation are often built by different people and never meet. The productive move is to close the loop between them:

  • Harvest eval cases from production failures. Every escalated session or budget anomaly becomes a test case in the regression suite — with its trace attached. The eval set grows from reality, not imagination.
  • A/B with traces as the arbiter. When comparing prompts or models, diff the trace distributions: steps per task, tool choice frequencies, failure layers. Score deltas without trace deltas often mean the eval set is too small to trust.
  • Attribute cost before optimizing. In most agent systems two or three stages dominate spend — usually a bulky retrieval step or an over-verbose planning loop. Metered per-stage costs turn "we should optimize the agent" into "cut the planning loop's tool descriptions in half", which is an afternoon of work with a measurable payoff.

A Minimal Starter Kit

  1. Structured trace logging (JSON lines or a tracing backend) with session/task/span IDs and per-call cost.
  2. Four dashboard panels: success rate, median steps, tool error rate, cost per task.
  3. One alert: repeated identical tool calls within a session.
  4. A weekly ritual: pull the five worst traces of the week, turn them into eval cases, add to the suite.

None of this requires exotic infrastructure. It requires deciding that "the agent did something weird" is not an acceptable incident report — and that every weird thing an agent does should be reconstructible, measurable, and converted into a test.

Key Takeaways

  • Full execution traces (LLM + tool calls, with cost per span) are the foundation; everything else aggregates from them.
  • Track success rate, steps per task, tool error rate, and cost per task — five series, most of the signal.
  • Loop detection belongs in real-time alerting, not post-mortems.
  • Feed production failures back into the eval suite with traces attached.
  • Cost attribution by stage turns vague optimization plans into an afternoon of work.

Related Articles