Back to Blog
Engineering

Monitoring LLM Agents: Prometheus & Grafana Setup

October 30, 20266 min read · 691 words
ObservabilityAI AgentsGrafana

Dashboards Nobody Built for LLMs

Grafana and Prometheus were built for services whose vital signs are request rate, error rate, and duration. An LLM pipeline has all of those — plus a second axis the dashboards in every tutorial ignore: tokens, which map directly to latency, capacity, and money. A team that monitors an agent with a generic RED dashboard is blind to the two failure modes that actually hurt: token-flow saturation (rate limits, cost) and silent quality drift.

This is the concrete Prometheus + Grafana setup we deploy for agent and RAG systems — metric names, labels, alert rules, and the panel layout. Adapt the prefixes; keep the semantics.

The Metric Set: RED Plus Tokens

Standard RED metrics, labeled per pipeline stage (retrieval, generation, rerank, tool): 

llm_requests_total{stage, model, query_class, outcome}
llm_request_duration_seconds{stage, model, query_class}  # histogram
llm_tool_calls_total{tool, outcome}

Then the LLM-specific series that generic setups miss: 

llm_tokens_total{direction="input|output", model, query_class}   # counter
llm_time_to_first_token_seconds{model}                            # histogram
llm_generation_tokens_per_second{model}                           # histogram
llm_cost_dollars_total{stage, model, query_class}                 # counter
llm_context_tokens{stage, query_class}                            # gauge: prompt size sampled per request
agent_loop_steps_total{query_class}                               # counter: calls per user action
agent_eval_pass_rate{query_class}                                 # gauge: from the eval suite, per scheduled run

The two non-obvious ones: llm_context_tokens as a gauge catches RAG context inflation early — if the p90 climbs after a "harmless" retrieval tweak, your TPM headroom just shrank. And agent_eval_pass_rate is exported by a scheduled job that runs the eval suite against production samples: it is the only series that catches silent quality drift, because every latency metric stays green while answers get worse.

Cost as a First-Class Counter

llm_cost_dollars_total is incremented per call: tokens × the model's price tier, computed in code where the token count is known. This single series powers the three Grafana panels that end invoice surprises:

  • Cost per query class, rate over 6h — shows which feature is burning budget right now.
  • Cost per completed task — total cost divided by successful tasks; the number that belongs in planning, not raw spend.
  • Cost by stage, stacked — the breakdown that tells you whether to fix retrieval, prompts, or routing.

This is the observability half of the cost methodology described in our LLM cost optimization guide — without per-span cost counters, that methodology has no data to act on.

Alerts That Earn Their Noise Budget

Most LLM alerting fails in one of two directions: silence during real incidents, or pager spam from a single slow provider response. These five rules survived our production usage: 

# 1. Provider saturation — the earliest warning you get
increase(llm_requests_total{outcome="rate_limited"}[5m]) > 0

# 2. Loop detection — same tool called 5+ times in one session
increase(agent_tool_calls_total{outcome="duplicate"}[10m]) > 3

# 3. Cost anomaly — burn rate 3x the 24h baseline
rate(llm_cost_dollars_total[1h]) > 3 * rate(llm_cost_dollars_total[24h])

# 4. Quality drift — eval pass rate dropped vs weekly baseline
agent_eval_pass_rate < 0.9 * agent_eval_pass_rate offset 1w

# 5. Context inflation — p90 prompt size grew 30% since yesterday
histogram_quantile(0.9, rate(llm_context_tokens_bucket[1h]))
  > 1.3 * histogram_quantile(0.9, rate(llm_context_tokens_bucket[1h] offset 1d))

Rules 4 and 5 are the pair nobody runs and everyone needs: they fire while the product still works, which is exactly when fixes are cheap.

The Dashboard Layout

One screen, four rows, in the order you investigate an incident:

  1. Health: success rate by stage, p95 duration by stage, 429/5xx rate, loop alerts.
  2. User experience: TTFT p50/p95, streaming tok/s, end-to-end completion time by query class.
  3. Economics: cost rate by stage (stacked), cost per task, tokens in/out by model.
  4. Quality: eval pass rate trend, eval pass rate by query class, count of production failures converted to eval cases this week.

Rows 3 and 4 are what make this an LLM dashboard rather than a web-service dashboard wearing an LLM costume. If your on-call engineer can answer "is it broken, is it slow, is it expensive, is it wrong" from one screen, the setup is done.

How This Fits the Broader System

The dashboards consume what the pipeline emits; the pipeline design and the eval suite feeding rule 4 come from our AI agent engineering practice. The conceptual foundation — traces as source of truth, feeding production failures back into evals — is covered in the observability companion piece. And if your bottleneck turns out to be capacity rather than correctness, performance engineering as a service is the same stack pointed at throughput.

Related Articles