How do you monitor an LLM application in production?

Trace every request end to end (prompt version, model, retrieved context, tool calls, tokens, latency, cost, outcome), watch operational metrics like any service, and add quality signals that ordinary monitoring lacks: automated grading of sampled traffic, user feedback, guardrail hits and drift in the kinds of input you get. Then close the loop by turning bad cases into evaluation tests.

1. Tracing

Record each request as a trace with a span per step: retrieval, reranking, each model call, each tool call. OpenTelemetry has semantic conventions for generative AI (spans, metrics and events for model calls, agents and MCP), so traces from different libraries and providers line up in one tool. Store with each model call:

• model and parameters, prompt template version,
• input, output and cached tokens, cost,
• time to first token and total latency,
• stop reason (finished, max tokens, refusal, tool use), errors and retries.

Mask personal data before it reaches the tracing system.

2. Operational metrics

• Error and timeout rates by provider and model; rate-limit hits.
• p50 and p95 latency per step.
• Cost per request, per feature and per customer, with budget alerts.
• Cache hit rate, if you use prompt caching.

3. Quality signals

• Sampled grading: run a share of live traffic through the same checks as your evaluation set (faithfulness, format, policy) with code and an LLM judge.
• User feedback: thumbs, corrections, "talk to a person", retries and abandoned sessions.
• Refusal and "don't know" rates: a sudden change usually means a prompt or model change went wrong.
• Guardrail hits: what was blocked, and how many were false alarms.
• Agents: turns per task, failed tool calls, tasks that hit limits.

4. Drift

Users start asking about new things, documents change, a provider updates a model. Track the topics or intents of incoming requests and retrieval scores over time, and re-check quality when they shift.

5. Alerts that matter

Cost spike, latency spike, error spike, a drop in the graded quality score, a jump in refusals or guardrail blocks. Alert on changes from the normal level, not only fixed thresholds.

6. Close the loop

Every bad production case gets reviewed, labelled and added to the evaluation set. Release changes behind a flag or to a slice of traffic, and compare quality and cost against the current version before full rollout.

Summary: "Traces tell me what happened, sampled grading tells me whether it was good, and the eval set makes sure it stays fixed."