What is tracing in an LLM app or an AI agent?
Recording every step an LLM app or agent takes for one request, each model call, tool call and retrieval with its input, output, tokens and time, so you can see why an answer went wrong.
intermediate · Also called LLM Tracing, Agent Tracing, Trace, LLM Observability
A trace is the full record of what an LLM app did for one request. Every step becomes an entry, called a span: the call to the model with the prompt as it was sent and the answer, each tool the AgentAgentA model that uses tools in a loop, deciding each next step from what the last one returned, until the task is done. called with its arguments and result, each search of the documents with what came back, and for every one of them the time it took, the tokensTokenThe small piece of text a language model reads and writes, often part of a word. Prices, limits and the context window are all counted in tokens. and the cost. The spans nest, so you can see that the slow part of a 9-second answer was the second search, not the model.
Ordinary logs say that something happened. A trace says why the answer came out the way it did.
Why an LLM app needs it
When a normal program gives a wrong result, you can usually run it again and step through it. An LLM app may not give the same answer twice, and an agent's path depends on what each tool returned. Without a record of that run, a complaint like "the assistant told me my order was cancelled" cannot be explained. With the trace you can see whether the tool returned the wrong order, the search found an old policy, or the model misread a correct answer.
Traces are also where evals start. You learn how an app fails by reading many traces, writing a note on each one that went wrong and grouping the notes, and the failures you find become the cases of your Eval SetEval SetThe list of test cases an LLM app is graded on after every change. Each case is an input and what a good answer must do, so a score going up or down shows whether the change helped.. Later, an LLM judgeLLM as a JudgeUsing a language model to grade another model's output against a rubric, for qualities code cannot check. It scales to thousands of answers, but must be checked against a person's verdicts before it is trusted. can grade a sample of live traces every day, and the failures it flags go back into the set.
What to record
For each model call: the model, the full prompt and the answer, the stop reason, the input and output tokens (cached ones included), the request id the provider returned, and the time. For each tool call: its name, arguments, result and any error. For each retrieval: the query and the ids of the chunks it returned. Tie them together under one id per request, and keep the user's id and the version of your prompt, so you can find every trace a change affected.
Traces hold what users typed and what the model saw, which can include personal data. Decide what is masked and how long traces are kept before you start storing them.
A standard shape
You can write traces to your own database, but a standard shape lets any tool read them. OpenTelemetry, the common standard for application traces, has conventions for generative AI, still marked as in development. A span for a model call is named for the operation and the model, such as chat claude-opus-5-5, with attributes like gen_ai.request.model and gen_ai.usage.input_tokens, and there are operations for tool calls, retrieval, embeddings and agents. Writing traces in it keeps you free to change tracing tools later.
Sources