Skip to the page
Tecyfy

OpenTelemetry for LLM apps: trace prompts, tools and cost

OpenTelemetry's GenAI conventions give every model call, tool use and agent step a standard span, with the tokens it used. How to turn tracing on for an LLM app, what the spans hold, how to keep prompts out of them, and how to add up tokens without counting them twice.

Tecyfy · Published · 8 min read

Woodcut of a glowing thread running through 3 small machine beads in a row, with the word Trace above them

OpenTelemetry traces an LLM app the way it traces any other service: each step becomes a span, and the spans of one request form a trace. What makes it work for model calls is a set of shared names, the GenAI semantic conventions. A model call becomes a span such as chat claude-sonnet-5-5 with gen_ai.usage.input_tokens and gen_ai.usage.output_tokens on it, a tool call becomes execute_tool search_tickets, and any platform that knows the conventions can show them without custom parsing.

That is the case for using it over a vendor's own tracing format: the same spans go to Datadog, Grafana, Honeycomb, Langfuse or your own collector, and your dashboards survive a change of model provider. Anthropic adopted it for Claude on 14 October 2026, when it launched OpenTelemetry export in beta for the Claude API and its SDKs.

What a trace of an LLM app looks like

A trace of one agent request nests like this, using the span names the conventions recommend:

invoke_agent support_agent                 3.5 s
├─ chat claude-sonnet-5-5                  1.3 s   3,200 in / 180 out   finish: tool_use
├─ execute_tool search_tickets             0.4 s
└─ chat claude-sonnet-5-5                  1.7 s   4,100 in / 260 out   finish: end_turn

From that you can read where the time went (the model, not the tool), what the request cost in tokens, and why the first call stopped (it asked for a tool). With content recorded, you can also read what the user asked and what the model said. That is what the TracingTracingRecording every step an LLM app or agent takes for one request, each model call, tool call and retrieval with its input, output, tokens and time, so you can see why an answer went wrong. concept means by a trace: the record of one request through every step, which you then turn into ObservationObservationA piece of information an agent receives after executing an action. data for debugging and evals.

The widget below is that trace. Press each span to see its attributes, switch content on and off, and add up its tokens 2 ways.

Try itOne agent request as a trace
Span
  1. invoke_agent support_agent3480 ms
  2. chat claude-sonnet-5-51310 mscounted
  3. anthropic.messages.create1240 mscounted
  4. execute_tool search_tickets420 ms
  5. chat claude-sonnet-5-51650 mscounted
  6. anthropic.messages.create1580 mscounted

chat claude-sonnet-5-5 · kind CLIENT

gen_ai.operation.name
chat
gen_ai.provider.name
anthropic
gen_ai.request.model
claude-sonnet-5-5
gen_ai.usage.input_tokens
3200
gen_ai.usage.output_tokens
180
gen_ai.response.finish_reasons
["tool_use"]
gen_ai.system_instructions
(not recorded)
gen_ai.input.messages
(not recorded)
gen_ai.output.messages
(not recorded)

No prompt text leaves

Summing every span that carries tokens gives 14,600 input tokens. That is twice the real figure: each model call has a client span from the SDK and a server span from the API, both with the same usage.

A made-up trace in the shape of OpenTelemetry's GenAI conventions and the Claude SDK's spans. Attribute names follow the conventions; values are invented.

The span names and attributes that matter

The GenAI conventions have Development status, so names can still change. They live in their own repository, open-telemetry/semantic-conventions-genai, with general conventions plus provider-specific ones for Anthropic, OpenAI, AWS Bedrock and Azure AI Inference, and pages for MCP and the A2A protocol.

Model calls. The client inference span should be named {gen_ai.operation.name} {gen_ai.request.model}, with kind CLIENT. Its attributes, by requirement level:

AttributeLevelHolds
gen_ai.operation.nameRequiredchat, execute_tool, invoke_agent and other known values
gen_ai.provider.nameRequiredThe provider, such as anthropic
gen_ai.request.modelIf availableThe model asked for
gen_ai.response.modelRecommendedThe model that answered
gen_ai.response.finish_reasonsRecommendedWhy it stopped
gen_ai.usage.input_tokensRecommendedInput tokens, cached ones included
gen_ai.usage.output_tokensRecommendedOutput tokens
gen_ai.usage.cache_read.input_tokensWhen availableTokens read from the prompt cache
gen_ai.usage.reasoning.output_tokensWhen availableTokens spent on reasoning
gen_ai.input.messages, gen_ai.output.messagesOpt-inThe prompt and the reply

Tool calls. An execute_tool {gen_ai.tool.name} span, with gen_ai.tool.name required and the tool call id when the model gave one. The arguments and the result are opt-in.

Agents. An agent run gets an invoke_agent {gen_ai.agent.name} span over its steps, and gen_ai.conversation.id ties the spans of one conversation together across requests.

Turn it on in Python

Most of the work is OpenTelemetry's ordinary setup: a tracer provider and an exporter. Install the SDK and the OTLP exporter:

pip install --upgrade anthropic opentelemetry-sdk opentelemetry-exporter-otlp-proto-http

Then register the provider before the first call you want traced. A current Claude SDK finds it and emits a span for each request, and one for each tool its tool runner runs:

from anthropic import Anthropic
from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor

provider = TracerProvider()
provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter(endpoint="http://localhost:4318/v1/traces")))
trace.set_tracer_provider(provider)

client = Anthropic()
tracer = trace.get_tracer("support-agent")

with tracer.start_as_current_span("invoke_agent support_agent") as span:
    span.set_attribute("gen_ai.operation.name", "invoke_agent")
    span.set_attribute("gen_ai.agent.name", "support_agent")
    message = client.messages.create(
        model="claude-sonnet-5-5",
        max_tokens=1024,
        messages=[{"role": "user", "content": "Why was I charged twice?"}],
    )

The SDK's span nests under your invoke_agent span, so the model call appears inside the request that caused it. The endpoint here is an OpenTelemetry Collector on its default HTTP port; point it at your platform's OTLP endpoint instead. Your own tools get spans the same way: wrap each in execute_tool <name> with gen_ai.tool.name set.

Anthropic's SDKs do the same in TypeScript, C#, Go, Java, PHP and Ruby. During the beta it warns that options and span names can change in a new SDK version, and advises pinning the version so a release does not break your dashboards.

For spans from inside the API, such as each model run within a request and the server tools it ran, an admin adds an OpenTelemetry destination in the Claude Console. With both on, the SDK and API spans of a request join in one trace.

Claude Code has its own tracing, also in beta and off by default: set CLAUDE_CODE_ENABLE_TELEMETRY=1, CLAUDE_CODE_ENHANCED_TELEMETRY_BETA=1 and OTEL_TRACES_EXPORTER. Our Claude Code costs page covers what it spends.

Keep prompts out unless you mean to store them

Prompts and replies are the most useful part of a trace and the most sensitive. The conventions make every attribute that holds them opt-in, and say plainly that they are likely to contain personal data. The tools follow:

  • The Claude SDK and API default to metadata only: identifiers, timing, status, token counts and the names you chose for agents, tools and MCP servers. Full content adds the system prompt, the messages and the reply.
  • Claude Code redacts prompt text, tool input and tool content in its spans unless you set OTEL_LOG_USER_PROMPTS, OTEL_LOG_TOOL_DETAILS or OTEL_LOG_TOOL_CONTENT.

Anthropic points out the catch with full content: your observability platform stores it under its own access and retention rules, so check those before you switch it on. The conversation id is exported in both modes, so keep personal data out of it too.

Size is the other limit. With full content, Anthropic caps each span at 0.75 MB by default, enough for a 200k-token context of English text, and below the 1 MB request limit of many platforms. A full 1M-token context needs about 4 MB, so raise the limit only if your platform accepts it. When content is cut, the span says so with anthropic.content.truncated.

Add up tokens once, then price them

A trace makes cost per request, per user or per feature easy to see, because the token counts sit on the spans next to everything else. The conventions record tokens, not money: multiply by the model's prices yourself, from our model price table.

2 traps make the sum wrong:

  1. The same call appears twice. With both SDK and API tracing on, each Claude request has a client span from the SDK and a server span from the API, both named anthropic.messages.create and both carrying the usage. Anthropic's rule is to sum one source per request: the API spans when you have both, told apart by span kind (CLIENT or SERVER). The conventions say the same for any provider: group inference spans by service.name when counting.
  2. Cached tokens are already inside input tokens. gen_ai.usage.input_tokens includes the tokens read from and written to the prompt cache. Adding cache_read on top counts them twice; instead, split them out and price them at the cache rates, which on Claude are lower for reads and higher for writes. The prompt caching calculator shows what that difference is worth.

Spans can also arrive twice when a delivery is retried, so count each by its trace and span id. And traces are not a bill: Anthropic says delivery is best effort and a span can be lost, and points to its Usage and Cost Analytics API for a complete record. Use traces to explain spend, and the provider's usage data to total it. Our worked AI agent cost per month shows what those totals add up to for a real agent.

Where traces pay off

The first win is usually debugging: a slow request shows whether the model, a tool or a retry took the time, and a wrong answer shows the tool result it was built on. The second is evals. With content recorded, failed traces become test cases, which is how our guide to evaluating an AI agent builds its test set. Start with metadata only, find the requests worth reading, and record content only where you need it.

Questions

What is OpenTelemetry for LLMs?
It is OpenTelemetry, the vendor-neutral standard for traces, metrics and logs, used with its semantic conventions for generative AI. The conventions name the spans and attributes for model calls, tool executions and agent steps (for example gen_ai.request.model and gen_ai.usage.input_tokens), so any observability platform that supports them can read an LLM app's traces.
Are the OpenTelemetry GenAI semantic conventions stable?
No. The GenAI semantic conventions have Development status, so names and attributes can still change. They now live in their own repository, open-telemetry/semantic-conventions-genai, which extends the core semantic conventions.
What span does an LLM call get in OpenTelemetry?
A client inference span, named after the operation and the model, such as chat claude-sonnet-5-5. It must carry gen_ai.operation.name and gen_ai.provider.name, and is recommended to carry the request and response model, finish reasons and the input and output token counts. A tool call gets a span named execute_tool followed by the tool's name.
Does OpenTelemetry record the prompt and response?
Not by default. In the GenAI conventions the attributes that hold prompts, responses, system instructions and tool arguments are opt-in, because they can hold personal data. The Claude SDK and Claude API follow the same idea: metadata only is the default, and full content has to be switched on.
How do I trace the Claude API with OpenTelemetry?
Register an OpenTelemetry tracer provider in your app and use a current Claude SDK, which then emits a span for each request and each tool the tool runner runs. For spans from the API itself, an admin adds an OpenTelemetry destination in the Claude Console. Both are in beta since 14 October 2026.
How do I get the cost of an LLM call from a trace?
Add up the token attributes of the model call spans, counting each call once, and multiply by the model's prices. Input tokens already include tokens read from and written to the prompt cache, which are priced differently, so split those out with the cache attributes before applying prices.