Distributed Tracing an Agent Across Models, Tools and Services

A single agent request rarely does one thing. It calls a model, reads the reply, calls a tool, calls another model with the tool’s output, then reaches a downstream service that calls a model of its own. When it is slow or wrong, your logs show you fragments and nothing that connects them.

This is the problem distributed tracing was built for, and it is more acute for agents than for the request-response services the technique grew up on. A conventional web request fans out to a handful of predictable services. An agent run fans out dynamically: the number of model calls and tool invocations is decided at runtime by the model itself. You cannot know in advance how many hops a task will take, which means you cannot debug it from per-service logs. You need one trace per agent task that ties every hop together, and you need the token counts and errors to hang off the specific span that produced them.

What a trace is, in this setting

A trace is a tree of spans sharing one trace identifier. Each span is a timed unit of work with a name, a start and end, a parent, and a bag of attributes. For an agent, the root span is the task. Under it sit the model calls, the tool invocations, and the outbound calls to downstream services. Each of those services, if it is instrumented, contributes its own spans under the same trace. The result is one connected picture: where the time went, what each model call cost in tokens, and which span threw the error that failed the run.

The thing that makes the picture connected rather than a pile of disconnected trees is context propagation. Every span needs to know its parent, and when work crosses a process boundary the parent identifier has to travel with the request. Within a single process the tracing library carries that context implicitly. Across a network hop you have to inject it into the outbound call and extract it on the other side. Get propagation wrong and you do not get a broken trace you can spot; you get two separate traces that look complete and tell you nothing about how they relate.

Standardise on OpenTelemetry and its GenAI conventions

Do not invent your own trace format. OpenTelemetry is the vendor-neutral standard, it exports to every serious backend, and it now carries a set of semantic conventions specifically for generative AI. Using them matters because it makes your model and tool spans legible to tooling that did not know about your agent in advance, and it stops each team naming the same thing five different ways.

The conventions to know are the gen_ai.* attributes. A model call records gen_ai.operation.name (for example chat), gen_ai.provider.name, gen_ai.request.model and gen_ai.response.model, and crucially gen_ai.usage.input_tokens and gen_ai.usage.output_tokens. A tool invocation records gen_ai.operation.name of execute_tool and gen_ai.tool.name. An agent-level span carries gen_ai.agent.name. Put the token counts on the model span, not the root, and cost attribution becomes arithmetic over your trace store rather than a separate accounting exercise. This is the same instrumentation discipline that makes token budgets and per-unit chargeback possible without bolting on a second system.

These conventions are still settling. Treat the attribute names as current-but-moving, pin the semantic-convention version you build against, and expect to revisit as they stabilise.

One root span, and propagation at every boundary

The shape that works is a single root span per agent task with every model call, tool call and service hop nested beneath it. The one operation people get wrong is propagating context across the outbound HTTP boundary to a downstream service. OpenTelemetry does this with the W3C Trace Context standard: the current span is serialised into a traceparent request header, and the receiving service extracts it to continue the same trace. The snippet below shows the whole path in one place.

# pip install opentelemetry-sdk opentelemetry-exporter-otlp httpx
from opentelemetry import trace
from opentelemetry.propagate import inject
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
import httpx

provider = TracerProvider()
provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter()))
trace.set_tracer_provider(provider)
tracer = trace.get_tracer("agent.runtime")

def run_agent(task_id, prompt):
    # One root span per task - every hop hangs off this.
    with tracer.start_as_current_span("invoke_agent research-agent") as root:
        root.set_attribute("gen_ai.operation.name", "invoke_agent")
        root.set_attribute("gen_ai.agent.name", "research-agent")
        root.set_attribute("agent.task.id", task_id)
        plan = call_model("gpt-4o", prompt)
        return call_tool("search_docs", plan)

def call_model(model, response=None):
    # Child span for the model call; token counts live here.
    with tracer.start_as_current_span(f"chat {model}") as span:
        span.set_attribute("gen_ai.operation.name", "chat")
        span.set_attribute("gen_ai.provider.name", "openai")
        span.set_attribute("gen_ai.request.model", model)
        response = provider_sdk.chat(model)  # call the model here
        span.set_attribute("gen_ai.response.model", response.model)
        span.set_attribute("gen_ai.usage.input_tokens", response.usage.input_tokens)
        span.set_attribute("gen_ai.usage.output_tokens", response.usage.output_tokens)
        return response

def call_tool(tool_name, args):
    # Child span; the active context is injected into the outbound call.
    with tracer.start_as_current_span(f"execute_tool {tool_name}") as span:
        span.set_attribute("gen_ai.operation.name", "execute_tool")
        span.set_attribute("gen_ai.tool.name", tool_name)
        headers = {}
        inject(headers)  # writes the W3C traceparent header
        return httpx.post("https://search.internal/query", json=args, headers=headers)

The inject call is the whole game at the boundary. It reads the currently active span and writes traceparent into the headers you are about to send. The downstream service, if it runs OpenTelemetry, extracts that header automatically and its spans appear under your trace. If the downstream is third-party and unins­trumented, the trace stops at your outbound span, which is honest: you can still see the call was made and how long it took.

The async trap

Agents run tools concurrently, and concurrency is where propagation quietly breaks. The active context is stored per execution unit. When you hand work to a thread pool, a background task, or a callback, that unit does not inherit the calling context automatically, and the spans it creates become orphans under a fresh trace. The fix is to capture the context where the work is scheduled and re-attach it inside the worker before you open the span. In Python that means capturing with contextvars.copy_context() or passing the context explicitly and attaching it; most async frameworks have an OpenTelemetry integration that does this for you, and you should use it rather than hand-rolling the plumbing. Test it deliberately: fire two tool calls in parallel and confirm both land under the same trace, because this is the failure that looks fine in the happy path and falls apart under the load the feature exists to handle.

Reading the trace

Once the trace is whole, the questions that were impossible from logs become a glance at a waterfall. A latency spike is the widest bar, and you can see whether it was a slow model call, a slow tool, or a downstream service stalling. Cost is the sum of the token attributes across the model spans in the trace, per task and per agent. Failure is the span carrying the error status, with its parent chain showing what led there. Mean-time-to-resolution on an agent incident collapses when the first thing an on-call engineer opens is one connected trace rather than five log streams they have to correlate by timestamp — the same argument that makes MTTR worth instrumenting at all.

For a regulated firm the trace is also evidence. It is the record of what the agent did, in what order, at what cost, and where it failed — the substrate underneath any serious claim to agent observability, auditability and reversibility. The instinct to reach for tracing when the bill or the latency surprises you is the right one. The mistake is treating it as a debugging luxury you add later, because the propagation you skipped in the async path is exactly the context you will wish you had the first time an agent run goes wrong in production.