Distributed Tracing for Your API With OpenTelemetry: Spans, Context Propagation and an OTLP Exporter

When a request is slow, the logs tell you every service was fine. The trace tells you which one was lying. Distributed tracing is the difference between “the checkout endpoint is slow sometimes” and “the checkout endpoint spends 340ms waiting on a single unindexed SQL query in the inventory service.” This is how you get there with OpenTelemetry on a FastAPI service, without buying anything.

OpenTelemetry (OTel) is the vendor-neutral standard for traces, metrics and logs. It is a CNCF project, and it matters because it decouples instrumentation from the backend: you instrument once, then point the data at Jaeger, Grafana Tempo, or a commercial platform, and swap between them without touching application code. The examples below use OpenTelemetry Python, FastAPI 0.115, Python 3.12 and an OTLP exporter into Jaeger.

The three things a trace is made of

Before the code, the vocabulary, because getting these wrong is where people flounder. A trace is one request’s journey through your estate. It is a tree of spans, where each span is a single unit of work with a start time, a duration, and a set of key-value attributes. Every span carries a context — a trace ID and its own span ID — and that context is what stitches spans together across process boundaries. The root span is created when the request arrives; child spans nest underneath it for the database call, the outbound HTTP call, the business logic you choose to measure.

The mechanism that carries context from one service to the next is propagation. OpenTelemetry’s default propagator implements the W3C Trace Context specification: it serialises the active context into a traceparent HTTP header on outbound requests and reads it back on inbound ones. That header is the entire trick. A downstream service that reads traceparent and continues the trace shows up as a child; one that ignores it starts a brand-new, disconnected trace — which is exactly the broken state most half-instrumented systems live in.

Wiring up the SDK and the exporter

Install the SDK, the OTLP gRPC exporter, and the auto-instrumentation packages for the libraries you actually use. Auto-instrumentation is the reason OTel is worth it: the FastAPI, HTTPX and SQLAlchemy packages create spans for every request, outbound call and query without you writing a line per endpoint.

pip install \
  opentelemetry-sdk \
  opentelemetry-exporter-otlp-proto-grpc \
  opentelemetry-instrumentation-fastapi \
  opentelemetry-instrumentation-httpx \
  opentelemetry-instrumentation-sqlalchemy

Now the tracing bootstrap. Keep this in its own module and import it before the app starts. The service.name resource attribute is not optional decoration — it is the name every backend groups spans under, and an unset one shows up as unknown_service, which makes a multi-service trace unreadable.

# telemetry.py
from opentelemetry import trace
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import (
    OTLPSpanExporter,
)


def configure_tracing() -> None:
    """Initialise the global tracer provider and OTLP exporter."""
    resource = Resource.create({"service.name": "checkout-api"})
    provider = TracerProvider(resource=resource)

    # BatchSpanProcessor buffers spans and flushes on a background
    # thread; never use SimpleSpanProcessor in production, it blocks
    # the request on every export.
    exporter = OTLPSpanExporter(endpoint="http://localhost:4317")
    provider.add_span_processor(BatchSpanProcessor(exporter))

    trace.set_tracer_provider(provider)

A note on the endpoint. 4317 is the OTLP/gRPC port. In production you would run an OpenTelemetry Collector as a sidecar or daemonset and export to that, rather than straight to a backend — the Collector handles batching, retries and fan-out to multiple destinations. But modern Jaeger (v1.35 and later) accepts OTLP natively on 4317, so for local work you can point the exporter directly at a Jaeger all-in-one container and skip the Collector entirely.

Instrumenting the app and the database

Call the instrumentors after the app and the database engine exist. FastAPIInstrumentor wraps the ASGI layer, so it creates the root server span and — critically — reads the inbound traceparent header for you. HTTPXClientInstrumentor injects that header on the way out. SQLAlchemyInstrumentor turns every query into a child span with the statement as an attribute.

# main.py
from fastapi import FastAPI
from sqlalchemy import create_engine
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor
from opentelemetry.instrumentation.httpx import HTTPXClientInstrumentor
from opentelemetry.instrumentation.sqlalchemy import SQLAlchemyInstrumentor

from telemetry import configure_tracing

configure_tracing()

app = FastAPI()
engine = create_engine("postgresql+psycopg://app:app@db/orders")

FastAPIInstrumentor.instrument_app(app)
HTTPXClientInstrumentor().instrument()
SQLAlchemyInstrumentor().instrument(engine=engine)

At this point you already have end-to-end traces: server span, SQL spans, outbound HTTP spans, all correctly parented, and context propagating to any downstream FastAPI service that is also instrumented. That is roughly twenty lines for a system that previously told you nothing about where latency went.

Custom spans for the steps that matter

Auto-instrumentation covers the plumbing. It cannot see your business logic — the fraud check, the pricing calculation, the call to a payment provider’s SDK. Those are the spans that make a trace tell a story, and you add them by hand with a tracer. Attach attributes that you will actually want to filter on later: the customer tier, the order value, the decision a branch took. Do not attach anything you would not want a supervisor or an engineer without data-access clearance to read — trace attributes are not the place for card numbers or personal data.

from opentelemetry import trace

tracer = trace.get_tracer(__name__)


async def process_order(order: Order) -> Receipt:
    with tracer.start_as_current_span("process_order") as span:
        span.set_attribute("order.id", order.id)
        span.set_attribute("order.value_pence", order.total_pence)

        with tracer.start_as_current_span("fraud_check") as fraud_span:
            decision = await run_fraud_check(order)
            fraud_span.set_attribute("fraud.decision", decision.outcome)

        if decision.outcome == "blocked":
            span.set_status(trace.Status(trace.StatusCode.ERROR))
            raise OrderBlocked(order.id)

        return await settle(order)

Because start_as_current_span uses the active context, the fraud_check span nests under process_order, which nests under the auto-created server span, which — if the request arrived from another instrumented service — nests under that caller’s span. The whole chain is one trace, and setting the span status to ERROR is what makes Jaeger colour the failed path red so you can find it in a haystack.

Where this pays off, and what to watch

Two operational realities. First, sample. Exporting every span from a high-throughput service is expensive and mostly redundant; configure a sampler (tail-based sampling in the Collector is the usual answer) so you keep all the errors and slow traces and a fraction of the rest. Second, tracing is one signal, not the whole picture — it tells you where a slow request went, while your delivery and reliability metrics tell you whether the system is getting better over time. Use both.

Getting this in early is cheap; retrofitting it during an incident is not. The same discipline applies to the other cross-cutting concerns on a FastAPI service — safe request retries and a baseline that runs in CI — none of which you want to be inventing under pressure. Instrument the service while it is calm, and the next time someone says the API is slow, you will have an answer instead of a theory.