Lesson 8 of 8 · 13 min · Logs and traces

Distributed tracing with OpenTelemetry

Covers: OpenTelemetry Python SDK 1.x with OTLP/HTTP export, Jaeger v2 (jaegertracing/jaeger), three Python services under Docker. Updated 13 Nov 2026. Part of the DevOps Engineer roadmap.

What you will learn

  • Why metrics and logs are not enough in microservices
  • Traces, spans, attributes and context propagation (W3C traceparent)
  • Instrumenting with OpenTelemetry: SDK, exporters, auto-instrumentation, the Collector
  • Reading a trace to find slow or failing calls (tested example)
  • Sampling, cost, and joining traces with logs and metrics

In simple words

One click by a user may pass through ten services. When it is slow, which one caused it? A trace follows that single request across all of them and shows the time spent in each, like a courier tracking page that lists every hub the parcel passed through.

Words to know

  • Trace: the full journey of one request through all the services it touched.
  • Span: one step of that journey, with its start time and how long it took.
  • OpenTelemetry: the open standard and toolkit for collecting traces, metrics and logs.
  • Sampling: keeping only a share of the traces, to control cost.

The problem

A request to GET /checkout takes 464 ms. It passed through a frontend, an orders service and a payments service, each with its own metrics and logs. Metrics say checkout is slow; logs in each service say little. Which service, and which call inside it, took the time? A distributed trace records the whole journey of one request across services, with timings.

Concepts

  • A trace is the end-to-end record of one request (one trace id).
  • A span is one unit of work within a trace: an incoming HTTP request handled by a service, a database query, a call to another service, a function. A span has a name, start time, duration, status (OK/ERROR), attributes (key-values like http.route, db.system, http.status_code) and events (timestamped notes such as exceptions).
  • Spans form a tree through parent span ids: the frontend's span is the root; its outgoing HTTP call is a child; the orders service's server span is a child of that call; and so on.
  • Context propagation: for spans in different processes to join one trace, the trace id and parent span id travel with the request. The standard is the W3C Trace Context header:
traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
             │  └──── trace id (16 bytes) ────────┘ └─ parent span id ─┘ └ flags (sampled)

Every service reads it from the incoming request, creates its spans as children, and injects it into outgoing calls. If any service in the chain does not propagate it, the trace breaks into pieces: the most common tracing problem. (Message queues need the context in message headers.)

OpenTelemetry

OpenTelemetry (OTel) is the CNCF standard for generating telemetry: vendor-neutral APIs and SDKs for most languages, the OTLP wire protocol, semantic conventions for attribute names, and the OpenTelemetry Collector. It covers traces, metrics and logs; tracing is its most mature use. You instrument once and send the data to any backend: Jaeger, Grafana Tempo, Zipkin, or commercial products (Datadog, New Relic, Honeycomb, AWS X-Ray via the ADOT distro, Google Cloud Trace).

Ways to instrument:

  1. Auto-instrumentation (zero or little code): agents/libraries that wrap common frameworks, HTTP clients and database drivers (Java agent -javaagent:opentelemetry-javaagent.jar; opentelemetry-instrument python app.py; Node --require @opentelemetry/auto-instrumentations-node/register; .NET and Go options). Start here: you get spans for HTTP in/out and DB calls immediately.
  2. Manual spans around business operations (process_payment, calculate_price) and custom attributes (order.id, cart.items, user.plan), which make traces searchable by what matters to you.
  3. The Collector: a standalone service that receives OTLP, processes (batching, sampling, attribute filtering, redaction) and exports to one or more backends. Apps send to a local or cluster collector; changing backends needs no code change.

In Kubernetes, the OpenTelemetry Operator can inject auto-instrumentation via annotations; service meshes (Istio/Linkerd) and eBPF tools (Beyla, Pixie) provide some tracing without code changes.

A tested example: three services

We wrote one small Python service run as three instances: frontend calls orders, which loads a cart from a database (simulated, 50 ms) and calls payments, which calls a card gateway (simulated, 400 ms). Each service sets up the SDK and exports spans to Jaeger over OTLP:

from opentelemetry import trace, propagate
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.instrumentation.requests import RequestsInstrumentor

provider = TracerProvider(resource=Resource.create({"service.name": NAME}))
provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter(endpoint="http://jaeger:4318/v1/traces")))
trace.set_tracer_provider(provider)
RequestsInstrumentor().instrument()          # outgoing HTTP calls: creates spans and injects traceparent
tracer = trace.get_tracer(NAME)

# in the request handler:
ctx = propagate.extract(dict(self.headers))   # read traceparent from the incoming request
with tracer.start_as_current_span(f"GET {self.path}", context=ctx, kind=trace.SpanKind.SERVER) as span:
    span.set_attribute("http.route", self.path)
    with tracer.start_as_current_span("call card gateway") as gw:     # manual span for a business operation
        ...
        gw.set_status(trace.Status(trace.StatusCode.ERROR, "gateway timeout"))

Key pieces: the resource names the service; the batch processor exports spans asynchronously (never block requests on telemetry); the requests instrumentation handles propagation on outgoing calls; propagate.extract handles incoming ones (web framework instrumentation does this automatically in real services).

We sent a normal request (/checkout) and a failing one (/charge-fail), then asked Jaeger for the traces. A successful trace, shown as a tree (service, span name, duration):

frontend  GET /checkout            464 ms
  frontend  GET                    462 ms        <- outgoing HTTP call to orders
    orders    GET /checkout        459 ms
      orders    load cart from db   50 ms
      orders    GET                407 ms        <- outgoing call to payments
        payments  GET /checkout    401 ms
          payments  call card gateway  400 ms

Reading it: of the 464 ms, 400 ms are spent in one span, call card gateway in the payments service. The database call (50 ms) and the network hops (a few ms) are minor. No amount of staring at frontend metrics would have shown that; the trace points at the exact component. The failing request:

frontend  GET /charge-fail          464 ms  ERROR
  frontend  GET                     463 ms  ERROR
    orders    GET /charge-fail      460 ms  ERROR
      orders    load cart from db    50 ms
      orders    GET                 407 ms  ERROR
        payments  GET /charge-fail  401 ms  ERROR
          payments  call card gateway 400 ms  ERROR

The error status propagates up from the gateway span to the root, so in the UI you can search "show traces with errors in service payments" and immediately see where the failure originates, instead of finding only a 502 at the top. (The data came from Jaeger's v3 query API; the Jaeger UI draws the same tree as a waterfall/Gantt chart, where the 400 ms bar dominates.)

Using traces

  • Find the slow part: open the slowest traces (UIs sort by duration), look for the widest span; check whether time is spent in a service, waiting on another, or in serial calls that could run in parallel (a staircase pattern).
  • Find where errors start: the deepest span marked ERROR is usually the origin.
  • Find N+1 query problems: a loop of dozens of identical DB spans under one request.
  • Service maps: tools build dependency graphs from traces (which service calls which, with error rates and latency).
  • Compare before/after deployments, or fast vs slow requests for the same endpoint (what differs: a cache miss, a particular customer?).

Sampling and cost

Recording every span of every request is expensive at scale. Sampling keeps a fraction:

  • Head sampling: decide at the start of the trace (for example 1% of traces, or keep everything for important routes). Cheap, simple, but you may miss the rare slow or failing request.
  • Tail sampling: the Collector buffers whole traces and decides at the end, keeping all error traces and slow traces plus a small share of normal ones. Much better for debugging, at the cost of Collector memory and complexity.
  • Always propagate the sampling decision (the flags in traceparent) so the whole trace is consistently kept or dropped.

Also: avoid high-cardinality or sensitive attributes in spans just as in logs (no passwords, tokens or full personal data; redaction processors in the Collector help), and set retention (traces are usually kept days, not months).

Joining the three pillars

The goal is to move between signals without searching blindly:

  1. Metric → trace: exemplars attach a trace id to a histogram bucket sample, so on the latency graph you can click a point and open the trace behind it (Prometheus and Grafana support this).
  2. Trace → logs: put the trace_id and span_id in every log line (the OTel logging integration adds them automatically; in Python, the LoggingInstrumentor), then in the tracing UI jump to logs filtered by that id (Grafana with Tempo and Loki "trace to logs").
  3. Logs → trace: a trace_id field in logs becomes a link via a Loki derived field.
  4. Same labels everywhere: service.name, env, version in all three signals.

This also means your correlation id (previous lesson) should simply be the trace id once tracing is on.

Backends

  • Jaeger (CNCF graduated): open source, good UI; v2 is built on the OTel Collector and ingests OTLP directly (our test used jaegertracing/jaeger, with the UI on port 16686 and OTLP on 4317/4318). Storage: in-memory for testing, Elasticsearch/OpenSearch, Cassandra or ClickHouse for production.
  • Grafana Tempo: trace storage on object storage, designed to be cheap, integrated with Grafana (TraceQL, trace to logs/metrics).
  • Zipkin: older, simpler.
  • Managed/commercial: AWS X-Ray, Google Cloud Trace, Datadog APM, Honeycomb, New Relic, Lightstep/ServiceNow, Elastic APM.

Common mistakes

  • Broken propagation: one service or a message queue that drops traceparent → fragmented traces.
  • Not exporting asynchronously, slowing requests or losing data on shutdown (flush on exit).
  • Instrumenting everything with tiny spans → cost and noise; instrument boundaries and business operations.
  • Sensitive data in span attributes.
  • Sampling too aggressively (missing errors) or not at all (huge bills).
  • No link between traces, logs and metrics.
  • Forgetting service naming consistency (service.name differing per environment).

Try it yourself

  1. Run Jaeger (docker run -p 16686:16686 -p 4318:4318 jaegertracing/jaeger) and a small service with OpenTelemetry auto-instrumentation; make requests and open the UI.
  2. Add two services where one calls the other and verify you see one trace across both.
  3. Add a custom span and attributes around a business operation, mark an error and find it by searching for error traces.
  4. Add trace_id to your JSON logs and use it to find the logs for one slow trace.

Check yourself

What is a span, and how do spans form a trace?

A span is one timed operation with attributes and status. Spans share a trace id and point to a parent span id, forming a tree that represents one request's journey.

What is context propagation and what breaks if it is missing?

Passing the trace id and parent span id between services (the traceparent header). If a service does not propagate it, the trace is split into separate unrelated pieces.

Why use the OpenTelemetry Collector?

It receives, batches, processes (sampling, redaction) and exports telemetry to any backend, decoupling applications from vendors and centralising policies.

What is the benefit of tail sampling?

It keeps whole traces that turned out to be interesting (errors, slow requests) and samples normal ones, so important traces are retained at lower cost.

Interview questions

How would you debug a slow request in a microservice architecture?

Open its trace (from a latency exemplar, a request id or a search by duration), find the widest or deepest-error span to identify the service and operation, check the span attributes, then jump to the logs for that trace id and the metrics of that service.

What is the difference between logs, metrics and traces?

Metrics are numeric time series for trends and alerts; logs are discrete event records with detail; traces show one request's path and timing across services. Together they let you detect, locate and explain problems.

How do you add tracing to an existing application with minimal code change?

Use OpenTelemetry auto-instrumentation (agent or library wrapper) for the language, configure the service name and OTLP endpoint through environment variables, deploy a Collector, and add manual spans only where business context is needed.

Summary

  • A trace is a request's end-to-end path made of spans with parent links; context propagation (traceparent) joins spans across services.
  • OpenTelemetry provides standard instrumentation, OTLP and the Collector; auto-instrument first, add manual spans for business operations.
  • A real three-service trace showed 400 of 464 ms in one payments span, and error status propagating up from the failing call.
  • Sample sensibly (tail sampling), keep sensitive data out, and join traces with logs and metrics using trace ids and exemplars.

Going deeper

See how much of this you can explain

A free mock interview asks you new questions on Monitoring and observability for DevOps Engineer and scores every answer, with what to fix.