Lesson 6 of 8 · 12 min · Logs and traces

Structured logging and correlation ids

Covers: Python 3 logging with a JSON formatter; nginx 1.27 generating and forwarding $request_id. Updated 13 Nov 2026. Part of the DevOps Engineer roadmap.

What you will learn

  • Why unstructured text logs do not scale
  • A structured (JSON) log format and the fields every line should have
  • Log levels and when to use each
  • What not to log (secrets, personal data) and how to avoid it
  • Correlation (request) ids: generating, propagating and logging them
  • Logging in containers: stdout, not files

In simple words

A log line such as "error happened" helps nobody. Structured logging writes each line as labelled fields (time, level, user, message), so a machine can search and filter it. A correlation ID is a tracking number given to each request, so you can follow that one request through every service.

Words to know

  • Structured log: a log line written as named fields, usually in JSON.
  • Log level: how serious a line is: DEBUG, INFO, WARNING or ERROR.
  • Correlation ID: one ID attached to everything that happens for a single request.
  • stdout: a program's standard output. Containers should write their logs there, not to files.

From text lines to structured events

A traditional log line:

2026-11-12 15:27:03 ERROR payment failed for user 3 on /checkout after 3012 ms

is fine for a human reading one file, but hard for a machine: to find all failures of a particular user or all requests slower than 3 s you need fragile regular expressions, which break when someone rephrases the message. A structured log writes each event as a record of named fields, usually one JSON object per line:

{"ts": "2026-11-12T15:27:03Z", "level": "ERROR", "service": "shop-api", "msg": "payment failed",
 "request_id": "cd347ab63fcc", "path": "/checkout", "status": 502, "duration_ms": 3012,
 "error": "Traceback (most recent call last): ... TimeoutError: payment gateway timed out after 3s"}

Now every field can be filtered, grouped and counted by the log system without parsing prose. Human readability suffers a little (use jq or the log UI), and it is worth it.

A Python formatter that produces exactly that (standard library only):

import json, logging, time

class JsonFormatter(logging.Formatter):
    def format(self, record):
        data = {
            "ts": time.strftime("%Y-%m-%dT%H:%M:%S", time.gmtime(record.created)) + "Z",
            "level": record.levelname,
            "service": "shop-api",
            "msg": record.getMessage(),
        }
        data.update(getattr(record, "ctx", {}))          # extra structured fields
        if record.exc_info:
            data["error"] = self.formatException(record.exc_info)
        return json.dumps(data)

log = logging.getLogger("shop")
...
log.info("request finished", extra={"ctx": {"request_id": rid, "path": path, "status": 200, "duration_ms": 41}})
log.error("payment failed", exc_info=True, extra={"ctx": {"request_id": rid, "status": 502}})

Most languages have a ready solution: structlog or python-json-logger (Python), logback with a JSON encoder or Log4j2 JSON layout (Java), pino/winston (Node), zap/slog (Go), and Serilog (.NET). Prefer a library that can bind context once (logger.bind(request_id=...)) so every line from a request carries it.

Fields every line should have

Field Purpose
ts UTC timestamp in ISO 8601 (include milliseconds)
level severity
service (and version, env, host/pod) where it came from; version helps correlate with deployments
msg a short, constant, human-readable message (put variable data in fields, not in the message, so messages can be grouped)
request_id / trace_id links lines of one request across services
user_id (or a hashed id) who, for support and debugging, subject to privacy rules
path, method, status, duration_ms for request logs
error (type, message, stack) exceptions as fields, one event per error

Keep field names consistent across services (agree on request_id, not reqId in one and correlation in another), and avoid changing a field's type (a number in one line and a string in the next breaks indexing).

Log levels

Level Use for In production
DEBUG detailed diagnostics for developers off by default; switchable at run time
INFO normal events worth recording: startup, request completed, job finished on
WARNING unexpected but handled: retry succeeded, deprecated call, nearly full resource on, review
ERROR an operation failed and was not completed: a request returned 5xx, a payment failed on, alert on rates
CRITICAL/FATAL the service cannot continue on, page

Rules of thumb: log events, not chatter (a log line per loop iteration will bury everything and cost money); log an error once, where it is handled, with the stack trace, not at every layer it passes through; make sure an ERROR is something a person could act on.

What not to log

Logs are copied to many places, retained for months and read by many people. Never log:

  • Secrets: passwords, API keys, tokens, session ids, private keys, full authorization headers, database connection strings with passwords.
  • Sensitive personal data: card numbers, government ids, health data, full email/phone/address (in India, the Digital Personal Data Protection Act makes personal data handling a legal matter), message bodies. Log an internal id instead, or a hash/partial value (last four digits).
  • Whole request/response bodies by default.
  • Anything you would not be comfortable seeing in a leak.

Controls: central redaction in the logging library (filters that mask known keys like password, authorization, token), allow-lists of fields to log instead of dumping objects, code review focus on log statements, scanners for secrets in logs, short retention and restricted access. Beware of exception messages that contain data (a failed SQL statement with values), and of debug logging left on.

Correlation ids

In one request flow through several services (nginx → web app → orders → payments), an error in the last service produces a log line far away from the first one. A correlation id (also called request id) is a unique id created when the request first enters the system and passed along with every call, and included in every log line. Then a single search for that id returns the full story across all services.

How it works:

  1. The edge (load balancer, ingress, API gateway, or first service) generates an id if the incoming request does not have one. nginx does it with the built-in $request_id variable.
  2. It is passed to the next hop in a header, conventionally X-Request-ID (or traceparent for distributed tracing; next lessons).
  3. Every service reads the header (or creates one), stores it in the request context, adds it to every log line and forwards it on outgoing calls.
  4. The response carries the id back to the client (X-Request-ID header), so a user can quote it to support.

We tested it with nginx in front of a small Python app. nginx configuration:

log_format with_id '$remote_addr "$request" $status rid=$request_id upstream_rt=$upstream_response_time';
access_log /dev/stdout with_id;
server {
  listen 80;
  location / {
    proxy_set_header X-Request-ID $request_id;
    proxy_pass http://app:9000;
  }
}

The app logs JSON including request_id taken from the header and echoes it in the response. Results:

$ curl -si localhost:18085/cart | grep -i x-request-id
X-Request-ID: b1ae69856d47d1837eaab07ce2124ace

nginx access log:  "GET /checkout HTTP/1.1" 200 rid=4d4f71f751a5e853339bcc87dcd4a2fe upstream_rt=0.001
app log:           {"level": "INFO", "msg": "handling request", "path": "/checkout", "request_id": "4d4f71f751a5e853339bcc87dcd4a2fe"}

The id in the nginx log and the application log was identical, so one search joins them. A real application uses middleware to do this automatically:

# sketch: Flask-like middleware
@app.before_request
def set_request_id():
    g.request_id = request.headers.get("X-Request-ID") or uuid.uuid4().hex
@app.after_request
def add_header(resp):
    resp.headers["X-Request-ID"] = g.request_id
    return resp
# outgoing calls: requests.get(url, headers={"X-Request-ID": g.request_id})

Use contextvars (Python), MDC (Java/SLF4J), AsyncLocalStorage (Node) or context.Context (Go) to carry the id without passing it through every function, including into background jobs spawned by the request.

Practical rules:

  • Never trust a client-supplied id blindly for security decisions; it is only a label. Validate length and characters, or generate a fresh one at the edge and keep the client's as a separate field.
  • Ids should be unique and unguessable enough (UUID or random hex), not sequential.
  • Also propagate through queues and async jobs: put the id in message metadata and restore it in the consumer.
  • When distributed tracing is in place, the trace id is the correlation id: log it in every line (the OpenTelemetry logging integration adds trace_id and span_id automatically), and a click goes from a log line to the trace.

Logging in containers and Kubernetes

Follow the twelve-factor rule: write logs to stdout/stderr, one event per line, and let the platform collect them (Docker logging driver, the Kubernetes node's log files, a collector such as Grafana Alloy, Fluent Bit or Vector). Do not write log files inside containers (they vanish with the container and fill its disk) and do not run rotation or shipping inside the app. Multi-line stack traces are why JSON (stack trace in one field) beats text, where each trace line would be a separate event. Add Kubernetes metadata (namespace, pod, container, node) at collection time.

Cost and volume

Logs are usually the largest observability bill. Decide retention by value (error logs 30-90 days, debug none), sample noisy successes, avoid logging in hot loops, drop health-check requests, and keep high-value audit logs separately and longer.

Common mistakes

  • Free-text messages with data embedded ("user 3 failed"), instead of constant messages with fields.
  • Logging secrets or personal data.
  • No correlation id, so cross-service debugging means guessing from timestamps.
  • Logging the same exception at every layer.
  • Inconsistent field names/types between services.
  • Debug logging in production permanently.
  • Writing logs to files in containers.
  • Timestamps in local time without a zone.

Try it yourself

  1. Convert a service's logging to JSON and add request_id, path, status and duration_ms to its request log.
  2. Put nginx (or any proxy) in front, generate $request_id, forward it and show the same id in both logs.
  3. Add a redaction filter that masks fields named password or authorization and test it.
  4. Make a background job inherit the request id from the message that triggered it.

Check yourself

What advantage does a structured log have over a text log?

Each event has named fields that tools can filter, group and count without fragile text parsing, and the fields stay stable when messages are reworded.

What is a correlation id?

A unique id created at the first entry point of a request and propagated through all services and into every log line, so all logs for one request can be found with one search.

Where should a containerised application write its logs?

To stdout/stderr, one event per line, so the platform can collect them.

Name three things that must never be logged.

Passwords or tokens, card or other sensitive identifiers, and complete personal data or request bodies containing them.

Interview questions

How do you trace a failing request through five microservices using logs?

Ensure the edge generates a request/trace id, every service logs it and forwards it, then search the central log system for that id across services and read the events in time order (or open the trace if tracing is enabled).

What belongs in a log message versus a field?

A short constant message describing the event, with variable data (ids, durations, status codes) in fields, so events can be grouped and queried.

How would you handle sensitive data in logs?

Avoid logging it, redact known fields centrally, log ids instead of values, restrict access and retention, and scan logs for leaks.

Summary

  • Log one JSON object per event with consistent fields (timestamp, level, service, message, ids, status, duration, error).
  • Use levels deliberately, log errors once, and never log secrets or sensitive personal data.
  • Generate a request id at the edge, forward it in X-Request-ID or trace headers, log it everywhere and return it to the client.
  • In containers, log to stdout and let the platform collect; control volume and retention.

Going deeper

See how much of this you can explain

A free mock interview asks you new questions on Monitoring and observability for DevOps Engineer and scores every answer, with what to fix.