Lesson 6 of 8 · 12 min · Logs and traces
Structured logging and correlation ids
What you will learn
- Why unstructured text logs do not scale
- A structured (JSON) log format and the fields every line should have
- Log levels and when to use each
- What not to log (secrets, personal data) and how to avoid it
- Correlation (request) ids: generating, propagating and logging them
- Logging in containers: stdout, not files
In simple words
A log line such as "error happened" helps nobody. Structured logging writes each line as labelled fields (time, level, user, message), so a machine can search and filter it. A correlation ID is a tracking number given to each request, so you can follow that one request through every service.
Words to know
- Structured log: a log line written as named fields, usually in JSON.
- Log level: how serious a line is: DEBUG, INFO, WARNING or ERROR.
- Correlation ID: one ID attached to everything that happens for a single request.
- stdout: a program's standard output. Containers should write their logs there, not to files.
From text lines to structured events
A traditional log line:
2026-11-12 15:27:03 ERROR payment failed for user 3 on /checkout after 3012 ms
is fine for a human reading one file, but hard for a machine: to find all failures of a particular user or all requests slower than 3 s you need fragile regular expressions, which break when someone rephrases the message. A structured log writes each event as a record of named fields, usually one JSON object per line:
{"ts": "2026-11-12T15:27:03Z", "level": "ERROR", "service": "shop-api", "msg": "payment failed",
"request_id": "cd347ab63fcc", "path": "/checkout", "status": 502, "duration_ms": 3012,
"error": "Traceback (most recent call last): ... TimeoutError: payment gateway timed out after 3s"}
Now every field can be filtered, grouped and counted by the log system without parsing prose. Human readability suffers a little (use jq or the log UI), and it is worth it.
A Python formatter that produces exactly that (standard library only):
import json, logging, time
class JsonFormatter(logging.Formatter):
def format(self, record):
data = {
"ts": time.strftime("%Y-%m-%dT%H:%M:%S", time.gmtime(record.created)) + "Z",
"level": record.levelname,
"service": "shop-api",
"msg": record.getMessage(),
}
data.update(getattr(record, "ctx", {})) # extra structured fields
if record.exc_info:
data["error"] = self.formatException(record.exc_info)
return json.dumps(data)
log = logging.getLogger("shop")
...
log.info("request finished", extra={"ctx": {"request_id": rid, "path": path, "status": 200, "duration_ms": 41}})
log.error("payment failed", exc_info=True, extra={"ctx": {"request_id": rid, "status": 502}})
Most languages have a ready solution: structlog or python-json-logger (Python), logback with a JSON encoder or Log4j2 JSON layout (Java), pino/winston (Node), zap/slog (Go), and Serilog (.NET). Prefer a library that can bind context once (logger.bind(request_id=...)) so every line from a request carries it.
Fields every line should have
| Field | Purpose |
|---|---|
ts |
UTC timestamp in ISO 8601 (include milliseconds) |
level |
severity |
service (and version, env, host/pod) |
where it came from; version helps correlate with deployments |
msg |
a short, constant, human-readable message (put variable data in fields, not in the message, so messages can be grouped) |
request_id / trace_id |
links lines of one request across services |
user_id (or a hashed id) |
who, for support and debugging, subject to privacy rules |
path, method, status, duration_ms |
for request logs |
error (type, message, stack) |
exceptions as fields, one event per error |
Keep field names consistent across services (agree on request_id, not reqId in one and correlation in another), and avoid changing a field's type (a number in one line and a string in the next breaks indexing).
Log levels
| Level | Use for | In production |
|---|---|---|
DEBUG |
detailed diagnostics for developers | off by default; switchable at run time |
INFO |
normal events worth recording: startup, request completed, job finished | on |
WARNING |
unexpected but handled: retry succeeded, deprecated call, nearly full resource | on, review |
ERROR |
an operation failed and was not completed: a request returned 5xx, a payment failed | on, alert on rates |
CRITICAL/FATAL |
the service cannot continue | on, page |
Rules of thumb: log events, not chatter (a log line per loop iteration will bury everything and cost money); log an error once, where it is handled, with the stack trace, not at every layer it passes through; make sure an ERROR is something a person could act on.
What not to log
Logs are copied to many places, retained for months and read by many people. Never log:
- Secrets: passwords, API keys, tokens, session ids, private keys, full authorization headers, database connection strings with passwords.
- Sensitive personal data: card numbers, government ids, health data, full email/phone/address (in India, the Digital Personal Data Protection Act makes personal data handling a legal matter), message bodies. Log an internal id instead, or a hash/partial value (last four digits).
- Whole request/response bodies by default.
- Anything you would not be comfortable seeing in a leak.
Controls: central redaction in the logging library (filters that mask known keys like password, authorization, token), allow-lists of fields to log instead of dumping objects, code review focus on log statements, scanners for secrets in logs, short retention and restricted access. Beware of exception messages that contain data (a failed SQL statement with values), and of debug logging left on.
Correlation ids
In one request flow through several services (nginx → web app → orders → payments), an error in the last service produces a log line far away from the first one. A correlation id (also called request id) is a unique id created when the request first enters the system and passed along with every call, and included in every log line. Then a single search for that id returns the full story across all services.
How it works:
- The edge (load balancer, ingress, API gateway, or first service) generates an id if the incoming request does not have one. nginx does it with the built-in
$request_idvariable. - It is passed to the next hop in a header, conventionally
X-Request-ID(ortraceparentfor distributed tracing; next lessons). - Every service reads the header (or creates one), stores it in the request context, adds it to every log line and forwards it on outgoing calls.
- The response carries the id back to the client (
X-Request-IDheader), so a user can quote it to support.
We tested it with nginx in front of a small Python app. nginx configuration:
log_format with_id '$remote_addr "$request" $status rid=$request_id upstream_rt=$upstream_response_time';
access_log /dev/stdout with_id;
server {
listen 80;
location / {
proxy_set_header X-Request-ID $request_id;
proxy_pass http://app:9000;
}
}
The app logs JSON including request_id taken from the header and echoes it in the response. Results:
$ curl -si localhost:18085/cart | grep -i x-request-id
X-Request-ID: b1ae69856d47d1837eaab07ce2124ace
nginx access log: "GET /checkout HTTP/1.1" 200 rid=4d4f71f751a5e853339bcc87dcd4a2fe upstream_rt=0.001
app log: {"level": "INFO", "msg": "handling request", "path": "/checkout", "request_id": "4d4f71f751a5e853339bcc87dcd4a2fe"}
The id in the nginx log and the application log was identical, so one search joins them. A real application uses middleware to do this automatically:
# sketch: Flask-like middleware
@app.before_request
def set_request_id():
g.request_id = request.headers.get("X-Request-ID") or uuid.uuid4().hex
@app.after_request
def add_header(resp):
resp.headers["X-Request-ID"] = g.request_id
return resp
# outgoing calls: requests.get(url, headers={"X-Request-ID": g.request_id})
Use contextvars (Python), MDC (Java/SLF4J), AsyncLocalStorage (Node) or context.Context (Go) to carry the id without passing it through every function, including into background jobs spawned by the request.
Practical rules:
- Never trust a client-supplied id blindly for security decisions; it is only a label. Validate length and characters, or generate a fresh one at the edge and keep the client's as a separate field.
- Ids should be unique and unguessable enough (UUID or random hex), not sequential.
- Also propagate through queues and async jobs: put the id in message metadata and restore it in the consumer.
- When distributed tracing is in place, the trace id is the correlation id: log it in every line (the OpenTelemetry logging integration adds
trace_idandspan_idautomatically), and a click goes from a log line to the trace.
Logging in containers and Kubernetes
Follow the twelve-factor rule: write logs to stdout/stderr, one event per line, and let the platform collect them (Docker logging driver, the Kubernetes node's log files, a collector such as Grafana Alloy, Fluent Bit or Vector). Do not write log files inside containers (they vanish with the container and fill its disk) and do not run rotation or shipping inside the app. Multi-line stack traces are why JSON (stack trace in one field) beats text, where each trace line would be a separate event. Add Kubernetes metadata (namespace, pod, container, node) at collection time.
Cost and volume
Logs are usually the largest observability bill. Decide retention by value (error logs 30-90 days, debug none), sample noisy successes, avoid logging in hot loops, drop health-check requests, and keep high-value audit logs separately and longer.
Common mistakes
- Free-text messages with data embedded (
"user 3 failed"), instead of constant messages with fields. - Logging secrets or personal data.
- No correlation id, so cross-service debugging means guessing from timestamps.
- Logging the same exception at every layer.
- Inconsistent field names/types between services.
- Debug logging in production permanently.
- Writing logs to files in containers.
- Timestamps in local time without a zone.
Try it yourself
- Convert a service's logging to JSON and add
request_id,path,statusandduration_msto its request log. - Put nginx (or any proxy) in front, generate
$request_id, forward it and show the same id in both logs. - Add a redaction filter that masks fields named
passwordorauthorizationand test it. - Make a background job inherit the request id from the message that triggered it.
Check yourself
What advantage does a structured log have over a text log?
Each event has named fields that tools can filter, group and count without fragile text parsing, and the fields stay stable when messages are reworded.
What is a correlation id?
A unique id created at the first entry point of a request and propagated through all services and into every log line, so all logs for one request can be found with one search.
Where should a containerised application write its logs?
To stdout/stderr, one event per line, so the platform can collect them.
Name three things that must never be logged.
Passwords or tokens, card or other sensitive identifiers, and complete personal data or request bodies containing them.
Interview questions
How do you trace a failing request through five microservices using logs?
Ensure the edge generates a request/trace id, every service logs it and forwards it, then search the central log system for that id across services and read the events in time order (or open the trace if tracing is enabled).
What belongs in a log message versus a field?
A short constant message describing the event, with variable data (ids, durations, status codes) in fields, so events can be grouped and queried.
How would you handle sensitive data in logs?
Avoid logging it, redact known fields centrally, log ids instead of values, restrict access and retention, and scan logs for leaks.
Summary
- Log one JSON object per event with consistent fields (timestamp, level, service, message, ids, status, duration, error).
- Use levels deliberately, log errors once, and never log secrets or sensitive personal data.
- Generate a request id at the edge, forward it in
X-Request-IDor trace headers, log it everywhere and return it to the client. - In containers, log to stdout and let the platform collect; control volume and retention.
Going deeper
- The Twelve-Factor App: Logs: logs as event streams.
- OWASP Logging Cheat Sheet: what to log and what not to.
- nginx: $request_id: the built-in unique request id.
- Python logging cookbook: custom formatters and structured logging.
- OpenTelemetry logs: correlating logs with traces.
See how much of this you can explain
A free mock interview asks you new questions on Monitoring and observability for DevOps Engineer and scores every answer, with what to fix.