Lesson 7 of 8 · 12 min · Logs and traces

Log aggregation with Loki and the ELK stack

Covers: Grafana Loki, Grafana Alloy, Docker Compose; ELK overview. Updated 13 Nov 2026. Part of the DevOps Engineer roadmap.

What you will learn

  • Why central log aggregation is needed
  • The pipeline: producer → collector → storage → query UI
  • Grafana Loki: labels, streams and LogQL, tested
  • The ELK/Elastic stack and how its approach differs
  • Choosing, sizing, retention and alerting on logs

In simple words

With fifty containers on ten machines, you can't log in to each one to read its logs. Log aggregation collects every log into one searchable place, like putting all the CCTV feeds on one screen. This lesson compares the two common tool sets and tries one of them out.

Words to know

  • Log aggregation: collecting logs from every machine into one central system.
  • Collector: the small program on each machine that ships its logs out.
  • Loki: Grafana's log storage. It is cheap because it indexes only the labels.
  • ELK: Elasticsearch, Logstash and Kibana: a powerful search-based set of log tools.
  • Retention: how long logs are kept before they are deleted.

Why aggregate logs

On one server you can grep /var/log/.... With dozens of containers on a changing set of machines it stops working: containers disappear with their logs, the right pod is unknown, and a request touches several services. A log aggregation system collects logs from everywhere into one searchable store with retention, access control and links to dashboards and traces.

The pipeline has four parts:

applications (stdout / files)  →  collector / agent  →  storage & index  →  query / UI & alerts
 (JSON lines, request_id)         Alloy, Fluent Bit,      Loki, Elasticsearch,    Grafana, Kibana
                                  Vector, Filebeat        OpenSearch, CloudWatch
  • Collectors tail files or container logs, add metadata (pod, namespace, node), parse or filter, and forward. In Kubernetes they run as a DaemonSet (one per node). Common ones: Grafana Alloy (successor to Promtail, which reached end of life in 2026), Fluent Bit, Vector, Filebeat/Elastic Agent, the OpenTelemetry Collector.
  • Storage differs mainly in what it indexes, the key cost/performance trade-off (below).
  • Cloud options: CloudWatch Logs, Azure Monitor Logs, Google Cloud Logging: convenient, managed, billed per GB ingested and stored.

Grafana Loki

Loki is "Prometheus for logs": it indexes only labels (small metadata like job, env, level, namespace, pod) and stores the log text itself compressed, unindexed, in chunks (on object storage such as S3 in production). Queries first pick streams by label, then scan their text. Consequences: very cheap to run and to store, quick for "show me the logs of this service for the last hour", slower for needle-in-a-haystack full-text across everything unless you narrow by labels and time. It integrates natively with Grafana and with Prometheus-style labels, so you can jump from a metric panel to the same service's logs.

A stream is the sequence of lines with one unique label set. Cardinality rules again apply (as with Prometheus): do not make request_id or user_id a label; keep them inside the log line and filter on them at query time.

The tested pipeline

We ran Loki and Grafana Alloy with Docker Compose; the app wrote JSON lines to a shared file. Alloy's configuration:

local.file_match "app" {
  path_targets = [{"__path__" = "/logs/*.log", "job" = "shop-api", "env" = "dev"}]
}

loki.source.file "app" {
  targets    = local.file_match.app.targets
  forward_to = [loki.process.json.receiver]
}

loki.process "json" {
  stage.json {
    expressions = { level = "level", request_id = "request_id", path = "path" }
  }
  stage.labels {
    values = { level = "" }          # promote only low-cardinality fields to labels
  }
  forward_to = [loki.write.default.receiver]
}

loki.write "default" {
  endpoint {
    url = "http://loki:3100/loki/api/v1/push"
  }
}

Read it as a pipeline: find files → tail them → parse JSON, promote level to a label (keep request_id and path in the line) → push to Loki. After the app wrote its logs, Loki's API listed the labels it had indexed:

['env', 'filename', 'job', 'level', 'service_name']
level = ['ERROR', 'INFO']

LogQL

LogQL has a log stream selector (labels, like PromQL) followed by optional filters and parsers:

{job="shop-api", level="ERROR"}                                  # stream selector: all error lines
{job="shop-api"} |= "payment failed"                             # line contains text (|~ regex, != not)
{job="shop-api"} | json | duration_ms > 3000                     # parse JSON fields, then filter on them
{job="shop-api"} | json | request_id="cd347ab63fcc"              # follow one request

The first returned the error events (payment failed, request_id, status 502); the second with the numeric filter matched 3 slow payment failures; and the request-id query returned the two lines of that request in order:

2026-11-12T15:27:01Z INFO  request started
2026-11-12T15:27:01Z ERROR payment failed 502

LogQL also has metric queries that turn logs into numbers, usable in dashboards and alerts:

sum by (path) (count_over_time({job="shop-api", level="ERROR"} | json [10m]))
{path="/checkout"}   7

(seven errors in ten minutes, all on /checkout). rate(), sum by, topk and quantile_over_time work too. In Grafana's Explore you pick the Loki data source, write these queries, and see lines with parsed fields; add derived fields that turn a trace_id in a log line into a link to the trace.

The ELK / Elastic stack

ELK is Elasticsearch + Logstash + Kibana (now often "Elastic Stack" with Beats/Elastic Agent; OpenSearch is the AWS-led open-source fork with similar components and Dashboards):

  • Elasticsearch stores log events as JSON documents in indices and builds an inverted index over all fields, giving fast full-text search and powerful aggregations over any field.
  • Logstash (heavy) or Beats/Elastic Agent/Fluent Bit collect and transform logs; ingest pipelines parse and enrich in Elasticsearch.
  • Kibana provides search, dashboards, alerting and machine learning features.
Loki Elasticsearch (ELK)
Index labels only full-text + all fields
Storage cost low (compressed text on object storage) high (index overhead often exceeds the raw data)
Query speed for arbitrary text search slower (scans), fast with good labels and time range fast across everything
Operations simpler, scales horizontally with object storage more complex (shards, mappings, JVM heap, cluster sizing)
Rich analytics (aggregations, ML, security use cases/SIEM) basic strong
Fit Kubernetes/Prometheus/Grafana shops, cost-sensitive volume search-heavy, analytics, security, compliance, free-form text

Elasticsearch specifics worth knowing: index lifecycle management (ILM) moves indices through hot/warm/cold/delete phases (daily or size-based indices), mappings define field types (a field cannot change type; mapping explosions from dynamic fields are a classic outage cause), shards and replicas affect performance and resilience, and licensing differs (Elastic's license vs OpenSearch Apache 2.0). Managed services: Elastic Cloud, Amazon OpenSearch Service.

Other options: Splunk and Datadog (commercial, feature-rich, priced by volume), CloudWatch Logs Insights, VictoriaLogs, ClickHouse-based tools (SigNoz, HyperDX). Choose by volume, query needs, team skills, budget and existing stack; whichever you pick, the principles in the previous lesson (structured JSON, consistent fields, correlation ids) matter more than the tool.

Retention, cost and operations

  1. Retention by value: application errors 30-90 days; access/debug logs 7-14 days; audit/security logs long (a year or more, immutable, separate store); regulatory requirements override.
  2. Reduce volume at the source: log levels, sampling, no health-check noise, no giant payloads; drop or aggregate in the collector.
  3. Cost model: ingestion volume (GB/day) is the main driver; storage tiers (hot SSD vs object storage); query cost for scans.
  4. Capacity and back-pressure: collectors buffer on disk when the backend is slow; set limits so logging problems cannot take down applications. Never let the app block on log delivery.
  5. Access control: logs contain sensitive operational data; separate teams/tenants (Loki multi-tenancy X-Scope-OrgID, Elastic spaces and roles).
  6. Monitor the logging pipeline itself: ingestion rate, dropped lines, collector errors, storage usage; alert if logs stop arriving from a service (absence alert).
  7. Time: ensure clocks are synchronised and timestamps are UTC; out-of-order and late logs complicate queries.

Alerting on logs

Prefer metrics for alerting (cheaper, more reliable); but some signals only exist in logs ("payment gateway timeout", security events). Both Loki (ruler) and Grafana can alert on LogQL metric queries: for example "more than 5 payment failed events in 5 minutes". Keep log-based alerts few and specific; turn recurring ones into proper metrics (a counter in code) over time.

Common mistakes

  • High-cardinality Loki labels (request id as a label).
  • Indexing everything in Elasticsearch without lifecycle policies, filling disks.
  • No retention policy (or infinite retention).
  • Logging blocking the application when the backend is down.
  • Unparsed multi-line text that splits stack traces into separate events.
  • Collecting everything, including debug noise and sensitive data.
  • No monitoring of the log pipeline.
  • Choosing the tool before the requirements.

Try it yourself

  1. Run Loki and Alloy with Docker Compose, write JSON logs to a file, and query errors with {job="..", level="ERROR"} in Grafana Explore.
  2. Write LogQL that counts errors per path over 10 minutes, and one that finds a specific request_id.
  3. Compare the storage used by the same logs in Loki and in a single-node Elasticsearch/OpenSearch.
  4. Define retention for your logs: what stays for 14 days, 90 days, a year?

Check yourself

What does Loki index, and why is that cheaper than Elasticsearch?

Only a small set of labels per stream; the log text is stored compressed and unindexed, so it avoids the large full-text index that Elasticsearch builds for every field.

Why must request_id not be a Loki label?

Every distinct value would create a new stream (high cardinality), degrading performance. Keep it in the log line and filter with | json | request_id="...".

When might you choose Elasticsearch over Loki?

When you need fast full-text search across all fields, rich aggregations, security analytics (SIEM) or complex ad-hoc queries, and accept the higher storage and operational cost.

Where do log collectors run in Kubernetes?

As a DaemonSet on every node (or sidecars in special cases), tailing container log files and adding pod metadata.

Interview questions

Design centralised logging for a Kubernetes platform.

Applications log JSON to stdout; a DaemonSet collector (Alloy/Fluent Bit/Vector) enriches with Kubernetes metadata, parses and ships to Loki or OpenSearch with object-storage backing; Grafana for query and links to metrics and traces; retention tiers, multi-tenant access control, redaction, alerts on log-derived metrics, and monitoring of the pipeline itself.

How do you control logging costs?

Reduce volume at the source (levels, sampling), filter in the collector, set retention per log class, use cheap tiers or object storage, avoid indexing unneeded fields, and prefer metrics for high-volume signals.

How do you find all logs for one failed checkout?

Take the request id (from the user, the response header or an error), query the log system for it across services, and read the events in time order; or jump from the trace in a tracing UI.

Summary

  • Centralise logs with a collector → storage → query pipeline so containers' logs outlive them and requests can be followed across services.
  • Loki indexes labels only (cheap, Grafana-native) and uses LogQL; Elasticsearch/OpenSearch index everything (powerful search, higher cost and operations).
  • Keep labels low-cardinality, keep ids in the line, set retention and sampling, and monitor the pipeline.
  • Prefer metrics for alerting; use log-based alerts sparingly and turn repeats into metrics.

Going deeper

See how much of this you can explain

A free mock interview asks you new questions on Monitoring and observability for DevOps Engineer and scores every answer, with what to fix.