Lesson 1 of 8 · 11 min · Metrics and dashboards

Observability basics, metrics and Prometheus

Covers: Prometheus (prom/prometheus), prometheus_client 0.21 for Python, Docker Compose. Updated 13 Nov 2026. Part of the DevOps Engineer roadmap.

What you will learn

  • Why monitoring matters and what observability adds
  • The three pillars: metrics, logs, traces
  • Prometheus' pull model, targets, labels and time series
  • Counters, gauges and histograms, and what they are for
  • Instrumenting an application and reading the /metrics format
  • The four golden signals and the RED and USE methods

In simple words

You can't fix what you can't see. Monitoring is the dashboard of your car: speed, fuel, engine warning. For software you watch numbers such as requests per second and errors. Prometheus is the tool that collects those numbers from your applications every few seconds and stores them.

Words to know

  • Metric: a number measured over time, such as requests per second.
  • Prometheus: a popular open-source tool that collects and stores metrics.
  • Counter: a metric that only goes up, such as total requests served.
  • Gauge: a metric that goes up and down, such as memory in use.
  • Label: a tag on a metric, such as status="500", so you can split and filter it.

Monitoring and observability

Monitoring collects predefined measurements and alerts when known things go wrong: "CPU above 90%", "error rate above 5%". Observability is the ability to understand a system's internal state from its outputs, so you can answer questions you did not think of in advance: "why are checkout requests from Hyderabad slow only for users on plan X?" It needs rich, correlated telemetry:

Pillar What it is Good for Cost
Metrics numbers over time (counts, rates, latencies) dashboards, alerting, trends, capacity cheap, aggregated
Logs timestamped records of events details of what happened, errors, audit more expensive, detailed
Traces the path of one request across services with timings finding where time or errors come from in distributed systems moderate, usually sampled

They answer different questions: metrics say that something is wrong and how big; traces say where; logs say why. The next lessons cover each; the common thread is to be able to move between them (a dashboard spike → traces of slow requests → logs of the failing call, joined by ids).

Why invest: you cannot run reliable systems blind; incidents are found by customers instead of by you; capacity and cost decisions need data; releases (canary, rollback: CI/CD course) need signals to decide on.

Prometheus

Prometheus (a CNCF graduated project) is the standard open-source metrics system, especially in Kubernetes. Its design:

  • Pull model: Prometheus scrapes (HTTP GETs) the /metrics endpoint of each target at an interval (we used 5 s; 15-30 s is typical), instead of applications pushing. It knows when a target is down (up == 0). Targets are found by static lists, service discovery (Kubernetes, EC2, Consul) or relabelling.
  • Time series database: every unique combination of metric name and labels is a series of (timestamp, value) points stored locally (by default 15 days; use remote storage such as Thanos, Mimir or Cortex for long-term and HA).
  • PromQL: a query language over those series (next lesson).
  • Rules and alerts: recording rules precompute expensive expressions; alerting rules fire alerts sent to Alertmanager (alerting lesson).
  • Ecosystem: exporters expose metrics for things that cannot be instrumented directly (node_exporter for Linux hosts, cAdvisor/kube-state-metrics for containers, blackbox_exporter for probes, database exporters); client libraries for most languages; Grafana for dashboards.

The scrape configuration of our test stack:

global:
  scrape_interval: 5s
scrape_configs:
  - job_name: shop-api
    static_configs:
      - targets: ["app:8000"]
        labels: {env: dev}

The target list in the API confirmed both targets up:

prometheus  up  http://localhost:9090/metrics
shop-api    up  http://app:8000/metrics

Prometheus automatically adds job and instance labels, plus the up series (1 when the last scrape worked), so you get a free "is it alive?" signal for everything.

Metric types

  1. Counter: a value that only goes up (resets to zero on restart): requests served, errors, bytes sent. You almost never look at the raw value; you take its rate.
  2. Gauge: a value that goes up and down: memory in use, queue length, requests in flight, temperature.
  3. Histogram: counts observations in configurable buckets (cumulative le = "less than or equal"), plus a _sum and _count. Used for latencies and sizes, because you can calculate percentiles and averages across many instances.
  4. Summary: like a histogram but computes quantiles client-side; cannot be aggregated across instances. Prefer histograms.

Instrumenting an app

With the Python client library (this is the app we ran):

from prometheus_client import Counter, Histogram, Gauge, generate_latest

REQUESTS = Counter("http_requests_total", "Total HTTP requests", ["method", "path", "status"])
LATENCY = Histogram("http_request_duration_seconds", "Request latency in seconds", ["path"],
                    buckets=(0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5))
INFLIGHT = Gauge("http_requests_in_flight", "Requests currently being served")

# in the request handler:
INFLIGHT.inc()
...
LATENCY.labels(path).observe(duration_seconds)
REQUESTS.labels("GET", path, str(status)).inc()
INFLIGHT.dec()

Frameworks have middleware that does this for you (for example prometheus-fastapi-instrumentator, Micrometer in Spring Boot, prom-client in Node). Exposing the metrics is just an endpoint that returns the text format:

http_requests_total{method="GET",path="/",status="200"} 249.0
http_requests_total{method="GET",path="/slow",status="200"} 85.0
http_requests_total{method="GET",path="/flaky",status="500"} 24.0
http_request_duration_seconds_bucket{le="0.1",path="/slow"} 13.0
http_request_duration_seconds_bucket{le="0.5",path="/slow"} 48.0
http_request_duration_seconds_bucket{le="+Inf",path="/slow"} 85.0
http_request_duration_seconds_count{path="/slow"} 85.0
http_request_duration_seconds_sum{path="/slow"} 40.598

Read the histogram: of 85 /slow requests, 13 took ≤ 0.1 s, 48 took ≤ 0.5 s (buckets are cumulative), and all 85 are ≤ +Inf. The sum of all durations is 40.6 s, so the average is 40.6 / 85 ≈ 0.48 s.

Labels and cardinality

Each label combination is a separate time series. Labels are powerful (slice by path, status, instance) but high-cardinality labels (user id, request id, full URL with ids, email) create millions of series and bring Prometheus down. Rules: label values should come from small, bounded sets; use route templates (/users/{id}) instead of raw URLs; put identifiers in logs and traces, not metric labels. A good estimate: cardinality = product of label value counts per metric.

Naming conventions: snake_case, base units (seconds, bytes), _total suffix for counters, _seconds/_bytes unit suffixes, prefix with the application area (http_, shop_).

What to measure: golden signals, RED and USE

Google's SRE book names the four golden signals for any user-facing service:

  1. Latency: how long requests take (distinguish successful from failed requests; use percentiles).
  2. Traffic: how much demand (requests per second).
  3. Errors: the rate of failed requests (explicit 5xx, wrong content, too slow).
  4. Saturation: how full the service is (CPU, memory, queue depth, connection pool, disk).

The RED method (for request-driven services): Rate, Errors, Duration. The USE method (for resources like CPU, disks, networks): Utilisation, Saturation, Errors. Together they cover almost everything: RED for services, USE for infrastructure. Start every service with RED metrics; our test app has exactly those: http_requests_total (rate, errors by status) and http_request_duration_seconds (duration), with an in-flight gauge for saturation.

Beyond technical metrics, add business metrics (orders placed, payments failed, sign-ups): they detect problems that all the technical metrics miss ("the site is up but nobody can check out").

Running it yourself

A minimal stack with Docker Compose: your app, Prometheus (with the scrape config above and a rules file), Alertmanager and Grafana. Start it, send traffic, open Prometheus at port 9090 (Status → Targets to verify scraping, Graph to run queries), then continue with PromQL. For hosts, run node_exporter; in Kubernetes install kube-prometheus-stack (Helm), which deploys Prometheus Operator, Grafana, Alertmanager, node exporters and ready-made dashboards and alerts.

Common mistakes

  • Alerting and dashboarding on raw counters instead of rates.
  • High-cardinality labels (user ids, URLs with ids).
  • Only infrastructure metrics with no request or business metrics.
  • Averages instead of percentiles for latency.
  • Pushing metrics from short-lived jobs without the Pushgateway (batch jobs are the one legitimate use) or forgetting that targets can disappear.
  • No retention/storage plan for long-term data.
  • Not monitoring the monitoring (is Prometheus itself up and scraping?).

Try it yourself

  1. Run the stack above (or any app with a client library), open /metrics and identify a counter, a gauge and a histogram.
  2. In the Prometheus UI, find the up series for each target. Stop the app and watch up become 0.
  3. Add a new metric (for example orders_total{payment_method=...}) and verify it appears after a scrape.
  4. List the golden signals for an application you know and the metric for each.

Check yourself

What is the difference between a counter and a gauge?

A counter only increases (it resets on restart) and is used through its rate; a gauge goes up and down and is read as a current value.

Why use a histogram for latency?

It records observations in buckets so you can compute percentiles and averages across many instances; averages alone hide slow requests.

What does the up metric tell you?

Whether Prometheus' last scrape of that target succeeded (1) or failed (0), giving a free liveness signal.

Why avoid user ids as metric labels?

Each distinct value creates a new time series, causing a cardinality explosion that exhausts memory and slows queries.

Interview questions

Explain how Prometheus works.

It periodically scrapes /metrics endpoints of targets (found via static config or service discovery), stores labelled samples in a local time series database, evaluates recording and alerting rules, answers PromQL queries (used by Grafana), and sends firing alerts to Alertmanager.

What are the four golden signals?

Latency, traffic, errors and saturation.

What is the difference between monitoring and observability?

Monitoring watches known failure conditions with predefined checks; observability gives enough rich, correlated data (metrics, logs, traces) to investigate unknown problems.

Summary

  • Metrics say that something is wrong, traces where, logs why; observability links them.
  • Prometheus scrapes labelled counters, gauges and histograms into a time series database and queries them with PromQL.
  • Keep label cardinality low; measure the golden signals (RED for services, USE for resources) plus business metrics.
  • Instrument with client libraries or framework middleware; use exporters for hosts and components.

Going deeper

See how much of this you can explain

A free mock interview asks you new questions on Monitoring and observability for DevOps Engineer and scores every answer, with what to fix.