Lesson 2 of 8 · 9 min · Metrics and dashboards

PromQL in practice: rates, ratios and percentiles

Covers: Prometheus scraping a Python app with three endpoints. Updated 13 Nov 2026. Part of the DevOps Engineer roadmap.

What you will learn

  • Instant and range vectors, selectors and label matchers
  • rate(), increase() and why counters are never used raw
  • Aggregation: sum by, avg, topk, and joining with on
  • Error ratios and request rates per path
  • Percentiles with histogram_quantile, and how bucket choice affects the answer
  • Recording rules and common mistakes

In simple words

Raw metrics are just long lists of numbers. PromQL is the language you use to ask questions of them, such as "how many errors per second in the last five minutes?" or "how slow is the slowest 1% of requests?". A handful of patterns answer most everyday questions.

Words to know

  • PromQL: the query language of Prometheus.
  • rate(): turns an always-growing counter into "how many per second".
  • Aggregation: combining many series into one, such as a total for each service.
  • Percentile: a value most requests are faster than. p99 means 99% were quicker than this.

Selecting series

A bare metric name returns every series (an instant vector: one current value per series). Narrow it with label matchers:

http_requests_total                                  # all series
http_requests_total{path="/flaky"}                   # equality
http_requests_total{status=~"5.."}                   # regex match: 500-599
http_requests_total{path!="/metrics", status!="200"} # negatives

Adding [5m] makes a range vector: all samples from the last 5 minutes per series. Range vectors feed functions like rate(); they cannot be graphed directly.

Rates, not raw counters

A counter's raw value (249, 250, 251...) says little and resets when the process restarts. What you want is how fast it is increasing:

rate(http_requests_total[1m])           # per-second average over the last minute
increase(http_requests_total[1m])       # total increase over the last minute

rate() handles counter resets automatically. The range ([1m]) must contain at least a couple of scrapes: use at least four times the scrape interval (with a 5 s scrape, [30s] or more; with 15 s scrapes use [1m] or [5m]). A longer range is smoother and slower to react; a shorter one is noisier.

Aggregating

rate() keeps all labels. Aggregate to answer a question:

sum by (path) (rate(http_requests_total[30s]))

Real result under load (requests per second):

{path="/"}      5.32
{path="/slow"}  1.52
{path="/flaky"} 2.92

sum by (path) adds all series sharing a path (across statuses and instances) and drops other labels. sum without (instance) is the opposite style (drop only the named labels). Others: avg, min, max, count, topk(3, ...), quantile. sum(...) with no by gives one total.

Error ratio

The classic "what share of requests fail?":

sum(rate(http_requests_total{status=~"5.."}[30s])) / sum(rate(http_requests_total[30s]))
{}  0.061475      (6.1% of requests are 5xx)

Division of two aggregated vectors with no remaining labels works directly. When you keep labels (per path) both sides need matching label sets: sum by (path) (rate(...{status=~"5.."}[1m])) / sum by (path) (rate(...[1m])). Beware paths with no errors: the numerator has no series for them, so they disappear from the result instead of showing 0; use or or unless patterns or alert on absence deliberately.

For the status codes over the last minute:

sum by (status) (increase(http_requests_total[1m]))
{status="200"}  426.9
{status="500"}  22.7

Note increase extrapolates, so values can be non-integers.

Gauges

Gauges are used directly or with functions: http_requests_in_flight, avg_over_time(queue_depth[5m]), max_over_time(memory_bytes[1h]), deriv(disk_free_bytes[1h]), predict_linear(node_filesystem_avail_bytes[6h], 4*3600) < 0 (will the disk fill within 4 hours?).

Latency: averages and percentiles

Average latency from a histogram's _sum and _count:

sum by (path) (rate(http_request_duration_seconds_sum[30s]))
  / sum by (path) (rate(http_request_duration_seconds_count[30s]))
{path="/"}      0.0185
{path="/slow"}  0.5151
{path="/flaky"} 0.0302

Averages hide the slow tail. Percentiles use the buckets:

histogram_quantile(0.95, sum by (le, path) (rate(http_request_duration_seconds_bucket[30s])))
{path="/"}      0.0454
{path="/slow"}  2.2625
{path="/flaky"} 0.0481

Three things to know about this very common query:

  1. The le (bucket boundary) label must stay in the aggregation (sum by (le, ...)); drop it and the result is meaningless.
  2. Aggregate the buckets across instances before taking the quantile; you cannot average percentiles from several instances.
  3. The result is an estimate assuming a uniform distribution inside the bucket where the quantile falls. Our /slow endpoint never took longer than 1.2 seconds, yet the p95 shows 2.26 s. The reason: our buckets are ... 0.5, 1, 2.5, and the 95th percentile falls in the wide 1-2.5 s bucket, so the interpolation lands at 2.26 s. Percentile accuracy depends on bucket design: put bucket boundaries around your SLO thresholds (for a 300 ms latency target include 0.25, 0.3, 0.35...) and keep them narrow where it matters. (Native histograms in newer Prometheus remove this limitation.)

Selectors in time, offsets and comparison

rate(http_requests_total[5m]) offset 1d          # the same query a day ago
sum(rate(http_requests_total[5m])) / sum(rate(http_requests_total[5m] offset 1w))   # compare with last week

Also absent(up{job="shop-api"}) is 1 when the series is missing entirely (alert for a target that vanished from discovery), and up == 0 for scrape failures.

Recording rules

Complex expressions evaluated on every dashboard refresh are expensive. A recording rule evaluates a query regularly and stores the result as a new series:

groups:
  - name: shop-api
    rules:
      - record: job:http_requests:rate1m
        expr: sum by (job) (rate(http_requests_total[1m]))
      - record: job:http_errors:ratio1m
        expr: sum by (job) (rate(http_requests_total{status=~"5.."}[1m])) / sum by (job) (rate(http_requests_total[1m]))

Querying job:http_errors:ratio1m returned {job="shop-api"} 0.0504. Naming convention: level:metric:operations. Use recording rules for dashboards and alerts that use heavy aggregations; check syntax with promtool check rules rules.yml (output SUCCESS: 4 rules found).

Exploring and debugging queries

  • Run queries in the Prometheus UI (Graph tab) or curl 'localhost:9090/api/v1/query?query=up'; the Table view shows labels and values.
  • Explore metric names: type a few letters in the UI, or query count by (__name__) ({job="shop-api"}).
  • If a query returns nothing: check the metric name and labels exactly, the time range, and that the target is up.
  • If a ratio is empty: label sets differ between numerator and denominator (use on(...)/ignoring(...)).
  • Use Grafana Explore to try queries side by side with logs (Loki) and traces.

Common mistakes

  • Graphing a raw counter, or applying rate() to a gauge.
  • rate() range shorter than two scrape intervals (returns nothing or noise).
  • Dropping le in a histogram_quantile.
  • Averaging percentiles across instances.
  • Using irate() for alerts (instant rate is spiky; use rate).
  • Trusting percentiles when buckets are poorly chosen.
  • Expensive queries on dashboards (wide regexes, long ranges) without recording rules.
  • sum(rate()) before histogram_quantile forgotten per instance.

Try it yourself

  1. Write the query for requests per second by status code and for the 99th percentile latency of one endpoint on your own app.
  2. Compute the error ratio per path and notice which paths vanish when they have no errors; fix it with an or vector(0) pattern or by alerting on the total.
  3. Change your histogram buckets so that your SLO threshold is a bucket boundary and compare the percentile before and after.
  4. Write a recording rule for your most expensive dashboard query and validate it with promtool check rules.

Check yourself

Why do you use rate() on counters?

Raw counters only grow and reset on restart; rate() gives the per-second increase, handling resets, which is the meaningful quantity.

What does sum by (le, path) (rate(..._bucket[5m])) do inside histogram_quantile?

Aggregates bucket counters over all instances per bucket boundary and path, keeping le so the quantile can be estimated from the cumulative distribution.

Why can a p95 estimate be higher than any request actually observed?

The quantile is interpolated within a bucket; with wide buckets (1-2.5 s here) the estimate falls inside that range even if real values were smaller.

Why use recording rules?

To precompute expensive aggregations once, making dashboards and alerts faster and consistent.

Interview questions

How would you calculate the error rate of a service from Prometheus metrics?

sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m])), optionally by service, with a recording rule for use in dashboards and alerts.

Why are averages a poor latency metric?

They hide the tail: a few very slow requests barely move the mean but ruin user experience. Use percentiles (p95/p99) from histograms.

What is the difference between rate, irate and increase?

rate is the average per-second increase over the range; irate uses only the last two samples (spiky, for fast-moving graphs); increase is the total increase over the range.

Summary

  • Select with labels, take rate() of counters, aggregate with sum by, and compute ratios of aggregates for error rates.
  • Histograms give averages (_sum / _count) and percentiles (histogram_quantile, keeping le); choose buckets around your SLOs.
  • Record expensive expressions with recording rules and validate with promtool.
  • Use ranges of several scrape intervals; beware empty vectors and label mismatches.

Going deeper

See how much of this you can explain

A free mock interview asks you new questions on Monitoring and observability for DevOps Engineer and scores every answer, with what to fix.