Lesson 2 of 8 · 9 min · Metrics and dashboards
PromQL in practice: rates, ratios and percentiles
What you will learn
- Instant and range vectors, selectors and label matchers
rate(),increase()and why counters are never used raw- Aggregation:
sum by,avg,topk, and joining withon - Error ratios and request rates per path
- Percentiles with
histogram_quantile, and how bucket choice affects the answer - Recording rules and common mistakes
In simple words
Raw metrics are just long lists of numbers. PromQL is the language you use to ask questions of them, such as "how many errors per second in the last five minutes?" or "how slow is the slowest 1% of requests?". A handful of patterns answer most everyday questions.
Words to know
- PromQL: the query language of Prometheus.
- rate(): turns an always-growing counter into "how many per second".
- Aggregation: combining many series into one, such as a total for each service.
- Percentile: a value most requests are faster than. p99 means 99% were quicker than this.
Selecting series
A bare metric name returns every series (an instant vector: one current value per series). Narrow it with label matchers:
http_requests_total # all series
http_requests_total{path="/flaky"} # equality
http_requests_total{status=~"5.."} # regex match: 500-599
http_requests_total{path!="/metrics", status!="200"} # negatives
Adding [5m] makes a range vector: all samples from the last 5 minutes per series. Range vectors feed functions like rate(); they cannot be graphed directly.
Rates, not raw counters
A counter's raw value (249, 250, 251...) says little and resets when the process restarts. What you want is how fast it is increasing:
rate(http_requests_total[1m]) # per-second average over the last minute
increase(http_requests_total[1m]) # total increase over the last minute
rate() handles counter resets automatically. The range ([1m]) must contain at least a couple of scrapes: use at least four times the scrape interval (with a 5 s scrape, [30s] or more; with 15 s scrapes use [1m] or [5m]). A longer range is smoother and slower to react; a shorter one is noisier.
Aggregating
rate() keeps all labels. Aggregate to answer a question:
sum by (path) (rate(http_requests_total[30s]))
Real result under load (requests per second):
{path="/"} 5.32
{path="/slow"} 1.52
{path="/flaky"} 2.92
sum by (path) adds all series sharing a path (across statuses and instances) and drops other labels. sum without (instance) is the opposite style (drop only the named labels). Others: avg, min, max, count, topk(3, ...), quantile. sum(...) with no by gives one total.
Error ratio
The classic "what share of requests fail?":
sum(rate(http_requests_total{status=~"5.."}[30s])) / sum(rate(http_requests_total[30s]))
{} 0.061475 (6.1% of requests are 5xx)
Division of two aggregated vectors with no remaining labels works directly. When you keep labels (per path) both sides need matching label sets: sum by (path) (rate(...{status=~"5.."}[1m])) / sum by (path) (rate(...[1m])). Beware paths with no errors: the numerator has no series for them, so they disappear from the result instead of showing 0; use or or unless patterns or alert on absence deliberately.
For the status codes over the last minute:
sum by (status) (increase(http_requests_total[1m]))
{status="200"} 426.9
{status="500"} 22.7
Note increase extrapolates, so values can be non-integers.
Gauges
Gauges are used directly or with functions: http_requests_in_flight, avg_over_time(queue_depth[5m]), max_over_time(memory_bytes[1h]), deriv(disk_free_bytes[1h]), predict_linear(node_filesystem_avail_bytes[6h], 4*3600) < 0 (will the disk fill within 4 hours?).
Latency: averages and percentiles
Average latency from a histogram's _sum and _count:
sum by (path) (rate(http_request_duration_seconds_sum[30s]))
/ sum by (path) (rate(http_request_duration_seconds_count[30s]))
{path="/"} 0.0185
{path="/slow"} 0.5151
{path="/flaky"} 0.0302
Averages hide the slow tail. Percentiles use the buckets:
histogram_quantile(0.95, sum by (le, path) (rate(http_request_duration_seconds_bucket[30s])))
{path="/"} 0.0454
{path="/slow"} 2.2625
{path="/flaky"} 0.0481
Three things to know about this very common query:
- The
le(bucket boundary) label must stay in the aggregation (sum by (le, ...)); drop it and the result is meaningless. - Aggregate the buckets across instances before taking the quantile; you cannot average percentiles from several instances.
- The result is an estimate assuming a uniform distribution inside the bucket where the quantile falls. Our
/slowendpoint never took longer than 1.2 seconds, yet the p95 shows 2.26 s. The reason: our buckets are... 0.5, 1, 2.5, and the 95th percentile falls in the wide 1-2.5 s bucket, so the interpolation lands at 2.26 s. Percentile accuracy depends on bucket design: put bucket boundaries around your SLO thresholds (for a 300 ms latency target include 0.25, 0.3, 0.35...) and keep them narrow where it matters. (Native histograms in newer Prometheus remove this limitation.)
Selectors in time, offsets and comparison
rate(http_requests_total[5m]) offset 1d # the same query a day ago
sum(rate(http_requests_total[5m])) / sum(rate(http_requests_total[5m] offset 1w)) # compare with last week
Also absent(up{job="shop-api"}) is 1 when the series is missing entirely (alert for a target that vanished from discovery), and up == 0 for scrape failures.
Recording rules
Complex expressions evaluated on every dashboard refresh are expensive. A recording rule evaluates a query regularly and stores the result as a new series:
groups:
- name: shop-api
rules:
- record: job:http_requests:rate1m
expr: sum by (job) (rate(http_requests_total[1m]))
- record: job:http_errors:ratio1m
expr: sum by (job) (rate(http_requests_total{status=~"5.."}[1m])) / sum by (job) (rate(http_requests_total[1m]))
Querying job:http_errors:ratio1m returned {job="shop-api"} 0.0504. Naming convention: level:metric:operations. Use recording rules for dashboards and alerts that use heavy aggregations; check syntax with promtool check rules rules.yml (output SUCCESS: 4 rules found).
Exploring and debugging queries
- Run queries in the Prometheus UI (Graph tab) or
curl 'localhost:9090/api/v1/query?query=up'; the Table view shows labels and values. - Explore metric names: type a few letters in the UI, or query
count by (__name__) ({job="shop-api"}). - If a query returns nothing: check the metric name and labels exactly, the time range, and that the target is
up. - If a ratio is empty: label sets differ between numerator and denominator (use
on(...)/ignoring(...)). - Use Grafana Explore to try queries side by side with logs (Loki) and traces.
Common mistakes
- Graphing a raw counter, or applying
rate()to a gauge. rate()range shorter than two scrape intervals (returns nothing or noise).- Dropping
lein ahistogram_quantile. - Averaging percentiles across instances.
- Using
irate()for alerts (instant rate is spiky; userate). - Trusting percentiles when buckets are poorly chosen.
- Expensive queries on dashboards (wide regexes, long ranges) without recording rules.
sum(rate())beforehistogram_quantileforgotten per instance.
Try it yourself
- Write the query for requests per second by status code and for the 99th percentile latency of one endpoint on your own app.
- Compute the error ratio per path and notice which paths vanish when they have no errors; fix it with an
or vector(0)pattern or by alerting on the total. - Change your histogram buckets so that your SLO threshold is a bucket boundary and compare the percentile before and after.
- Write a recording rule for your most expensive dashboard query and validate it with
promtool check rules.
Check yourself
Why do you use rate() on counters?
Raw counters only grow and reset on restart; rate() gives the per-second increase, handling resets, which is the meaningful quantity.
What does sum by (le, path) (rate(..._bucket[5m])) do inside histogram_quantile?
Aggregates bucket counters over all instances per bucket boundary and path, keeping le so the quantile can be estimated from the cumulative distribution.
Why can a p95 estimate be higher than any request actually observed?
The quantile is interpolated within a bucket; with wide buckets (1-2.5 s here) the estimate falls inside that range even if real values were smaller.
Why use recording rules?
To precompute expensive aggregations once, making dashboards and alerts faster and consistent.
Interview questions
How would you calculate the error rate of a service from Prometheus metrics?
sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m])), optionally by service, with a recording rule for use in dashboards and alerts.
Why are averages a poor latency metric?
They hide the tail: a few very slow requests barely move the mean but ruin user experience. Use percentiles (p95/p99) from histograms.
What is the difference between rate, irate and increase?
rate is the average per-second increase over the range; irate uses only the last two samples (spiky, for fast-moving graphs); increase is the total increase over the range.
Summary
- Select with labels, take
rate()of counters, aggregate withsum by, and compute ratios of aggregates for error rates. - Histograms give averages (
_sum / _count) and percentiles (histogram_quantile, keepingle); choose buckets around your SLOs. - Record expensive expressions with recording rules and validate with
promtool. - Use ranges of several scrape intervals; beware empty vectors and label mismatches.
Going deeper
- Querying Prometheus: basics: selectors, vectors and matchers.
- Query functions:
rate,increase,histogram_quantile,predict_linearand more. - Query operators: aggregation and vector matching.
- Histograms and summaries: choosing buckets and accuracy.
- Recording rules: syntax and naming.
See how much of this you can explain
A free mock interview asks you new questions on Monitoring and observability for DevOps Engineer and scores every answer, with what to fix.