Lesson 4 of 8 · 11 min · Alerting and reliability
Alerting rules and Alertmanager
What you will learn
- Anatomy of an alerting rule: expression,
for, labels, annotations - How alerts move through pending and firing
- Alertmanager: grouping, routing, inhibition, silences and receivers
- Testing rules with
promtool test rules, and validating configuration - What deserves a page, and how to avoid alert fatigue
In simple words
Nobody watches dashboards all day. An alert is a rule that watches for you and sends a message when something is wrong. The difficult part is alerting on the right things: too many alerts and people start ignoring them, the way people ignore a car alarm that never stops.
Words to know
- Alerting rule: a condition which, when true for long enough, raises an alert.
- Alertmanager: the tool that groups alerts and sends them to the right people.
- Silence: muting an alert for a while, for example during planned work.
- Alert fatigue: being sent so many alerts that the important ones get missed.
Alerting rules
An alerting rule is a PromQL expression that, when it returns any series, creates alerts. From our test stack:
groups:
- name: shop-api
rules:
- alert: HighErrorRate
expr: job:http_errors:ratio1m > 0.05
for: 15s
labels:
severity: page
annotations:
summary: "Error ratio is {{ $value | humanizePercentage }} on {{ $labels.job }}"
- alert: TargetDown
expr: up == 0
for: 10s
labels:
severity: page
annotations:
summary: "{{ $labels.instance }} is down"
expr: the condition; each returned series (with its labels) is an alert instance.for: the condition must hold continuously for this long before the alert fires. Until then the alert is pending. This filters out blips: a one-off scrape failure should not wake anyone. (Our 10-15 seconds is for the demo; real values are minutes, such asfor: 5m.)labels: added to the alert; used for routing (severity,team,service).annotations: human-readable text with templating:{{ $labels.x }}for labels and{{ $value }}for the current value; put a runbook URL in an annotation so the person paged knows what to do.
Alert state machine: inactive → pending → firing → resolved. The Prometheus API at /api/v1/alerts lists them. During our load test HighErrorRate showed pending with the summary Error ratio is 5.038% on shop-api; when we stopped the app, TargetDown went to firing (app:8000 is down) after 10 seconds.
Prometheus only evaluates rules and sends alerts; Alertmanager decides who gets notified and how.
Alertmanager
route:
receiver: default
group_by: [alertname, job]
group_wait: 5s
group_interval: 10s
repeat_interval: 1h
receivers:
- name: default
webhook_configs:
- url: http://hook:8080/
Its jobs:
- Deduplicate: many Prometheus servers or rule evaluations sending the same alert become one.
- Group: alerts with the same
group_bylabels are combined into a single notification, so 200 pods failing produce one message ("200 PodCrashLooping in namespace shop") rather than 200 pages.group_waitwaits a few seconds to collect related alerts first;group_intervalthrottles updates to an existing group;repeat_intervalre-notifies while the alert keeps firing. - Route: a tree of matchers sends alerts to receivers by label:
severity=page→ PagerDuty/Opsgenie,severity=ticket→ Jira/email/Slack channel,team=payments→ the payments channel. Child routes override parents;continue: truelets an alert match several routes. - Inhibit: suppress alerts made redundant by others: when
ClusterDownfires, silence the hundred service alerts it causes (inhibit_ruleswith source and target matchers andequallabels). - Silence: temporary mutes created in the UI or with
amtool silence add, for planned maintenance. Always with an expiry and a reason. - Notify: receivers for email, Slack, Microsoft Teams, PagerDuty, Opsgenie, VictorOps, webhooks and more, with message templates.
A realistic routing tree:
route:
receiver: slack-default
group_by: [alertname, service]
routes:
- matchers: [severity="page"]
receiver: pagerduty-oncall
group_wait: 10s
continue: false
- matchers: [severity="ticket"]
receiver: slack-tickets
inhibit_rules:
- source_matchers: [alertname="ClusterDown"]
target_matchers: [severity=~"page|ticket"]
equal: [cluster]
End to end, our webhook receiver got the notification as JSON from Alertmanager: status firing, grouped by {'alertname': 'TargetDown', 'job': 'shop-api'}, with the annotation app:8000 is down; after we restarted the app a second message arrived with status resolved. Send "resolved" notifications: they tell responders the problem went away. Alertmanager's own API (/api/v2/alerts) showed the alert as active.
Alertmanager is made highly available by running 2-3 instances in a gossip cluster; Prometheus sends alerts to all. Do not load balance them.
Test your rules
Alert rules are code: test them before production. Syntax check:
promtool check rules rules.yml
# SUCCESS: 4 rules found
amtool check-config alertmanager.yml
# SUCCESS (found: global config, route, 0 inhibit rules, 1 receivers)
And unit tests with synthetic series, so you know the alert fires when it should and stays quiet when it should:
rule_files: [rules.yml]
evaluation_interval: 5s
tests:
- interval: 5s
input_series:
- series: 'http_requests_total{job="shop-api",status="200"}'
values: '0+10x20' # +10 each 5s step
- series: 'http_requests_total{job="shop-api",status="500"}'
values: '0+2x20' # +2 each step: 2/12 = 16.67% errors
alert_rule_test:
- eval_time: 90s
alertname: HighErrorRate
exp_alerts:
- exp_labels: {severity: page, job: shop-api}
exp_annotations:
summary: "Error ratio is 16.67% on shop-api"
- eval_time: 3s
alertname: HighErrorRate
exp_alerts: [] # not yet: still pending
promtool test rules rules_test.yml printed SUCCESS. Put these checks in CI (CI/CD course) so a typo in a rule is caught in the pull request. In Kubernetes the Prometheus Operator uses PrometheusRule objects and the same promtool checks.
What should page a human
Alerts interrupt people, sleep and weekends. Principles from the Google SRE book:
- Alert on symptoms, not causes. Page when users are affected (high error ratio, high latency, an SLO burning) rather than on "CPU at 85%" or "pod restarted once". Causes belong on dashboards and in ticket-level alerts.
- Every page must be urgent, actionable and need a human. If the answer is "wait and see", it should not be a page. If the fix is always the same script, automate it.
- Include context and a runbook: what it means, how to check, who to escalate to.
- Fewer, better alerts. Review alerts that fire often without action and delete or tune them; track pages per on-call shift as a health metric. Alert fatigue makes people ignore the one that matters.
- Use severities and routing: pages for user-impacting, urgent issues; tickets/Slack for things to fix within days (disk will fill in 3 days, certificate expires in 14 days).
- Avoid flapping: use
for, hysteresis and rate windows long enough to avoid noise. - Alert on absence and on the monitoring itself:
absent(...)for vanished series, a "dead man's switch" alert that always fires and a heartbeat service that notifies you if it stops (so you know Alertmanager and the path to PagerDuty work), alerts on Prometheus being down or failing to scrape. - Prefer SLO burn-rate alerts for service health (next lesson) over static thresholds.
- Blameless post-incident reviews feed back into the alert set.
Useful infrastructure alerts that are actionable: node or volume will run out of space within N hours (predict_linear), certificate expiry within 14 days, pod crash-looping for 10 minutes, deployment replicas not available, backup older than 26 hours, queue backlog growing for 15 minutes.
Common mistakes
- Alerting on every cause (CPU, memory, restarts) → noise.
- No
forclause → flapping pages from one blip. - Alerts without runbooks or owners.
- Rules never tested until the incident.
- Routing everything to one channel; no severity separation.
- No silences discipline, or silences without expiry.
- Assuming the alert path works without a heartbeat test.
- Static thresholds copied from blog posts instead of derived from SLOs.
Try it yourself
- Write a
HighErrorRaterule for your service and unit-test it withpromtool test rulesincluding a "does not fire" case. - Configure Alertmanager with a webhook receiver and watch the JSON of a firing and a resolved notification.
- Add routing by
severityand an inhibit rule so a cluster-wide alert suppresses per-service ones. - Add a dead man's switch alert and a runbook URL annotation to every page-level alert.
Check yourself
What does the for clause in an alerting rule do?
It requires the condition to hold continuously for that duration before the alert moves from pending to firing, filtering short blips.
Where do grouping, routing and silencing happen: Prometheus or Alertmanager?
Alertmanager. Prometheus evaluates rules and sends alerts; Alertmanager groups, routes, inhibits, silences and notifies.
Why alert on symptoms rather than causes?
Symptoms (errors, latency affecting users) are what matter and need human action; causes (CPU high, a pod restarted) often have no user impact and generate noise.
How do you check that an alerting rule fires correctly before production?
Write a promtool test rules unit test with synthetic input series and expected alerts, and run it in CI along with promtool check rules.
Interview questions
How do you reduce alert fatigue?
Alert only on actionable, user-impacting symptoms (SLO burn), group and route alerts by severity, add for durations, remove or demote noisy alerts, add runbooks, automate known fixes, and review pages per shift regularly.
How would you know if your alerting itself is broken?
A dead man's switch/heartbeat alert that must always be firing and notifies an external service if it stops, alerts on Prometheus and Alertmanager health, and periodic end-to-end tests of the notification path.
What is the difference between a silence and an inhibition?
A silence is a manual temporary mute of alerts matching labels (for maintenance). An inhibition rule automatically suppresses some alerts while another, more fundamental alert is firing.
Summary
- Alerting rules turn PromQL conditions into alerts with
for, labels and annotations (with runbook links); Alertmanager groups, routes, inhibits, silences and notifies. - Test rules with
promtool check rulesandpromtool test rules, and configuration withamtool check-config, in CI. - Page only on urgent, actionable, user-visible symptoms; send the rest to tickets; monitor the monitoring.
- Fewer, better, tested alerts keep on-call sustainable.
Going deeper
- Alerting rules: syntax and templating.
- Alertmanager: grouping, inhibition, silences and high availability.
- Alertmanager configuration: routes and receivers.
- Unit testing for rules:
promtool test rules. - Google SRE book: Monitoring Distributed Systems: what to alert on.
- Google SRE book: Being On-Call: sustainable on-call practice.
See how much of this you can explain
A free mock interview asks you new questions on Monitoring and observability for DevOps Engineer and scores every answer, with what to fix.