Lesson 5 of 8 · 13 min · Alerting and reliability

SLIs, SLOs and error budgets

Covers: Prometheus rules; promtool. Updated 13 Nov 2026. Part of the DevOps Engineer roadmap.

What you will learn

  • SLI, SLO and SLA: what each means and how they differ
  • Choosing good SLIs for different kinds of systems
  • Computing the error budget
  • Burn rate and multi-window, multi-burn-rate alerts (tested)
  • Using error budgets in decisions and process

In simple words

"The site should always work" is not a target you can measure. An SLO is: for example, "99.9% of requests succeed each month". The small share that is allowed to fail is your error budget. While budget is left you can release quickly. When it is used up, you slow down and fix reliability.

Words to know

  • SLI: the measurement, such as the percentage of requests that succeed.
  • SLO: the target you set for that measurement, such as 99.9%.
  • SLA: a promise to customers, with a penalty if it is broken.
  • Error budget: how much failure the SLO allows. 99.9% leaves about 43 minutes a month.

The vocabulary

  • SLI (service level indicator): a measurement of service behaviour from the user's point of view, expressed as a ratio of good events to valid events: "the proportion of requests answered successfully in under 300 ms".
  • SLO (service level objective): a target for an SLI over a time window: "99.9% of requests are good over 30 days".
  • SLA (service level agreement): a contract with customers, with consequences (credits, penalties) if missed. SLAs are looser than internal SLOs: if the contract promises 99.9%, aim your SLO at 99.95%.
  • Error budget: 1 - SLO: the amount of unreliability you are allowed. A 99.9% SLO allows 0.1% of requests (or time) to be bad.

Why bother? "As reliable as possible" is an unbounded and expensive goal. Users cannot tell 99.99% from 99.999%, but you pay dearly for each extra nine, and perfect reliability means never shipping anything. SLOs state how reliable is reliable enough, give engineering and product a shared, numeric basis for decisions, and replace opinion-driven alerting with budget-driven alerting.

Choosing SLIs

Pick indicators that reflect what users experience, measured as close to the user as practical (load balancer or client-side, not only the application).

System type Typical SLIs
Request/response service (web, API) availability = good responses / all valid requests (non-5xx); latency = share of requests faster than a threshold (for example 300 ms)
Data pipeline / batch freshness (data not older than X), correctness, throughput, completion within a deadline
Storage durability, availability, latency
Streaming/queues end-to-end latency, lag
Mobile/web front end page load, crash-free sessions, successful key journeys

Design rules:

  • Express every SLI as good events / total events, between 0 and 100%.
  • Define "good" carefully: a 200 response with an error body is not good; a 4xx caused by the client (404 on a missing id) usually is not counted against you, but a 429 from your own rate limiter may be.
  • Latency as a threshold ratio, not an average: "95% of requests under 300 ms" (and a second objective like "99% under 1 s").
  • Measure critical user journeys (checkout, login, search), not every endpoint equally.
  • Keep the list short: 2 to 4 SLIs per service.

An availability SLI in PromQL (using our test metrics):

1 - (
  sum(rate(http_requests_total{status=~"5.."}[30d])) / sum(rate(http_requests_total[30d]))
)

and a latency SLI with histogram buckets (the share of requests at or below 0.5 s; make 0.5 a bucket boundary):

sum(rate(http_request_duration_seconds_bucket{le="0.5"}[5m])) / sum(rate(http_request_duration_seconds_count[5m]))

Setting SLOs

Start from reality and users, not aspiration:

  1. Measure what you deliver today and how users react (complaints, churn, support tickets).
  2. Choose a target that is achievable and sufficient; ask product what users would notice. Internal tools can target 99%, consumer checkout 99.9%, critical infrastructure higher.
  3. Use a window such as 28 or 30 days rolling (and perhaps a quarterly view).
  4. Write it down with owners; revisit quarterly.
  5. Remember dependencies: your availability cannot exceed that of the components in your critical path (0.999 × 0.999 × 0.999 ≈ 99.7%).

What an SLO means in time (computed):

SLO Allowed bad time per 30 days Per year
99% 432 minutes (7.2 hours) 87.6 hours
99.5% 216 minutes 43.8 hours
99.9% 43.2 minutes 8.76 hours
99.95% 21.6 minutes 4.38 hours
99.99% 4.3 minutes 0.88 hours

(For request-based SLIs the budget is a share of requests rather than minutes, but the proportions are the same.)

The error budget

For a 99.9% SLO the error budget is 0.1% of requests in the window. If you serve 100 million requests in 30 days you can fail 100,000 of them and still meet the SLO. The budget is a resource to spend: on risky releases, experiments, maintenance, infrastructure migrations, or on actual incidents.

Budget policy turns it into action:

  • Budget remaining: ship features, run experiments, move fast.
  • Budget nearly exhausted or burned: slow down; prioritise reliability work; freeze risky launches until it recovers.
  • Chronically unspent budget: you may be too reliable: ship faster or tighten the SLO only if users benefit.

This aligns developers (who want to ship) and operations (who want stability) around one number instead of an argument. Write the policy down and get product and engineering management to agree before the first breach.

Burn rate

Burn rate says how fast you are consuming the budget compared with an even spend. Burn rate = observed error ratio / allowed error ratio.

  • Burn rate 1 → you'll use exactly the whole 30-day budget in 30 days.
  • Burn rate 14.4 → you'll use it all in 30 days / 14.4 ≈ 2.08 days (50 hours).
  • Burn rate 6 → budget gone in 5 days.
  • Burn rate 3 → 10 days; burn rate 1 → 30 days.

If a 99.9% SLO's allowed error ratio is 0.001 and you currently see 2% errors, the burn rate is 0.02 / 0.001 = 20: you are on course to lose the month's budget in 36 hours. A one-hour burn at 14.4× consumes 14.4 × (1/720) = 2% of the monthly budget.

Alerting on burn rate

Static thresholds ("error rate above 1%") are either too noisy or too slow relative to your SLO. The approach recommended in the Google SRE Workbook is multi-window, multi-burn-rate alerts:

  • A long window shows the problem is significant (it has already consumed meaningful budget).
  • A short window (about 1/12 of the long one) shows it is still happening (so the alert resolves quickly after a fix and doesn't page for a problem that ended).
  • Different burn rates for different urgency: fast burn pages immediately; slow burn creates a ticket.
Severity Burn rate Long window Short window Budget consumed before alert Response
Page 14.4 1 h 5 min 2% page now
Page 6 6 h 30 min 5% page
Ticket 1 3 days 6 h 10% fix during working hours

Prometheus rules for a 99.9% SLO (the first two rows), with recorded error ratios at several windows:

groups:
  - name: slo-shop-api
    rules:
      - record: sli:errors:ratio_rate5m
        expr: sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m]))
      - record: sli:errors:ratio_rate30m
        expr: sum(rate(http_requests_total{status=~"5.."}[30m])) / sum(rate(http_requests_total[30m]))
      - record: sli:errors:ratio_rate1h
        expr: sum(rate(http_requests_total{status=~"5.."}[1h])) / sum(rate(http_requests_total[1h]))
      - record: sli:errors:ratio_rate6h
        expr: sum(rate(http_requests_total{status=~"5.."}[6h])) / sum(rate(http_requests_total[6h]))
      - alert: ErrorBudgetFastBurn
        expr: sli:errors:ratio_rate1h > (14.4 * 0.001) and sli:errors:ratio_rate5m > (14.4 * 0.001)
        labels: {severity: page}
        annotations:
          summary: "Burning the 30-day error budget 14.4x too fast (budget gone in ~2 days)"
      - alert: ErrorBudgetSlowBurn
        expr: sli:errors:ratio_rate6h > (6 * 0.001) and sli:errors:ratio_rate30m > (6 * 0.001)
        labels: {severity: ticket}
        annotations:
          summary: "Burning the 30-day error budget 6x too fast"

We validated the file (promtool check rules: SUCCESS: 6 rules found) and unit-tested it with synthetic traffic at 10 requests per second:

  • With 2% errors for 3 hours (burn rate 20) the fast-burn page and the slow-burn ticket both fired, with the expected labels and summaries.
  • With exactly 0.1% errors (burn rate 1, spending the budget at the planned pace) neither alert fired.

Both tests passed. This is the value of testing alert logic: you know the thresholds behave as designed. Tools like Sloth, Pyrra and OpenSLO generate these rules (and dashboards) from a short SLO definition so you do not hand-write them for every service.

Dashboards and reporting

Show for each SLO: the current SLI, the target, the remaining error budget (percentage and minutes), the burn rate over several windows, and the events that consumed budget (incidents, deploys). Review SLO status in a regular reliability meeting, and use budget exhaustion to drive priorities.

Limits and traps

  • Aggregate SLOs hide pain: 99.9% overall may mean one important customer or region suffers badly; add per-segment SLIs for key customers.
  • Low-traffic services produce noisy ratios (one error among ten requests = 10%); use longer windows or synthetic probes.
  • Don't count what users don't feel (health checks, bots) and do count what they do (timeouts that never reached your app: measure at the load balancer or with external probes).
  • SLO targets need honesty: if you always exceed the SLO by a wide margin, it is not driving decisions.
  • Perverse incentives: a budget policy must not punish teams for incidents; it informs priorities.

Common mistakes

  • No SLO at all, or one chosen as "as many nines as possible".
  • SLIs from internal metrics users do not feel (CPU, queue depth).
  • Targets set without data.
  • Alerting on raw error thresholds instead of burn rates.
  • Only a long window (alerts long after the problem ended) or only a short window (flapping).
  • Budget policy with no enforcement, so the number is decorative.
  • Treating the SLA as the SLO.

Try it yourself

  1. Pick a service you know and write two SLIs (availability, latency) as good/total ratios with a clear definition of "good".
  2. Choose SLO targets, compute the allowed bad minutes per 30 days, and write what you would do when 50% and 100% of the budget are gone.
  3. Write the fast- and slow-burn rules for your SLO and unit-test them with promtool test rules.
  4. Build a dashboard panel showing remaining error budget over the last 30 days.

Check yourself

What is the difference between an SLI, an SLO and an SLA?

The SLI is a measurement of service quality; the SLO is the internal target for that measurement over a window; the SLA is a contractual commitment to customers with consequences if breached.

How much downtime does a 99.9% availability SLO allow per 30 days?

About 43.2 minutes (0.1% of 30 days), or 8.76 hours per year.

What does a burn rate of 14.4 mean?

Errors are occurring 14.4 times faster than the rate that would just use up the whole error budget by the end of the window; the 30-day budget would be exhausted in about 2 days.

Why combine a long and a short window in burn-rate alerts?

The long window ensures the problem is significant; the short window ensures it is still happening, so the alert fires quickly and clears quickly after recovery.

Interview questions

How would you introduce SLOs in a team that has none?

Choose a critical user journey, define SLIs as good/total ratios with the team and product, measure for a few weeks to set realistic targets, agree an error-budget policy, build a dashboard, replace threshold alerts with burn-rate alerts, and review the SLO regularly.

What do you do when the error budget is exhausted?

Follow the agreed policy: pause risky feature releases, prioritise reliability fixes and the causes of the budget burn, and resume normal velocity when the budget recovers.

Why is 100% availability a bad target?

It is unattainable and increasingly expensive, users often cannot perceive the difference, their own networks and devices are less reliable anyway, and it leaves no room to change the system.

Summary

  • SLIs measure user-visible quality as good/total ratios; SLOs set internal targets over a window; SLAs are contracts looser than SLOs.
  • The error budget (1 − SLO) is a spendable resource that balances reliability and feature velocity through an agreed policy.
  • Alert on burn rate with multi-window rules (fast burn pages, slow burn tickets); test the rules with promtool.
  • Keep SLOs few, honest and reviewed; watch for segment-level pain and noisy low-traffic ratios.

Going deeper

See how much of this you can explain

A free mock interview asks you new questions on Monitoring and observability for DevOps Engineer and scores every answer, with what to fix.