Lesson 3 of 8 · 11 min · Metrics and dashboards

Dashboards in Grafana

Covers: Grafana 13 with a provisioned Prometheus data source and dashboard. Updated 13 Nov 2026. Part of the DevOps Engineer roadmap.

What you will learn

  • What Grafana does and how data sources work
  • Panels, queries, units, thresholds and variables
  • Provisioning data sources and dashboards from files (dashboards as code)
  • Dashboard design: what to show and in what order
  • Explore, annotations, and moving between metrics, logs and traces
  • Access and operations

In simple words

Numbers in a table are hard to read at 2 a.m. Grafana turns them into charts on a dashboard, so you can see in one look whether things are normal. A good dashboard answers "is it healthy?" first and gives detail only after that. This lesson builds one and shows how to keep it in files.

Words to know

  • Grafana: a tool that draws charts and dashboards from your data.
  • Data source: where Grafana reads data from, such as Prometheus.
  • Panel: one chart or number on a dashboard.
  • Dashboards as code: keeping dashboards in files, so they can be reviewed and restored.

What Grafana is

Grafana is the open-source visualisation and dashboarding tool for observability. It stores no metrics itself. It connects to data sources (Prometheus, Loki, Tempo/Jaeger, Elasticsearch/OpenSearch, CloudWatch, Azure Monitor, SQL databases, InfluxDB and many more), sends them queries, and draws the results. A dashboard is a collection of panels (time series, stat, gauge, bar, table, heatmap, logs, traces) arranged on a grid, with a shared time range and optional variables.

Grafana also provides alerting (Grafana-managed alert rules over any data source), Explore (ad-hoc querying), users, teams, folders and permissions, annotations and a plugin ecosystem. Alerts on Prometheus data are usually kept in Prometheus rules (next lesson); Grafana alerting is handy for non-Prometheus sources.

Data sources

Add one in the UI (Connections → Data sources) or provision it from a file (below). A Prometheus data source needs its URL (http://prometheus:9090) and, in production, authentication and TLS. Use access: proxy so the Grafana server makes the requests (browsers do not need network access to Prometheus, and credentials stay on the server). The uid is a stable id referenced by dashboards.

Panels and queries

A panel contains one or more queries in the data source's language, a visualisation type and display options:

  • Time series for rates and latencies over time. Stat for a single current number (error ratio) with colour thresholds. Gauge/bar gauge for utilisation. Table for lists. Heatmap for histogram buckets (latency distributions over time). Logs and Traces panels for Loki and Jaeger/Tempo.
  • Legend format such as {{path}} names each series from its labels.
  • Units matter: say seconds (s), percent (0.0-1.0 → percentunit), bytes, requests/sec. Grafana then formats 0.0615 as "6.15%" and 2.26 as "2.26 s". Always set units.
  • Thresholds colour values (green below 1%, orange above, red above 5%).
  • Min step / interval: Grafana chooses $__rate_interval (use it in rate() ranges: rate(http_requests_total[$__rate_interval])) so the range always fits the scrape interval and the graph resolution.

Our dashboard's three panels used these queries:

sum by (path) (rate(http_requests_total[1m]))                                  # Requests per second  (legend {{path}})
job:http_errors:ratio1m                                                        # Error ratio          (stat, percentunit)
histogram_quantile(0.95, sum by (le, path) (rate(http_request_duration_seconds_bucket[1m])))   # p95 latency (unit s)

Variables

Dashboard variables add drop-downs ($env, $service, $instance) so one dashboard serves many services. Define a query variable such as label_values(http_requests_total, job) and use {job="$job"} in panels; set "multi-value" and "include All" where helpful. Repeating rows/panels per variable value show one panel per service. Variables are how you avoid copying dashboards.

Dashboards as code: provisioning

Dashboards edited by hand in the UI are lost when Grafana is rebuilt and cannot be reviewed. Treat them as code: keep the JSON (or generate it with Grafonnet/Jsonnet, Terraform's Grafana provider, or Grafana's Git Sync) in Git and provision them at start-up.

provisioning/datasources/prom.yml:

apiVersion: 1
datasources:
  - name: Prometheus
    type: prometheus
    uid: prom
    access: proxy
    url: http://prometheus:9090
    isDefault: true

provisioning/dashboards/dash.yml (tells Grafana where to find dashboard files):

apiVersion: 1
providers:
  - name: default
    folder: Shop
    type: file
    options:
      path: /var/lib/grafana/dashboards

and a dashboard JSON file in that directory (abridged):

{
  "title": "Shop API overview",
  "uid": "shop-api",
  "time": {"from": "now-15m", "to": "now"},
  "refresh": "10s",
  "panels": [
    {"id": 1, "type": "timeseries", "title": "Requests per second",
     "datasource": {"type": "prometheus", "uid": "prom"},
     "targets": [{"refId": "A", "expr": "sum by (path) (rate(http_requests_total[1m]))", "legendFormat": "{{path}}"}]},
    {"id": 2, "type": "stat", "title": "Error ratio",
     "datasource": {"type": "prometheus", "uid": "prom"},
     "fieldConfig": {"defaults": {"unit": "percentunit",
        "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}, {"color": "orange", "value": 0.01}, {"color": "red", "value": 0.05}]}}},
     "targets": [{"refId": "A", "expr": "job:http_errors:ratio1m"}]}
  ]
}

Mounting these directories into the container (./provisioning:/etc/grafana/provisioning, ./dashboards:/var/lib/grafana/dashboards) and starting Grafana, we verified through its HTTP API:

GET /api/health                  -> database: ok, version 13.3.1
GET /api/datasources             -> Prometheus  prometheus  http://prometheus:9090
GET /api/search?type=dash-db     -> Shop API overview  shop-api  (folder Shop)

and a query through Grafana (POST /api/ds/query, the same call a panel makes) for up returned two series, the two Prometheus targets with their labels. A rebuilt Grafana therefore comes up with its data source and dashboard already in place. In Kubernetes the kube-prometheus-stack chart provisions dashboards from ConfigMaps via a sidecar. You can also export any dashboard's JSON from the UI (Share → Export) and commit it; community dashboards at grafana.com/grafana/dashboards (for example Node Exporter Full) are good starting points.

Change the default admin password (we used admin in a lab); use SSO (OIDC/LDAP/GitHub), teams and folder permissions, and viewer roles for most people.

Designing dashboards that work

A dashboard that nobody reads or understands is worse than none. Principles:

  1. Start from questions and audiences. "Is the service healthy right now?" (on-call), "Why is it slow?" (debugging), "How are we doing this quarter?" (management). One dashboard per purpose.
  2. Top row: the golden signals / RED for the service: request rate, error ratio, latency percentiles, saturation. If the top row is green, the service is healthy.
  3. Put the most important at top-left; details below; link to deeper dashboards (drill-down) and runbooks.
  4. Show SLOs and thresholds (a line at the latency target, colour the stat red above the error budget burn). Context turns numbers into judgements.
  5. Use units, titles and descriptions that a new on-call engineer understands at 3 a.m.
  6. Keep it small: 6 to 12 panels; resist dashboard sprawl. Avoid panels that show averages only or dozens of unlabeled lines.
  7. Consistent layout across services (same rows, same colours) so people learn once. Use variables and templates.
  8. Compare: show deploy markers (annotations) and last week's line, so a change is visible.
  9. Make dashboards load fast: recording rules for heavy queries, sensible time ranges and refresh rates (not 1 s).
  10. Dashboards are for humans noticing problems; alerts are for machines noticing them. Never rely on someone watching a screen.

Typical dashboard set: a service overview (RED + saturation + dependencies), infrastructure (node/pod USE metrics), database, business KPIs, and one "SLO" dashboard with error budget.

Operating Grafana

  • Run it highly available if it matters (shared database instead of local SQLite), back up its database or, better, keep everything provisioned from Git.
  • Version dashboards through pull requests and review them like code.
  • Monitor Grafana and your data sources (it has /api/health and its own metrics).
  • Managed options: Grafana Cloud, Amazon Managed Grafana, Azure Managed Grafana.

Common mistakes

  • Dashboards edited only in the UI, unreviewed and unreproducible.
  • Missing units or misleading axes.
  • Dozens of panels with no hierarchy (nobody knows where to look).
  • Averages only.
  • Heavy queries on every refresh (no recording rules).
  • No link from dashboard to runbook/logs/traces.
  • Default credentials left in place; world-readable dashboards exposing internals.
  • Using dashboards instead of alerts for detection.

Try it yourself

  1. Provision a Prometheus data source and a dashboard from files, start Grafana in Docker and confirm both via the API (/api/datasources, /api/search).
  2. Build a RED dashboard for your own service with a service variable, units and thresholds.
  3. Add a deployment annotation from a script or pipeline step.
  4. Import the "Node Exporter Full" dashboard (ID 1860) and identify the USE metrics on it.

Check yourself

What does Grafana store, and what does it not store?

It stores dashboards, users, data source configuration and alert rules, but not the metrics, logs or traces themselves; those stay in the data sources.

Why provision dashboards and data sources from files?

So they are versioned, reviewable and reproducible: a rebuilt Grafana has the same configuration without manual steps.

What is $__rate_interval for?

It lets Grafana pick a rate() range that matches the scrape interval and the panel's resolution, avoiding empty or overly noisy graphs.

What belongs in the top row of a service dashboard?

The golden signals: request rate, error ratio, latency percentiles and a saturation measure, ideally with SLO thresholds.

Interview questions

How would you design the monitoring dashboards for a new microservice?

A standard RED dashboard from a template (rate, errors, duration percentiles, saturation, dependency health), with variables for environment and instance, SLO lines, deployment annotations, links to logs, traces and the runbook, provisioned from Git, plus alerts on the same signals.

How do you move from a metric anomaly to the root cause in Grafana?

Use consistent labels and time ranges to jump from the panel to Loki logs (filtered by service and time) and to traces via exemplars or trace ids in logs, then compare with deployment annotations.

How do you keep dashboards from becoming unmanageable?

Templates and variables, dashboards as code with reviews, ownership per dashboard, regular clean-up of unused ones, and a limit on panels per dashboard.

Summary

  • Grafana visualises data from many sources; panels hold queries, units and thresholds; variables make dashboards reusable.
  • Provision data sources and dashboards from files in Git so Grafana is reproducible; verify via its API.
  • Design for questions: golden signals first, SLO context, links to runbooks, logs and traces, annotations for changes.
  • Dashboards help humans investigate; alerts must detect problems automatically.

Going deeper

See how much of this you can explain

A free mock interview asks you new questions on Monitoring and observability for DevOps Engineer and scores every answer, with what to fix.