Every team accumulates Grafana dashboards the way old codebases accumulate unused functions. There is the one the previous SRE built, the one the founder demanded for the board meeting, the one auto-generated by a vendor integration, and the one that everyone actually opens during incidents. The first three are clutter. The fourth is the one that matters.
Good dashboards are an act of editorial judgment. They answer specific questions a specific person needs answered while their service is broken or about to be. This guide is about the practical patterns that distinguish dashboards people use from dashboards people scroll past.
The On-Call Test
The single best filter for dashboard design: can someone on-call, woken up at 3 AM, use it to figure out what is broken?
The test rules out a lot of what dashboards typically contain:
- Vanity charts showing growth over the last year. Irrelevant in the moment.
- Single-stat panels with green check marks. Useless when nothing is fine.
- Dashboards with 40+ panels that take 30 seconds to load. Too slow to read in an incident.
- Charts whose units are unclear or whose thresholds are not visible.
It also surfaces what dashboards should contain:
- The top of the dashboard answers "is anything wrong?" in three seconds.
- The rest answers "what kind of wrong?" by working downward.
- Each panel is interpretable without reference documentation.
Apply this test as you build, not after.
Layout: USE, RED, or Four Golden Signals
The layout decision is upstream of every individual panel. Three frameworks have proven useful.
Four Golden Signals (Google SRE) — latency, traffic, errors, saturation. Best for user-facing services.
RED (Tom Wilkie) — rate, errors, duration. A simpler subset, best for request-driven services.
USE (Brendan Gregg) — utilization, saturation, errors. Best for infrastructure resources (CPU, memory, disk).
A typical service dashboard pairs RED at the top (user-facing health) with USE at the bottom (resource health). Infrastructure dashboards lead with USE.
The value of these frameworks is consistency. When every dashboard follows the same model, an engineer paged into an unfamiliar service still knows where to look.
Row Structure
Long dashboards work better in rows than in a single wall of panels. A pattern that holds up:
- Row 1 — Overview. Five or six top-line numbers. Error rate, request rate, p95 latency, saturation. Each as a stat panel with a threshold-driven color.
- Row 2 — Request-side detail. Latency by endpoint, error rate by endpoint, request volume. Time-series.
- Row 3 — Resource-side detail. CPU, memory, network. Time-series.
- Row 4 — Dependency health. Database latency, cache hit rate, queue depth.
- Row 5 — Logs and events. A panel of recent error logs filtered to the service. A panel of deploys and incidents.
Rows can be collapsed by default so the dashboard loads quickly and shows the top-line view first. Expand only what you need.
Variables and Templating
The difference between a dashboard for one service and a dashboard for fifty services is templating. Grafana variables let one dashboard cover an entire fleet.
Minimal viable templating:
$environment— production, staging, dev.$service— pulled from a label likeservice_name.$instance— for drilling into a specific pod or host.
Advanced templating to add when the dashboard sees real use:
$datasource— for users who run dashboards across multiple Grafana datasources.$interval— auto-adjusted bucket size based on the dashboard time range.
The payoff: instead of fifty service-specific dashboards drifting out of sync, one canonical dashboard with a service dropdown. New services automatically appear when their metrics start flowing.
Units and Thresholds
A chart without units is decoration. A chart without thresholds tells you what happened but not whether it was bad.
Grafana panel options expose both:
- Units — set explicitly. "requests per second" for rate, "milliseconds" for latency, "bytes" for size. The default "None" is a bug.
- Thresholds — color the panel. Green under target, yellow approaching, red over. The threshold should match the actual SLO; arbitrary thresholds train operators to ignore the color.
Numerically display thresholds on time-series too, as horizontal lines. "Latency p95 is 240ms, target is 300ms" is a richer story than just the line.
Queries that Survive Cardinality
Dashboards die when their queries get slow. The usual culprit is high-cardinality aggregations — sum by (request_id) rather than sum by (status_code). Grafana renders the query, the backend chokes, and you wait fifteen seconds for the panel.
A few rules:
- Aggregate over time first, dimensions second.
rate(http_requests_total[5m])beforesum by (...). - Choose dimensions with bounded cardinality. Status code: bounded. Endpoint path: usually bounded. Request ID: unbounded — never aggregate.
- Use recording rules for expensive queries that appear in dashboards. The query runs once per scrape interval and is fast to render.
- Test panel queries against a long time range (24h, 7d) at build time. If they are slow, fix the query, not the dashboard.
Annotations and Context
A spike on a time-series is information. A spike with an annotation saying "deploy v1.42 — added new caching layer" is actionable.
Grafana supports annotations from data sources (events, logs) or from manual entries. The two annotations every service dashboard benefits from:
- Deploys. From CI events, the deployment platform, or a
deploymentslog stream. - Incidents. From the alerting system or an on-call tool.
With both wired up, every dashboard tells the story "latency rose at 14:32, right after the v1.42 deploy." Diagnosis is twenty seconds instead of twenty minutes.
Links Between Dashboards
Dashboards are not standalone. A service dashboard should link to:
- The host dashboard for its underlying nodes.
- The dependency dashboard for downstream services.
- The relevant Loki / log search filtered to the service.
- The runbook for the team that owns the service.
Grafana's dashboard links and panel-level data links handle all of these. A link from a latency spike directly to the logs for that time window saves the on-call from navigating manually.
What Not to Put on a Dashboard
Dashboards are not for everything. Some signals belong elsewhere:
- Anomalies and trends — alerting, not dashboards. A dashboard nobody opens cannot fire an alert.
- Long-term capacity planning — separate dashboards, not mixed in with operational ones. The time scales and audiences are different.
- Business metrics for execs — separate dashboards, or just send a PDF report. The on-call dashboard is not the place for ARR.
- Everything available about a service — the temptation is real. Resist it. A 50-panel dashboard is unread.
A useful test: if a panel has been on the dashboard for three months and you cannot remember a time it actually surfaced a useful insight, remove it.
Versioning and Code
Dashboards drift. A panel gets adjusted during an incident, an obscure feature changes, six months later the dashboard is broken and nobody knows why.
The answer is to treat dashboards as code. Export JSON. Commit to a repository. Deploy through CI to Grafana via the API or Terraform's Grafana provider. Edits in the UI either get merged back to git or get reverted.
This is more discipline than most teams have appetite for on day one. But the dashboards that survive years of team turnover are the ones where the source of truth is a YAML or JSON file, not a UI state.
A Minimum Viable Dashboard
For a service with no dashboard today:
- One row: error rate, request rate, p95 latency, p99 latency, saturation. Five stat panels.
- One row: per-endpoint latency time-series.
- One row: dependency health (database, cache, downstream services).
- Annotations from your deploy system.
- A link to the team runbook.
This is a one-hour build. It will get used. Expand from this baseline only when the team has a specific question the dashboard does not yet answer.
Building dashboards your on-call will actually use? We help teams design Grafana setups that answer real questions in real incidents. scopeforged.com