Every Kubernetes deployment eventually wants observability. The decision tree is exhausting. There are at least six metrics backends, four log stacks, three tracing systems, and a wide range of opinions about which to use. Most teams pick one of two paths: pay a vendor and move on, or assemble an open-source stack and operate it.
This guide is about the open-source path. It covers a specific, opinionated stack — Prometheus, Loki, Tempo, Grafana — and the practical considerations for running it on real workloads.
Why This Stack
The Grafana ecosystem stack (Prometheus, Loki, Tempo, Grafana) wins on three dimensions:
- Each component is best-in-class. Prometheus for metrics, Loki for logs, Tempo for traces. None are second-rate.
- Shared label conventions. A label like
service="checkout"means the same thing in all three. Cross-signal navigation works. - One UI. Grafana renders all three. The on-call does not switch tools to pivot between metrics, logs, and traces.
The alternatives — ELK, OpenSearch, Jaeger standalone — each have merit but require more integration work to feel cohesive. Datadog or New Relic deliver the same coherence as a paid product. Pick this stack when you want open standards, predictable cost, and operational control.
Deployment Topology
A typical mid-size deployment:
- One Prometheus per cluster. With Thanos sidecar for long-term storage in object storage.
- One Loki cluster with distributors, ingesters, queriers, and compactor. Backed by S3 or equivalent.
- One Tempo cluster with distributors, ingesters, and queriers. Also backed by S3.
- One Grafana pointing at all three datasources.
- OpenTelemetry Collector DaemonSet as the universal data shipper.
All components run inside Kubernetes. The Grafana operator and the various Helm charts (kube-prometheus-stack, loki, tempo) handle most of the deployment mechanics.
For smaller teams, every component except Prometheus runs as Grafana Cloud or a similar managed offering. The trade is predictable cost vs operational ownership.
Sizing
A few rough numbers for capacity planning:
- Prometheus — one instance handles 1-5M active series comfortably. Above that, shard or move to Mimir/Thanos receive.
- Loki — disk-attached ingesters handle 50-200 GB/day per instance. Above that, scale horizontally. Object storage scales infinitely.
- Tempo — span ingest scales with ingester count. 100K spans/sec per ingester is a reasonable starting point.
- OpenTelemetry Collector — one DaemonSet pod per node. Memory limit of 512 MB-1 GB depending on traffic.
Numbers vary enormously by workload. Start with defaults from the official Helm charts; tune based on actual usage after a month.
Instrumenting Applications
Every application emits to the same pipeline: OpenTelemetry to the local Collector. The Collector handles routing.
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
processors:
batch:
resource:
attributes:
- key: k8s.cluster.name
value: prod-us-east
action: upsert
exporters:
prometheusremotewrite:
endpoint: http://prometheus.monitoring:9090/api/v1/write
loki:
endpoint: http://loki.monitoring:3100/loki/api/v1/push
otlp/tempo:
endpoint: tempo.monitoring:4317
tls:
insecure: true
service:
pipelines:
metrics:
receivers: [otlp]
processors: [batch, resource]
exporters: [prometheusremotewrite]
logs:
receivers: [otlp]
processors: [batch, resource]
exporters: [loki]
traces:
receivers: [otlp]
processors: [batch, resource]
exporters: [otlp/tempo]
Applications send OTLP. The Collector splits the streams to the appropriate backend. Routing is a config change, not a code change.
Labels and Conventions
The value of the stack collapses if labels are inconsistent across signals. A few conventions that pay off:
- Service identity. Use
service.namefrom OTel semantic conventions. Translates toservicelabel in Prometheus,servicelabel in Loki, andservice.nameattribute in Tempo. - Environment.
deployment.environment— prod / staging / dev. Always present. - Cluster.
k8s.cluster.name— important once multiple clusters feed the same backends. - Version.
service.version— for correlating issues with deploys.
Grafana's Explore feature lets you pivot between metrics, logs, and traces using these shared labels. Without them, you have three independent backends in a shared UI — which is most of the cost and little of the benefit.
Dashboards Worth Building
Four dashboards that earn their keep:
Service overview — RED metrics (rate, errors, duration), per-endpoint detail, dependency health, recent deploys. One per service. Linked from runbooks.
Cluster overview — node CPU/memory, pod count, restarts, namespace resource usage. One per cluster. Used by platform engineers, not service teams.
SLO dashboard — per-service SLO compliance over rolling 30-day windows. Burn rate visualization. Linked from the service overview.
Incident dashboard — wide-time-range view of all services, color-coded by status. Used during major incidents to see the blast radius at a glance.
Resist the urge to build per-team or per-business-unit dashboards. They drift, fragment, and rot. The four above survive team turnover.
Alert Policy
Alerts come from Prometheus rules via Alertmanager. A defensible alert policy:
- Page-grade alerts — fire to PagerDuty/Opsgenie. Reserved for SLO violations, complete-service-down events, and customer-visible degradation. Aim for fewer than one per week per service.
- Ticket-grade alerts — fire to a backlog channel or ticketing system. Trends that need attention but not at 3 AM.
- Info-grade alerts — fire to a low-attention Slack channel. Useful for visibility without obligation.
The failure mode is page-grade alerts that fire frequently. They get ignored, then disabled, then a real incident is missed. Keep page-grade rare and use the other tiers freely.
Cost Profile
A rough monthly cost estimate for a mid-size workload (50 services, 100K req/s total, 1 TB logs/day, sampled tracing):
- Compute — Prometheus, Loki, Tempo, Collectors. $1-3K/month depending on cluster pricing.
- Object storage — for Thanos blocks, Loki chunks, Tempo blocks. $500-1500/month.
- Egress — usually negligible if everything is co-located.
- Operator effort — half an engineer for the first six months; less ongoing.
Compared to Datadog at the same workload (often $30-50K/month), the self-hosted path saves significant money. The break-even with vendors is usually around $5K-10K/month of equivalent vendor spend.
Common Failure Modes
Three patterns recur in teams running this stack.
Cardinality blowup. A developer adds user_id as a metric label. Prometheus series count explodes. Queries get slow. Storage fills. The fix is metric_relabel_configs to drop the offending label and a process to catch this in code review.
Single-tenant Loki struggling with multi-team workloads. Without per-tenant limits, one team's logging volume affects everyone. Run Loki in multi-tenant mode (header-based tenancy) or accept the noise budget.
Tempo missing spans. Tail-based sampling has a memory ceiling. If span volume exceeds the ingester's buffer, spans are dropped silently. Watch tempo_distributor_spans_received_total against tempo_ingester_traces_created_total for drops.
Each has a known fix; each costs an afternoon of debugging the first time.
A Migration Path
For a team currently on a paid vendor that wants to move to this stack:
- Deploy the open-source stack in parallel. Send a copy of all signals from the OTel Collector to both destinations.
- Rebuild dashboards in Grafana over a quarter. Verify parity.
- Migrate alerts to Prometheus / Alertmanager. Run both alert pipelines side by side for two weeks.
- Cut over team-by-team. Vendor's costs scale with usage; reduce vendor scope as teams migrate.
- Decommission vendor after a quarter of full operation on the new stack.
The migration is six to nine months for a 50-service organization. The cost savings start showing in month three.
When To Stay Off This Stack
The stack is not a fit when:
- You have under 10 services and zero SRE time. The operational overhead is not worth it. Pay a vendor.
- Your team strongly prefers SQL-like queries (NRQL, KQL). Grafana's PromQL/LogQL/TraceQL are different and the learning curve is real.
- You need session replay, real-time security analytics, or business analytics. The stack does not solve those; pair with specialized tools.
For the cases it does fit — moderate to large workloads, willingness to operate, preference for open standards — it is the most predictable observability platform available.
Designing a Kubernetes observability stack that fits your team size and budget? We help engineering teams stand up the layer of visibility that scales with their growth. scopeforged.com