A Practical Kubernetes Observability Stack

Philip Rehberger Nov 21, 2026 7 min read

Pick a K8s observability stack that fits your team size. Covers Prometheus, Loki, Tempo, and Grafana.

Every Kubernetes deployment eventually wants observability. The decision tree is exhausting. There are at least six metrics backends, four log stacks, three tracing systems, and a wide range of opinions about which to use. Most teams pick one of two paths: pay a vendor and move on, or assemble an open-source stack and operate it.

This guide is about the open-source path. It covers a specific, opinionated stack — Prometheus, Loki, Tempo, Grafana — and the practical considerations for running it on real workloads.

Why This Stack

The Grafana ecosystem stack (Prometheus, Loki, Tempo, Grafana) wins on three dimensions:

  1. Each component is best-in-class. Prometheus for metrics, Loki for logs, Tempo for traces. None are second-rate.
  2. Shared label conventions. A label like service="checkout" means the same thing in all three. Cross-signal navigation works.
  3. One UI. Grafana renders all three. The on-call does not switch tools to pivot between metrics, logs, and traces.

The alternatives — ELK, OpenSearch, Jaeger standalone — each have merit but require more integration work to feel cohesive. Datadog or New Relic deliver the same coherence as a paid product. Pick this stack when you want open standards, predictable cost, and operational control.

Deployment Topology

A typical mid-size deployment:

  • One Prometheus per cluster. With Thanos sidecar for long-term storage in object storage.
  • One Loki cluster with distributors, ingesters, queriers, and compactor. Backed by S3 or equivalent.
  • One Tempo cluster with distributors, ingesters, and queriers. Also backed by S3.
  • One Grafana pointing at all three datasources.
  • OpenTelemetry Collector DaemonSet as the universal data shipper.

All components run inside Kubernetes. The Grafana operator and the various Helm charts (kube-prometheus-stack, loki, tempo) handle most of the deployment mechanics.

For smaller teams, every component except Prometheus runs as Grafana Cloud or a similar managed offering. The trade is predictable cost vs operational ownership.

Sizing

A few rough numbers for capacity planning:

  • Prometheus — one instance handles 1-5M active series comfortably. Above that, shard or move to Mimir/Thanos receive.
  • Loki — disk-attached ingesters handle 50-200 GB/day per instance. Above that, scale horizontally. Object storage scales infinitely.
  • Tempo — span ingest scales with ingester count. 100K spans/sec per ingester is a reasonable starting point.
  • OpenTelemetry Collector — one DaemonSet pod per node. Memory limit of 512 MB-1 GB depending on traffic.

Numbers vary enormously by workload. Start with defaults from the official Helm charts; tune based on actual usage after a month.

Instrumenting Applications

Every application emits to the same pipeline: OpenTelemetry to the local Collector. The Collector handles routing.

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

processors:
  batch:
  resource:
    attributes:
      - key: k8s.cluster.name
        value: prod-us-east
        action: upsert

exporters:
  prometheusremotewrite:
    endpoint: http://prometheus.monitoring:9090/api/v1/write
  loki:
    endpoint: http://loki.monitoring:3100/loki/api/v1/push
  otlp/tempo:
    endpoint: tempo.monitoring:4317
    tls:
      insecure: true

service:
  pipelines:
    metrics:
      receivers: [otlp]
      processors: [batch, resource]
      exporters: [prometheusremotewrite]
    logs:
      receivers: [otlp]
      processors: [batch, resource]
      exporters: [loki]
    traces:
      receivers: [otlp]
      processors: [batch, resource]
      exporters: [otlp/tempo]

Applications send OTLP. The Collector splits the streams to the appropriate backend. Routing is a config change, not a code change.

Labels and Conventions

The value of the stack collapses if labels are inconsistent across signals. A few conventions that pay off:

  • Service identity. Use service.name from OTel semantic conventions. Translates to service label in Prometheus, service label in Loki, and service.name attribute in Tempo.
  • Environment. deployment.environment — prod / staging / dev. Always present.
  • Cluster. k8s.cluster.name — important once multiple clusters feed the same backends.
  • Version. service.version — for correlating issues with deploys.

Grafana's Explore feature lets you pivot between metrics, logs, and traces using these shared labels. Without them, you have three independent backends in a shared UI — which is most of the cost and little of the benefit.

Dashboards Worth Building

Four dashboards that earn their keep:

Service overview — RED metrics (rate, errors, duration), per-endpoint detail, dependency health, recent deploys. One per service. Linked from runbooks.

Cluster overview — node CPU/memory, pod count, restarts, namespace resource usage. One per cluster. Used by platform engineers, not service teams.

SLO dashboard — per-service SLO compliance over rolling 30-day windows. Burn rate visualization. Linked from the service overview.

Incident dashboard — wide-time-range view of all services, color-coded by status. Used during major incidents to see the blast radius at a glance.

Resist the urge to build per-team or per-business-unit dashboards. They drift, fragment, and rot. The four above survive team turnover.

Alert Policy

Alerts come from Prometheus rules via Alertmanager. A defensible alert policy:

  • Page-grade alerts — fire to PagerDuty/Opsgenie. Reserved for SLO violations, complete-service-down events, and customer-visible degradation. Aim for fewer than one per week per service.
  • Ticket-grade alerts — fire to a backlog channel or ticketing system. Trends that need attention but not at 3 AM.
  • Info-grade alerts — fire to a low-attention Slack channel. Useful for visibility without obligation.

The failure mode is page-grade alerts that fire frequently. They get ignored, then disabled, then a real incident is missed. Keep page-grade rare and use the other tiers freely.

Cost Profile

A rough monthly cost estimate for a mid-size workload (50 services, 100K req/s total, 1 TB logs/day, sampled tracing):

  • Compute — Prometheus, Loki, Tempo, Collectors. $1-3K/month depending on cluster pricing.
  • Object storage — for Thanos blocks, Loki chunks, Tempo blocks. $500-1500/month.
  • Egress — usually negligible if everything is co-located.
  • Operator effort — half an engineer for the first six months; less ongoing.

Compared to Datadog at the same workload (often $30-50K/month), the self-hosted path saves significant money. The break-even with vendors is usually around $5K-10K/month of equivalent vendor spend.

Common Failure Modes

Three patterns recur in teams running this stack.

Cardinality blowup. A developer adds user_id as a metric label. Prometheus series count explodes. Queries get slow. Storage fills. The fix is metric_relabel_configs to drop the offending label and a process to catch this in code review.

Single-tenant Loki struggling with multi-team workloads. Without per-tenant limits, one team's logging volume affects everyone. Run Loki in multi-tenant mode (header-based tenancy) or accept the noise budget.

Tempo missing spans. Tail-based sampling has a memory ceiling. If span volume exceeds the ingester's buffer, spans are dropped silently. Watch tempo_distributor_spans_received_total against tempo_ingester_traces_created_total for drops.

Each has a known fix; each costs an afternoon of debugging the first time.

A Migration Path

For a team currently on a paid vendor that wants to move to this stack:

  1. Deploy the open-source stack in parallel. Send a copy of all signals from the OTel Collector to both destinations.
  2. Rebuild dashboards in Grafana over a quarter. Verify parity.
  3. Migrate alerts to Prometheus / Alertmanager. Run both alert pipelines side by side for two weeks.
  4. Cut over team-by-team. Vendor's costs scale with usage; reduce vendor scope as teams migrate.
  5. Decommission vendor after a quarter of full operation on the new stack.

The migration is six to nine months for a 50-service organization. The cost savings start showing in month three.

When To Stay Off This Stack

The stack is not a fit when:

  • You have under 10 services and zero SRE time. The operational overhead is not worth it. Pay a vendor.
  • Your team strongly prefers SQL-like queries (NRQL, KQL). Grafana's PromQL/LogQL/TraceQL are different and the learning curve is real.
  • You need session replay, real-time security analytics, or business analytics. The stack does not solve those; pair with specialized tools.

For the cases it does fit — moderate to large workloads, willingness to operate, preference for open standards — it is the most predictable observability platform available.


Designing a Kubernetes observability stack that fits your team size and budget? We help engineering teams stand up the layer of visibility that scales with their growth. scopeforged.com

Share this article

Related Articles

Need help with your project?

Let's discuss how we can help you build reliable software.