Prometheus Federation: Scaling Metrics Across Clusters

Philip Rehberger Nov 12, 2026 7 min read

Use federation, remote write, or sharding to scale Prometheus past one cluster.

Prometheus was designed for a single environment. Run one instance per cluster, scrape your services, evaluate alerts. For most teams that is exactly enough — until it isn't. The moment you have two clusters with overlapping operational concerns, or a Prometheus instance that has outgrown what one VM can store, scaling becomes the question.

There is no single answer. Prometheus offers three distinct scaling models — federation, remote write, and sharding — and each fits a different problem. Picking the wrong one means rebuilding the metrics pipeline twelve months later.

This guide explains what each approach actually does, where it breaks down, and how to choose.

What Federation Does (and Doesn't)

Federation is the original Prometheus scaling pattern. A parent Prometheus scrapes the /federate endpoint of one or more child Prometheus instances, pulling a subset of their metrics into its own database. The pattern was designed for a specific use: aggregating high-level summaries from many environments into a global view.

scrape_configs:
  - job_name: 'federate'
    scrape_interval: 60s
    honor_labels: true
    metrics_path: '/federate'
    params:
      'match[]':
        - '{job=~"node-exporter|apiserver"}'
        - 'job:request_duration:p99'
    static_configs:
      - targets:
          - 'prometheus-us-east.example.com'
          - 'prometheus-eu-west.example.com'

A few things worth noticing:

  • Federation is pull-based, not push. The parent reaches out to the children on its own schedule.
  • It is best for pre-aggregated data. Pulling raw, high-cardinality metrics through federation does not scale — the parent ends up with the same volume problem.
  • The default scrape interval is long. 60 seconds is typical for federation. The parent is not the source of truth; the children are.

Federation is the right answer when you want a roll-up view across regions or clusters and accept lower resolution at the top. It is not the right answer when you want a single, queryable view of all metrics at full resolution.

Remote Write

Remote write reverses the relationship. Each Prometheus instance forwards its samples to a centralized store — Thanos, Cortex, Mimir, VictoriaMetrics, or a vendor's hosted backend. The local Prometheus keeps short-term storage for alerting; the central store handles long-term retention and global queries.

remote_write:
  - url: "https://mimir.example.com/api/v1/push"
    queue_config:
      max_samples_per_send: 10000
      max_shards: 50

Why this scales better than federation:

  • Volume is not duplicated by aggregation. Every sample is sent exactly once to a system designed to handle billions of them.
  • Backends are horizontally scalable. Cortex / Mimir partition data across many ingesters and queriers. Storage is in object stores (S3, GCS).
  • Global queries hit the central store directly. No parent-child hierarchy to maintain.

The cost is operational complexity. You now run a distributed metrics database. Mimir or Cortex requires zookeeper-equivalent coordination, object storage, multiple service types, and tuning. VictoriaMetrics is simpler operationally but commercially licensed for some features.

For most teams beyond the single-Prometheus phase, remote write to one of these backends is the right model.

Sharding

Sharding is the third option, distinct from the first two. A single logical Prometheus is split across multiple physical instances, each responsible for a subset of targets.

scrape_configs:
  - job_name: 'sharded'
    relabel_configs:
      - source_labels: [__address__]
        modulus: 4
        target_label: __tmp_hash
        action: hashmod
      - source_labels: [__tmp_hash]
        regex: '0'
        action: keep

Each instance keeps only the targets whose hash matches its assigned bucket. The queries are still split across instances; you query each shard and merge results (Thanos Query does this).

Sharding solves the problem of a single Prometheus running out of CPU or memory because it cannot scrape all its targets. It does not solve global queries or long-term retention. You typically combine it with remote write or Thanos sidecar.

Sharding is the right call when one Prometheus cannot keep up with the scrape volume and you do not want to move to a remote-write backend yet.

Combining Patterns

Mature setups frequently combine all three.

Per-cluster Prometheus + remote write to a global store + Thanos sidecars on each Prometheus.

The Prometheus instances handle local scraping and short-term storage. They forward to a central backend (Mimir, Cortex, or Thanos receiver) for global, long-term queries. Thanos sidecars upload Prometheus's blocks to object storage as a backup path.

This is the canonical setup for organizations with multiple production environments and a need for both fast local alerting and global retrospective queries.

When Federation Still Wins

Despite the scaling limitations, federation has a niche it keeps winning:

  • Org-level rollups. A single dashboard at the C-suite level: total requests across all regions, total error rate, total saturation. Pre-aggregated, low resolution, low cost.
  • Air-gapped environments. A parent Prometheus in a less-restricted network pulls a metrics summary from a child in a restricted one. Push-based remote write often is not permitted.
  • Quick proof of concept. Setting up federation takes minutes. Setting up Mimir takes weeks.

For these, federation is still the simplest path.

Cardinality and Cost

The most common reason any of these patterns falls over is cardinality. A label like user_id or request_id produces unbounded time series. Each unique combination of label values becomes a new series in storage.

At scale, this destroys every Prometheus backend, federated or not. The mitigations:

  • Audit cardinality regularly. prometheus_tsdb_head_series and the cardinality endpoint show you the worst offenders.
  • Drop high-cardinality labels at scrape time. metric_relabel_configs with action: labeldrop removes a label before storage.
  • Cap per-tenant cardinality at the backend. Mimir and Cortex both support per-tenant series limits that prevent one bad metric from filling shared storage.

A Prometheus that stays under control on cardinality scales much further than one that does not, regardless of the topology.

A Decision Lens

Walk through these in order:

  1. Is a single Prometheus instance running out of resources? If yes, shard the scrape load. If no, skip to the next step.
  2. Do you need a global view across multiple clusters or regions? If no, stay on per-cluster Prometheus. If yes:
  3. Is the global view a high-level rollup (a small number of pre-aggregated metrics)? Federation is enough.
  4. Is the global view a full, queryable copy of all metrics? You need remote write to Mimir, Cortex, VictoriaMetrics, or a vendor.
  5. Do you need years of retention? Add Thanos sidecar or remote write to a backend with object-store retention.

The usual mistake is jumping straight to Mimir for a setup that two federation hops would have solved. The reverse — building federation that strains under cardinality you should have remote-written — is just as common.

Migration Notes

If you outgrow federation, the migration to remote write is mechanically simple but operationally careful.

  • Stand up the remote backend alongside existing Prometheus.
  • Add remote_write to each Prometheus. Monitor the local queue size and remote ingester health.
  • Mirror the federation dashboards against the remote backend. Verify parity over a week.
  • Cut over alerts and dashboards. Decommission federation.

The rollback path is keeping the federation hierarchy as a fallback during the cutover week. After that, the remote backend is canonical and federation is removed.


Figuring out which Prometheus scaling pattern fits your traffic? We help teams design metrics pipelines that scale without surprise rebuilds. scopeforged.com

Share this article

Related Articles

Need help with your project?

Let's discuss how we can help you build reliable software.