Prometheus was designed for a single environment. Run one instance per cluster, scrape your services, evaluate alerts. For most teams that is exactly enough — until it isn't. The moment you have two clusters with overlapping operational concerns, or a Prometheus instance that has outgrown what one VM can store, scaling becomes the question.
There is no single answer. Prometheus offers three distinct scaling models — federation, remote write, and sharding — and each fits a different problem. Picking the wrong one means rebuilding the metrics pipeline twelve months later.
This guide explains what each approach actually does, where it breaks down, and how to choose.
What Federation Does (and Doesn't)
Federation is the original Prometheus scaling pattern. A parent Prometheus scrapes the /federate endpoint of one or more child Prometheus instances, pulling a subset of their metrics into its own database. The pattern was designed for a specific use: aggregating high-level summaries from many environments into a global view.
scrape_configs:
- job_name: 'federate'
scrape_interval: 60s
honor_labels: true
metrics_path: '/federate'
params:
'match[]':
- '{job=~"node-exporter|apiserver"}'
- 'job:request_duration:p99'
static_configs:
- targets:
- 'prometheus-us-east.example.com'
- 'prometheus-eu-west.example.com'
A few things worth noticing:
- Federation is pull-based, not push. The parent reaches out to the children on its own schedule.
- It is best for pre-aggregated data. Pulling raw, high-cardinality metrics through federation does not scale — the parent ends up with the same volume problem.
- The default scrape interval is long. 60 seconds is typical for federation. The parent is not the source of truth; the children are.
Federation is the right answer when you want a roll-up view across regions or clusters and accept lower resolution at the top. It is not the right answer when you want a single, queryable view of all metrics at full resolution.
Remote Write
Remote write reverses the relationship. Each Prometheus instance forwards its samples to a centralized store — Thanos, Cortex, Mimir, VictoriaMetrics, or a vendor's hosted backend. The local Prometheus keeps short-term storage for alerting; the central store handles long-term retention and global queries.
remote_write:
- url: "https://mimir.example.com/api/v1/push"
queue_config:
max_samples_per_send: 10000
max_shards: 50
Why this scales better than federation:
- Volume is not duplicated by aggregation. Every sample is sent exactly once to a system designed to handle billions of them.
- Backends are horizontally scalable. Cortex / Mimir partition data across many ingesters and queriers. Storage is in object stores (S3, GCS).
- Global queries hit the central store directly. No parent-child hierarchy to maintain.
The cost is operational complexity. You now run a distributed metrics database. Mimir or Cortex requires zookeeper-equivalent coordination, object storage, multiple service types, and tuning. VictoriaMetrics is simpler operationally but commercially licensed for some features.
For most teams beyond the single-Prometheus phase, remote write to one of these backends is the right model.
Sharding
Sharding is the third option, distinct from the first two. A single logical Prometheus is split across multiple physical instances, each responsible for a subset of targets.
scrape_configs:
- job_name: 'sharded'
relabel_configs:
- source_labels: [__address__]
modulus: 4
target_label: __tmp_hash
action: hashmod
- source_labels: [__tmp_hash]
regex: '0'
action: keep
Each instance keeps only the targets whose hash matches its assigned bucket. The queries are still split across instances; you query each shard and merge results (Thanos Query does this).
Sharding solves the problem of a single Prometheus running out of CPU or memory because it cannot scrape all its targets. It does not solve global queries or long-term retention. You typically combine it with remote write or Thanos sidecar.
Sharding is the right call when one Prometheus cannot keep up with the scrape volume and you do not want to move to a remote-write backend yet.
Combining Patterns
Mature setups frequently combine all three.
Per-cluster Prometheus + remote write to a global store + Thanos sidecars on each Prometheus.
The Prometheus instances handle local scraping and short-term storage. They forward to a central backend (Mimir, Cortex, or Thanos receiver) for global, long-term queries. Thanos sidecars upload Prometheus's blocks to object storage as a backup path.
This is the canonical setup for organizations with multiple production environments and a need for both fast local alerting and global retrospective queries.
When Federation Still Wins
Despite the scaling limitations, federation has a niche it keeps winning:
- Org-level rollups. A single dashboard at the C-suite level: total requests across all regions, total error rate, total saturation. Pre-aggregated, low resolution, low cost.
- Air-gapped environments. A parent Prometheus in a less-restricted network pulls a metrics summary from a child in a restricted one. Push-based remote write often is not permitted.
- Quick proof of concept. Setting up federation takes minutes. Setting up Mimir takes weeks.
For these, federation is still the simplest path.
Cardinality and Cost
The most common reason any of these patterns falls over is cardinality. A label like user_id or request_id produces unbounded time series. Each unique combination of label values becomes a new series in storage.
At scale, this destroys every Prometheus backend, federated or not. The mitigations:
- Audit cardinality regularly.
prometheus_tsdb_head_seriesand the cardinality endpoint show you the worst offenders. - Drop high-cardinality labels at scrape time.
metric_relabel_configswithaction: labeldropremoves a label before storage. - Cap per-tenant cardinality at the backend. Mimir and Cortex both support per-tenant series limits that prevent one bad metric from filling shared storage.
A Prometheus that stays under control on cardinality scales much further than one that does not, regardless of the topology.
A Decision Lens
Walk through these in order:
- Is a single Prometheus instance running out of resources? If yes, shard the scrape load. If no, skip to the next step.
- Do you need a global view across multiple clusters or regions? If no, stay on per-cluster Prometheus. If yes:
- Is the global view a high-level rollup (a small number of pre-aggregated metrics)? Federation is enough.
- Is the global view a full, queryable copy of all metrics? You need remote write to Mimir, Cortex, VictoriaMetrics, or a vendor.
- Do you need years of retention? Add Thanos sidecar or remote write to a backend with object-store retention.
The usual mistake is jumping straight to Mimir for a setup that two federation hops would have solved. The reverse — building federation that strains under cardinality you should have remote-written — is just as common.
Migration Notes
If you outgrow federation, the migration to remote write is mechanically simple but operationally careful.
- Stand up the remote backend alongside existing Prometheus.
- Add
remote_writeto each Prometheus. Monitor the local queue size and remote ingester health. - Mirror the federation dashboards against the remote backend. Verify parity over a week.
- Cut over alerts and dashboards. Decommission federation.
The rollback path is keeping the federation hierarchy as a fallback during the cutover week. After that, the remote backend is canonical and federation is removed.
Figuring out which Prometheus scaling pattern fits your traffic? We help teams design metrics pipelines that scale without surprise rebuilds. scopeforged.com