Anomaly Detection in Alerting: Beyond Static Thresholds

Philip Rehberger Nov 17, 2026 7 min read

Move from "CPU > 80%" to alerts that fire on real deviation. Covers prediction-based and statistical methods.

The default way to set up alerting is to pick a threshold. CPU above 80%, error rate above 1%, latency above 500ms. Easy to write, easy to explain, and almost always wrong as the system grows.

Static thresholds fail in two directions. They alert too often when load is normal but seasonally elevated — every Monday morning, every Black Friday. They fail to alert when something subtle has shifted but the metric hasn't crossed the line — a request rate dropping by 30% on a service that usually shows steady traffic. Both failures cost time and trust in the alerting pipeline.

Anomaly detection is the cluster of techniques that fix this. The idea is the same across approaches: alert when a metric deviates from its expected behavior, not from an arbitrary number. This guide covers the methods that actually work in production, where they fit, and where they fail.

When Static Thresholds Are Fine

Not every metric needs anomaly detection. A few cases where a static threshold is the right answer:

  • Hard physical limits. Disk space at 95%, memory pressure with OOM imminent, certificate expiry in under a day. These have actual ceilings.
  • SLO violation alerts. "Error rate over 1% sustained for 10 minutes" maps directly to a contractual commitment. The threshold is the meaningful number.
  • Anomalies that the math cannot detect. A doubling of error rate from 0.1% to 0.2% is statistically significant but operationally inconsequential.

The rule of thumb: if there is a specific number that means "this is broken," use it. If there is a pattern that means "this looks different from normal," anomaly detection is the better tool.

Methods That Work

Three categories of anomaly detection see production use.

Statistical

The simplest approach. Compute a moving baseline (mean, median) and a measure of variance (standard deviation, MAD), then alert when the current value is more than N standard deviations away.

alert: HighLatencyAnomaly
expr: |
  (
    histogram_quantile(0.95, rate(http_duration_seconds_bucket[5m]))
    - avg_over_time(
        histogram_quantile(0.95, rate(http_duration_seconds_bucket[5m]))[1h:5m]
      )
  ) / stddev_over_time(
    histogram_quantile(0.95, rate(http_duration_seconds_bucket[5m]))[1h:5m]
  ) > 3

Fires when the current p95 latency is more than 3 standard deviations above the past hour's average.

This works well for metrics with mostly stationary behavior on the timescale of the baseline window. It fails on metrics with strong periodic patterns — a 9am traffic spike that happens every weekday looks like an anomaly each time.

Fix the periodicity problem with offset 1d or offset 7d baselines that compare to the same hour yesterday or last week:

rate(http_requests_total[5m]) /
rate(http_requests_total[5m] offset 7d)

Prediction-based (forecasting)

A forecasting model (Holt-Winters, ARIMA, Prophet, or a neural model) predicts the expected value at each time step. Alerts fire when the actual value deviates significantly from the prediction.

Prometheus's holt_winters function supports the simplest form:

holt_winters(rate(http_requests_total[5m])[1h:5m], 0.5, 0.1)

Forecasting handles seasonality natively. A model trained on a week of data will predict the Monday morning spike and not alert on it.

The practical issue is operational. Forecast models need parameters tuned per metric. A model that fits CPU well does not fit request rate. Production deployments usually pick one or two robust methods and apply them broadly rather than per-metric tuning.

Multivariate / change-point

The most sophisticated approach. Look for a structural change in a metric or in the relationship between metrics. "Latency and error rate moved together in a way they normally don't" is the kind of signal change-point detection catches that single-metric methods miss.

Tools in this space: Anodot, AWS Lookout for Metrics, Datadog Watchdog, vendor offerings that bake in proprietary models.

These are not usually self-hosted. Teams that want change-point detection either buy it or build it on top of a stream processing system with custom logic. Adoption is concentrated among teams that have specific multivariate scenarios they need to catch.

A Pragmatic Stack

For most teams, the alerting layer that delivers the best balance of value and effort:

  1. Static thresholds for SLO violations and hard limits. Clear, actionable, easy to maintain.
  2. Year-over-year or week-over-week comparisons for periodic metrics. Cheap to compute, handles seasonality.
  3. Statistical anomaly detection (Z-score over a moving baseline) for a small number of high-value metrics. Request rate, error rate, key business metrics.
  4. Forecasting only for specific metrics that need it. Often capacity planning more than alerting.
  5. Vendor anomaly detection (Datadog Watchdog, etc.) if you already pay for it. Free signal worth tuning into existing routing.

Layering avoids the trap of replacing all thresholds with anomaly detection. Some thresholds are valuable. The goal is to remove the bad ones, not all of them.

Where Anomaly Detection Fails

A few failure modes worth understanding before betting on it.

Slow drift is invisible. If the metric drifts up by 1% per day, a baseline that uses the past week of data drifts with it. The anomaly score never crosses the threshold. Drift detection requires a longer-horizon comparison.

Step changes look like anomalies forever. A genuine improvement — a deploy that drops latency by 50% — registers as a deviation from the baseline. The system alerts on the good news. Mitigation is including the recent deploy in the baseline; some tools handle this with manual baseline reset.

Low-frequency events lack statistics. A metric that only changes a few times per hour does not have enough samples for stddev or forecast methods to be meaningful. Static or count-based thresholds are better.

False positives compound. A noisy anomaly detector that fires three times per week gets ignored — its real signal disappears into the noise. Tuning matters more than method.

What Drives False Positives

The single most common cause is comparison window mismatch. If you compare "now" to "last 1 hour," any change that lasts longer than the window looks normal. If you compare to "same hour last week," anything that didn't happen last week is anomalous.

Useful patterns:

  • Short-term baselines for fast detection. 30-60 minutes of recent data, used to detect sudden spikes.
  • Long-term baselines for periodicity. Same hour last week or last few weeks averaged. Used to detect deviations from the normal pattern.
  • Two-baseline alerts. Fire only when both "now is anomalous vs recent" and "now is anomalous vs same time last week" are true.

The two-baseline pattern is conservative — it misses some real anomalies — but produces dramatically fewer false positives.

Roll Out Carefully

Anomaly detection that fires once per week is useful; one that fires three times per day is worse than no alert.

A staged rollout:

  1. Shadow mode. Run the anomaly rule. Send notifications to a low-priority channel. Do not page anyone.
  2. Tune for a month. Track precision and recall. Adjust baselines and thresholds.
  3. Promote to a real alert. Route to the on-call channel after the rule has fired only on genuine anomalies for several weeks.
  4. Maintain. Review the rule's firing history quarterly. Service behavior changes; baselines drift; rules need attention.

Most teams skip the shadow phase, route the new rule straight to PagerDuty, and disable it after the third 3 AM false page. The shadow phase is what turns anomaly detection into a tool that lasts.

A Closing Heuristic

If static thresholds are firing too often or missing too much, the answer is rarely "buy a magic anomaly detection product." It is to understand what the metric actually looks like over time and pick the simplest method that fits.

For most metrics, a week-over-week comparison plus a static SLO threshold catches what matters. The exotic methods earn their keep on a small number of high-value, high-variance metrics. Knowing which is which saves a lot of alert fatigue.


Drowning in alert noise from static thresholds? We help teams pick the anomaly detection methods that match each metric's shape. scopeforged.com

Share this article

Related Articles

Need help with your project?

Let's discuss how we can help you build reliable software.