Spot instances offer up to 90% discount on cloud compute. The catch is that the cloud can take them back with a few minutes' warning. For workloads that handle this gracefully, the savings are massive. For workloads that do not, spot instances produce production incidents.
This post is how to use spot instances safely in production — what fits, what does not, and the patterns that keep the savings without the surprises.
How Spot Pricing Works
Cloud providers have excess capacity at any given moment. Spot instances let you bid on that capacity at a discount. When the cloud needs the capacity back (other customers want on-demand), spot instances are reclaimed with a short notice.
- AWS Spot: 90% discount possible. 2-minute reclamation notice. Sometimes longer warning via Capacity Rebalancing.
- GCP Preemptible / Spot: Similar discounts. 30-second notice. Maximum 24-hour lifetime for preemptible.
- Azure Spot: Similar. 30-second notice.
Spot prices fluctuate based on supply and demand. A common instance type might be 60-70% off on average; a specific type might spike to on-demand prices in periods of high demand.
What Spot Is Good For
The defining property of spot-suitable workloads: they tolerate being terminated.
Good fits:
- Stateless application servers behind a load balancer. If one instance dies, others handle the traffic. Auto-scaling replaces the lost instance.
- Batch processing. Jobs that can resume from checkpoints. A failed batch retries.
- Test and staging environments. Lower stakes than production; cost matters more than uptime.
- CI runners. A killed CI job can be restarted automatically.
- Background workers. Queue consumers that can have their work redistributed.
- Data processing. Spark, Hadoop, Flink — these are designed to handle worker failures.
In all these, the architecture already handles failure. Spot just makes failures more frequent without adding new failure modes.
What Spot Is Bad For
Workloads that do not tolerate instance termination:
- Database primaries. Killing a primary causes data integrity work.
- Stateful services without replication. A killed instance means real state loss.
- Workloads with strict latency SLAs during reclamation. Reclaimed instances cause traffic drops or slowdowns.
- Services with long-running operations that cannot resume. A 10-hour render job that has no checkpointing is bad on spot.
In short: anything where "instance disappears" is a real problem.
Diversification
The single most important spot strategy: do not rely on one instance type. Use many.
Each spot instance type has its own supply/demand. When the cloud reclaims spot, it usually reclaims a specific type. A workload using 10 different instance types has independent reclamation risk; a workload using one type can lose them all simultaneously.
Kubernetes with Karpenter:
apiVersion: karpenter.sh/v1beta1
kind: NodePool
metadata:
name: spot
spec:
template:
spec:
requirements:
- key: karpenter.sh/capacity-type
operator: In
values: [spot]
- key: kubernetes.io/arch
operator: In
values: [amd64, arm64]
- key: node.kubernetes.io/instance-type
operator: In
values: [m5.large, m5.xlarge, m5a.large, m5a.xlarge,
m5n.large, m5n.xlarge, m6i.large, m6i.xlarge,
m6a.large, m6a.xlarge, m7i.large, m7i.xlarge]
Karpenter picks the cheapest available across all these types. If one type becomes unavailable, it shifts to another.
EC2 Auto Scaling Groups with mixed instances policy:
MixedInstancesPolicy:
LaunchTemplate:
LaunchTemplateSpecification:
LaunchTemplateName: my-template
Overrides:
- InstanceType: m5.large
- InstanceType: m5a.large
- InstanceType: m5n.large
- InstanceType: m6i.large
InstancesDistribution:
OnDemandPercentageAboveBaseCapacity: 0
SpotAllocationStrategy: capacity-optimized
The capacity-optimized strategy picks instances from pools with the deepest capacity (less likely to be reclaimed).
For both: more diversity = lower reclamation rate.
Mixing Spot and On-Demand
A common pattern: critical baseline on-demand, burst capacity on spot.
Always running: 5 on-demand instances (baseline capacity)
Scale up to: 50 spot instances (handle traffic spikes)
If spot capacity goes away, the workload falls back to its on-demand baseline. Slower than ideal, but not down.
Setting OnDemandPercentageAboveBaseCapacity controls this. A baseline of 5 on-demand instances plus all-spot scaling means the 6th-50th instances are spot.
For workloads where some uptime guarantee matters: mix. For purely best-effort workloads: 100% spot is fine.
Reclamation Handling
When spot reclamation happens, the workload has 30 seconds to 2 minutes to handle it gracefully.
For Kubernetes pods:
- The node receives a termination signal
- Kubernetes drains the node (evicts pods cleanly with PreStop hooks)
- Pods land elsewhere
- The old node is terminated
This works if:
- Pods have proper
terminationGracePeriodSeconds - Services tolerate brief unavailability of one replica
- Health checks remove the draining node from load balancer rotation
For non-Kubernetes workloads:
- Listen for the spot reclamation notification (metadata endpoint)
- On notification, drain the work (finish current request, exit cleanly)
- The orchestrator replaces the instance
# AWS Spot reclamation notice
while true; do
if curl -s http://169.254.169.254/latest/meta-data/spot/instance-action; then
# Reclamation notice received
/opt/drain-and-exit.sh
exit 0
fi
sleep 5
done
Without reclamation handling, workloads experience hard termination at the OS level. In-flight work is lost.
Measuring the Savings
Spot's savings depend on workload tolerance and instance choice. Rough averages:
- Spot prices average: 60-90% off on-demand
- Practical savings after diversification and on-demand mix: 50-70%
- Worst case (high-demand types in busy periods): 30% savings
For a workload spending $10K/month on compute, moving 70% of it to spot saves $4K-$6K/month. The savings compound across multiple environments.
When Spot Is Not Worth It
Some situations where spot creates more problems than savings:
- Small total spend. If you spend $200/month on compute, 70% savings is $140. Not worth the operational complexity.
- Workloads with bespoke setup. If each instance takes 20 minutes to provision and configure, reclamation churn becomes painful.
- Strict SLAs without redundancy. A workload that needs 99.99% uptime with one instance cannot use spot.
For most teams: anything spending $1K+/month on compute is worth optimizing with spot. Below that, on-demand simplicity wins.
Common Mistakes
- Single instance type. Easy to set up, hardest to keep running.
- Spot for stateful databases. Just no.
- No reclamation handling. Workloads experience hard kills.
- Spot in EKS without Karpenter or Auto Scaling. Manual replacement does not scale.
- Not monitoring reclamation rate. A workload that gets reclaimed every hour has different economics than one reclaimed weekly.
What to Monitor
Once spot is in production, watch:
- Reclamation rate. How often spot instances are reclaimed. A few per hour is fine; constant churn is a problem.
- Average runtime per instance. Longer is better — instances that survive longer amortize their startup cost.
- Cost savings. Is the savings what you projected? Spot pricing fluctuates; the actual savings can vary month to month.
- Service-level impact. Are reclamations affecting latency or error rates?
If reclamation rate is high, diversify more or move some workload to on-demand.
A Practical Adoption Path
For a team new to spot:
- Start with non-production. Move test, staging, CI to spot. Low risk, real savings.
- Move stateless production workloads. Application servers behind load balancers. Use mixed instances with diversification.
- Move batch and background workers. Already designed to be interruptible.
- Tune. Watch reclamation patterns, adjust instance type diversity and on-demand baseline.
The migration is incremental. The team learns spot's failure modes on lower-stakes workloads before exposing production.
What Stays On-Demand
After all the optimization, some things remain on-demand:
- Database primaries (managed services use their own underlying instances)
- Stateful services without good replication
- Critical baseline capacity that needs uptime guarantees
- Workloads with strict latency requirements during reclamation events
Mixing is the right answer. 100% spot is for the most fault-tolerant workloads; 100% on-demand is for the least. Most production setups use both.
Evaluating spot instances for production workloads and not sure where the risk-versus-savings line is? We help teams design spot strategies that capture the cost benefits without producing reliability incidents. scopeforged.com