Kubernetes Cluster Upgrades Without Downtime

Kubernetes upgrades are the kind of work that everybody understands needs to happen and most teams put off until the running version is two years behind. The cluster falls out of support, security patches stop coming, new features pile up unavailable. Eventually the upgrade is no longer optional, and the longer you waited the more painful it gets.

This post is what a real Kubernetes upgrade involves, the patterns that minimize disruption, and the discipline that prevents falling years behind in the first place.

What Actually Needs Upgrading

A Kubernetes "version upgrade" touches three layers:

Control plane. The API server, controller manager, scheduler, etcd. Managed services (EKS, GKE, AKS) handle this for you. Self-managed clusters do it yourself.

Node operating system and kubelet. Each node runs a kubelet and a container runtime. Upgrading the cluster means upgrading every node's kubelet to match.

Workloads. Pods, Deployments, StatefulSets, custom resources. Most workloads are version-tolerant, but some APIs change between Kubernetes versions and break workloads that depend on them.

A successful upgrade means all three layers move to the new version without disrupting application traffic.

The Cadence Question

Kubernetes releases a new minor version every four months. Each version is supported for ~14 months (one year plus extended).

Recommended cadence:

• Aggressive: Upgrade within 1-3 months of release. Stay near the leading edge.
• Standard: Upgrade within 6 months. Still in support.
• Lagging: Upgrade once per year. Risky.
• Falling behind: Two or more versions behind. Real risk of running out of support.

Most teams should aim for the standard cadence. Aggressive is for teams that have a good upgrade process; lagging accumulates technical debt fast.

Pre-Upgrade Checks

Before any upgrade, run through:

Deprecated APIs. Kubernetes deprecates and removes APIs over time. The kube-no-trouble tool (or pluto) scans manifests and identifies APIs that will be removed.

kubectl get all -A -o yaml | kubent

If you have manifests using removed APIs, fix them before the upgrade. Otherwise the upgrade rejects valid-looking resources.

Operator and CRD compatibility. Operators may have version constraints. Check the operator's compatibility matrix against the target Kubernetes version.

Helm chart compatibility. Charts that work on 1.28 may use deprecated APIs in 1.29. Bump chart versions.

Add-on compatibility. Cilium, Calico, ingress controllers, cert-manager — each has its own version compatibility. Plan upgrades together.

Application API usage. Applications that talk to the Kubernetes API directly (operators you wrote, custom controllers) may use deprecated APIs.

Doing these checks one minor version at a time is easier than catching up across several.

Control Plane Upgrade

For managed services:

• EKS: A few clicks in the console or aws eks update-cluster-version. Takes ~30 minutes per minor version. Cannot skip versions.
• GKE: Either auto-upgrade (the default for Autopilot) or manual via gcloud container clusters upgrade. Similar duration.
• AKS: az aks upgrade. Similar duration.

Managed services typically pre-validate the upgrade for known compatibility issues. They will refuse to upgrade if the cluster has obvious problems.

For self-managed (kubeadm-based):

# On the control plane
kubeadm upgrade plan
kubeadm upgrade apply v1.29.0

# Then upgrade kubelet and kubectl on the control plane
apt-mark unhold kubelet kubectl
apt-get update && apt-get install -y kubelet=1.29.0-1.1 kubectl=1.29.0-1.1
apt-mark hold kubelet kubectl
systemctl daemon-reload
systemctl restart kubelet

Self-managed upgrades are more involved. Each step is documented but mistakes are easier to make.

Node Upgrades

Once the control plane is on the new version, nodes need to follow. The constraint is that all kubelet versions must be within one minor of the control plane.

Two patterns:

In-place upgrade. Upgrade kubelet on each node. Drains the node, upgrades, joins back, repeats.

# Per node
kubectl drain my-node --ignore-daemonsets --delete-emptydir-data
# SSH to node
apt-mark unhold kubelet
apt-get update && apt-get install -y kubelet=1.29.0-1.1
apt-mark hold kubelet
systemctl daemon-reload
systemctl restart kubelet
kubectl uncordon my-node

Works but is slow and risky. One bad node update during the rolling process can break pods.

Replace-the-node pattern. Spin up new nodes with the new version. Cordon and drain old nodes. Delete old nodes. The cluster gradually replaces all old nodes with new ones.

# Create new node group with new version
# Cordon old nodes
kubectl cordon <old-node>
kubectl drain <old-node> --ignore-daemonsets --delete-emptydir-data
# Delete the old node

This is the dominant pattern in 2026. For managed Kubernetes:

• EKS: create a new managed node group with the new version, drain the old group.
• GKE: node upgrades happen automatically (Autopilot) or via gcloud container node-pools upgrade.
• AKS: similar via node pool upgrade.

For Karpenter-managed nodes: bumping the node pool's AMI or instance template causes Karpenter to gradually replace nodes. Drift detection handles it automatically.

The Pod Disruption Budget

When draining nodes, Pod Disruption Budgets (PDBs) prevent the upgrade from taking down too many pods at once.

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: api-pdb
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: api

The drain operation respects PDBs — it will not evict a pod if doing so violates the budget. This ensures rolling upgrades stay within HA constraints.

Set PDBs for every critical workload before starting an upgrade. Without them, the upgrade can take down all replicas of a service simultaneously.

Etcd Backups

Before any upgrade: back up etcd. Even managed services where you do not directly access etcd often expose a backup mechanism. For self-managed:

ETCDCTL_API=3 etcdctl snapshot save /backup/snap-$(date +%Y%m%d).db \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key

A pre-upgrade etcd snapshot is your only path back if things go wrong. Test the restore before you need it.

Testing in Lower Environments

Always upgrade non-production clusters first. The pattern:

1. Upgrade dev → run smoke tests → verify
2. Upgrade staging → run integration tests → verify
3. Wait at least a day to catch latent issues
4. Upgrade production

The wait between staging and production is important. Some issues only show up under sustained traffic — a memory leak in the new kubelet, a regression in scheduling behavior.

Rolling Back

Kubernetes does not officially support rolling back a minor version upgrade. The supported recovery is to restore from etcd backup.

In practice:

• Managed services can sometimes downgrade if you catch it fast enough.
• Self-managed downgrade is risky; the API server may have written data the older version cannot read.

The realistic answer: forward only. Test enough that rollback is not needed.

Workload Migration Considerations

Some Kubernetes versions deprecate APIs that workloads use. For example:

• extensions/v1beta1 Ingress removed in 1.22 (use networking.k8s.io/v1)
• policy/v1beta1 PodDisruptionBudget removed in 1.25
• batch/v1beta1 CronJob removed in 1.25

Tools like pluto and kubent scan for these. Fix manifests before the upgrade or things fail to apply.

For Helm charts using deprecated APIs: bump the chart version, which usually includes the fix.

A Practical Upgrade Cadence

For a single managed cluster (EKS, GKE, AKS):

• Every 4 months: read the new version's release notes
• Within 2 months of release: upgrade dev
• Within 4 months: upgrade staging
• Within 6 months: upgrade production
• Total commitment: 1-2 days of engineering time per upgrade

For self-managed clusters: roughly double the time, with more attention to add-on compatibility.

This cadence keeps the cluster supported, captures security patches, and prevents the "we are five versions behind" situation that becomes a months-long project.

The Honest Reality

Most teams fall behind. The discipline of regular upgrades is one of those things that always loses to feature work — until the security team or the support ending forces a rushed catch-up.

The teams that handle it well have one of two patterns:

• Automation. Auto-upgrade for managed services. Karpenter handles node rotation. Operators stay current.
• Calendar discipline. Quarterly upgrade weeks. Treated as engineering work that gets scheduled.

Pick one. Both work. Neither happens by accident.

Working on a Kubernetes upgrade plan for a cluster that has fallen behind, or wanting to set up a sustainable upgrade cadence? We help teams modernize cluster operations without breaking application teams. scopeforged.com