Cluster upgrades stop being an event when you do them monthly

There are two upgrade cultures. In the first, the cluster is upgraded every few weeks, nobody mentions it, and the change log is a line in a pull request. In the second, the cluster is four versions behind, the upgrade is a project with a name and a maintenance window, three people are nervous about it, and it gets postponed again because there is a launch next month.

The second culture is not more careful. It is the consequence of the first postponement, and the risk compounds. Kubernetes releases roughly three times a year with about fourteen months of upstream patch support per minor version, and managed providers layer their own windows on top, with extended support available at a price and eventually a forced upgrade. Falling behind means you eventually upgrade on the provider's schedule, during whatever else is happening that week.

Why behind is more expensive than current

Skipping versions is not supported for the control plane: you upgrade one minor version at a time, so being four behind means four sequential upgrades, each with its own validation.

Each version also carries its own deprecations, and the further you travel in one project the more of them you hit at once. A single-version upgrade usually has a short list of changes you can reason about. A four-version upgrade is a research task.

Meanwhile the estate keeps moving. New manifests get written against the old API, more add-ons accumulate, and the gap between your version and the version your tooling assumes grows. Extended support fees are the visible cost and the smallest part of it.

Find the deprecated APIs before the upgrade finds them

This is the single highest-value preparation step, and it is mechanical.

Removed API versions are the classic breakage: a manifest that applies fine today fails after the upgrade because the version it names no longer exists. The workloads already running are unaffected until something reapplies them, which is why the failure often surfaces days later in a pipeline rather than during the upgrade window.

Check three places, because they hold different things:

• The cluster's live objects, via the deprecation metrics the API server exposes and its audit log, which together tell you what is still being requested and by whom.
• Your manifests and Helm charts in git, including the ones for environments you rarely deploy.
• Third-party charts and operators, which are the most common source of a surprise.

Tools that diff your manifests against a target version catch most of it in a single run. Do this check in CI against the next version, permanently, so the list is never a surprise.

Also read the provider's own release notes, not just upstream. Managed providers change defaults, node image contents and add-on versions in ways upstream notes do not mention.

The drain is where upgrades actually stall

Upgrading the control plane is usually uneventful. Replacing nodes is where it stops.

A node drain evicts pods and waits. It waits forever if a PodDisruptionBudget cannot be satisfied, and misconfigured budgets are extremely common:

• A budget with minAvailable equal to the replica count, which permits zero disruption and blocks the drain permanently.
• A budget attached to a Deployment with one replica, which is a contradiction the API accepts.
• A StatefulSet where eviction is blocked by a volume that cannot detach because the pod has not terminated cleanly.

Then there are pods nothing will evict: bare pods with no controller, which are never recreated, and long-running Jobs that drain will wait on politely. Both need a decision before the upgrade, not during it.

Set a drain timeout and know what you will do when it is hit. Test the drain on one node in production during working hours, well before the upgrade, because that single test finds nearly all of this.

Version skew, add-ons and the order of operations

The control plane goes first. Nodes follow and may run older than the control plane within the supported skew window, never newer. Your kubectl and your CI tooling have their own supported range, and a pipeline using a client two versions behind will fail in ways that look like cluster problems.

Add-ons are the part teams forget, and they are the part that breaks. The CNI, CSI drivers, the ingress or gateway controller, cert-manager, external-dns, the metrics server, the autoscaler, the service mesh, admission webhooks and any operator each have their own compatibility matrix. Some must be upgraded before the control plane, some after.

Write the matrix down once, as a table of component against supported Kubernetes versions, and keep it in the repository next to the cluster definition. It turns a research exercise into a checklist.

Admission webhooks deserve special caution. A webhook that fails closed and is incompatible with the new version blocks every deployment in the cluster, including the deployment that would fix it. Check failurePolicy and the certificate expiry on every webhook before you start.

Blue-green node pools beat in-place rolling

For the data plane, the pattern that minimises risk is not an in-place rolling replacement. It is creating a new node pool on the new version alongside the old one, cordoning the old pool, draining it gradually, and watching.

The advantage is that rollback is trivial. If workloads misbehave on the new node image, uncordon the old pool and drain back. With in-place rolling, the old nodes are gone and rollback means another full cycle.

It costs you the price of running both pools for a few hours, which is a small price for an exit. Move one non-critical workload first and let it soak, rather than draining everything in sequence on the first attempt.

The rhythm that makes this boring

Upgrade the non-production clusters on a schedule, automatically, and let them run the new version for a week or two before production follows. The lag catches real problems with no stakes attached.

Then upgrade production monthly or at least every quarter, in working hours, as normal work rather than as an event. Same person does not have to do it every time; that is the point of writing the checklist.

Two supporting habits make this hold. Keep the cluster definition in code so a version bump is a reviewable diff, as with the state and pipeline patterns in Terraform on AWS. And treat the upgrade as a change like any other, with the same validation and the same rollback expectation, which is exactly the framing in putting AKS into production and OKE in production.

The things people forget

• etcd backups before the control plane upgrade. Managed providers take them, but verify and know the restore path.
• Node image changes carry kernel and runtime changes. A workload depending on a specific kernel feature or an old container runtime behaviour can break with no Kubernetes API involved.
• Cluster-scoped custom resources are shared. Upgrading one controller can change CRDs that another controller depends on, which is acute in a shared cluster.
• Quota for the new node pool. A blue-green upgrade needs headroom, and hitting an instance quota mid-upgrade is a bad moment to discover it.
• Feature gates and beta APIs. Anything you enabled that was beta may be promoted with changes, or removed.

What to do this week

Run a deprecated-API check against the next minor version, on your production cluster and on your manifests in git. It takes ten minutes and produces a concrete list. If that list is empty, schedule the upgrade for a Tuesday morning and stop treating it as a project. We put this check into the pipeline during the platform phase of a cloud engagement.