Stateful workloads in Kubernetes, and the question you should ask first

The cluster runs everything else, so the database moves in too. Six months later a node is drained during an upgrade, the volume will not detach because the pod has not terminated cleanly, the drain hangs, and someone discovers at two in the morning that the replica they thought existed was scheduled in the same availability zone as the primary.

None of that is an argument against running stateful workloads in Kubernetes. It is an argument for knowing what the platform actually guarantees, because the gap between what a StatefulSet promises and what people believe it promises is where these incidents live.

What a StatefulSet actually gives you

Three things, and they are narrower than the name suggests.

Stable network identity. Each pod gets a predictable name and a stable DNS entry that survives rescheduling. This is what lets cluster members find each other, and it is the main reason StatefulSets exist.

Stable storage. Each pod gets its own PersistentVolumeClaim, and when the pod is rescheduled it reattaches to the same volume. The claim is not deleted when the pod is, which is deliberate and is also why orphaned volumes accumulate.

Ordered operations. Pods are created and scaled down in order, and rolling updates proceed one at a time, waiting for each to be ready.

What it does not give you: any understanding of your data. It will not promote a replica, will not check replication lag, will not stop you scaling down the pod holding the only current copy, and will not coordinate a backup. Ordering is not consensus. If your database needs a leader election, something has to perform it, and that something is either the database itself or an operator.

Storage classes and the constraints you inherit

The access mode is the first constraint. Most cloud block storage supports attachment to a single node at a time, which is correct for a database and means a pod cannot start on a new node until the volume detaches from the old one. Shared filesystems support multi-node access at higher cost and lower performance. Choosing a shared filesystem so that two pods can write to the same directory is usually a sign the architecture needs revisiting rather than a storage decision.

Set the reclaim policy deliberately. Delete removes the underlying volume when the claim goes, which is right for a cache and catastrophic for a database. Retain keeps it, which is right for a database and is why estates accumulate orphaned disks that bill forever, as noted in Azure storage and data transfer.

Volume expansion works on most modern storage classes and is one-way: you can grow a volume, not shrink it. Check the class has expansion enabled before you need it at midnight.

Performance is provisioned, and the default class is usually the wrong one for a database. A volume's throughput and operations per second are a setting with a price, and a database on a general-purpose volume that runs out of burst credit produces a latency incident that looks like an application problem.

The zone pin is the constraint people forget

A block volume exists in one availability zone. A pod using it can only be scheduled in that zone. That single fact has consequences people discover during failures rather than during design.

Your replicas are not distributed unless you made them so. If three database pods all ended up in the same zone because that is where capacity was, you have three copies with one failure domain. Use topology spread constraints, and verify the result rather than assuming, because the scheduler satisfies constraints where it can and proceeds where it cannot unless you make them hard.

Losing a zone means the pods in it cannot reschedule elsewhere, because their volumes are stranded. Recovery is a restore from backup or a promotion of a replica in another zone, not a reschedule. This is exactly the scenario the zone-loss experiment in the first five chaos experiments is designed to surface.

Snapshots are not backups

The CSI snapshot API gives you point-in-time volume snapshots, and they are genuinely useful for fast rollback of a bad migration.

They are not a backup, for three reasons. A volume snapshot of a running database may be crash-consistent rather than application-consistent, so restoring it is equivalent to recovering from a power cut, which most databases handle and some do not. Snapshots usually live in the same account and the same failure domain as the thing they protect. And they do not capture the cluster state needed to reconstruct the workload.

Use the database's own backup mechanism, write it somewhere outside the cluster's blast radius, and test the restore. The reasoning is in a backup the attacker can delete and the discipline in RTO, RPO and the restore test.

Operators, and what you are buying

A database operator encodes operational knowledge as a controller: provisioning, replication topology, failover, backup scheduling, minor version upgrades, and connection routing to the current primary.

The good ones are genuinely valuable, and they are the only way running a database in the cluster makes sense at any scale, because the alternative is a StatefulSet plus a collection of scripts and a person who remembers how failover works.

What you are buying is the operator's opinion about your database, and what you are taking on is a dependency with its own upgrade cycle, its own bugs and its own failure modes. Evaluate on the things that matter at three in the morning: does it handle failover automatically, does it verify backups, how does it behave during a node drain, what happens when the operator itself is down, and who maintains it.

The honest question

Should this database be in the cluster at all.

The case for a managed database service is strong and teams under-weight it. You get automated backups with tested restore, patching, failover, point-in-time recovery and a support contract, and you do not own any of it at two in the morning. The premium over raw compute is real and it is usually less than the cost of the engineering time it replaces, particularly for a team that does not have a database specialist.

The genuine reasons to run it yourself: a database with no managed equivalent, a cost profile at large scale where the managed premium exceeds the operational cost, a data residency or sovereignty requirement the managed service cannot meet, or a need for extensions and configuration the managed service forbids. The comparison for the most common case is in tuning managed PostgreSQL.

Our default recommendation: managed for the primary transactional database, in-cluster for caches, queues, search indexes and anything where losing the data is an inconvenience rather than an incident.

The things people forget

• Drain blocks on stateful pods. A PodDisruptionBudget that permits zero disruption hangs a node drain forever, which is the most common cause of a stalled cluster upgrade, as described in cluster upgrades.
• Scaling down deletes pods, not claims. The volumes remain and keep billing.
• Readiness is not the same as ready to serve. A replica still catching up passes a naive readiness probe and receives traffic it cannot answer correctly.
• Resource limits kill databases. A memory limit slightly below a legitimate working set produces an out-of-memory kill that looks random.
• Connection pools do not know about failover. After a promotion, clients holding connections to the old primary need to be forced to reconnect.

What to do this week

For your most important stateful workload, run one command: list the pods with the zone each one is in. If two replicas share a zone, you have found a single point of failure that your architecture diagram says does not exist, and a topology spread constraint fixes it. We check this in the platform phase of a cloud engagement.