A Kubernetes Secret is base64, and everyone learns this too late

Someone runs kubectl get secret -o yaml, pipes the value through a base64 decoder, and the database password appears on screen. This surprises people, and the surprise reveals the misconception: a Kubernetes Secret is not encrypted, it is encoded. The distinction matters because the whole security model follows from it.

Encoding protects against accidental display of binary data. It protects against nothing else. Anyone who can read Secrets in a namespace can read every credential in it, and in a default cluster the values sit in etcd in plaintext.

The three things to fix

Encryption at rest in etcd. Most managed providers now encrypt etcd contents by default or offer it as a setting backed by a key you control. Check yours; it is a checkbox and it removes the case where a disk or a backup exposes every secret in the cluster.

Who can read Secrets. This is the control that matters most and the one people skip. Permission to read Secrets in a namespace is permission to read every credential in it. Grant it per service account for the specific secrets needed, never as a blanket namespace-level read. Remember that permission to create pods is effectively permission to read any secret a service account in that namespace can mount, which is the reasoning in several teams on one cluster.

Where the value comes from. A secret committed to git is a leak from the moment it is pushed, and rewriting history does not unpublish it. This is what the operator and driver patterns solve.

External Secrets Operator, which syncs

The operator watches a custom resource describing where a secret lives in your cloud's secret manager, fetches it, and creates a normal Kubernetes Secret from it. Applications consume it as an ordinary Secret with no code change.

The virtue is that nothing changes for the workload. Every application, every Helm chart and every library that expects a Secret keeps working, which makes adoption incremental rather than a migration.

The trade is that the secret does exist in etcd, so you are relying on the RBAC and encryption above rather than removing the exposure. You are solving the git problem and the rotation problem, not the etcd problem.

Two features worth using deliberately. A ClusterSecretStore lets the platform team define the connection to the vault once, so application teams reference it without holding the credentials to reach it. And PushSecret works in the other direction, writing a generated value out to the vault, which is useful when the cluster creates the credential.

Set the refresh interval consciously. Too frequent and you are paying per API call to your secret manager at a rate that adds up, which is one of the surprise costs noted in Vault or your cloud's secret manager. Too infrequent and a rotation takes hours to propagate.

The CSI driver, which mounts

The Secrets Store CSI driver takes the other approach: it mounts secrets directly into the pod's filesystem as a volume, fetched at pod start and refreshed on an interval, without a Kubernetes Secret existing at all.

The virtue is that the value is never in etcd and never visible to anyone with namespace read access. For a genuinely sensitive credential this is the stronger position.

The trade is that the application must read from a file rather than an environment variable, which is a code change for most applications, and tooling that expects a Secret does not work. The driver can optionally also create a Secret, which restores compatibility and gives up the main benefit, so choose deliberately rather than enabling it by default.

The practical split we recommend: the operator for the bulk of credentials, because adoption is free, and the CSI driver for the small set where etcd exposure is genuinely unacceptable.

Workload identity is the prerequisite

Both patterns need the cluster to authenticate to the secret manager, and the wrong way to do that is a static cloud credential stored in a Secret, which is the problem you were solving.

Every managed Kubernetes service now supports federating a Kubernetes service account to a cloud identity, so the pod receives a short-lived token and no static credential exists. Configure it per workload rather than one identity for the whole cluster, because a single identity with read access to every secret recreates the blast radius you were trying to reduce.

This is the same mechanism as getting long-lived keys out of CI, applied inside the cluster.

Rotation is where this actually pays off, or does not

Syncing a new value into a Secret does not restart anything. If your application read the value at startup into a connection string, it will keep using the old one until it restarts, and then fail if the old credential was revoked.

Three ways to handle it, in increasing order of quality. A reloader controller that watches the Secret and triggers a rolling restart of the Deployment, which is simple and causes a restart. The application refetching on a schedule or on an authentication failure, which is the correct answer. Or dynamic short-lived credentials, where the whole problem disappears because the credential was never long-lived.

Whichever you choose, this is the part that decides whether your rotation programme works at all, and it is application work rather than platform work, which is why it stalls. The full argument is in rotation nobody does, because it breaks things.

GitOps, where the manifest references but does not contain

The pattern that makes this work with a git-driven deployment is that the repository contains the reference and never the value.

An ExternalSecret resource naming a key in the vault is safe to commit, reviewable in a pull request, and produces no leak. The reconciliation model of Argo CD in production then applies to secrets the same way it applies to everything else.

The alternative approach, encrypting values in the repository with a sealed-secrets or SOPS pattern, also works and is a reasonable choice for a small team with no cloud secret manager. Its weakness is that rotation means re-encrypting and committing, and the encrypted history is still public within the repository, so a compromised key exposes every past value.

The things people forget

• Secrets appear in more places than the Secret. Environment variables show in a pod spec description, values end up in logs, and a crash dump can contain them.
• Image pull secrets are secrets too, and are often the most widely shared credential in a cluster.
• A failing sync is silent. If the operator cannot reach the vault, the existing Secret stays in place and looks fine. Alert on reconciliation failure.
• Deleting the custom resource may delete the Secret, depending on the deletion policy. Set it deliberately.
• The vault becomes a hard dependency of pod startup. If it is unreachable during a mass restart, nothing comes up. Know what that looks like before it happens.

What to do this week

Pick a namespace and list who can read Secrets in it, then decode one and see what it contains. If the answer is a production database password readable by everyone with namespace access, that is the finding, and tightening the RBAC is a faster win than installing any operator. We check exactly this in a security engagement.