Rotation nobody does, because it breaks things

Every security policy says credentials are rotated every ninety days. Almost none of them are. When you ask why, the answer is always the same incident: someone rotated a database password, three services that had read it at startup kept using the old one, and the resulting outage was worse than the risk the rotation was addressing. After that, rotation became a quarterly checkbox that nobody ticks honestly.

The policy was not wrong. The order was. Rotation is not a scheduling problem, it is an application design problem, and until the application can pick up a new credential without a restart, no amount of automation is safe to turn on.

The pattern that makes rotation boring

The mechanism that works is two valid credentials at the same time.

For a database, this means two users rather than one password being swapped. The rotation job creates or updates the second user with a fresh password, waits for applications to pick it up, verifies that nothing is still authenticating as the first, and only then invalidates it. Cloud secret managers implement this as an alternating-user strategy, and the reason it is the default in their templates is that the single-user version causes outages.

For an API key with a provider that supports multiple active keys, the same shape: create the new key, deploy it, confirm traffic has moved, revoke the old one. The revocation is a separate step, days later, gated on evidence rather than on a timer.

The invariant to hold on to: there is always a window where both the old and the new credential authenticate successfully. If your provider only allows one active credential, you cannot rotate without downtime, and that is a procurement finding worth raising.

The application has to reread, and most do not

This is the actual work, and it is unglamorous.

A service that reads a connection string from an environment variable at process start will never see a rotation. Environment variables are fixed at exec time. The fix is one of three, in increasing order of effort:

• Fetch on a schedule and refresh in place. A background task that reloads the secret every few minutes and swaps it into the connection factory. Simple, and adequate for most services.
• Fail and retry with a refetch. On an authentication error, refetch the secret and retry once before surfacing the failure. This handles the case where rotation happened between refreshes, and it is worth having even alongside a schedule.
• A sidecar or CSI driver that writes the secret to a mounted file, with the application watching the file. This moves the refresh out of application code and is the right answer when you have many services in many languages.

Connection pools are the specific trap. A pool that authenticates once and holds connections open for hours will keep working after rotation and then fail all at once when the pool recycles, which looks like a random outage at 4am with no deploy to blame. Set a maximum connection lifetime so credentials are re-presented regularly, and test rotation with the pool under load rather than on an idle service.

Some credentials should be deleted, not rotated

A meaningful share of what teams plan to rotate should not exist.

Long-lived cloud access keys for CI belong in this category. Rotating them on a schedule is managing a risk you can eliminate: federate the pipeline to the cloud with short-lived tokens and the credential disappears entirely, which is the argument in getting long-lived keys out of CI.

The same applies to service account keys, which are the credential that most reliably leaks, as covered in service account keys in GCP. Workload identity federation replaces them. So do personal access tokens used as machine credentials, shared accounts with a password in a vault, and long-lived tokens held by third-party integrations that support an authorisation-code flow instead.

Run this filter over the inventory before you build any rotation automation. Every credential you delete is one you never have to rotate, monitor or explain to an auditor.

At the far end of this reasoning is dynamic generation: a credential minted per session with a short lease, so rotation becomes continuous and invisible. That capability is the strongest argument for the choice discussed in Vault or your cloud's secret manager.

You cannot rotate what you have not found

The inventory is the step teams skip, and it is why rotation programmes stall. A secret manager holds the secrets someone put in it, which is not the same as all of them.

Build the list from several directions at once. Enumerate what is in the secret managers and key vaults. Scan the source repositories and their full history. Scan container images, which frequently carry build-time credentials in layers. Check CI and CD variable stores, infrastructure state files, which hold values in plaintext, and configuration management data. Then ask each team what they use that is not on the list, because there is always something in a wiki page.

For each entry record four things: what it authenticates to, who or what uses it, when it was created, and whether it can be rotated without downtime today. That last column is your work queue, and it is usually the shortest one.

Priority, because you cannot do them all at once

Rank by what an attacker gets, not by age. A read-write production database credential used by four services outranks a ninety-day-old token for a staging webhook.

The credentials to handle first are those with broad write access to production data, those that grant access to the identity or secrets system itself, those shared between more than one system, those held by a third party, and those that have ever been in a place they should not have been.

Anything in the last group is not a rotation, it is an incident.

When a scanner finds one

The minute a secret is found in a repository, a log or an image, the sequence is fixed and the order matters.

Revoke first. Not rotate, revoke: assume it is compromised. Then check what it did. Pull the audit logs for that credential and look for use from unexpected addresses, at unexpected times, or for unusual operations. This step is what turns a leak into a scoped assessment rather than a guess, and it depends on having the audit logging in place beforehand.

Then issue the replacement, then clean up the exposure. Removing the secret from the repository is the last step and the least important, because git history is distributed and you should assume the value is public from the moment it was pushed. Rewriting history does not unpublish it. The tooling and the workflow are covered in secret scanning with Gitleaks and TruffleHog.

Finally, record it. A leaked credential that could read personal data may be a notifiable event, and that assessment belongs in the incident process with a named decision-maker.

The things people forget

• Rotation needs an audit trail. If you cannot prove when a credential was last changed, you cannot answer the auditor's actual question.
• Encryption keys rotate differently. Rotating a key encryption key is cheap; re-encrypting the data under it is not. Use envelope encryption so the two are separable.
• Certificates are credentials too. Short-lived automated certificates remove a whole class of outage, and expiry is a top cause of real incidents.
• Test rotation in staging with production-shaped load. An idle service always survives rotation. A busy one is where the connection pool bites.
• Alert on rotation failure. A rotation job that silently stops is worse than no rotation, because the dashboard says ninety days and reality says three years.

What to do this week

Take your most important production database credential and answer one question: if it changed right now, which services would break? Do not reason about it, check the code for where the secret is read. If the answer is "at startup", you have found why rotation has never happened, and fixing that one service is the beginning of the programme. We start the credential workstream of a security engagement with exactly that check.