Policy in the cluster, without the webhook that takes it down

The incident is always the same. A policy is written to require resource limits on every pod. It is applied in enforce mode across the cluster on a Thursday. Everything already running keeps running, so it looks fine. Then a node is replaced overnight, the system DaemonSet that had no limits cannot be admitted, and the cluster loses its logging agent and its CNI pods on that node. Nobody connects the outage to a policy applied fourteen hours earlier.

Admission control is powerful precisely because it sits in the path of every write to the API server. That is also why the rollout matters more than the tool.

What admission control is for

Three jobs, and they are worth separating because they need different rules.

Validation rejects objects that violate a rule: no privileged containers, no latest tags, no public load balancers in this namespace, required labels for cost attribution.

Mutation fills things in: default resource requests, a sidecar injection, a standard set of labels, an imagePullSecret. Mutation is underrated and often the kinder option, because it fixes the manifest instead of rejecting it.

Background scanning reports on objects that already exist. Enforcement only ever applies to new writes, so without background scanning you have no idea how much of the cluster would fail the policy today.

Before writing anything custom, turn on Pod Security Admission at the baseline or restricted level. It is built into Kubernetes, needs no controller, and covers the majority of what teams write their first ten policies for. Policy engines are for what it does not cover.

Choosing between the two

Kyverno expresses policies as Kubernetes resources in YAML. There is no separate language: you write match conditions and a validation, mutation or generation rule. Generation is a distinguishing feature, creating a default NetworkPolicy or a Secret copy in every new namespace, which is genuinely useful for the namespace-provisioning problem. The cost of the YAML approach is that complex logic gets verbose.

OPA Gatekeeper uses Rego, a real policy language, with constraint templates defining logic and constraints instantiating it with parameters. Rego is more capable for complex rules and is used far beyond Kubernetes, so an organisation already using OPA for API authorisation gets one language everywhere. The cost is that Rego is a genuine learning curve and debugging it is not pleasant for someone who writes YAML all day.

The honest recommendation: if your team is Kubernetes-native and your policies are mostly shaped like "require this field, forbid that value", Kyverno gets you there faster and more people on the team can maintain it. If you already run OPA elsewhere, or your policies need real logic across multiple resources, Gatekeeper. Both are CNCF projects with active communities and both work. The rollout discipline below matters more than the pick.

The rollout that does not cause an outage

Four stages, and skipping any of them is how the opening story happens.

1. Audit. Deploy the policy in audit or dry-run mode and turn on background scanning. Do nothing else for a week. You now have a report of every existing object that violates the rule, which is the number you actually needed before deciding whether the policy is reasonable.

2. Fix the backlog. Work through the violations with the owning teams. This is the slow part and it is where the value is. A policy applied to a clean estate is a guardrail; a policy applied to a dirty one is an outage generator.

3. Warn. Move to a mode that admits the object but surfaces a warning to the user applying it. People see the message in their pipeline output and fix things before enforcement arrives.

4. Enforce, by namespace. Not cluster-wide on day one. Start with one team that has agreed, then widen. Keep the audit-mode policy for the namespaces not yet enforcing, so you can see progress.

Announce each stage. A policy that silently starts rejecting deployments generates support tickets that blame the platform team for a mysterious failure.

Exclusions are not optional

Exclude the system namespaces from the start, and be explicit rather than relying on a default. kube-system contains workloads you do not control and whose manifests are set by the provider, and a policy that blocks them can prevent a node from joining the cluster.

Beyond that, exclude the namespaces of components that must survive to fix problems: your CNI, your CSI drivers, your monitoring, your policy engine itself. A policy engine that blocks its own upgrade is a genuinely awkward afternoon.

Write exclusions as an explicit list in the policy rather than as a cluster-wide setting, so a reader can see what is exempt and why. Review the list quarterly, because exclusions added during an incident tend to become permanent.

The failure policy is the real risk

Every admission webhook has a failure policy that decides what happens when the webhook cannot be reached. This is the single most consequential setting in the whole system.

Fail means the API server rejects the write. That is correct from a security standpoint: a policy that can be bypassed by making it unavailable is not a control. It also means that if your policy engine is down, nothing can be deployed, including the fix, and in the worst case new pods cannot be created at all.

Ignore means writes proceed unchecked. Safer operationally, and it means an attacker who can disrupt the webhook can bypass your policies.

The workable position is Fail for the policies that genuinely matter for security, Ignore for the hygiene ones, plus the operational measures that make failure unlikely: multiple replicas across nodes and zones, a PodDisruptionBudget, generous timeouts, and an alert on webhook latency and error rate. Watch the certificate expiry too, because an expired webhook certificate produces exactly the same outage as a crashed controller.

Know your break-glass procedure. Deleting the ValidatingWebhookConfiguration restores service immediately and disables your policies, and someone should have practised it.

Policies as code, reviewed like code

Policies belong in git, deployed by the same pipeline as everything else, with the same review. That is what makes them evidence: an auditor asking how you prevent privileged containers gets a versioned file, an approval record and a report of current compliance, which is a far better answer than a screenshot.

Test them. Both engines have a CLI that evaluates a policy against a manifest, so a pull request that changes a policy can run a fixture of manifests that should pass and should fail. Without that, the first test of a policy change is production.

This also feeds the control mapping directly: one policy can satisfy a requirement in several frameworks at once, as argued in one control set across frameworks. The same instinct applies at the cloud layer, where the deployment discipline in Azure Policy as code is the same shape.

The things people forget

• Enforcement does not touch what already exists. Background scanning is what tells you the truth about the estate.
• Mutation ordering matters. Multiple mutating webhooks apply in sequence and can fight. Check the result, not the intent.
• Latency is on every write. A slow policy adds latency to every object creation in the cluster, which shows up as slow deployments and slow scaling.
• Policies need an owner. A rule nobody maintains becomes an exclusion list that grows until the rule means nothing.
• Start with fewer policies than you want. Ten well-understood, enforced policies beat sixty in audit mode that nobody reads.

What to do this week

Deploy your policy engine with background scanning and one policy in audit mode, requiring resource requests on every pod. Do not enforce it. Read the report. The number of violations tells you exactly how far the estate is from where you thought it was, and that number is the argument for the work. We run this in the platform phase of a security engagement.