Your first five chaos experiments, and the ones to run in staging

The way chaos engineering usually arrives is that someone installs a fault injection tool, kills a pod in production on a Wednesday afternoon, takes down checkout for nine minutes, and the practice is banned for two years. The idea was right and the sequence was wrong.

Chaos engineering is not breaking things to see what happens. It is stating what you believe the system will do, then checking whether it does. If you cannot write down the expected behaviour beforehand, you are not running an experiment, you are causing an incident.

The shape of a real experiment

Four things, written down before anything runs.

A hypothesis in steady-state terms. Not "the system will survive". Something measurable: "if one of the three API pods is terminated, the error rate stays below 0.1 percent and p99 latency stays under 400 ms". You need a steady-state metric you already trust, which is why this practice depends on having service level objectives first.

A blast radius. One pod, one availability zone, one dependency, one percent of traffic. Never "the cluster".

A stop condition and a way to stop. Define the metric threshold at which you abort, and confirm the abort mechanism works before you start. An experiment you cannot stop is an outage with paperwork.

A prediction of what you will learn either way. If the hypothesis holds, you have evidence. If it does not, you have a finding with a reproduction.

Run in staging first, and be honest that staging findings are weaker: fewer instances, no real traffic, mocked dependencies. Staging tells you whether the mechanism works. Production tells you whether it works at scale. Move to production only after the same experiment has been clean in staging several times, during working hours, with the team watching and the owner of the service present.

The five that always find something

1. Terminate one instance. Kill a single pod, container or virtual machine in a replicated service. You are testing whether the replacement starts before the remaining capacity saturates, whether in-flight requests drain or are dropped, and whether the load balancer notices promptly. The classic finding is a service with two replicas and a readiness probe that passes before the application can serve, so traffic arrives at a process that is not ready and the error rate spikes for fifteen seconds.

2. Add latency to a dependency. Inject 500 ms of delay into calls to a database, a cache or a third-party API. This is the most productive experiment in the list because it exposes missing timeouts. The common finding is a client with no timeout at all, or a timeout longer than the caller's own, so a slow dependency exhausts the connection pool and a degraded dependency becomes a total outage. Retries without backoff turn it into a self-inflicted denial of service.

3. Fill a disk. Write until a volume reaches 95 percent on one node. You are testing whether monitoring catches it before the application does, and what the application does when it cannot write. Logging libraries that block on write are a frequent and unpleasant discovery.

4. Lose an availability zone. Make one zone's instances unreachable, or drain the nodes in it. You are testing whether capacity in the remaining zones is genuinely sufficient rather than nominally sufficient, whether stateful workloads pinned to that zone's volumes can reschedule, and whether the database failover happens and how long it takes. This is the experiment that most often invalidates a multi-zone claim, and it connects directly to the arithmetic in multi-region failover and what it costs.

5. Expire a certificate. In staging, set a certificate to expire, or block the renewal path. Certificate expiry causes a surprising share of real outages, the failure mode is total, and the recovery path is often undocumented because it works automatically until the day it does not.

Game days beat automation, at first

The instinct after the first successful experiment is to automate it on a schedule. Resist it for a while.

A game day is a scheduled session where the team runs an experiment together, with someone playing the role of on-call, working only from the alerts and the runbooks. It teaches more than automation because the thing being tested is not only the system but the response: whether the alert fired, whether it said something useful, whether the runbook was accurate, whether anyone knew who to escalate to.

Most first game days find that the technical resilience is better than expected and the human process is worse. The alert fires into a channel nobody watches, the runbook references a dashboard that was renamed, the escalation contact left the company. None of that appears in an automated experiment that checks a metric.

Automate later, once experiments have been boring several times in a row.

Running it without losing the room

• Announce it. Silent chaos in production destroys trust and produces a real incident response, wasting everyone's evening.
• Never during a freeze, a launch or a peak. The value of the finding does not offset the risk during Black Friday.
• One variable at a time. Two simultaneous faults give you an interesting story and no attributable cause.
• Record the result even when nothing happened. "We killed a pod and nothing happened" is the evidence that the design works, and it is what earns permission for the next experiment.
• Fix what you find before running the next one. A backlog of unactioned chaos findings turns the practice into theatre.

Tooling, briefly

For Kubernetes, the CNCF options cover pod, network and stress faults declaratively as custom resources, which means experiments live in git and get reviewed. The clouds have their own managed fault injection services that reach infrastructure-level faults such as zone disruption and API throttling, which the in-cluster tools cannot reach.

You need none of it to start. The first three experiments in the list are a kubectl delete pod, a sidecar or proxy rule adding delay, and a dd command. Buy tooling when the manual version has proven the practice, not before.

What to do this week

Pick your most important service and write down one hypothesis about it, with a number: what happens to error rate and latency if one instance disappears. Then check whether you could measure that today. If you cannot, the finding arrived before the experiment did, and fixing the measurement is the first job. We run the first game day with the team during the resilience phase of a cloud engagement.