The runbook you use at 3am, not the one you show the auditor

Two documents usually exist. One is the incident response policy: twenty pages, approved annually, written to satisfy a framework, and never opened during an incident. The other is the thing people actually do, which lives in the heads of three engineers and changes depending on who is awake.

The gap between them is where incidents go badly. Not because people are incompetent, but because at 3am, under pressure, with partial information, humans need structure they have practised. Anything they have to read for the first time is not going to get read.

Severity by criteria, not by feel

The first failure in most incidents is arguing about how bad it is. Severity has to be decidable in thirty seconds by the person who noticed, without asking permission.

Write criteria, not adjectives. Something like: highest severity means customer-facing functionality is unavailable, or data confidentiality or integrity is compromised, or a regulatory notification clock may have started. Next level down means degraded service with a workaround, or a security event contained but not yet understood. Lowest means single-customer impact or an internal system.

Two rules keep it honest. Anyone can declare, and the declaration is never punished. It is cheaper to stand down a false alarm than to spend forty minutes deciding whether to start. And a security incident with potential data exposure goes to the top severity by default, because the notification clock is unforgiving and downgrading later is easy.

Three roles, and the first one is not technical

The single most effective change in incident handling is separating the person coordinating from the people investigating.

Incident commander. Owns the process, not the fix. Decides severity, assigns work, keeps the timeline, calls for escalation, and decides when the incident ends. Crucially, they do not debug. The most common failure mode in a small team is that the best engineer is simultaneously running the response and head-down in a terminal, and communication stops entirely.

Investigators. One or more people doing the technical work, reporting findings to the commander rather than narrating into a channel nobody is aggregating.

Communications. Handles the status page, customer messaging and internal updates on a fixed cadence. In a small company this can be the commander, but it must be an explicit hat, because "somebody should tell the customers" is a sentence that gets said forty minutes too late.

For a small team, two people is the practical minimum. If you are one person, the structure still helps: write the timeline as you go, because you will not remember it.

One channel, one timeline

Open a dedicated channel per incident rather than using the team channel. Everything goes there: what was observed, what was tried, what changed, who was told, with timestamps.

Keep the timeline as you go. This is the discipline people abandon under pressure and regret afterwards, because the timeline is what the post-incident review needs, what a regulator asks for, and what tells you whether the thing you changed at 02:14 caused the improvement at 02:20 or coincided with it.

Update on a cadence even when there is nothing new. Every thirty minutes, a message that says "still investigating, no change" prevents six people asking for an update individually and pulling the investigators away.

Escalation is a list with names and phone numbers

Write down, per system, who to call and in what order, and include the paths people forget: the cloud provider's support channel and the severity level that gets a human, the vendor whose outage is causing yours, the person with the credentials nobody else has, legal and the data protection contact, and the executive who decides on customer communication.

Include the break-glass procedure. If the identity provider is down, how does anyone log in? If that answer does not exist, you have found the next incident, and it is worth stopping to fix.

Test the contacts. An escalation list where two people have left the company is common and only discovered at the worst moment.

The notification clock, and who watches it

This is where a technical incident becomes a regulatory one, and where teams lose time they cannot recover.

The obligations differ but the shape is the same: a short deadline that starts at awareness rather than at resolution. GDPR runs to 72 hours to the supervisory authority for a personal data breach, with individual notification when the risk to people is high. NIS2 requires an early warning within 24 hours and a fuller notification after, for entities in scope. DORA sets its own reporting path for financial entities and their critical suppliers. US state privacy laws and sector rules add their own. Verify the current text for the ones that bind you rather than the version in the policy document.

The engineering consequence is that someone must be assigned, during the incident, to assess whether a clock has started. Put it on the commander's checklist as an explicit item with a name next to it. And note that the clock runs whether or not anyone has thought about it, which is why detection and audit logging are compliance controls: if you were not logging reads, you cannot establish what was accessed, and an inability to scope the breach tends to widen the notification rather than narrow it. The obligation side is covered in NIS2 scope and obligations and GDPR as engineering controls.

The review that changes something

Hold it within a week, while memory is fresh, and hold it blameless in the real sense: the question is what made the mistake reasonable at the time, not who made it. An engineer who fears the review will not tell you what they actually did, and then you learn nothing.

The output is a small number of actions with owners and dates, in the normal backlog, reviewed like any other work. A review that produces fourteen action items produces zero completed ones. Two or three, done, is a system that improves.

Track one number over time: how many incidents were caused by something a previous review identified and nobody fixed. If that number is not zero, the review process is the problem, not the systems.

What people forget

• Alerts have to reach the commander. Detection quality decides how early the process starts, which is why this depends on alerts people answer.
• The status page is a system too. Hosting it in the infrastructure that just failed is a mistake everyone makes exactly once.
• Practise it. A tabletop exercise once a quarter finds the stale contact and the missing runbook link at no cost.
• Hand over properly. Incidents outlast shifts. Write a handover summary with current state, what has been ruled out, and what is in flight.
• Preserve evidence for security incidents. Snapshot before you remediate. Rebuilding the compromised host destroys the forensic trail.

What to do this week

Take your last real incident and try to reconstruct its timeline from what was written down at the time. If you cannot establish when it started, when you knew, and who was told, that is the finding. The fix is a one-page template pinned in the incident channel, and it costs an afternoon. We build this with teams in the resilience phase of a security engagement.