Alerts people answer, and the ones they learned to ignore

Ask an on-call engineer what happens when their phone buzzes at 2am. In a healthy team, they read the alert and know within ten seconds whether to open a laptop. In most teams, they read it, think "that one again", silence it, and go back to sleep. Sometimes the one they silenced was real.

That is not an individual failure. It is the predictable result of a rota where most pages are not actionable. Once the base rate of false alarms passes roughly one in three, human attention collapses, and no amount of process fixes it. The only repair is deleting alerts.

Alert on symptoms, not on causes

The single highest-value change is moving from cause-based alerts to symptom-based ones.

A cause-based alert fires on CPU above 80 percent, memory above 90, a queue longer than 1000, disk at 85 percent. Each of these can be fine. A batch job at 100 percent CPU for twenty minutes is a batch job working. A queue of 5000 that drains in ninety seconds is normal. Every one of these alerts will fire when nothing is wrong, which is the definition of the problem.

A symptom-based alert fires on what the user experiences: the checkout error rate is above the threshold, the p99 latency on the search endpoint has doubled, orders are not being processed. If none of your users can tell, it is not a page.

The counter-argument is always the same: "but I want to know before it hurts users". Two answers. First, the things that reliably predict user pain are few, and disk filling is the honest example, so keep those and delete the rest. Second, that is what dashboards and tickets are for. Prediction belongs on a screen someone looks at during working hours, not on a phone at night.

Burn rate beats static thresholds

If you have service level objectives, alert on error budget burn rate rather than on a raw threshold. The mechanism is simple: compare how fast you are consuming the budget against how fast you would consume it if the entire period were uniformly bad.

The reason it works is that it scales with severity automatically. A total outage burns a month of budget in an hour and pages immediately. A slow degradation burns the budget over days and produces a ticket, not a page. One rule replaces the tangle of "5 percent for 5 minutes, or 2 percent for 30 minutes" thresholds that teams accrete over years.

The standard implementation uses two windows at once, a fast one to catch severe events quickly and a slow one to avoid firing on a brief blip, and both must be burning before it pages. It cuts false pages substantially in most estates. This only works if the objectives are chosen well in the first place, which is the argument in SLOs that stop a deploy.

Page, ticket, or dashboard

Every alert must be classified into exactly one of three destinations, and the classification is a decision about human time.

Page means wake someone now. The criterion is strict: user-visible impact, or imminent irreversible harm such as a disk that will fill before morning, and a human can do something about it right now. If the automated remediation will handle it, it is not a page.

Ticket means it needs work but not tonight. Certificate expiring in three weeks, a node pool consistently overcommitted, a rising error rate that is not yet breaching the objective. Ticketed alerts must have a real queue with a real owner, or they are just a slower way of ignoring things.

Dashboard means it is context, not a notification. Most infrastructure metrics belong here.

The rule that keeps this honest: if an alert fires and the response is "we should look at that some time", it was never a page. Reclassify it the morning after.

Grouping, inhibition and the one incident that pages twelve times

When a database fails, twenty services report errors. Twenty pages for one incident is how a rota burns out.

Alertmanager and its equivalents handle this, but only if configured. Group alerts by a common label such as cluster or service so one notification carries many instances. Use inhibition rules so that a firing "database down" alert suppresses the dependent "service cannot connect" alerts. Set a grouping wait so a burst of related alerts arrives as one notification rather than twelve. And route by team ownership from a label on the alert, not from a static list in the config that drifts the moment someone changes team.

Silences need an expiry. A permanent silence is a deleted alert wearing a disguise, except nobody can see it in the code review. Audit silences monthly and delete the underlying alert if the silence keeps getting renewed.

Every alert links to a runbook, and the runbook is honest

An alert that says HighMemoryUsage and nothing else hands the on-call engineer a research project. Attach a link to a runbook in the alert annotation, and require it before the alert can be merged.

The runbook needs four things and no more: what this actually means in user terms, how to confirm it is real in under two minutes, the first two mitigations, and who to escalate to. It should fit on one screen. Long runbooks do not get read at 3am. This is the same document that the incident response runbook points at for the technical layer.

Runbooks rot. Link them from the alert definition so the reviewer sees them together, and check them during game days.

The review that actually deletes things

Once a month, pull every alert that fired and classify what happened: a real incident, an automatic recovery, or nothing. Then apply a blunt rule. An alert that has fired more than three times and produced no human action gets deleted or downgraded to a ticket. Not tuned, not silenced. Deleted.

Teams resist this because each alert was added by someone who had a reason. The reason was usually a single past incident, and the alert is a memorial to it rather than a control. If the failure recurs, you will find out through the symptom-based alert on the user-visible effect, which is the one that would have caught it the first time too.

Track two numbers and put them somewhere the team sees: the number of pages per on-call shift, and the percentage that led to action. Under about two pages per shift with most of them actionable is a rota that stays healthy. Everything else is a plan for turnover.

What people forget

• Alerts on a metric that disappears never fire. If a service stops reporting entirely, there is no series to evaluate. Alert on the absence explicitly.
• The monitoring system needs monitoring. A dead scraper is silent, which looks exactly like everything being fine. Use a heartbeat alert to an external service.
• Nobody tests alerts. They are code, and code that has never run is broken. Fire them deliberately in a game day.
• Business hours are not the same everywhere. A team spread across time zones needs routing by time, not one rota that quietly means one person.
• Notification fatigue includes chat. A channel with 400 automated messages a day is exactly as ignored as a noisy pager.

What to do this week

Export every alert that fired in the last 30 days, sorted by count. Take the top five and ask, for each, what a human did about it. If the answer for any of them is nothing, delete it today. In most estates the top five are more than half of all pages, and deleting two of them changes the experience of being on call more than any tool purchase. We do this pass in the first fortnight of a cloud engagement.