SLOs that stop a deploy, and SLOs that decorate a dashboard

The usual sequence is this. Someone decides the team needs SLOs. They open the monitoring tool, find a metric that already exists, and it is host uptime. They set the target at 99.9 percent because three nines sounds serious. The number goes on a dashboard, stays green for eleven months, and in the twelfth there is an outage that the dashboard also shows as green, because the hosts were up the whole time.

The failure is not the number. It is that the indicator was chosen from what was easy to measure rather than from what the user feels, and that nothing in the delivery process ever consults it. An SLO that cannot change a decision is a decoration with a maintenance cost.

The indicator starts with the user, not with the metric you already have

Ask what a user is trying to do and what "it worked" means to them. For an API, the user is a client application and the answer is "my request got a correct response quickly enough". For a batch pipeline, it is "yesterday's data was in the warehouse before I opened the report".

Server availability answers none of those. A host can be up while the application returns 500s, while its dependency is down, while responses take nine seconds, or while the queue behind it is four hours behind. Uptime is a property of your infrastructure. The indicator has to be a property of the interaction.

Measure at the point closest to the user that you control. The gateway is usually right; the application's own instrumentation is second best, because it cannot report the requests that never reached it.

Define the good event and the bad event with painful precision

This is where most SLO documents stay vague and become unusable six months later. Write it as a ratio and say exactly what is in the numerator and the denominator.

Bad: "99.9 percent availability". Good: "the proportion of HTTP requests to /api/v1/*, excluding /health, returning a status other than 5xx and completing in under 400 ms, measured at the gateway, per minute".

Then settle the edge cases in writing, because they surface during the first incident. Are 429s bad events? Are 4xx excluded, and does that exclusion hide the 400s you caused with a breaking schema change? Does a maintenance window count, given that "planned" is a property of your calendar and not of the user's day? Is synthetic traffic in the denominator, keeping the ratio healthy at 3am when there are no real users?

The target comes from what the business tolerates, not from a pretty number

Do not start with nines. Start with measured performance over the last quarter, and with the question: below what level do customers notice, complain or leave?

Then work out what the next nine buys. Going from 99.9 to 99.99 cuts allowable badness by a factor of ten, which normally means multi-region active-active, a harder deployment process and a different on-call rota. Worth paying for a payments authorisation path; not worth paying for an internal admin panel.

Two rules that save arguments. The target must be meaningfully below your best month, or there is no budget to spend and the mechanism is dead on arrival. And if you have an external SLA, the internal objective sits stricter than it, so you find out before your customers invoice you.

The error budget is the decision mechanism, or the whole thing is theatre

The budget is the arithmetic complement of the target: 99.9 percent over 30 days is roughly 43 minutes of allowed badness. What makes it useful is agreeing in advance what changes when it runs low.

The policy we install with clients is short. Above half the budget remaining, ship normally and run the risky experiments. Below a quarter, no non-essential deploys to that service and reliability work goes to the top of the backlog. Exhausted, feature freeze until it recovers, with one named person who can override and a record of every override.

The point is not punishment. It is that "should we ship this risky change" stops being a personality contest between the person who wants velocity and the person who wants stability, and becomes a lookup against a number both of them agreed to last quarter.

Use a rolling window, 28 or 30 days, not a calendar month. A calendar month grants an amnesty at midnight on the first, so a bad week on the 29th is forgiven two days later. A rolling window keeps the incident in the denominator for four weeks, which is roughly how long the lesson should last.

When the budget runs out

Three cases, three answers. Burned in one large incident: the budget did its job, so freeze, fix the cause, resume. Burned in a steady drip of small failures: the cause is usually a dependency or a saturated resource, and the fix is capacity or a circuit breaker rather than heroics. Burned every month: the target is wrong or the architecture cannot support it, so either lower it with the business in the room or fund the work that makes it reachable. Quietly ignoring a permanently exhausted budget teaches the organisation that the objective is fiction.

The things people forget

• Three or four objectives beat thirty. An SLO per endpoint is a spreadsheet, not a policy. Budget the journeys that generate revenue or block work, and monitor the rest without budgeting them.
• The indicator needs its own reliability. If the metrics pipeline drops data, the SLO silently improves. Alert on missing data as its own condition.
• Dependencies eat your budget. On a managed service with a 99.9 SLA, a 99.99 objective above it is arithmetic you cannot win without redundancy across regions or providers.
• Latency needs a threshold, not an average. A mean of 200 ms hides a tenth of users at four seconds. Count the requests over the line.
• Review targets quarterly. An objective nobody has revisited in two years is a fossil.

Once the objectives exist they should drive the paging policy, which is the subject of alerting that people actually answer: burn rate against a budget is a far better trigger than a static threshold on a graph. We set both up in the reliability phase of a cloud engagement.

What to do this week

Take your most important user journey and write one sentence defining the good event and the bad event, naming the measurement point. Then query the last 30 days and compute the actual ratio. You will know immediately whether the target you were about to pick is ambitious, trivial, or already broken, and that hour of work is the whole foundation.