The pattern is familiar enough to be boring. A change goes out on the ninth. On the twelfth someone notices the daily cost graph looks odd but assumes it is a reporting lag. On the first of the following month the invoice arrives with an extra five figures on it, and two days of forensics establish that a retry loop in a data pipeline has been re-reading the same objects several million times a day for three weeks.
Every part of that is preventable, and none of it is prevented by a budget.
Why budgets and forecasts are not enough
A budget is an alert threshold, not a control. It does not stop spending, because stopping spending in a cloud account means disabling production. Budgets are still worth setting, at subscription, project or account level with an alert at a percentage of forecast, because they catch the slow drift. They do not catch a spike, because a spike that starts on the third of the month will not breach a monthly budget until the twentieth.
A forecast alert is better and still late, because the forecast is built from a trend and a spike takes days to bend the trend.
What catches a spike is comparing today against what today normally looks like. Day-over-day and same-day-last-week comparison on a per-service, per-account basis, with a threshold on percentage change and a floor on absolute change so you are not paged about a service that went from two euros to eight.
Know your data latency before you design the alert
This is the constraint that shapes everything else, and teams design around an assumption that is wrong.
Billing data is not real time. Depending on the provider and the service, usage records can take hours to appear and a day or more to settle, and some line items are reconciled later still. If your detection depends on final billing data, your minimum detection time is already most of a day.
So use two signals. Billing data for accuracy and for the cost attribution. And operational metrics as the early warning, because they are real time: requests per second on an API, bytes scanned by a query engine, node count in a cluster, egress bytes on a gateway, log ingestion volume. A cluster that doubled its node count at 14:00 is visible at 14:01 in Prometheus and on Thursday in the bill.
The practical design is an alert on the operational metric that predicts the cost, and a daily reconciliation against billing data to catch what the metrics did not cover.
The native detectors, and what they miss
Every major provider now offers anomaly detection that learns a spending pattern per service or account and alerts on deviation. Turn it on. It is free or near free, it requires no engineering, and it catches the large obvious events.
Its limits are consistent across providers. The model needs history, so a new account or a new service produces false positives for weeks. Granularity is usually per service and per account, so a single team's runaway inside a shared account can be lost in the noise of a large bill. And the detection is bounded by the billing data latency described above.
Treat the native detector as a backstop and build the specific checks for your top three or four cost drivers yourself. Those are where the money is, and you know their shape better than a generic model does.
The shapes a spike actually takes
Almost every cost incident we have investigated is one of these, and knowing the list lets you write targeted detection rather than generic anomaly alerts.
- A retry loop without backoff. A failing dependency and an aggressive retry turns one operation into thousands. Shows up as request count, not as anything that looks like a cost problem.
- A query without a filter. A scheduled job that scans an entire table instead of a partition, in a system priced per byte scanned. This is the single most expensive category we see, and the mechanism and the controls are in BigQuery bills that triple overnight.
- Logging turned up and left on. Debug logging enabled during an incident, never reverted, in a platform priced per gigabyte ingested. Covered from the structural side in logs are the line that grows.
- An autoscaler with a high ceiling and a bad signal. A memory leak or a stuck queue drives scaling to the maximum and holds it there.
- Data transfer added by a topology change. A service moved to another zone or region, and now every call crosses a billed boundary.
- Storage with no lifecycle rule. Slow, not a spike, and the largest cumulative loser. Versioning with no expiry is the usual specific cause.
- Leaked credentials used for mining. The malicious case, which looks like a sudden appearance of large compute instances in regions you do not use. It is also a security incident, and the cost alert is often the first detection.
- A non-production environment left running. A load test cluster from a Friday afternoon.
A detection rule for each of these, written against the operational metric that drives it, will catch more than any general model.
The response loop is the part that is missing
Most organisations that have this problem do have some alerting. What they do not have is a person.
The alert needs a named owner, per account or per service, derived from the same ownership tags that make cost attribution work at all. It needs a channel that is monitored rather than a mailbox. It needs a triage question that can be answered in five minutes: what changed, and can we revert it. And it needs an escalation path for the case where the answer is that the spend is legitimate and the forecast now needs updating.
Keep a short log of cost incidents with cause and resolution, the same way you would for outages. After a handful of entries the pattern is obvious, and the patterns repeat across teams.
The other half of the loop is prevention, and it lives upstream in the pull request. A cost delta shown at review time stops a proportion of these before they exist, which is the argument in cost is decided in the pull request.
Attribution decides whether the alert is useful
An anomaly alert that says the account spent more tells you nothing you can act on. An alert that says a specific service in a specific environment owned by a specific team spent more is a ticket.
That requires tag coverage, which is the same prerequisite as everything else in this area, and the same first step as in where the AWS bill hides and the Azure bill. If a meaningful share of cost lands in "untagged", your anomaly detection resolves to "something, somewhere, cost more", and people stop reading it.
The things people forget
- Alert on the rate, not only the total. A service that doubled is interesting at any absolute level if it is a leading indicator.
- Suppress the known. A monthly batch job that always spikes should be excluded, or it trains everyone to ignore the channel.
- Commitments distort the signal. When a reservation expires, cost jumps with no change in usage. Model that so it does not look like an incident.
- Currency and tax. Cross-currency estates see swings that are not usage at all.
- Test the alert. Create a small deliberate spike in a sandbox and confirm the alert fires and reaches a human. Most alerting is never tested.
What to do this week
Pull daily cost by service for the last 90 days and plot it. Look for the step changes rather than the trend. Every step is a change someone made, and for each one ask whether anybody noticed at the time. The number of unnoticed steps is the size of the gap, and it is usually the most persuasive chart you can put in front of a platform team. We build this view in the cost phase of a cloud engagement.