Prometheus stops working the way you first built it

The failure is always the same shape. Prometheus has run fine for eighteen months on a 4 GB pod. Someone adds a user_id label to a request counter because it would be handy for debugging. Two days later the pod is OOM-killing every twenty minutes, queries time out, and the on-call engineer has no monitoring during the incident that the monitoring change caused.

Prometheus is superb at the scale you start at and unforgiving about how you grow it. Almost everything that goes wrong traces back to one number.

Cardinality is the cause, and you can measure it directly

A Prometheus time series is a unique combination of metric name and label values. Every distinct combination is a separate series held in memory with its own chunk. A counter with three labels of ten values each is a thousand series. Add a fourth label with ten thousand values and it is ten million, and the process dies.

You do not have to guess where it is. Prometheus exposes it:

• topk(20, count by (__name__)({__name__=~".+"})) gives you the twenty metric names with the most series. This is the query to run first, every time.
• prometheus_tsdb_head_series is your total active series, the number to graph and alert on.
• count(count by (label_name) (your_metric)) tells you how many distinct values a specific label actually has in practice, which is nearly always more than the developer assumed.
• The TSDB status page in the Prometheus UI shows top series by metric and by label pair without writing any query at all. Send people there before they add a label.

Set a hard alert on total active series with a threshold you have actually capacity-planned for, and a second one on the growth rate. Cardinality problems are far cheaper to catch at plus-15-percent-in-an-hour than at the OOM.

Labels that must never carry an identifier

The rule is simple and worth writing into a lint check: a label value must come from a small, bounded, slowly changing set.

Never put in a label: user ID, customer ID, request ID, trace ID, session ID, email address, full URL path with IDs in it, raw error message, timestamp, container ID, pod IP, or any UUID. Every one of these is unbounded, and unbounded means the series count grows with your traffic instead of with your architecture.

The specific one that catches teams is the HTTP path. path="/api/orders/8814/items" is one series per order. Instrument the route template, path="/api/orders/:id/items", not the resolved URL. Most frameworks give you the template; the auto-instrumentation just has to be configured to use it.

If you genuinely need per-user or per-request detail, that is what traces and logs are for. Put the high-cardinality identifier on a span attribute where it belongs, and use the trace to get from a metric anomaly to the specific request. That handoff is the entire argument for using distributed traces to debug.

Enforce it at the collection point too. metric_relabel_configs in the scrape config can drop a label or an entire metric before it is ever stored, which means you can stop a bad deploy from hurting you without waiting for a code fix.

Retention and remote write are two different problems

Local Prometheus retention is disk-bound and typically set somewhere between 15 and 90 days. That is fine for operations and useless for capacity planning, quarterly trends and anything an SLO report needs over a year.

The answer is remote write to long-term storage, but be clear about what you are buying. Remote write ships every sample over the network continuously, so it adds CPU, memory for the write-ahead-log queue, and egress. Tune max_samples_per_send and queue capacity, and watch prometheus_remote_storage_samples_failed_total and the queue's shard count, because a backing store that slows down applies backpressure directly to your Prometheus.

Reduce what you send. You almost never need one-second raw resolution of every series in cold storage. Filter with write_relabel_configs to send the series you actually query historically, and let the rest expire locally.

Running two Prometheus servers is not high availability

The standard HA pattern is two identical Prometheus instances scraping the same targets. It gets you survival of one node failure, and that is all it gets you. It does not get you:

• A consistent view. The two instances scrape at slightly different times, so their samples differ. Query one and then the other during an incident and the numbers do not match, which erodes trust exactly when you need it.
• Deduplicated alerts. You need Alertmanager in cluster mode, with all instances gossiping, or every alert fires twice. Alertmanager handles this well, but it is a separate thing you have to configure.
• A global view across clusters. Two replicas of one cluster's Prometheus tell you nothing about the other four clusters.

That last gap is the honest reason to reach for something bigger.

Thanos or Mimir, and when you actually need one

Do not deploy either because a conference talk said to. You need one when at least one of these is true: you have to query across multiple Prometheus instances or clusters in a single dashboard; you need retention measured in years with sane query performance; or a single Prometheus no longer fits in a machine you are willing to run.

Thanos bolts onto existing Prometheus servers. A sidecar uploads TSDB blocks to object storage, a querier fans out across sidecars and store gateways, and a compactor downsamples old blocks. It is incremental, which is its main virtue: you keep the Prometheus servers you have.

Mimir is a horizontally scalable, multi-tenant backend you remote-write into. Prometheus becomes a thin scraper and forwarder. It is the better fit when you have genuine multi-tenancy, many teams, and want one operated system rather than a fleet of servers plus a query layer.

Rough guidance: Thanos if you are adding long-term storage and a global view to an estate of per-cluster Prometheus servers you are happy with. Mimir if you are consolidating a sprawl of them into a platform. Both need object storage, both need real operational attention, and both are a step change in complexity. Object storage costs are not the concern; the compaction and query resources are.

The operator and ServiceMonitor are how this stays maintainable in Kubernetes

Hand-editing prometheus.yml in a ConfigMap does not survive contact with a multi-team cluster. The Prometheus Operator replaces it with Kubernetes objects: Prometheus for the server, ServiceMonitor and PodMonitor for scrape targets selected by label, PrometheusRule for recording and alerting rules, and Alertmanager for routing.

The practical effect is that a team ships its own ServiceMonitor in its own Helm chart, and the platform team never touches a scrape config again. Two things to get right: the label selectors on the Prometheus object determine which ServiceMonitor objects are picked up across namespaces, and getting that wrong is the usual reason a new service silently is not scraped. And PrometheusRule objects should be validated in CI with promtool check rules, because a syntax error in a rule file can stop the whole rule group from loading.

Recording rules for the queries your dashboards run every 30 seconds

A dashboard panel with a rate() over a five-minute window across 50,000 series, refreshed every 30 seconds by twelve people, is a self-inflicted denial of service. Recording rules precompute the expression on a schedule and store the result as a new series, so the panel reads one cheap series instead of aggregating fifty thousand.

Find the candidates the same way you find cardinality: look at prometheus_engine_query_duration_seconds and at the slowest panels in Grafana. Anything aggregating across a large label set, anything used in more than three dashboards, and anything an alert evaluates frequently is a recording rule. Name them by the convention level:metric:operation, for example namespace:container_cpu_usage:rate5m, so it is obvious in a dashboard that you are reading a precomputed series.

This also makes your alert rules faster and more stable, which matters because slow rule evaluation delays alerts precisely when the cluster is under load.

The things people forget

• Scrape interval is a cardinality multiplier for storage, not for series count. Halving the interval doubles samples and disk, but does not change active series. People conflate the two and tune the wrong knob.
• kube-state-metrics and cAdvisor are the biggest series producers in most clusters. Before blaming application metrics, count them. Dropping the container metrics you never query is often the single largest win.
• Staleness and absent(). An alert on a metric that disappears entirely will never fire, because there is no series to evaluate. Use absent() or up for "the thing stopped reporting" conditions.
• Federation is not a scaling strategy. It was designed for pulling a small set of aggregated series between servers. Using it to pull everything from ten clusters into one Prometheus produces exactly the overload you were trying to avoid.
• The write-ahead log replays on restart. A very large head block means a slow startup, and a Prometheus that takes eight minutes to come back is eight minutes of blindness after every restart.

Right-sizing a monitoring stack tends to run alongside right-sizing the cluster it watches, which is why it usually comes up in the same conversation as why an EKS cluster costs twice what it should. We work through both in a cloud engagement.

What to do this week

Open the TSDB status page on your production Prometheus, or run topk(20, count by (__name__)({__name__=~".+"})), and look at the top five metric names. Then run count(count by (label) (metric)) on the worst one for each of its labels. In under an hour you will know exactly which label is costing you the most memory, and in most estates it is one you can drop with a metric_relabel_configs entry today.