A team investigating an unexpected AWS increase looks at CloudWatch, sees log ingestion, sets a shorter retention on the biggest log group, and moves on. The bill barely changes, because retention was never the expensive part.
CloudWatch charges separately for log ingestion, log storage, custom metrics, metric API calls, dashboards, alarms and Logs Insights queries. Each has its own unit and its own failure mode, and the two that generate most of the surprise are custom metrics and application logging left at debug level.
Find out which axis you are paying on
Before changing anything, split the CloudWatch line by usage type in Cost Explorer. The usage types name the axis directly, distinguishing data ingested, timed storage, metrics and API requests.
Then get the per-log-group breakdown, because the distribution is always top-heavy. The metric IncomingBytes per log group over the last 30 days tells you where ingestion is going, and three or four groups will usually carry most of it.
Do both before touching anything. Teams routinely optimise the third-largest line because it was the one they understood.
Custom metrics and the dimension explosion
This is the one that produces genuinely shocking numbers, and it is arithmetic rather than misfortune.
A custom metric is billed per unique combination of metric name and dimension values. Publish a metric with a Service dimension of ten values and an Endpoint dimension of twenty, and you have two hundred metrics. Add a CustomerId dimension with five thousand values and you have a million, each billed monthly.
The rule is the same as for Prometheus labels: a dimension value must come from a small, bounded set. Never use customer ID, request ID, user ID, a full URL path, an instance ID on a fleet that churns, or a version string that changes every deploy. The route template, not the resolved path, exactly as in Prometheus at scale.
Two specifics worth knowing. Embedded metric format lets you write metrics inside a structured log event, and CloudWatch extracts them, which changes the cost shape and lets you keep high-cardinality fields as log properties while only the low-cardinality ones become metrics. That is usually the right architecture. And metric filters on log groups create metrics from log content, which is convenient and quietly creates cardinality if the filter captures a variable value.
Also check for metrics nobody consumes. A metric published since 2022 that no dashboard or alarm references is a pure monthly charge, and enumerating metrics against the alarms and dashboards that use them is an afternoon that frequently pays for itself.
Ingestion, and the log level nobody reverted
Ingestion is priced per gigabyte and it is where an application's behaviour shows up directly.
The usual causes, in the order we find them:
- Debug logging enabled in production. Turned on during an incident, never turned off. This is the single most common cause of a step change in the CloudWatch line.
- Lambda logging every invocation. At high invocation counts, the function's own log output can cost more than the function's compute.
- VPC flow logs at full verbosity on busy interfaces, which is an enormous volume that is almost never queried in raw form.
- Load balancer access logs including health check requests, which are a large fraction of requests and are never read.
- Multi-line stack traces counted as many events unless the agent joins them.
The structural fixes are the ones in logs are the line that grows: structured events, levels used as designed, and a dynamic log level so an engineer can raise verbosity for one function without a deploy and without it becoming permanent.
Where the data is needed but not at analytics prices, subscription filters can route logs elsewhere, and the infrequent access log class costs less to ingest in exchange for reduced query features. Check what you lose before moving a group, particularly metric filters and alarms.
Storage, retention and the default that costs the most
Log groups default to never expiring. That default is the reason most accounts hold years of application logs nobody has queried since the week they were written.
Set retention per log group by class rather than one number for everything: audit and security logs to whatever the compliance requirement is, application logs to somewhere between one and three months, debug and access logs to days. Enforce it with a policy so new log groups do not inherit the infinite default, because a manual sweep is undone by the next deploy.
For anything that genuinely needs long retention, export to object storage and apply lifecycle rules there. The storage price is much lower, and the query path is slower in a way that matches how rarely you will use it, which is the same reasoning as in S3 and data transfer costs.
Queries, dashboards and alarms
Logs Insights is billed per gigabyte of data scanned. A query over a large log group with a wide time range scans everything in it, and a dashboard widget backed by an Insights query re-runs on every refresh, for every viewer. That is a scheduled expense disguised as a convenience. Narrow the time range, filter early in the query, and be careful about putting Insights queries on a dashboard that is open on a wall screen all day.
Dashboards are billed above a free allowance, which is minor unless the estate has accumulated dozens that nobody opens. Alarms are billed per alarm, with composite and high-resolution alarms priced differently, and high-resolution alarms are noticeably more expensive. Use standard resolution unless you genuinely need sub-minute detection.
The API is charged per request, and GetMetricData from a third-party observability tool polling thousands of metrics every minute is a real line. If you run an external monitoring platform, check what it is pulling and how often. Metric streams are usually cheaper than polling at scale.
The things people forget
- Container Insights and enhanced monitoring are custom metrics. Turning them on across a large cluster creates a lot of metrics per node and per pod. Useful, and not free.
- Every AWS service writes logs somewhere. Log groups appear without anyone creating them deliberately, and nothing deletes them.
- Cross-account log destinations still bill on ingestion, in the receiving account.
- Retention change is not retroactive in the way people assume. Reducing retention deletes older data, which is the point, but verify nothing compliance-relevant is in the group first.
- Alarms on absent metrics. Configure the treatment of missing data deliberately, or an alarm silently never fires.
What to do this week
Run a query listing your log groups sorted by ingested bytes over the last 30 days, and next to it list the log groups with no retention set. The intersection of those two lists is where the money is, and setting retention on the top five is a ten-minute change with an immediate effect on the next invoice. It is the same first pass we run in the cost phase of a cloud engagement, alongside the layers in where the AWS bill hides.