Why your EKS cluster costs twice what it should

A cluster with 40 nodes at 22 percent average CPU is not a capacity problem. It is a requests problem, a bin-packing problem and a scaling-policy problem stacked on top of each other, and the dashboard will show all forty nodes as "in use" the whole time.

Here is how we take a cluster apart, in the order that finds money fastest.

Measure four things first

Requests versus usage, per namespace. The single most useful number in a Kubernetes cluster is the ratio of requested CPU and memory to actually used CPU and memory. Above 3x, you are buying three nodes to run one node's worth of work. Get it from kubectl top aggregated by namespace, or properly from Prometheus with kube_pod_container_resource_requests against container_cpu_usage_seconds_total.

Allocatable versus requested, per node. If requests are reasonable but nodes are still empty, the problem is bin packing: a few pods with large requests, anti-affinity rules, or a DaemonSet that reserves more than you think. Look for nodes where the largest unschedulable gap is bigger than the largest pending pod.

Cost per namespace. Kubecost or OpenCost, or a homemade split of the node bill by requests. Without this, no team believes the number is theirs and nothing changes.

Cross-AZ traffic. In a three-AZ cluster with topology-unaware services, roughly two thirds of pod-to-pod traffic crosses a zone boundary and gets billed. On chatty microservice estates this is a genuine line item, not a rounding error — the same one that shows up in layer four of the AWS bill.

Three changes that usually halve it

Set requests from observed usage, and set limits sparingly. Requests should sit near the p95 of real usage, not at a round number someone typed in 2023. Vertical Pod Autoscaler in recommendation mode gives you the numbers without acting on them; run it for two weeks and apply the output by hand. On limits: set memory limits, because the alternative is a node-wide OOM, but be careful with CPU limits, which cause throttling at p99 while the node sits idle. A CPU request with no CPU limit is the right default for most latency-sensitive services.

Replace Cluster Autoscaler with Karpenter. Cluster Autoscaler scales node groups you defined in advance, so your bin packing is limited by the instance types you guessed. Karpenter provisions the instance that fits the pending pods, consolidates workloads onto fewer nodes when they shrink, and will use Spot with a diversified pool. Consolidation alone typically removes 20 to 30 percent of nodes from a cluster that has been running for a year. Set a disruption budget and do-not-disrupt annotations for the stateful workloads before you turn it on.

Put the right things on Spot. Stateless, replicated, restart-tolerant workloads on Spot with a diversified instance pool and a 15-node minimum on-demand core for the control-plane-adjacent pieces. Spot is 60 to 90 percent cheaper and the interruption story is manageable when the workload is genuinely stateless. What kills teams is putting a single-replica stateful service on Spot and discovering the reclaim notice at 3am.

The things people forget

• Control plane and NAT. Each cluster is a fixed monthly control plane fee plus the NAT gateway it uses. Four small clusters where one would do is pure overhead.
• Idle dev clusters. Scale node groups to zero outside working hours. A cron and a Lambda; an afternoon of work.
• Over-provisioned DaemonSets. A logging agent requesting 500m CPU on 40 nodes is 20 cores you are paying for and not using.
• Old, unattached EBS volumes from deleted PersistentVolumeClaims with a Retain reclaim policy.

What good looks like

A healthy cluster runs at 55 to 70 percent average CPU allocation with requests within 1.5x of usage, autoscales in under two minutes, and can lose any single node without a page. Getting there is two or three weeks of work and it does not require rewriting anything — which is why it is usually the first thing we touch in a cloud engagement.

What to do this week

Compute the requests-to-usage ratio for your three largest namespaces. If it is above 2.5x, you have found your answer, and Vertical Pod Autoscaler in recommendation mode will hand you the corrected numbers within a fortnight.