Putting AKS into production: the decisions that are hard to change later
Network plugin, identity model, node pool layout and upgrade strategy are all chosen in the first hour and all painful to change afterwards. Here is what we pick and why.
An AKS cluster takes four minutes to create and about two years to regret. Most of the regret comes from four decisions made in the creation wizard by someone who needed a cluster that afternoon, and all four require recreating the cluster to change.
1. Network plugin: Azure CNI Overlay
The choice is between kubenet (legacy), Azure CNI (a VNet IP per pod) and Azure CNI Overlay (pod IPs from a separate overlay CIDR, node IPs from the VNet).
Azure CNI with VNet IPs per pod is the one that causes the outage. A /22 subnet gives you about a thousand addresses, which sounds like plenty until you account for a pod per IP plus surge capacity during upgrades, and then you cannot scale. Resizing the subnet of a running cluster is not a thing you do casually.
Azure CNI Overlay is the right default now: pods get addresses from a private overlay range that does not consume VNet space, nodes get VNet IPs, and you keep network policy support. Use it unless you have a specific requirement for pods to be directly addressable from outside the cluster.
Pick a network policy engine at creation too — Cilium is the strongest option and is offered as Azure CNI powered by Cilium. Retrofitting network policy onto a cluster with no policy means auditing every flow in production.
2. Identity: Workload Identity, and nothing else
Pod-managed identities are deprecated. Use Microsoft Entra Workload ID: a Kubernetes service account is federated to an Entra application, and pods get short-lived tokens with no secret anywhere. Enable the OIDC issuer and workload identity at cluster creation — both are cluster-level settings.
For the cluster's own identity, a user-assigned managed identity, not a service principal with a secret you will have to rotate.
For humans, Entra ID integration with Azure RBAC for Kubernetes authorization, so kubectl access is governed by the same groups and PIM elevation as everything else. Local accounts disabled. Otherwise you have a cluster with a static admin kubeconfig circulating in a wiki.
3. Node pools: at least three, with taints
A system node pool for CoreDNS, metrics-server and the rest of the control-plane-adjacent components, tainted CriticalAddonsOnly so workloads do not land on it. Then user pools per workload class: general, memory-optimised, Spot.
Spot node pools carry an eviction taint by default, which is correct — put stateless replicated workloads there with a toleration and keep everything else off. On an estate with batch or CI workloads, Spot is the largest remaining discount after rightsizing.
Cluster autoscaler per pool, and — more importantly — requests set from observed usage, which is the same lesson as why your EKS cluster costs twice what it should and applies identically here. Node Autoprovisioning (Karpenter for AKS) is the better answer where it is available for your configuration.
4. Upgrades: automatic, with a maintenance window
AKS versions fall out of support on a schedule, and a cluster three versions behind is an urgent project rather than a routine one. Set an auto-upgrade channel (stable or patch), configure a planned maintenance window, use Pod Disruption Budgets so the drain respects your availability requirements, and set a max surge so upgrades are not glacial.
Test the upgrade in a non-production cluster of the same version first. The failure mode is almost always an API deprecation in a Helm chart, not the node image.
The things that bite in month two
- Log Analytics ingestion from Container Insights can be a top-three line on the bill. Configure the data collection rule to exclude namespaces you do not need, and avoid collecting stdout from chatty sidecars at full volume.
- Private cluster or not. A private API server is the right answer for production and means your CI needs a self-hosted runner in the VNet, or an authorised IP range. Decide before, not after.
- Ingress. Application Gateway for Containers or an ingress-nginx controller with an internal load balancer. Either works; what does not work is three teams each installing their own.
- Key Vault CSI driver for secrets, so nothing lives in a Kubernetes Secret in etcd unencrypted by your own key.
- Defender for Containers, which is the AKS-aware detection layer and is worth the per-node cost in production.
All four opening decisions are cluster-recreate operations, which is why we spend real time on them at the start of a cloud engagement rather than discovering them later.
What to do this week
Check your cluster's network plugin and pod subnet size. az aks show will tell you in one command. If you are on Azure CNI with VNet pod IPs and less than a /21, work out how many pods you can run before you exhaust it — that number is your real scaling ceiling, and it is better known now than during an incident.