Terraform on AWS: state layout, pipelines and the drift you will find

The two Terraform repositories we are handed look the same from the outside and fail differently. One is a single root module holding four hundred resources across every environment, where terraform plan takes eleven minutes and everyone is afraid of it. The other is two hundred tiny modules with no clear ownership, where changing a subnet CIDR touches nine repositories.

Both are state layout problems. Get that right and the rest of Terraform is ordinary software engineering.

Split state by blast radius and rate of change

The rule we use: one state per account, per layer, where a layer is a set of resources that change together and would be destroyed together without it being a catastrophe.

• Foundation — VPC, subnets, transit gateway attachments, Route 53 zones, KMS keys. Changes monthly. Its destruction is an outage.
• Platform — EKS cluster, RDS, shared load balancers, IAM roles for workloads. Changes weekly.
• Workload — one state per service: task definitions, autoscaling, alarms, the service's own IAM role. Changes daily.

A change to a workload should never plan against the VPC. When it does, the eleven-minute plan is the least of your problems — the real risk is a hurried apply touching something it should not.

Cross-layer references go through data sources or SSM parameters that the lower layer publishes, never terraform_remote_state reading another team's file. Remote state reads create a hidden coupling that nothing in the code makes visible, and they break the day someone moves a resource.

The backend

S3 with versioning and DynamoDB locking, the bucket in the account it serves, encrypted with a KMS key whose policy allows only the pipeline role and the break-glass role. Turn on S3 versioning before you need it; recovering a state file someone truncated is a five-minute job with versioning and a very bad week without it.

Pipelines, and humans with read-only

The single change that improves a Terraform estate most: nobody applies from a laptop in production. The pipeline assumes a per-account role via OIDC from your CI provider — no static keys anywhere — and it is the only principal with write permissions. Humans get ReadOnlyAccess plus a break-glass role that pages when assumed.

The pipeline shape we deploy:

1. fmt, validate, and a policy check — tfsec or checkov for security, plus conftest for your own rules like "no security group with 0.0.0.0/0 on port 22".
2. plan on pull request, posted as a comment. The plan runs with a read-only role, so a plan cannot change anything.
3. Manual approval for production, automatic for staging.
4. apply with the write role, and the plan file from step 2 — applying a re-planned change is how surprises get in.

The drift you will find

Every estate older than a year has drift, and the first plan after you introduce the pipeline is the moment you discover how much. Three categories, handled differently:

• Resources changed by hand in the console. The plan wants to revert them. Before you let it, ask why someone changed it — often the console change was correct and the code is wrong. Fix the code.
• Resources created by hand that Terraform does not know about. These do not show in a plan at all, which is what makes them dangerous. Find them by comparing terraform state list against the live inventory; Skyline draws exactly this diff, and it is the view that makes the orphaned-resource conversation short.
• Resources Terraform created and someone deleted. The plan wants to recreate them. Usually correct, occasionally catastrophic — check what depends on them first.

Import the ones worth keeping with import blocks in the configuration rather than the CLI command, so the import is reviewable in a pull request like everything else.

Modules, sparingly

Write a module when the same pattern appears three times, not before. A module with fourteen boolean feature flags is a sign that two different things were forced into one abstraction. Pin module versions, including registry modules, and upgrade deliberately — an unpinned module is a change that arrives without a pull request.

This layout is what we put in place at the start of most cloud engagements, usually alongside the account map from our landing zone piece.

What to do this week

Run terraform plan against production and count the resources it wants to change. If the answer is not zero, you have drift, and the list of what it wants to revert is the most honest document about your estate that exists.