The testing pyramid for infrastructure code
Formatting and validation are free, static analysis is nearly free, policy on the plan is cheap, and real apply-and-destroy tests are expensive and slow. Spend accordingly, and know what each layer cannot catch.
Infrastructure code has an unusual property: the thing it produces is expensive to create, expensive to destroy, and impossible to fully simulate. That makes the usual advice about test coverage unhelpful, and it is why most teams end up with either nothing or an elaborate integration suite that takes forty minutes and gets disabled.
The workable version is a pyramid, where each layer is roughly ten times cheaper than the one above it and catches a different class of mistake.
Layer one: formatting and validation, free
fmt -check and validate in CI, on every pull request. Formatting removes an entire category of diff noise and the argument about style. Validation catches syntax errors, type mismatches, undeclared variables and missing required arguments without touching a cloud provider.
This costs seconds and no credentials. There is no reason not to have it, and a surprising number of repositories do not.
Add variable validation blocks in the code itself while you are here. A constraint that an environment name must be one of three values, or that an instance count must be within a range, turns a runtime failure into an immediate one with a readable message.
Layer two: static analysis, nearly free
Security and correctness scanners read the configuration and flag known-bad patterns: a bucket without encryption, a security group open to the world, a database without deletion protection, an unencrypted volume.
Checkov and its peers cover the common cloud misconfigurations well and are part of the standard set in eight open-source tools for cloud posture. They run in seconds, need no credentials, and catch the mistakes that make up most real findings.
Two practices decide whether this layer survives. Start in report-only mode and fix the backlog before enforcing, or the first run produces four hundred findings and everyone learns to ignore the tool. And handle suppressions properly: an inline exception with a written reason is fine, a global skip list is how the scanner becomes decorative.
Layer three: policy on the plan, cheap and underused
This is the layer most teams are missing, and it is where the best return is.
The plan output as JSON is a complete description of what is about to change, and you can assert against it without applying anything. That lets you check things static analysis cannot see, because they depend on the resolved values: that no resource is being destroyed in production, that the instance type is on the approved list, that every resource carries the required tags, that the change does not exceed a cost threshold.
Write the rules with a policy engine, in the same language as your admission control if you already have one, which is the argument for consistency in Kyverno or Gatekeeper. Or assert against the JSON directly with a script; the logic matters more than the tool.
The single highest-value rule to start with: fail the build if the plan destroys or replaces anything in a production workspace without an explicit override. That one check prevents the most expensive mistake this tooling is capable of making, and it takes an afternoon.
Cost belongs in this layer too, as a delta on the pull request rather than a surprise on the invoice, which is the argument in cost is decided in the pull request.
Layer four: unit tests on module logic
Terraform's native test framework runs plan-based assertions against a module with given inputs, checking that the resulting plan has the properties you expect. No infrastructure is created.
This is worth doing for modules with real logic: conditional resource creation, complex for_each expressions, computed names, or a variable that changes several downstream values. It is not worth doing for a module that creates one resource from one variable, where the test would restate the code.
Write the tests as fixtures the module's example directory can share, so the examples stay working and the tests stay realistic.
Layer five: integration tests that actually apply
Create the infrastructure, assert against the real thing, destroy it. This is the only layer that catches the failures that matter most: a provider behaviour that differs from the documentation, a permission missing at apply time, two resources that cannot coexist, an eventual consistency problem that makes an apply flaky.
It is also slow and expensive. A full apply-and-destroy cycle on a networking module is minutes, sometimes tens of minutes, and it costs real money on every run.
So reserve it for the small number of foundational modules where a defect is expensive: the network module, the cluster module, the identity module. Run it on merge or nightly rather than on every push.
Two things make the difference between a suite that survives and one that is disabled. Reliable cleanup, because a failed test that leaves resources behind costs money and eventually exhausts a quota; use a defer-style teardown and run a scheduled sweeper that deletes anything in the test account older than a day. And a dedicated test account or subscription, never a shared environment, so the sweeper can be ruthless.
Ephemeral environments, and their real cost
Spinning up a full environment per pull request is attractive and works well for application changes on top of stable infrastructure.
For infrastructure changes it is harder than it looks. Some resources take a long time to create, some have account-level quotas that a dozen concurrent environments exhaust, some have names that must be globally unique, and some cannot be destroyed cleanly without manual intervention.
Where it fits, it fits well. Where it does not, a shared long-lived staging environment that receives changes first is the honest alternative, and it is what most estates end up with.
Drift is the test that runs in production
None of the above tells you whether reality still matches the code. A scheduled plan against every production workspace, alerting when the plan is not empty, is the cheapest ongoing test you can run and it catches the console change nobody mentioned.
Expect noise initially from resources that legitimately change outside Terraform. Suppress those deliberately with ignore_changes rather than ignoring the alert, and the signal becomes trustworthy. The state and pipeline patterns this depends on are in Terraform state and pipelines on AWS.
The things people forget
- The plan is the most important review artefact. Post it on the pull request. A human reading a plan catches things no rule anticipates, which is the argument in reading a Terraform plan is not the same as seeing it.
- Tests need credentials, which is a supply chain risk. A test suite with permission to create infrastructure is a target. Use short-lived federated credentials, as in getting long-lived keys out of CI.
- Provider version pinning is part of correctness. An unpinned provider upgrading itself changes behaviour between a passing test and an apply.
- Test the destroy path. Many modules apply cleanly and fail to destroy, which you discover when you need to tear down an environment.
- Flaky infrastructure tests get deleted. Invest in retries and in cleanup, or the suite will not survive its second month.
What to do this week
Add one check: fail the pipeline if a plan against a production workspace contains a destroy or a replace, unless a specific label is on the pull request. It is an afternoon of work with the plan JSON, it needs no new tooling, and it prevents the single most expensive category of infrastructure mistake. We wire this in during the platform phase of a cloud engagement.