RTO, RPO and the restore test almost nobody has run

Ask an engineering team whether they have backups and the answer is yes, immediately and confidently. Ask when they last restored one into a working environment, end to end, with a stopwatch running, and the room goes quiet. In the estates we audit, the honest answer is usually "never", or "eighteen months ago, and it was a single database".

That gap is where outages become week-long incidents. The backup job succeeding is a green tick in a console. The restore is forty steps involving IAM, DNS, secrets, TLS certificates, a third-party payment provider and a person who left the company. Nobody has walked that sequence, so nobody knows it takes eleven hours instead of the two the disaster recovery document claims. The fix is not more backup. It is targets you can defend and a test that proves them.

Set the targets per service, not for the company

A single company-wide "RTO of 4 hours, RPO of 15 minutes" is a document written to satisfy a questionnaire. It is unaffordable where it is strict and useless where it is loose, because your checkout path and your internal expense tool do not deserve the same money.

Recovery time objective is how long the service may be down. Recovery point objective is how much data you accept losing. Both are business decisions with an engineering price tag, and both belong to a named service owner.

Do it as a table, one row per service:

The justification column is the one that matters. It stops a team asking for a one-minute RPO on everything and then funding none of it. Tier 3 with an honest five-day RTO is a legitimate answer, and it frees budget for tier 1.

Then check the targets against reality. An RPO of one minute means near-synchronous replication, not nightly snapshots. An RTO of one hour means the standby exists now, not that someone will terraform it during the incident.

A backup that exists is not a backup you can restore

The distinction is boring and it is the whole article. Common ways a backup that "exists" fails at the moment of truth, several of which are also what an attacker will target deliberately, as covered in backups that survive ransomware:

• The snapshot is in the same account, region and encryption key as the thing it protects.
• The database dump is logically corrupt because it was taken with an inconsistent read.
• The restore requires a KMS key whose policy only granted access to the deleted role.
• The backup contains data but not schema version, and the application refuses to start against it.
• The retention window is 7 days and the corruption started 12 days ago.
• Restoring takes 9 hours because the volume is 8 TB and the storage tier throttles first-read throughput.

That last one is the reliable surprise. A cold archive tier does not restore at the speed of a warm snapshot, and large volumes rehydrated from object storage are often "available" long before they perform. Measure it once on your own data and you will never quote a restore time from a vendor page again.

The quarterly restore test, and what it produces

Once a quarter, per tier-1 service, restore into an isolated environment and run the application against it. Not a checksum. Not a "the snapshot mounted". The application, answering a real request, with the data present.

Run it as a timed exercise with a scribe. Record:

• Wall-clock start, first successful read, first successful write, full service restored.
• Every manual step, including the ones somebody improvised.
• Every credential, key or approval that had to be found rather than looked up.
• The measured RPO, which is the timestamp of the newest record you actually recovered.

The output is two artefacts: an updated one-page runbook and a list of defects with owners. The defects are the point. A test that finds nothing was not a real test, usually because someone prepared the environment in advance.

Chaos experiments are the same discipline applied to failure modes other than data loss, and the two programmes should share a calendar. We cover the entry point in first chaos engineering experiments that do not break production.

The dependencies nobody writes down

• DNS. Restoring the service in a new place is worthless if the record still points at the old one and the TTL is 24 hours. Lower TTLs on failover-critical records before you need to, and know who can change them at 3am.
• Secrets and configuration. The restored instance needs API keys, database passwords and OAuth client secrets. If your secret store is inside the failed blast radius, the restore is blocked on the thing that broke.
• TLS certificates. Private CA chains and pinned certificates expire, and the renewal automation usually lives on the host you just lost.
• Third parties. Payment gateways, identity providers and partner APIs often allowlist source IPs. Your recovery environment has different ones. Get them registered in advance.
• Licences and quotas. A restore into a fresh account hits default service quotas immediately. Request the increases before the incident, not during it.
• The people. One person who knows the sequence is a single point of failure with a holiday calendar.

Write the plan for 3am

The recovery plan most organisations own is a 40-page Word document with a change-history table. Nobody reads it during an incident.

What works is one page per service: preconditions, the ordered commands, the verification step, the rollback and two phone numbers. Copy-pasteable, and stored somewhere reachable when the primary environment is down, which means not only in the wiki that runs on the affected cluster. Print one copy. It sounds absurd until the identity provider is the outage.

We build this table and run the first restore test in the resilience phase of a cloud engagement. That first test is nearly always the one that reveals the real numbers.

What to do this week

Pick your most important database. Restore last night's backup into a scratch environment and time it, from clicking start to the application serving a real query. Write the number down next to the RTO in your disaster recovery document. If the two numbers disagree, you now have the most useful ticket in your backlog.