Define the drill boundary
Choose one representative environment and state exactly what will be restored:
- database records;
- uploaded files or object storage;
- application configuration;
- identity mappings;
- encryption or signing dependencies; and
- the application revision compatible with that data.
Run the first drill in an isolated destination. Do not overwrite production. Use test credentials and block outbound email, payments, webhooks, and other business effects.
The NIST contingency-planning guide is written for federal systems, but its cycle of planning, testing, training, and maintaining recovery procedures is broadly useful. This guide is a small operational exercise, not a compliance claim.
Step 1: record expectations
Before touching the backup, write:
- recovery point objective: the maximum acceptable data age;
- recovery time objective: the target time to resume the named workflow;
- latest backup timestamp and location;
- operator and observer;
- required accounts, keys, tools, and versions; and
- stop conditions.
If the team has never set objectives, record current business assumptions instead of inventing precision.
Step 2: restore into isolation
Create a clean destination. Retrieve the backup using the documented process, verify its expected size or checksum when available, and restore each state store in dependency order.
Record every undocumented decision. If the process requires a credential owned by an absent person, an unpinned tool, or a manual database edit, treat that as a drill finding rather than improvising it away.
Step 3: verify useful integrity
Do not stop at “restore command exited zero.” Check:
- expected record counts or control totals;
- a sample of parent-child relationships;
- representative uploaded files;
- the ability of a test identity to load allowed data;
- absence of another identity's data;
- required scheduled-job state; and
- the named critical workflow.
Use synthetic or approved test records. Do not expose customer data to an unapproved environment.
Step 4: measure and decide
Record:
| Measure | Expected | Observed | Result |
|---|---|---|---|
| Backup age | |||
| Time to retrieve | |||
| Time to restore | |||
| Time to verify workflow | |||
| Missing dependencies | None | ||
| Integrity exceptions | None |
Decide whether the recovery is acceptable, conditionally acceptable, or failed. A slow but correct restore and a fast incomplete restore require different repairs.
Step 5: clean up and improve
Destroy or lock the isolated environment according to its data policy. Rotate temporary credentials. Update the runbook with exact commands and owners, then schedule the next drill after the findings are fixed.
Rollback for this exercise means abandoning the isolated destination—not writing partially restored state back to production. A real incident needs a separate return-to-service decision identifying which state becomes authoritative.
Verify the result
The drill is complete only when the dated record names the backup, destination, operator, elapsed times, integrity checks, failures, cleanup, and next owner. That record is more useful than a green backup icon.