Writing a Disaster Recovery Runbook You Can Actually Follow at 3am

A disaster recovery plan that lives in a slide deck fails on the night you need it. The person rebuilding the service at 3am is tired, frightened, and usually not the person who wrote the plan.

Most DR documentation is written to pass an audit, not to be executed. It describes strategy, names responsible parties, and reassures the reader that backups exist. What it does not contain is the one thing that matters at 3am: the exact, ordered, copy-pasteable commands that rebuild a specific service, and a test that tells you when you are actually done. This piece builds that document for a real service, so that someone unfamiliar with the system can execute it under pressure and know when to stop.

The worked example is a checkout API backed by PostgreSQL, with application files on disk. The tooling is deliberately boring and current: pgbackrest 2.53 for the database, restic 0.17 for the files, systemd to run the service, and git to hold the runbook. Swap the specifics for yours; keep the shape.

The header block: what you promised, and when you last proved it

Every runbook opens with three numbers and a date. The recovery point objective (RPO) is how much data you have agreed to lose. The recovery time objective (RTO) is how long you have agreed to be down. Those are business commitments, not technical ones, and if nobody signed them the runbook is building to a target that was never agreed — which is a decision I have argued belongs to the board, not the on-call engineer, in RTO and RPO as business decisions.

The fourth field is the one most runbooks omit and the one a supervisor will ask about first: the date you last rehearsed the restore. A runbook that has never been executed end-to-end is a hypothesis, not a control. Put the date at the top, keep it honest, and let it embarrass you into scheduling the next rehearsal.

The restore order is the runbook

The single most valuable thing the document holds is the order of recovery. Data before application. Secrets before the process that needs them. DNS last, once the service is proven, so you never cut users over to a host that is still restoring. Write it as commands, not prose, with the expected output stated inline so the operator knows whether each step worked.

# ===== RUNBOOK: restore checkout-api =====
# RPO 15 min | RTO 60 min | last rehearsed: 2026-06-30 (see git log)
# Tested against: postgresql-16, pgbackrest 2.53, restic 0.17

# 0. Declare the incident. Page the on-call DBA. Start the clock.

# 1. Provision a clean host (or reuse the warm standby) and export config.
export PGBACKREST_STANZA=checkout
export RESTIC_REPOSITORY=s3:https://s3.eu-central-1.amazonaws.com/acme-backups/checkout
export RESTIC_PASSWORD_FILE=/etc/restic/checkout.pass

# 2. Restore the database to the latest consistent state.
systemctl stop postgresql@16-main
pgbackrest --stanza="$PGBACKREST_STANZA" --delta --type=default restore
systemctl start postgresql@16-main
# Confirm recovery has finished before continuing:
sudo -u postgres psql -Atc "select pg_is_in_recovery();"   # expect: f

# 2a. CORRUPTION / BAD DEPLOY variant — restore to just before the event
#     instead of the latest (poisoned) state:
# pgbackrest --stanza="$PGBACKREST_STANZA" --delta \
#   --type=time --target="2026-10-14 15:52:00+00" restore

# 3. Restore application files (uploads, rendered config) from restic.
restic restore latest:/srv/checkout/data \
  --target /srv/checkout/data --verify

# 4. Restore secrets. Never store them in this repo. Pull from the vault
#    or decrypt the sealed env file the service reads at boot:
age --decrypt -i /root/keys/checkout.age \
  -o /srv/checkout/.env /srv/checkout/.env.age

# 5. Start the service and confirm it is listening locally.
systemctl start checkout-api
systemctl is-active checkout-api                     # expect: active

# 6. Run the smoke test (next section). ONLY move DNS after it passes.
/opt/checkout/smoke-test.sh https://checkout-recovery.internal.acme.example

# 7. Cut DNS over to the recovered host. Record the time. Stop the clock.

Two details earn their place. The --delta flag on pgBackRest restores only the files that differ from what is already on disk, which turns an hour-long restore of a large cluster into minutes when you are reusing a standby — the difference between hitting your RTO and blowing through it. And restic restore uses the snapshot:subpath syntax to pull back just the data directory rather than the whole snapshot, with --verify so restic re-reads what it wrote and tells you if a restored file does not match its stored hash. A restore that silently produces corrupt files is worse than no restore at all.

Secrets deserve their own line of discipline. They never live in the runbook repository. Step 4 assumes an age-sealed env file or a vault fetch; the mechanics of rotating and recovering credentials cleanly are their own exercise, worked through in a key rotation runbook you can actually run.

The smoke test that says “it is actually back”

“The process is running” is not “the service is back”. A checkout API can bind to its port with an empty database, a broken read path, or a write path that fails on the first order. The smoke test exists to distinguish listening from working, and to give a frightened operator a single unambiguous verdict: PASS or FAIL. Keep it in the same repository and call it directly from step 6.

#!/usr/bin/env bash
# smoke-test.sh — confirm checkout-api is genuinely back, not just listening.
set -euo pipefail

BASE_URL="${1:?usage: smoke-test.sh BASE_URL}"
fail() { echo "SMOKE FAIL: $1" >&2; exit 1; }

# 1. Health endpoint returns 200.
code=$(curl -s -o /dev/null -w '%{http_code}' "$BASE_URL/healthz")
[[ "$code" == "200" ]] || fail "healthz returned $code"

# 2. Database is reachable AND holds real data, not an empty schema.
rows=$(sudo -u postgres psql -Atc "select count(*) from orders;")
[[ "$rows" -gt 0 ]] || fail "orders table empty after restore"

# 3. A read path that touches the database returns a known-good record.
curl -fsS "$BASE_URL/api/orders/self-test" >/dev/null \
  || fail "order read path broken"

# 4. A write path round-trips (idempotent probe order).
curl -fsS -X POST "$BASE_URL/api/orders/smoke" \
  -H 'content-type: application/json' -d '{"probe":true}' >/dev/null \
  || fail "order write path broken"

echo "SMOKE PASS: $BASE_URL healthy at $(date -u +%FT%TZ)"

The test checks the four things that actually break after a restore: the process, the data, a read that exercises the database, and a write that proves the service is not read-only or wedged. It exits non-zero on any failure, which means the runbook can gate the DNS cutover on it and a tired operator cannot skip ahead by accident.

Keep it in version control, next to the scripts it calls

A runbook in a wiki drifts from the system it describes; a runbook in a shared drive is unreachable when the identity provider is the thing that is down. Put it in git, in the same repository as the smoke test and any helper scripts it invokes, so the commands and the code they run are versioned together and reviewed together. When someone changes the backup layout, the pull request that changes it should change the runbook in the same commit, or the review should fail.

Version control also gives you the rehearsal record for free. Each time you run the drill, commit the smoke-test output and update the “last rehearsed” line; the git log becomes contemporaneous evidence that the control operates, which is precisely what a resilience supervisor asks for and what a paper plan can never produce. It is the same discipline that separates a real cloud exit from a stated one, which I have written about in DORA exit strategy testing. And because the restore leans on your backups actually being restorable, the retention policy behind them is not a side issue — a restic retention policy that keeps the snapshots you need is what makes step 3 possible at all.

The test of a disaster recovery runbook is not whether it reads well in a committee. It is whether the most junior person on the rota can open it at 3am, run it top to bottom, and reach a PASS — without phoning the author, because the author is on a plane. Write it for that person, rehearse it until it is boring, and the night you need it will be boring too.

Free interactive tool

Interactive deadline calculator

Check which regulations apply to you and when

Regulation across the EU, UK, US and Asia-Pacific has moved considerably in the past eighteen months, and several headline dates have shifted more than once. Twelve questions, about three minutes.

Results are shown on screen — no email required. A dated summary is available to download, and can be sent on if that's more useful. What we do with your answers.

Governance is what happens when nobody is watching.

Policies are easy. Consistent decision-making is harder. Understand where governance exists and where it has quietly become assumed.

Full Governance by Sixteen Pillars

Govern your business. Prove your compliance.

A board assurance cockpit for EU-regulated financial firms — tamper-evident, hash-chained proof of governance across DORA, GDPR, NIS2, ISO 27001, the EU AI Act and MiCA. In development.

See what's coming