Ransomware Recovery Runbooks as Code: Rehearsing the Worst Day in CI

The recovery plan you have never executed will fail on the day you need it. A ransomware runbook that lives in a PDF is a hypothesis, not a capability — and the incident is a bad time to discover which.

Most disaster-recovery documentation is written to be approved, not to be run. It reads well in a board pack, it satisfies the auditor’s request for evidence that a plan exists, and it has never once been executed end to end under time pressure. Then the encryption event lands on a Friday night, and the people opening that document for the first time in anger discover that the restore command in step nine references a bucket that was decommissioned eighteen months ago. The fix is not a better document. It is to express the runbook as automation you can dry-run repeatedly, time honestly, and break in rehearsal rather than in production.

The plan you have never run is fiction

A written recovery plan carries a hidden assumption: that every step still works, in order, when the environment around it has drifted for months. That assumption is almost always wrong. Credentials rotate. Backup targets move. The clean-image build that worked last year now pulls a base image that no longer exists. Nobody notices, because nobody runs the plan — the whole point of a disaster-recovery document is that you hope never to open it.

Recovery time is the metric that actually matters, and it is the one a paper plan cannot tell you. You do not know your recovery time objective is achievable until you have hit it. Guessing four hours because it sounds defensible is not the same as having restored a critical service in four hours, from a clean host, against an immutable backup, with credentials rotated, more than once. This is recovery-time engineering, and I have argued the same point in the context of healthcare ransomware resilience: the objective is not a number in a policy, it is a rehearsed, measured capability.

What a supervisor will ask you to prove

For EU financial entities, DORA’s Article 12 makes this concrete rather than aspirational. It requires backup policies that specify the scope and minimum frequency of backups by data criticality, and — the part firms routinely underweight — it requires that restoration and recovery procedures be tested periodically. It also requires that when you restore, you do so onto ICT systems that are physically and logically segregated from the source system, so that a compromise of production cannot follow the restore into your recovery environment. NIS2 pushes essential and important entities in the same direction through its backup and crisis-management obligations.

Read that as an engineering instruction, not a legal one. A supervisor is entitled to ask when you last exercised the restore, how long it took, and what evidence you hold that it succeeded. “We have a documented procedure” is a weak answer. “We rehearse it on every change to the recovery tooling and here is the timing history” is a strong one. The gap between those two answers is exactly the gap between a document and code.

The runbook as executable steps

The shape that works is a small, ordered set of steps with explicit dependencies, each carrying both the real command and a safe rehearsal stand-in, run by a harness that can operate in two modes. Isolate first, so you are not rebuilding into a live threat. Rebuild the host from a known-good image rather than trusting a machine that was on the network during the compromise. Rotate every credential the attacker may hold. Restore data from the newest immutable, verified backup — not the newest backup, the newest verified one. Validate before you declare recovery, because a restore that brings back corrupted or half-written data is not a recovery. The harness times each step and fails the job if the total breaches your objective.

"""Ransomware recovery runbook as rehearsable automation.

Dry-run in CI to time recovery against your RTO without touching production:
    python recover.py --rto-minutes 240
Run for real during a declared incident (guarded):
    python recover.py --execute --confirm
"""
import argparse, subprocess, sys, time
from dataclasses import dataclass, field

@dataclass
class Step:
    name: str
    run: str        # command executed during a real recovery
    rehearse: str   # safe stand-in exercised in the dry-run / staging restore
    depends_on: list = field(default_factory=list)

# Ordered by dependency: isolate first, restore late, validate before "done".
RUNBOOK = [
    Step("isolate",      "firewall-cut --tag prod --deny-all",
                         "echo isolate: revoke all ingress and egress"),
    Step("rebuild-host", "packer build clean-image.pkr.hcl",
                         "echo rebuild host from known-good image", ["isolate"]),
    Step("rotate-creds", "vault rotate --mount prod --all",
                         "echo rotate every credential the attacker may hold", ["isolate"]),
    Step("restore-data", "restic restore latest --repo s3:immutable --target /srv",
                         "restic restore latest --repo s3:staging --target /tmp/rehearsal",
                         ["rebuild-host"]),
    Step("validate",     "pytest tests/smoke -q",
                         "pytest tests/smoke -q", ["restore-data", "rotate-creds"]),
]

def order(steps):
    """Topological sort so dependencies always run before their dependants."""
    done, seq = set(), []
    while len(seq) < len(steps):
        progressed = False
        for s in steps:
            if s.name not in done and all(d in done for d in s.depends_on):
                seq.append(s); done.add(s.name); progressed = True
        if not progressed:
            sys.exit("cycle or missing dependency in runbook")
    return seq

def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("--execute", action="store_true", help="run for real; needs --confirm")
    ap.add_argument("--confirm", action="store_true")
    ap.add_argument("--rto-minutes", type=float, default=240.0, help="recovery-time objective")
    args = ap.parse_args()
    live = args.execute and args.confirm

    start = time.monotonic()
    for step in order(RUNBOOK):
        cmd = step.run if live else step.rehearse
        t0 = time.monotonic()
        rc = subprocess.run(cmd, shell=True).returncode
        dt = time.monotonic() - t0
        print(f"[{'LIVE' if live else 'REHEARSE'}] {step.name}: {dt:6.1f}s rc={rc}")
        if rc != 0:
            sys.exit(f"step {step.name!r} failed; recovery halted for investigation")

    total = (time.monotonic() - start) / 60
    print(f"total recovery: {total:.1f} min against RTO {args.rto_minutes:.0f} min")
    sys.exit(0 if total <= args.rto_minutes else 2)  # non-zero breaches the objective, fails CI

if __name__ == "__main__":
    main()

Rehearsing safely, and what the rehearsal actually proves

The rehearsal mode is the whole value, so be honest about what it does and does not measure. The echo stand-ins above prove the sequence still parses, the dependency ordering holds, and the harness runs — cheap checks that catch a surprising amount of drift when wired into CI on every change to the recovery repository. But the steps that carry real timing risk — restoring data and validating it — should point at genuine infrastructure. Restore the newest immutable backup into a throwaway, segregated staging target, run the smoke suite against it, and let the harness time the real restore. That is the number worth trusting, and it satisfies DORA’s segregation requirement precisely because the rehearsal never touches production.

Wire the exit code into the pipeline. A restore that succeeds but breaches the objective should fail the job the same way a broken build does, because a recovery that arrives after your RTO is a failed recovery. The timing history the pipeline accumulates becomes the evidence trail — much like automating a regulatory SLA as code produces the audit record as a by-product of doing the work. And the mean time to recovery you extract from those runs is a real, defensible operational metric rather than a figure someone once guessed in a policy review.

Where rehearsals break, which is the point

The first few honest rehearsals are uncomfortable, and that is exactly why you run them in advance. The backup you were relying on turns out never to have been verified, so it restores garbage. The clean image takes forty minutes to build, not the five you assumed, and it dominates your recovery time. The credential rotation locks out the very service you are trying to restore because a dependency order was wrong. Every one of those is a finding you want in a green-and-red CI log on a Tuesday, not in an incident bridge at two in the morning with the business offline and the clock running against a reporting deadline.

Codifying the runbook does not make the worst day pleasant. It makes it rehearsed. The difference between a plan and a capability is not the quality of the writing — it is the number of times you have actually run it, and the honesty of the clock while you did.