Writing a Production Cutover Runbook That Survives the Night

A cutover almost never fails on the migration itself. It fails on the step nobody wrote down: who was going to re-enable the writers, what the rollback command actually was, whether anyone checked the new database was taking traffic before the DNS change went out.

Moving a live system onto new infrastructure at two in the morning is not a technical problem so much as a coordination problem. The technical part you have usually solved in staging. What breaks on the night is the choreography: an owner who went to bed, a check that was assumed rather than run, a point of no return that was crossed without anyone deciding to cross it. A cutover runbook exists to remove that improvisation. This is how I write one that survives contact with a tired team and a ticking clock.

A runbook is a script, not a checklist

A checklist is a list of things to do. A runbook is a timed sequence of things to do, each with a single named owner and an explicit expected result. The difference matters at 2 a.m. Under pressure, a checklist invites the question "has someone done this?" and the answer is often a confident "yes" that turns out to mean "I assumed so." A runbook with a T-time and an owner column removes the ambiguity: at T-30 the application owner runs the pre-cutover smoke test, and either the box is ticked with a PASS or the cutover does not move forward.

Three properties make the difference between a runbook that holds and one that falls apart. It is timed — every step is anchored to T-minus or T-plus a reference moment, not to "after the previous thing." It is owned — one person per row, with a named backup, reachable for the whole window. And it is reversible — there is a rehearsed path back from every step up to a single, explicit point of no return.

The five sections every cutover runbook needs

Strip a good runbook down and the same five sections are always there:

• The freeze. Deploys off, schema changes off, feature-flag changes off. State it as a step with a time and an owner, because an unannounced freeze is not a freeze.
• Pre-cutover checks. A backup taken and its restore point recorded, and a smoke test run against the old stack so you have a known-good baseline to compare against.
• The point-of-no-return gate. An explicit GO / NO-GO with written criteria, discussed below.
• Verification. The same smoke test run against the new stack, plus queries that prove data is actually landing.
• Rollback. A numbered, rehearsed path back, with its own trigger conditions so nobody has to argue about whether to pull it.

Here is a template I hand teams as a starting point. It is deliberately plain Markdown so it lives in the repository next to the code, gets reviewed in a pull request, and can be diffed against the last cutover rather than reinvented each time. That review discipline is the same one worth applying to your everyday CI baseline — the runbook is code, so treat it like code.

# Cutover Runbook — <system> to <new-infra>
Date: 2026-__-__   Window: 22:00–02:00   Cutover lead: <name>

## Roles (one owner per row, reachable for the whole window)
| Role           | Owner | Backup | Contact |
|----------------|-------|--------|---------|
| Cutover lead   |       |        | phone   |
| Database       |       |        | phone   |
| Application    |       |        | phone   |
| Comms / status |       |        | phone   |

## Timeline (every step has a time, an owner and a tick box)
| T-time | Step                                   | Owner | Done | Notes         |
|--------|----------------------------------------|-------|------|---------------|
| T-60   | PagerDuty maintenance window opened    | Comms |      |               |
| T-45   | Announce freeze; deploys disabled      | Lead  |      |               |
| T-30   | Pre-cutover smoke test on OLD stack    | App   |      | baseline PASS |
| T-20   | Final backup + record restore point    | DB    |      | note LSN/GTID |
| T-10   | Scale down writers; drain queues       | App   |      |               |
| T-0    | GO / NO-GO gate (criteria below)       | Lead  |      |               |
| T+5    | Repoint DNS / flip feature flag        | App   |      |               |
| T+15   | Smoke test on NEW stack                | App   |      | must PASS     |
| T+30   | Ramp traffic 10% -> 100%            | App   |      |               |
| T+60   | Close window; announce complete        | Comms |      |               |

## GO / NO-GO criteria (ALL must be true to pass T-0)
- Final backup completed and restore point recorded
- Pre-cutover smoke test on OLD stack: PASS
- New stack health endpoint returns 200
- Every role present and acknowledged in the channel
- Rollback owner confirms the rollback path is ready

## Rollback (rehearsed, not hoped for)
Trigger: any smoke failure at T+15, or error rate > 2% sustained for 5 minutes.
1. Flip feature flag / DNS back to OLD stack (owner: App)
2. Re-enable writers on OLD stack (owner: DB)
3. Confirm OLD-stack smoke test PASS
4. Announce rollback; keep the window open for the post-mortem

The point-of-no-return gate

Most bad cutover nights share one feature: nobody can say exactly when the system became unrecoverable. Writers were scaled down, the new database took a few writes, and by the time the smoke test failed there was no clean way back because the old and new stacks had both taken live data. The fix is to name the point of no return and put a decision in front of it.

The GO / NO-GO gate at T-0 is that decision. Before it, everything is reversible and cheap: you can abort, re-open deploys, and go home. After it, you are committed to finishing forward. The criteria are written down in advance so the call is mechanical, not political — a final backup with a recorded restore point, a passing baseline smoke test, a healthy new stack, every owner present, and a rollback owner who confirms the path back is ready. If any line is false, it is a NO-GO, and a NO-GO at T-0 is a successful outcome, not a failure. The failures are the ones where somebody waved the gate through because the change window was closing.

Design the sequence so that the irreversible steps happen after the gate and are as few as possible. If a POST can be replayed safely because the endpoint is idempotent, a mid-cutover retry is a non-event rather than a duplicate charge — which is one more reason making writes safe to retry pays off long before cutover night.

Verification that proves the cutover worked

"It looks up" is not verification. The new stack can return a 200 on its health endpoint while its database is read-only, its auth chain is broken, or writes are silently failing. Verification means running the same scripted smoke test before and after, so a PASS on the old stack and a PASS on the new stack mean the same thing. Keep it small, deterministic and fast — it will be run under stress, possibly several times.

#!/usr/bin/env bash
# smoke.sh — prove a stack is actually serving before and after cutover.
# Usage: BASE_URL=... DB_URL=... SMOKE_TOKEN=... ./smoke.sh
set -euo pipefail

BASE_URL="${BASE_URL:?set BASE_URL}"
DB_URL="${DB_URL:?set DB_URL}"
SMOKE_TOKEN="${SMOKE_TOKEN:?set SMOKE_TOKEN}"
MAX_TIME=5

fail() { echo "SMOKE FAIL: $*" >&2; exit 1; }

# 1. Health endpoint returns 200 within MAX_TIME seconds.
code=$(curl -sS -o /dev/null -w '%{http_code}' --max-time "$MAX_TIME" \
  "$BASE_URL/healthz") || fail "health endpoint unreachable"
[ "$code" = "200" ] || fail "health returned $code"

# 2. An authenticated read path round-trips. --fail (-f) makes curl
#    exit 22 on any 4xx/5xx, so a broken auth chain fails the script.
curl -fsS --max-time "$MAX_TIME" \
  -H "Authorization: Bearer $SMOKE_TOKEN" \
  "$BASE_URL/api/v1/account/me" >/dev/null \
  || fail "authenticated read failed"

# 3. The database is reachable and recent writes are landing.
#    -X ignores ~/.psqlrc, -tA gives one bare value, ON_ERROR_STOP=1
#    turns any SQL error into a non-zero exit.
lag=$(psql "$DB_URL" -X -tA -v ON_ERROR_STOP=1 \
  -c "SELECT extract(epoch FROM now() - max(created_at))::int FROM orders;") \
  || fail "freshness query failed"
[ "$lag" -lt 900 ] || fail "newest order is ${lag}s old, expected under 900"

echo "SMOKE PASS"

Three checks, each proving a different failure mode: the service answers, an authenticated read path round-trips end to end, and recent writes are actually landing in the database. The freshness query is the one people skip and the one that catches a replica that stopped replicating. Bind these smoke results to the same signals you already watch in production — a spike in change-failure rate or a jump in time-to-restore is exactly what the DORA delivery metrics are there to surface, and a cutover is the highest-stakes change you will make all quarter.

Rehearse it against a staging clone

A runbook you have never executed is a hypothesis. The single highest-value thing you can do before the real night is a dry run against a staging environment restored from a production-shaped snapshot — same data volume, same rough topology. You are not testing whether the migration works; you are testing whether the runbook does. You will find that a command has the wrong flag, that a step assumed a credential nobody has, that the rollback takes eleven minutes when you budgeted three, that two steps you wrote as sequential can safely run in parallel and buy you slack.

Time the dry run. The number that matters is not "does it work" but "how long does the irreversible section take, and is the rollback genuinely faster than fixing forward." If rollback is slower than repair, your gate criteria are wrong and you should know that in a rehearsal, not at T+20 with real customers timing out.

The runbook that survives the night is boring by design: every step timed, every step owned, one honest gate, and a rollback you have actually run. If your next cutover plan is a wiki page written the afternoon before, you do not have a runbook — you have a wish, and wishes do not tick their own boxes at 2 a.m.