A Key Rotation Runbook You Can Actually Run

Key rotation breaks production because most teams treat it as a single flip: revoke the old credential, issue the new one, and pray the deploy lands before the next request. It never lands cleanly. The fix is to stop flipping and start overlapping.

The reason rotation gets deferred for eighteen months at a time is that everyone remembers the outage. Someone rotated a database password on a Friday, a connection pool three services away was still holding the old one, and the on-call engineer spent the evening working out which of forty containers had cached what. So the credential ages, the auditor flags it, and the cycle repeats. The way out is a runbook that never has a moment where exactly one credential is valid. You issue the new one, you run both at once, you confirm the new one is actually carrying traffic, and only then do you kill the old one. No maintenance window, no held breath.

The overlap principle

Every safe rotation has four states, in order, and the third is the one people skip:

1. Issue. Create the new credential. The old one is still live and still serving.
2. Deploy alongside. Roll the new credential out so both are accepted at once. Nothing has been taken away yet.
3. Verify carriage. Prove from logs that live traffic is now using the new credential, not just that the new credential works.
4. Revoke. Deactivate the old credential, watch for fallout, then delete it.

The property that makes this safe is that the two credentials are valid simultaneously for a defined grace window. On AWS IAM this is a first-class feature: an IAM user can hold two active access keys at once, which is the entire mechanism the built-in rotation guidance relies on. The application-level equivalent is that your service has to accept two valid keys for the same principal during the window. If your auth layer can only ever hold one key per caller, that is the thing to fix before you rotate anything.

Rotating an AWS IAM access key

Here is the overlap runbook for the most common case, an IAM access key used by a service or a CI runner. It creates the second key, and only deactivates and deletes the first once the last-used data confirms the old key has gone quiet.

#!/usr/bin/env bash
# Overlap rotation for an IAM user's access key. British spelling throughout.
set -euo pipefail

USER="svc-payments-worker"
OLD_KEY="AKIAOLDKEYEXAMPLE"

# Step 1 - Issue. IAM allows a maximum of two active keys per user,
# so this succeeds only if the user currently holds one.
aws iam create-access-key --user-name "$USER" > new-key.json
echo "New key issued. Distribute it to the secret store, then redeploy."

# Step 2 happens outside this script: push new-key.json into your secret
# manager and redeploy so both keys are live. Then come back and run step 3.

# Step 3 - Verify carriage before you touch the old key.
aws iam get-access-key-last-used --access-key-id "$OLD_KEY" \
  --query 'AccessKeyLastUsed.LastUsedDate'
# If that timestamp has stopped advancing, the old key is idle.

# Step 4 - Deactivate first (reversible), watch, then delete (final).
aws iam update-access-key --user-name "$USER" \
  --access-key-id "$OLD_KEY" --status Inactive
echo "Old key deactivated. If anything breaks, re-activate with --status Active."
# After a clean grace window:
# aws iam delete-access-key --user-name "$USER" --access-key-id "$OLD_KEY"

The discipline that matters is deactivate-then-delete. update-access-key --status Inactive is reversible; if you missed a consumer, you flip it back to Active and nothing is lost. delete-access-key is not reversible. Never delete in the same step as deactivate. That single habit turns a potential incident into a shrug.

Dual-credential support in your own config

For a key your own service issues to callers, the overlap has to live in configuration. The pattern is a primary and a secondary slot, both checked on every request. Constant-time comparison, so you are not leaking timing information about which key matched.

# config: both keys valid during the grace window
API_KEY_PRIMARY="pk_live_9f2c..."    # the newly issued key
API_KEY_SECONDARY="pk_live_4a71..."  # the outgoing key, still accepted

# verification (Python) - accept either, compare in constant time
import hmac, os

def key_is_valid(presented: str) -> bool:
    for env in ("API_KEY_PRIMARY", "API_KEY_SECONDARY"):
        expected = os.environ.get(env)
        if expected and hmac.compare_digest(presented, expected):
            return True
    return False

Rotation then becomes: write the new key into the secondary slot, deploy, promote it to primary, deploy again, blank the secondary. At no point is there one valid key. This same two-slot idea covers the other credential classes. A database password rotates the same way if your framework lets each instance read its connection string from the secret store on start; AWS Secrets Manager formalises it with staging labels, moving a new value through AWSPENDING to AWSCURRENT while the previous value keeps the AWSPREVIOUS label for a window. Signing keys, for JWTs or webhooks, overlap by publishing both public keys in the JWKS and switching which private key signs, so verifiers accept both until the old one is retired.

Logging the rotation for audit

A rotation nobody can prove happened is, to a supervisor or a customer’s security team, a rotation that did not happen. Every step above emits an event you should be capturing. On AWS, CreateAccessKey, UpdateAccessKey and DeleteAccessKey are all recorded in CloudTrail with the caller identity and timestamp, which is exactly the evidence trail an auditor wants to see. For keys your own service issues, log the issue, the promotion and the revoke as structured events with the key identifier prefix, never the secret itself, and the operator who ran it. This is the same instinct behind a proper logging architecture you can actually audit: capture the action, the actor and the time in a form you can query later, because you will be asked. When a key customer asks to audit your technology, “we rotate quarterly” is a claim; the CloudTrail export and the rotation log are the proof.

The quarterly checklist

Run this every quarter, per credential, with no maintenance window:

1. Confirm the consumer accepts two credentials at once. If not, stop and fix that first.
2. Issue the new credential into the secondary slot and deploy.
3. Verify from logs or last-used data that live traffic is on the new credential.
4. Promote new to primary; deactivate the old (do not delete).
5. Watch for the grace window; delete the old credential and record the event.

Rotation only stays cheap if it is boring and routine rather than a rare, frightening event. The overlap is what makes it boring, and boring is what makes it get done. If your rotations are still one-shot flips, that is the real finding, not the age of the key itself, and it is the loop worth closing before the next audit lands. It sits alongside the wider discipline of secret scanning and rotation that closes the loop and treating API keys as something other than a liability in the first place.