Spot GPUs Without Losing Work: Preemption-Safe Training and Batch Inference

Preemptible GPUs are the cheapest compute in the building, and the most dangerous place to run a job you cannot afford to restart.

Spot capacity — Google’s Spot VMs, AWS’s Spot Instances, and the equivalents on Azure and the specialist GPU clouds — sells you the same A100 or H100 as the on-demand pool at a fraction of the price, on one condition: the provider can take the machine back with almost no notice. For a lot of teams that condition reads as a dealbreaker, so they pay full price for training runs and overnight batch-inference jobs that would happily tolerate an interruption if the code were written to expect one. The saving is real — commonly 60 to 90 per cent off on-demand — but it only materialises if a killed job resumes instead of starting over. The engineering that makes that true is unglamorous and almost entirely about state.

The economics only work if the job survives being killed

A training run that loses six hours of progress every time a node is reclaimed is not cheaper on spot; it is more expensive, because you pay for the wasted GPU-hours and the wall-clock delay on top of them. The break-even is a function of two numbers: how often you get preempted, and how much work you lose each time. You do not control the first — preemption rates move with regional demand and are not something you get to negotiate — so the whole discipline reduces to making the second number small. If the most you ever lose is the work since your last checkpoint, and checkpoints are cheap and frequent, the eviction rate almost stops mattering.

What the preemption signal actually gives you

Every provider warns you before it pulls the node, and the length of that warning sets your entire design budget. On GKE, a Spot VM receives an ACPI G2 Soft Off signal and the default graceful shutdown window is 30 seconds — split, in practice, into roughly 15 seconds for your pods and 15 for system pods, with Google recommending that workloads terminate within about 25 seconds to avoid corrupting attached volumes. Recent GKE versions let you extend that to 120 seconds, but do not design as though you will always get it. On AWS, an EC2 Spot Instance gets a two-minute interruption notice, readable from the instance metadata service at /latest/meta-data/spot/instance-action and also delivered as an EventBridge event.

Two minutes sounds generous and 30 seconds sounds tight, but the honest conclusion from both is identical: this is not enough time to serialise and upload a large model from cold. A multi-billion-parameter checkpoint can run to tens of gigabytes, and you are not writing that to object storage in 25 seconds. So the signal does not mean “now save everything”. It means “flush whatever you have already been saving, and get out cleanly”.

Checkpoint as though you expect to be killed

The design that works treats checkpointing as a routine event on a timer, not a reaction to the shutdown signal. You write the model weights, the optimiser state, the current step or epoch, and — the part people forget — the data-loader position to object storage every N steps, where N is tuned so that a lost interval costs less than a single checkpoint. When the SIGTERM arrives, the handler does not start a fresh save from scratch; it sets a flag, lets the current step finish, writes one last checkpoint, and exits. Object storage matters here because it outlives the node: a bucket in GCS or S3 is still there when the job reschedules onto entirely different hardware, which local disk and node-attached volumes are not.

Two rules keep this safe. Write each checkpoint under a step-numbered key and treat the whole-object PUT as the commit — a half-written object must never be able to masquerade as a good checkpoint, so upload under a temporary name and rename, or lean on the fact that object stores only expose an object once the PUT completes. And keep the last few checkpoints, not just the latest, so that a corrupt or partial final write does not strand the run.

# --- GKE Spot node pool for GPU training (gcloud) ---
gcloud container node-pools create spot-a100 \
  --cluster training-cluster \
  --spot \
  --accelerator type=nvidia-tesla-a100,count=1 \
  --machine-type a2-highgpu-1g \
  --node-taints spot=true:NoSchedule \
  --enable-autoscaling --min-nodes 0 --max-nodes 8

# --- Job manifest: schedules onto spot, reschedules after eviction ---
apiVersion: batch/v1
kind: Job
metadata:
  name: train-resumable
spec:
  backoffLimit: 100                    # keep rescheduling after each eviction
  template:
    spec:
      terminationGracePeriodSeconds: 25  # stay inside the ~30s spot window
      restartPolicy: Never
      tolerations:
        - key: spot
          operator: Equal
          value: "true"
          effect: NoSchedule
      containers:
        - name: trainer
          image: registry.example.com/trainer:1.4
          command: ["python", "train.py",
                    "--resume-from", "gs://ckpt-bucket/run-42"]

# --- train.py: checkpoint on a timer, flush cleanly on preemption ---
import os, signal, sys, torch, gcsfs

STOP = False
def on_preempt(signum, frame):
    global STOP
    STOP = True                        # only set a flag; never save in the handler
signal.signal(signal.SIGTERM, on_preempt)

fs = gcsfs.GCSFileSystem()
RUN = "gs://ckpt-bucket/run-42"
CKPT_EVERY = 200                        # tune so a lost interval costs less than a save

def save(state, step):
    tmp = f"/tmp/step-{step}.pt"
    torch.save(state, tmp)
    # whole-object PUT: the checkpoint only becomes visible once the upload completes
    fs.put(tmp, f"{RUN}/step-{step}.pt")

for step, batch in enumerate(loader, start=resume_step):
    train_step(batch)
    if STOP or step % CKPT_EVERY == 0:
        save({"model":   model.state_dict(),
              "optim":   opt.state_dict(),
              "step":    step,
              "sampler": loader.sampler.state_dict()},  # data-loader position matters
             step)
        if STOP:
            sys.exit(0)                 # exit clean before the node is reclaimed

Resumption is where most of the bugs live

Saving is the easy half. The failure mode that wastes real money is a job that restarts, loads its weights, and then silently redoes work or skips data. Getting resumption right means three things. The data loader has to resume from where it stopped, not from the top of the epoch — persist the sampler or shuffle state, or you will quietly retrain on some examples and never present others. Every step that touches the outside world — writing a row, calling an API, emitting an inference result — has to be idempotent, so that replaying a step after a mid-flight kill produces the same outcome rather than a duplicate. This is the same discipline that makes a POST safe to retry, and it is worth borrowing that thinking wholesale. And partial batches have to be handled explicitly: a batch-inference job that dies halfway through a shard must know, on resume, which records are already written and which are still outstanding — which usually means keying outputs by input ID and skipping what already exists, rather than trusting a running counter.

Where this sits in the cost conversation

For a regulated firm the attraction is obvious — GPU spend is one of the fastest-growing lines in the technology budget, and spot capacity is the single largest lever on it — but the governance point is worth stating plainly. Spot is appropriate for interruptible, restartable work: training, fine-tuning, offline batch inference, evaluation sweeps. It is not appropriate for latency-sensitive online serving, where an eviction is simply an outage. The mistake I see is teams either avoiding spot entirely out of caution, or reaching for it on workloads that cannot tolerate interruption, when the real answer is to sort workloads by whether they can be checkpointed and route each to the right pool. That sorting — and the chargeback that makes each team feel the cost of its own GPU-hours — is where the savings become durable rather than a one-off. It belongs in the same FinOps discipline for AI and GPU spend as everything else you run on a GPU.

The cheap GPU and the expensive GPU are the same silicon. The only difference is whether your job was written to expect the interruption — and that is a property of your code, not your cloud bill.