This Fixes: CrashLoopBackOff and OOMKilled Pods

CrashLoopBackOff and OOMKilled are the two failures every on-call engineer sees most, and both are symptoms rather than causes. The status tells you the pod keeps dying and being restarted; it tells you almost nothing about why. This is how you get to the cause quickly, without guessing.

The mistake I see most often is treating CrashLoopBackOff as a diagnosis. It is not. It is Kubernetes reporting that a container has exited, been restarted, exited again, and is now being held back before the next attempt. The restart delay starts at ten seconds, doubles each time, and caps at five minutes. That backoff is the kubelet protecting the node, not a clue about the fault. The fault is one layer down, in why the container exited at all.

Read the events before you touch anything

The first command is always kubectl describe pod, and the part that matters is the Events block at the bottom and the container’s Last State. Do not start editing manifests or bouncing the deployment. Read what the cluster is already telling you.

# 1. See why the pod is unhappy - the Events and Last State matter most
kubectl describe pod payments-api-7d9f8-abcde

# 2. Pull the container's last terminated state as structured JSON
kubectl get pod payments-api-7d9f8-abcde 
  -o jsonpath='{.status.containerStatuses[0].lastState.terminated}'

# 3. Read logs from the instance that DIED, not the one now restarting
kubectl logs payments-api-7d9f8-abcde --previous

# 4. Confirm the memory ceiling the container was actually given
kubectl get pod payments-api-7d9f8-abcde 
  -o jsonpath='{.spec.containers[0].resources.limits.memory}'

Those four commands answer the two questions that decide everything else: what killed the container, and what was it working with when it died. The lastState.terminated object is the single most useful field on the whole pod. It carries the exit code, the reason, and the timestamps. Everything below is about reading it correctly.

Tell OOMKilled apart from an application crash

Here is the distinction that saves the most time. An exitCode of 137 means the process received SIGKILL: 128 plus signal 9. The kernel’s out-of-memory killer uses SIGKILL, which is why 137 and OOM are associated in everyone’s memory. But 137 on its own is not proof of an out-of-memory event. A liveness probe that fails past its grace period, or any external SIGKILL, produces the same code.

The field that actually confirms it is reason. If lastState.terminated shows reason: OOMKilled alongside exitCode: 137, the container breached its memory limit and the kernel killed it. If you see exitCode: 137 with no OOMKilled reason, something else sent the kill signal, and you should be looking at your liveness probe, not your memory limit. If the exit code is 1, 2, or an application-specific number, it is a plain crash and the answer is in the logs, not the resource block.

So: exit code plus reason, read together, tell you which of three worlds you are in. Out of memory. Killed by a probe. Or crashed on its own. Each has a different fix and there is no point applying one to the wrong world.

The previous instance holds the evidence

When a pod is looping, kubectl logs without arguments gives you the container that is currently starting, which by definition has not failed yet and usually prints almost nothing. The evidence is in the instance that already died. That is what --previous (short flag -p) retrieves: the logs of the last terminated container, including the stack trace, the failed database connection, or the config key it could not find.

For a crash on startup, the previous logs almost always contain the answer on the last few lines. A missing environment variable, an unreachable dependency at boot, a migration that failed, a malformed config file. This is also where an init-container failure shows up. If a pod is stuck in Init:CrashLoopBackOff, the failing container is the init container, and you have to name it explicitly to see its logs, because the main container has not run at all yet.

# A pod stuck in Init:CrashLoopBackOff - target the init container by name
kubectl logs payments-api-7d9f8-abcde -c run-migrations --previous

Now fix the actual cause

Once you know which world you are in, the fix is specific.

OOMKilled. The container needs more memory than its limit allows, or it has a leak. Raise the limit if the working set is genuinely larger than you assumed; do not raise it blindly if the usage climbs without bound, because that is a leak and a higher ceiling only delays the next kill.

resources:
  requests:
    memory: "512Mi"   # scheduled against this
  limits:
    memory: "1Gi"     # OOMKilled above this

Killed by a probe. The liveness probe is firing before the application is ready to answer it, so Kubernetes restarts a container that was merely slow to start. Give it an initialDelaySeconds that matches real startup, or move slow-start work behind a startupProbe so the liveness check does not begin until the app is actually up.

Plain crash. Fix what the previous logs point at. A bad config value, a missing secret, a dependency that is not reachable from inside the cluster. This is the category most likely to have been caught before it ever shipped: a pre-commit baseline that validates manifests and config catches a class of these at the point of change rather than at three in the morning.

Why “it works locally” hides the memory ceiling

The reason OOMKilled surprises teams is that a laptop has no memory limit worth mentioning. The container runs against the machine’s whole allocation, never approaches a ceiling, and looks healthy. In the cluster it is boxed into whatever limits.memory the manifest set, and the JVM heap, the Node heap, or the language runtime’s default arena sizing does not know about that box. The process sizes itself to the node, hits the pod limit, and is killed. The application did not change; the ceiling appeared.

This is why measuring restart rate and time-to-recovery matters more than treating each incident as a one-off. If you are already instrumenting MTTR and change-failure rate, a spike in pod restarts after a deploy is a signal you can see before a user does. And where a crash interrupts work mid-request, whether that request is safe to replay is a separate discipline worth getting right in advance rather than during the incident, which is the case for idempotent, retry-safe endpoints.

The rule to carry out of this: never fix a looping pod from the status field. Read the exit code and the reason, pull the previous instance’s logs, and act on what killed it. The backoff timer is buying you the time to do exactly that.