Why Did systemd OOM-Kill My Service (and How to Stop It)

Your service didn’t crash. There is no stack trace, no core dump, no application error in the log — it was there, and then it was gone. That signature almost always means one thing: the kernel out-of-memory killer took it, and it did so silently.

An OOM kill is a SIGKILL from the kernel. The process gets no chance to flush, log, or clean up, which is exactly why it looks like the service simply evaporated. The good news is that the kernel and systemd both record what happened, in two different places, and once you have read those records you can decide between three real fixes rather than guessing. This is the sequence I use on Ubuntu 24.04, which ships systemd 255 on a unified cgroups v2 hierarchy.

First: confirm it actually was the OOM killer

Do not assume. A service that disappears could have been killed by a failed health check, a bad deploy, or a segfault. systemd is explicit when the cause is memory, so start with the unit’s own journal.

# What did systemd think happened to the unit this boot?
journalctl -u myapp.service -b --no-pager | tail -n 40

# When a process is OOM-killed, systemd records it in three lines:
#   myapp.service: A process of this unit has been killed by the OOM killer.
#   myapp.service: Main process exited, code=killed, status=9/KILL
#   myapp.service: Failed with result 'oom-kill'.

The result string oom-kill is the confirmation. status=9/KILL alone is not enough — plenty of things send SIGKILL — but the explicit “killed by the OOM killer” line is systemd telling you the kernel did it. If you see that, you have your answer and can move on to the numbers.

Read the oom-kill line in dmesg — it tells you the real pressure

systemd tells you that it happened; the kernel ring buffer tells you why. This is the single most useful artefact and most people skip it.

# Timestamped kernel log around the kill.
dmesg -T | grep -iE 'oom-kill|out of memory|killed process' | tail -n 20

You are looking for two lines that read roughly like this:

oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=...,
  oom_memcg=/system.slice/myapp.service,
  task_memcg=/system.slice/myapp.service,task=python3,pid=2451,uid=997

Memory cgroup out of memory: Killed process 2451 (python3)
  total-vm:2941120kB, anon-rss:1988284kB, file-rss:12736kB,
  shmem-rss:0kB, UID:997 pgtables:4212kB oom_score_adj:0

Two fields decide the fix. The constraint tells you the scope of the shortage. CONSTRAINT_MEMCG means a cgroup hit its own limit — the failure was contained to that service. CONSTRAINT_NONE means the whole machine ran out of physical memory and the kernel went hunting for a victim across every process on the box, which is how one leaky service takes down an unrelated one. The anon-rss figure is the anonymous (non-reclaimable) resident memory the process actually held when it died — here nearly 2 GB. That is your real working-set number, not a guess, and it is what you size limits against.

Find out who actually eats the memory

Before you bound a service, confirm it is the one at fault. cgroups v2 accounts memory per unit, and systemd-cgtop reads that accounting live. On systemd 255 memory accounting is on by default, so this works without any extra configuration.

# Order the cgroup tree by memory; watch which slice climbs under load.
systemd-cgtop -m

# Point-in-time current usage for one unit.
systemctl show myapp.service -p MemoryCurrent

If the guilty unit climbs steadily and never plateaus, you have a leak and no limit will “fix” it — it will just change which number the process dies at. If it plateaus at a sensible ceiling and only occasionally spikes past available RAM, you have a sizing problem, which limits genuinely do solve.

The fix: contain the service, don’t destabilise the box

The core move is to give the unit its own memory ceiling so any future shortage is a CONSTRAINT_MEMCG event — a contained kill of one service — rather than a CONSTRAINT_NONE event that lets the kernel pick any process on the host. Two directives do the work, and they are not the same thing. MemoryHigh is a soft throttle: cross it and the kernel puts the cgroup under heavy reclaim pressure, slowing it down but not killing it. MemoryMax is the hard wall: reach it and the cgroup is OOM-killed. Set MemoryHigh below MemoryMax so the service is throttled and given a chance to shed memory before it hits the wall.

# /etc/systemd/system/myapp.service.d/memory.conf
[Service]
# Soft limit: sustained use above this triggers aggressive reclaim (throttle).
MemoryHigh=1500M
# Hard limit: the cgroup is OOM-killed if it reaches this. Leave a gap above High.
MemoryMax=1800M
# Bound swap so a leak cannot quietly consume the whole swap device.
MemorySwapMax=512M
# Bring it back automatically if it is killed anyway.
Restart=on-failure
RestartSec=2s

sudo systemctl daemon-reload
sudo systemctl restart myapp.service

# Verify the limits took effect.
systemctl show myapp.service \
  -p MemoryHigh -p MemoryMax -p MemorySwapMax -p MemoryCurrent

One directive worth checking is OOMPolicy=, which controls what systemd does after a process in the unit is OOM-killed. Its values are continue (leave the remaining processes running), stop (stop the whole unit cleanly), and kill (kill every process in the cgroup). The default is continue. For a multi-worker service that can be misleading — one worker dies, the unit stays “active”, and capacity silently drops. Confirm the current value with systemctl show myapp.service -p OOMPolicy and set it deliberately rather than inheriting it by accident.

Adding Restart=on-failure matters because a bounded, auto-restarting service pushes recovery from a manual page-out into an automatic one — the difference shows up directly in your mean time to recovery. It also changes your client contract: a restarted service will replay whatever was in flight, so any endpoint that mutates state needs to be safe to retry. If yours are not, fix that with idempotency keys before you lean on automatic restarts.

Pick the right fix, not the reflex one

The numbers you have now decide it. If anon-rss at death is close to a legitimate working set and the box is genuinely tight, add RAM or swap headroom and set limits that reflect reality. If the service plateaus normally but spikes on a specific operation, cap it with MemoryMax and handle the spike in code. If systemd-cgtop shows a line that only ever climbs, no limit is a fix — that is a leak, and the limit only decides how often you get paged. Setting explicit per-unit ceilings is also how you turn shared infrastructure into a set of bounded, accountable budgets, the same discipline that makes resource cost allocation possible at all.

The mistake is to raise the limit until the paging stops and call it solved. An unbounded service is not a service with plenty of headroom — it is a service that has been granted the right to take the whole machine down with it.