Docker

Docker Container Memory Limit Reached, Killed by OOMKiller

A container killed by the OOMKiller (Out-Of-Memory Killer) means it tried to use more memory than its configured limit allows, and the Linux kernel forcibly terminated it rather than letting it consume unbounded memory and potentially destabilize the whole host.

The Problem

A container stops unexpectedly, and checking its exit status reveals it was killed rather than exiting normally:

docker inspect your-container --format='{{.State.OOMKilled}}'
true
docker ps -a
STATUS: Exited (137) About a minute ago

Exit code 137 specifically corresponds to the container being killed by signal 9 (SIGKILL), which is exactly what the kernel's OOMKiller sends when forcibly terminating a process for excessive memory use.

Why It Happens

When a container's memory usage exceeds its configured limit (set explicitly via --memory or a Compose mem_limit, or implicitly via the host's total available memory if no limit is set), the kernel intervenes to prevent that container from consuming resources needed by the rest of the system. Common underlying causes:

  • The configured memory limit is genuinely too low for the application's actual needs, particularly common with JVM-based applications, which need explicit heap size configuration that accounts for the container's memory limit rather than assuming access to the full host's memory
  • A genuine memory leak in the application, where memory usage grows steadily over time rather than stabilizing, eventually exceeding any reasonable limit no matter how generous
  • A sudden spike in memory usage from processing an unusually large request or dataset, exceeding a limit that's normally sufficient for typical workloads
  • Multiple processes running inside the same container (unusual, but happens with certain base images or supervisor-based setups) collectively exceeding a limit that was sized assuming only a single process

The Fix

First, confirm the container's configured limit and compare it against actual observed usage before the kill, using docker stats during normal operation to understand typical memory consumption:

docker stats your-container

If the application genuinely needs more memory than currently allocated, and the host has capacity to spare, raise the limit:

docker run --memory=1g --memory-swap=1g your-image

In Compose:

services:
  app:
    deploy:
      resources:
        limits:
          memory: 1g

For JVM-based applications specifically, explicitly configure heap size to fit comfortably within the container's memory limit, since the JVM's default heap sizing heuristics historically didn't account for container memory limits correctly (though this has improved significantly in recent JVM versions with container-awareness enabled by default):

java -XX:MaxRAMPercentage=75.0 -jar app.jar

Setting max heap to a percentage of available container memory (rather than a fixed value) ensures the JVM leaves headroom for non-heap memory usage (thread stacks, native memory, metaspace) within the same container limit, reducing the chance of hitting exit code 137 from memory the JVM itself didn't account for.

If you suspect a genuine memory leak rather than a legitimately high baseline need, monitor memory usage over an extended period rather than a single snapshot, looking specifically for steady, unbounded growth rather than usage that rises and then stabilizes:

while true; do docker stats --no-stream your-container; sleep 60; done

Steadily climbing memory with no plateau, especially correlated with specific operations (processing more requests, running longer), points toward an application-level leak that needs to be found and fixed in code, rather than solved by simply raising the container's memory limit repeatedly.

Still Not Working?

If the container is being killed even though docker stats shows memory usage staying comfortably under the configured limit right up until the kill, check whether the host itself is under memory pressure independently of this specific container's limit — the OOMKiller can act at the host level too, and a host running low on overall memory can kill processes based on the kernel's own scoring system, not strictly tied to any individual container's configured --memory limit. Checking host-level memory pressure directly clarifies whether the fix needs to happen at the container level or at the broader host/cluster resource allocation level instead:

free -h
dmesg | grep -i "killed process"

You might also like