cgroup Memory Limits
How memory.max, memory.high and memory.events work in cgroup v2 (and the v1 equivalents), and why a container can be OOM-killed while the host looks fine.
The problem
A container keeps getting killed at 512 MB while the node has 40 GB free, or `docker stats` and `free` disagree, or a service under `MemoryMax=` slows to a crawl without dying. The limit lives in a cgroup, not in the host's global counters, so you have to read the cgroup's own files to see how close it is and what it is spending memory on.
How it works
- Find out which version you have: `stat -fc %T /sys/fs/cgroup` prints `cgroup2fs` on a v2 (unified) host and `tmpfs` on v1 or hybrid hosts. The cgroup of a process is in `/proc/<pid>/cgroup`; on v2 it is the path after `0::`, appended to `/sys/fs/cgroup`. Container runtimes create one cgroup per container.
- cgroup v2 limits: `memory.max` is the hard limit; when usage reaches it the kernel reclaims inside the cgroup first and only then OOM-kills a task in it. `memory.high` is a throttle: above it the cgroup's allocations are slowed and reclaimed hard, but it is never OOM-killed for crossing it. `memory.min` and `memory.low` protect memory from reclaim. `memory.swap.max` limits swap separately.
- What `memory.current` includes: anonymous memory, page cache (`file`), kernel memory such as slab and socket buffers, and `shmem`/tmpfs. A cache-heavy cgroup sitting at 100% of its limit is often fine because clean cache is reclaimable; dirty pages, tmpfs data and anonymous memory are not. Break it down with `memory.stat` (`anon`, `file`, `file_dirty`, `shmem`, `slab`, `pgmajfault`).
- `memory.events` holds counters: `low`, `high` (times throttled), `max` (times the limit was hit), `oom` (limit reached and an allocation was about to fail), `oom_kill` (tasks killed). Read it as a story: rising `high` or `max` with no `oom_kill` means the workload is paying with latency at its limit (reclaim, major faults); rising `oom_kill` means tasks died. `memory.events.local` excludes children, and `memory.peak` (newer kernels) shows the high-water mark.
- Why the host looks fine: the OOM is local to the cgroup, so global `MemFree` and `MemAvailable` stay high, and `free` inside the container reports host totals. The evidence is the kernel line `Memory cgroup out of memory: Killed process ...` with `constraint=CONSTRAINT_MEMCG` and `oom_memcg=` naming the cgroup, plus the cgroup's own `memory.events`.
- cgroup v1 equivalents: `memory.limit_in_bytes` (hard limit), `memory.usage_in_bytes` (includes cache), `memory.max_usage_in_bytes`, `memory.failcnt` (times the limit was hit), `memory.soft_limit_in_bytes`, `memory.stat` with `total_` prefixed hierarchical counters, `memory.oom_control` with an `oom_kill` count, and `memory.memsw.*` for memory plus swap combined. All of these were observed on this (v1 memory controller) host.
- Runtimes map their flags onto these files: Docker `--memory` and Kubernetes `limits.memory` become `memory.max` (or the v1 limit). Runtimes that read the cgroup size themselves behave better: modern JVMs derive their heap from the cgroup limit (`-XX:MaxRAMPercentage`), and Go supports `GOMEMLIMIT`; older runtimes that read `/proc/meminfo` size themselves to the host and get killed.
- Sizing rule of thumb: the limit must cover the application's peak anonymous memory plus runtime overhead (thread stacks, metaspace or native memory, allocator arenas) plus a margin for page cache and kernel memory. A limit equal to the heap size is a common cause of repeated OOM kills.
Examples
cgroup v2: locate a process's cgroup and read limit, usage, events and the biggest consumers.
stat -fc %T /sys/fs/cgroup
cg=$(sed -n 's/^0:://p' "/proc/<pid>/cgroup")
d="/sys/fs/cgroup$cg"
cat "$d/memory.max" "$d/memory.high" "$d/memory.current"
cat "$d/memory.events"
# oom_kill 3 means three tasks were killed inside this cgroup
grep -E '^(anon|file|file_dirty|shmem|slab|pgmajfault) ' "$d/memory.stat"cgroup v1 (memory controller) equivalents; failcnt counts times the limit was hit.
d="/sys/fs/cgroup/memory<cgroup-path>"
cat "$d/memory.limit_in_bytes" "$d/memory.usage_in_bytes" "$d/memory.max_usage_in_bytes"
cat "$d/memory.failcnt"
grep -E '^(total_rss|total_cache|total_dirty|total_shmem) ' "$d/memory.stat"
grep oom_kill "$d/memory.oom_control"Memory pressure of one cgroup (v2, needs PSI); some/full above zero means tasks are stalled on memory.
cat "$d/memory.pressure"
# some avg10=0.00 avg60=0.00 avg300=0.00 total=0
# full avg10=0.00 avg60=0.00 avg300=0.00 total=0In practice
- Linux kernel docs (admin-guide/cgroup-v2, admin-guide/cgroup-v1/memory) — The reference for memory.max, memory.high, memory.min/low, memory.events, memory.stat and the v1 files.
- systemd resource control — MemoryMax=, MemoryHigh= and MemorySwapMax= in a unit or drop-in map directly to the cgroup v2 files; systemd-cgtop shows live per-cgroup usage.
- Docker and Kubernetes — docker run --memory and Kubernetes limits.memory set the cgroup limit; pod-level triage of OOMKilled is covered on the Kubernetes sheet.
- In practice — Monitoring tools often chart a 'working set' (usage minus inactive file cache) rather than raw usage so that reclaimable cache does not trigger alerts.
Trade-offs
Pros
- A per-workload limit isolates a leaking or greedy service from its neighbours and from the host.
- `memory.high` gives a soft ceiling that applies back-pressure instead of killing.
Cons
- Cache, kernel memory and tmpfs count against the limit, so the number you must size for is larger than the heap.
- Hitting `memory.max` after reclaim fails is abrupt, and pressure just below it shows up as latency rather than an error.
- cgroup v1 and v2 expose different files and semantics, so scripts must handle both.