Docker Cheatsheet

How images and containers actually work, and how to run them without surprises: layers and overlayfs, namespaces and cgroups, Dockerfile and BuildKit caching, networking and storage, Compose, hardening, and step-by-step playbooks for containers that exit at once, exit codes 137 and 143, full disks and unreachable ports. Each entry gives the problem, how it works, commands and files verified on a real engine, real-world notes and trade-offs.

Updated: 2026-10-09

29 entries

Quick tables

Reference and comparison tables for quick scanning.

Docker CLI by task

The commands that answer a specific question about a container or image, grouped by what you are trying to find out. Notes record behaviour checked on Docker Engine 29.8 with the containerd image store on cgroup v1.

Docker CLI by task
AreaTaskCommandNotes
FindList containers, including stopped ones`docker ps -a --filter status=exited`Valid `status` values: created, running, paused, restarting, removing, exited, dead. Add `--format '{{.Names}} {{.Status}}'` for scripts and `-s` for sizes.
FindRead one field from the full config`docker inspect -f '{{.State.Status}}' <c>`Go template over the JSON; `{{json .State}}` prints a subtree. Top-level `.NetworkSettings.IPAddress` is absent on Engine 29.8 (deprecated since API 1.21; observed: `map has no entry for key IPAddress`), so use `.NetworkSettings.Networks.<net>.IPAddress`.
FindShow published ports`docker port <c>`Prints `80/tcp -> 127.0.0.1:18081`. Empty output means nothing is published, even if the image has `EXPOSE`.
ObserveFollow logs with timestamps`docker logs -f --tail 100 -t <c>`Reads what the logging driver kept; `--since 10m` limits by time. Works after the container exits until it is removed.
ObserveLive CPU, memory, PIDs`docker stats --no-stream`Without `--no-stream` it refreshes until Ctrl-C. Values come from the container's cgroup, not from `/proc` inside it.
ObserveList processes of a container`docker top <c> -o pid,args`Shows host PIDs, not the PIDs the container sees; ps options go after the container name.
ObserveWatch lifecycle events`docker events --since 10m --filter container=<c>`Actions include create, start, kill, die, stop, destroy. `--until` makes it return; `--format '{{.Action}}'` for scripts.
ActRun a command in a running container`docker exec -it <c> sh``-it` fails with `cannot attach stdin to a TTY-enabled container because stdin is not a terminal` when run without a terminal. Add `-u` and `-w` as needed.
ActStop with a custom grace period`docker stop -t 30 <c>`SIGTERM, then SIGKILL after the timeout (default 10 s, measured 10.17 s). Exit code 137 means the kill was needed.
ActBlock until exit and get the code`docker wait <c>`Prints the exit code (4 for `exit 4`) after the container stops.
FilesCopy files in or out`docker cp <c>:/path ./dest`Works on stopped containers and in images without a shell. Directions: `docker cp ./f <c>:/tmp/f` to copy in.
FilesSee what the container changed`docker diff <c>``A` added, `C` changed, `D` deleted relative to the image. Volumes and bind mounts never show up.
FilesExport the flat filesystem`docker export <c> | tar t`Writes the merged filesystem as one tar with no layers or image config; includes `.dockerenv`.
ImageShow how an image was built`docker image history --no-trunc <img>`Lists each step with its size. `<missing>` for lower layers is normal with the containerd image store.
ImageArchive and restore an image`docker save -o img.tar <img>` and `docker load -i img.tar`Engine 29 writes an OCI layout (`index.json`, `oci-layout`, `manifest.json`, `blobs/sha256/`). Tags are kept.
DiskWhere is the space going`docker system df`Columns TYPE, TOTAL, ACTIVE, SIZE, RECLAIMABLE for images, containers, local volumes and build cache; `-v` lists each object.
DiskWritable layer and image size of a container`docker inspect --size -f '{{.SizeRw}} {{.SizeRootFs}}' <c>``SizeRw` is the upper layer only (4096 for a nearly idle container); `docker ps -s` shows the same as `size (virtual ...)`.
DaemonChoose which daemon the CLI talks to`docker context ls` and `docker --context <name> ps``default` uses `DOCKER_HOST` or `unix:///var/run/docker.sock`. Remote contexts over SSH are created with `docker context create <name> --docker host=ssh://user@host` (not run here).

Most read-only commands (`ps`, `inspect`, `logs`, `top`, `diff`, `events`) only talk to dockerd and are safe to run on a production container. `exec`, `cp` into a container, `stop`, `kill` and `rm` change state.

Dockerfile instructions compared

Observed on BuildKit with Docker Engine 29. 'Layer' means a filesystem layer with a size in `docker history`; metadata instructions show 0B but still change the image config and the cache key.

Dockerfile instructions compared
InstructionEffectCreates a layer?Persists to runtime?Common mistake
FROMStarts a stage from a base image (or `scratch`)No new layer; reuses the base's layersYes: base content and config are inheritedFloating tag (`latest`) changes the base silently; ARG before FROM is not visible inside the stage
RUNExecutes a command in a temporary container and commits the filesystem diffYes (size = what the command wrote)Files persist; environment and working state of the command do notDeleting files in a later RUN; shell state (`cd`, `export`) does not carry to the next RUN
CMDDefault command or default arguments of ENTRYPOINTNo (0B)Yes, as `Config.Cmd`; replaced by arguments after the image nameShell form makes `/bin/sh -c` PID 1; several CMD lines: only the last counts
ENTRYPOINTFixed executable; CMD and `docker run` args are appendedNo (0B)Yes, as `Config.Entrypoint`; replaced only by `--entrypoint`, which also resets the image's CMDShell form ignores CMD and run arguments; script without `exec "$@"` swallows signals
COPYCopies files from the context or `--from` stage verbatimYesFiles persist`COPY . .` before dependency install; ownership stays root unless `--chown`; chown in a later RUN doubles the data
ADDLike COPY, plus unpacks local tar archives and downloads URLsYesFiles persistUsing ADD where COPY would do; archive silently extracted (observed: `a.tar` became `a.txt`)
ENVSets environment variables in the image configNo (0B)Yes: in `Config.Env` and every containerPutting secrets in ENV; `ENV key value` legacy syntax
ARGBuild-time variable, settable with `--build-arg`No (0B)No (empty at runtime), but its value appears in `docker history`Secrets in ARG; a changed ARG invalidates every RUN after it; ARG before FROM needs redeclaring
EXPOSEDocuments a port; used by `-P` and toolingNo (0B)Yes, as `Config.ExposedPorts`; publishes nothing by itselfAssuming it publishes the port (observed: `docker port` empty without -p/-P)
USERSets the user for later RUN and for the container's processNo (0B)Yes, as `Config.User`Expecting COPY to follow USER (files stay root-owned); a name that does not exist in `/etc/passwd`
WORKDIRSets the directory for later instructions and creates it if missingOnly if it creates the directory (observed 12.3kB)Yes, as `Config.WorkingDir`Using `RUN cd` instead; relative paths stacking up across several WORKDIR lines
VOLUMEDeclares the path as an anonymous volume at run timeNo (0B)Yes, as `Config.Volumes`; image content at the path is copied into the new volumeWriting data to the path in later RUN steps expecting it to be ignored (BuildKit kept the files)
HEALTHCHECKDefines a command the engine runs periodically to set health statusNo (0B)Yes, as `Config.Healthcheck` (durations in nanoseconds)Needs a tool such as wget or curl in the image; a slow check without `--timeout` and `--start-period`
LABELAdds key=value metadataNo (0B)Yes, as `Config.Labels`; inherited from the baseA changing value early in the file busts the cache for everything after it
SHELLChanges the shell used by shell-form RUN, CMD and ENTRYPOINTNo (0B)Yes, as `Config.Shell`Setting a shell that does not exist in the image; assuming exec-form commands use it
STOPSIGNALSignal `docker stop` sends first (default SIGTERM)No (0B)Yes, as `Config.StopSignal`Setting it while the process (PID 1) has no handler for that signal

Not run here: ADD from a URL or git repository (needs network access beyond Docker Hub) and the classic builder's VOLUME behavior (documented, not observed).

docker run resource and lifecycle flags

What each flag writes on the host. cgroup v1 file names were read on a real engine; the cgroup v2 column is from the kernel documentation and was not run here. Flags without a cgroup file show the kernel or daemon mechanism instead.

docker run resource and lifecycle flags
Flagcgroup v1 file / mechanismcgroup v2 fileDefaultGotcha
`--memory` / `-m``memory/memory.limit_in_bytes``memory.max`Unlimited (`9223372036854771712` on v1)Minimum 6m. Counts page cache too. A victim that is not PID 1 can leave exit 0 with `State.OOMKilled=true`
`--memory-swap``memory/memory.memsw.limit_in_bytes` (memory + swap)`memory.swap.max` (swap only, the difference)2 x `--memory` when `--memory` is setRejected without `--memory` or below it. Equal to `--memory` disables swap, `-1` is unlimited. No effect on a host without swap
`--memory-reservation``memory/memory.soft_limit_in_bytes``memory.low`None (soft limit equals unlimited)Only enforced under host memory pressure; must be lower than `--memory`
`--cpus``cpu/cpu.cfs_quota_us` per `cpu.cfs_period_us` (100000)`cpu.max` (`quota period`)Unlimited (quota -1)A time quota, not core count: throttling shows in `cpu.stat`. Cannot exceed host CPUs or be combined with `--cpu-quota`. `nproc` still shows host CPUs
`--cpu-shares``cpu/cpu.shares``cpu.weight` (converted)1024A relative weight under contention only; never a cap
`--cpuset-cpus``cpuset/cpuset.cpus``cpuset.cpus`All CPUs (`0-3` on a 4-core host)Pins the process; the only CPU flag that changes `nproc`. Error if the CPU does not exist
`--pids-limit``pids/pids.max``pids.max``max` (unlimited)At the limit `fork` fails with `Resource temporarily unavailable`; zombies count
`--ulimit`Not a cgroup: `setrlimit` per process (`nofile=soft:hard`)Same (not a cgroup)Inherited from dockerd (20000 for nofile on the test host)Per process, not per container; `nproc` is per user ID across the host
`--shm-size`Size option of the tmpfs mounted at `/dev/shm`Same (not a cgroup)`64m`Browsers and databases hit the 64m cap; shm pages count toward the memory limit
`--oom-kill-disable``memory/memory.oom_control` (`oom_kill_disable 1`)Typically not supported on v2OffNeeds `--memory` or Docker warns. The allocating process freezes (`under_oom 1`) instead of dying
`--oom-score-adj``/proc/<pid>/oom_score_adj`Same (not a cgroup)`0`-1000 to 1000. Negative values failed here with `Permission denied` (checked: -500 and -1000), so they may need extra privileges in restricted environments; it only ranks victims at host-level OOM
`--init`Runs `/sbin/docker-init` (tini) as PID 1, with your command as childSame (not a cgroup)Off (daemon `init` setting can turn on)Forwards signals and reaps zombies; `docker stop` then ends with 143 instead of waiting 10 s for 137
`--stop-timeout` / `docker stop -t`Daemon timer between the stop signal and SIGKILLSame (not a cgroup)10 secondsA stop that times out means exit 137. Compose: `stop_grace_period`
`--restart`Daemon restart manager (not a cgroup file)Same (not a cgroup)`no`Backoff starts at 100 ms and doubles; `docker stop` and `docker kill` are not restarted; cannot be used with `--rm`

Check which cgroup version you run with `docker info -f '{{.CgroupVersion}}'`. `docker update` changes memory, CPU, pids and restart settings on a running container; the others need a new container.

Network modes compared

What `--network` attaches the container to. DNS by name means the embedded resolver at 127.0.0.11 answers for other containers; behaviour was checked on Engine 29, except macvlan/ipvlan performance and overlay.

Network modes compared
ModeIsolationDNS by name-p needed to reach from outsidePerformanceUse case
bridge (default `docker0`, 172.17.0.0/16)Own namespace; all default-bridge containers can reach each other by IPNo (`--link` is deprecated)Yes (DNAT + docker-proxy)veth + NAT hopQuick one-offs; avoid for multi-container apps
User-defined bridge (`br-<id>`)Own namespace; a separate bridge per network, others dropped by iptablesYes: name, alias, short IDYes, only for access from outside the hostveth + NAT hopDefault choice for an application stack (Compose uses it)
hostNone: shares the host namespaceHost resolver onlyNo: `-p` is discarded with a warningNo NAT or veth; lowest overheadExporters, packet capture, very many ports
noneTotal: only `lo`, no routeNoNot applicable (nothing to publish)n/aOffline batch jobs, manual network wiring
container:<id>Shares another container's namespace, same IP and `127.0.0.1`Whatever the target hasPublish on the target container; `-p`, `--hostname`, `--dns` are rejected on the joinerSame as the targetSidecars, debugging a container's network from a tools image
macvlanOwn MAC and LAN address; host usually cannot reach it via the parent NICYes (embedded resolver works on any user-defined network)No: the container is directly on the LANNo NAT; depends on NIC and switchLegacy apps that need a LAN IP, DHCP, broadcast
ipvlan (l2 / l3)Own IP, shares parent MAC; l3 mode routes instead of bridgingYes, as for macvlan (not tested: this kernel rejected ipvlan)No: the container is directly addressableNo NAT; fewer MACs on the switchSame as macvlan where the switch limits MACs per port
overlayVXLAN network spanning hosts, encrypted optionallyYes, via the Swarm DNS (needs Swarm)Only through Swarm's routing mesh or host portsVXLAN encapsulation costMulti-host Swarm services; fails on a non-swarm engine

Verified on Engine 29: default-bridge NXDOMAIN, cross-network ping by IP lost 100%, host/none/container:<id> behaviour and their error messages, macvlan creation and name resolution on it, and the overlay error. Not verified: macvlan/ipvlan performance, anything about ipvlan (`failed to create the ipvlan port: operation not supported` on this kernel), Swarm overlay behaviour.

Storage types compared

Where container data can live, who creates the path, and what is still there afterwards. The pitfall column lists what bit during testing on Engine 29.

Storage types compared
TypeLifecycleWho creates the pathPerformanceSurvives docker rmTypical usePitfall
Named volume (`-v name:/path`)Until `docker volume rm`, `down -v` or pruneDocker, at `/var/lib/docker/volumes/<name>/_data`, on first useNative filesystem speed of the Docker rootYesDatabase data, anything that must outlive the containerCopy-up runs whenever the volume is empty at mount, and takes the image directory's owner; a missing image path makes a root-owned volume
Bind mount (`-v /host:/c`, `--mount type=bind`)Independent of Docker: the host path just existsYou; `-v` silently creates a missing source as root:root 755, `--mount` errorsNative speed of the host filesystemYes (not Docker's data)Source code, config files, host sockets, shared dev dataHides the image's files at that path (0 files seen); host ownership and modes apply unchanged
tmpfs (`--tmpfs`, `--mount type=tmpfs`)Until the container stopsDocker mounts it; `uid=`, `gid=`, `mode=`, `size=` optionsRAM speed; counts toward the memory cgroupNo (not even across restart)Secrets, scratch space, caches that must not hit diskWrites beyond `size=` fail; default owner is root with mode 1777
Anonymous volume (`-v /path`, Dockerfile `VOLUME`)Until removed by hand or with `rm -v` / `run --rm`Docker, with a 64-hex nameSame as a named volumeYes, unless `docker rm -v` or `--rm`Data directories of images that declare VOLUMEAccumulates unnoticed after plain `docker rm`; `volume rm` fails while a container uses it
Container writable layer (no mount)Until `docker rm`; survives stop/start/restartCreated by the storage driver (overlayfs upper dir)Copy-on-write; slower for write-heavy random IONoTemporary files, logs not worth keepingLost on `docker rm` and on Compose recreate; grows `docker system df` Containers size
Volume driver (`--driver local --opt ...`, plugins)Defined by the driver; the volume object is managed like a named volumeThe driver (local passes `type`, `device`, `o` to mount)Depends on the backend (tmpfs, NFS, block storage)Yes, as with any named volumeNFS or cloud-backed shared data, size-limited volumesMount options are driver specific; `local` with `type=nfs` was not tested here

All rows except NFS were exercised on Engine 29 with the overlayfs snapshotter. Memory accounting of tmpfs is how the kernel normally behaves; it was not measured here.

Container exit codes and where they come from

Every row below was reproduced on Docker Engine 29.8.2 unless the note says otherwise. The same number can come from two places (127 from runc or from a shell), so the third column matters: `State.Error` is non-empty only when Docker or runc failed before the app ran.

Container exit codes and where they come from
CodeMeaningSet byReproduced withFirst thing to check
`0`Normal exit. Not proof of success: a wrapper that daemonizes or a shell that lost its stdin also exits 0App`docker run -d alpine:3.20 sh` (no TTY, stdin closed): `Exited (0)` in 0.13 sIs the main process the foreground one? `docker logs`, then `-it` or a real foreground command
`1`Generic application failure (uncaught exception, failed command)App`python -c 'import nosuchmod'`, `ls /nonexistent``docker logs`: the reason is printed by the app
`2`Usage or misuse error, by convention (shell builtin misuse, missing file argument)App or shell`python /nonexist.py` returns 2The command line and the files it opens, in `docker logs`
`125`The docker CLI or daemon failed; no container process randocker`docker run --badflag busybox`, pulling an image that does not existstderr of `docker run`: flag, image name, name or port conflict. `docker ps -a` may show nothing
`126`Command found but cannot be executedruncEntrypoint file without the execute bit, or a directory as the command`State.Error` says `permission denied`; `ls -l` the file in the image, `chmod +x` before COPY
`127`Command not foundrunc, or a shell inside the container`docker run busybox /nonexistent-cmd` (runc, `State.Status=created`); `sh -c nonexistent_cmd` (shell, `exited`)Path, PATH and `--entrypoint`. Non-empty `State.Error` means runc, empty means the app's own shell
`128+n`Process killed by signal n; Docker reports 128 + signal numberKernel signal, reported by the runtimeSee the 130, 137, 139 and 143 rows`kill -l n` for the signal name; who sent it is the next question
`130`SIGINT (2), usually Ctrl-CSignal`sh -c 'kill -INT $$'` returns 130An interactive session or `docker kill -s INT`
`137`SIGKILL (9): cannot be handledKernel OOM killer, `docker kill`, or `docker stop` after its timeout`docker stop` on a PID 1 that ignores SIGTERM: 10.18 s then 137; `--memory 32m` with a 200 MB allocation: 137 and `OOMKilled=true``State.OOMKilled`: true means the memory limit; false means someone or the stop timeout. A killed child can leave exit 0 with `OOMKilled=true`
`139`SIGSEGV (11): a real segmentation faultKernel, on a memory fault`python -c 'import ctypes; ctypes.string_at(0)'` returns 139Native extension or libc mismatch (musl vs glibc); core dump or `dmesg` line for the faulting address
`143`SIGTERM (15) was not handled and killed the processSignal from `docker stop``docker run -d --init busybox sleep 1000` then `docker stop`: 143 in 0.2 sNormal for a stop of a program with no SIGTERM handler; if you expected a clean 0, add a handler
`255`Exec failure after the container started (runc init exits 255), or an application that exits with -1runc init, or app`exec /x.sh: no such file or directory` (CRLF shebang, missing interpreter), `exec format error` (no shebang, wrong arch)Container log line starting `exec `; `head -1 file | od -c` for `\r`, check the interpreter exists in the image

Exit statuses are 8 bits: `exit 300` in a shell returned 44 (300 mod 256). A signal sent from inside the container to its own PID 1 is ignored unless PID 1 installed a handler (`kill -SEGV $$` as PID 1 returned 0), so only a real fault or a signal from outside produces 139 or 137.

Fundamentals

A container is a host process wrapped in namespaces, cgroups and a filtered syscall surface, not a machine: what that shares with the host kernel, what it costs, and where the boundary is weaker.

The problem

A team moves a service from VMs to containers and expects the same things to hold: `free -m` shows the 512 MB it was given, `nproc` shows its CPU quota, a kernel module can be loaded, and a compromised process cannot reach the host. None of that is true. A container is a process on the host kernel with a restricted view of it, so resource numbers inside are the host's, the kernel is shared with every neighbour, and the isolation depends on kernel features and filters rather than on a hypervisor.

How it works

  • A container is a normal process tree on the host. `docker top` shows its processes with host PIDs, `ps` on the host lists them, and `uname -r` inside returns the host kernel release (verified: identical output inside and outside). There is no guest kernel, no virtual hardware and no boot sequence.
  • What the process can see is set by namespaces. The default runc spec for a Docker container creates new mount, network, UTS, PID and IPC namespaces, plus a time namespace on Engine 29.5 and later (checked on 29.8 in `config.json` and in `/proc/<pid>/ns/*`); the user and cgroup namespaces are not unshared, so root in the container is UID 0 on the host unless you enable user-namespace remapping.
  • What the process can use is set by cgroups. `--memory 64m` writes 67108864 to `memory.limit_in_bytes` (cgroup v1, verified) or `memory.max` (cgroup v2). The limit is enforced by the kernel, but `/proc/meminfo` and `/proc/cpuinfo` are not namespaced by cgroups, so `free -m` and `nproc` inside a limited container still print the host's totals (verified: 16 GB and 4 CPUs under `--memory 64m --cpus 0.5`). Runtimes and libraries that size themselves from those files over-allocate.
  • What the process may ask the kernel to do is filtered. Docker drops all but 14 capabilities by default (CapEff `00000000a80425fb` in the verified run), applies a default seccomp profile and an AppArmor or SELinux profile where the host has one. `unshare -u`, `dmesg` and `date -s` fail with `Operation not permitted` in a default container (verified), but every permitted syscall still enters the shared kernel directly.
  • Startup is process creation, not boot: `docker run --rm busybox:1.37 true` finished in about 0.35 s here including create, network setup and teardown. An idle `sleep` loop showed 976 KiB in `docker stats`. A VM pays for a guest kernel, init, and reserved RAM before the application starts.
  • The boundary is weaker than a hypervisor. A kernel bug, a leaked capability, `--privileged`, a mounted `/var/run/docker.sock`, or `--pid=host` (which showed the host's PID 1 and kernel threads in the verified run) gives the container reach into the host. For hostile multi-tenant code, sandboxed runtimes such as gVisor or Kata Containers add a second boundary at the cost of compatibility and speed.
  • The kernel is a dependency of the image. Image architecture must match the CPU (an `amd64` image on `arm64` needs emulation and is typically much slower), and features such as cgroup v1 versus v2, overlayfs, or specific syscalls come from the host, so the same image can behave differently across hosts.
  • State is separate from the instance. The writable layer disappears with `docker rm`, which suits disposable instances, and anything that must survive goes to a volume. A VM keeps its disk across reboots by default, which is why snapshots and in-place patching are normal there and rare for containers.

Examples

Shared kernel, unchanged /proc totals, limit in the cgroup, and the container process on the host.

bash
# Same kernel inside and outside
uname -r
docker run --rm busybox:1.37 uname -r

# /proc is not cgroup-aware: limits do not change what free and nproc print
docker run --rm --memory 64m --cpus 0.5 busybox:1.37 sh -c 'free -m | head -2; nproc'

# The limit itself lives in the cgroup (cgroup v1 path; cgroup v2 uses /sys/fs/cgroup/memory.max)
docker run --rm --memory 64m busybox:1.37 cat /sys/fs/cgroup/memory/memory.limit_in_bytes

# The container's process is visible on the host
docker run -d --name demo-vm busybox:1.37 sleep 300
ps -eo pid,ppid,cmd | grep '[s]leep 300'
docker top demo-vm
docker rm -f demo-vm

Filtered syscalls and capabilities, and which namespaces are actually separate by default.

bash
# Default containers cannot do kernel-level things: capability and seccomp filters
docker run --rm busybox:1.37 unshare -u true; echo "exit=$?"
docker run --rm busybox:1.37 dmesg 2>&1 | head -1
docker run --rm busybox:1.37 date -s "2030-01-01" 2>&1 | grep -i permitted

# Namespaces: the container's mnt/pid/net/uts/ipc/time differ from the host's (time only on Engine 29.5+), user and cgroup do not
for n in cgroup user mnt pid net uts ipc time; do
  echo "host $n $(readlink /proc/self/ns/$n)"
done
docker run --rm busybox:1.37 sh -c 'for n in cgroup user mnt pid net uts ipc time; do echo "ctr $n $(readlink /proc/self/ns/$n)"; done'

In practice

  • Docker Engine, containerd — Create the namespaces and cgroups and hand the process to runc; the container is a child of a shim, not of a hypervisor.
  • Kubernetes — Pods are groups of containers sharing a network namespace; node-level kernel upgrades affect every container on the node at once.
  • gVisor, Kata Containers, Firecracker — Add a user-space kernel or a micro-VM under an OCI-compatible runtime when the shared-kernel boundary is not enough.
  • AWS Lambda, Fly.io — Run customer code in micro-VMs (Firecracker) even though the packaging format is a container image, because the isolation requirement is VM-grade.
  • Podman — Same kernel primitives without a central daemon, commonly run rootless.

Trade-offs

Pros

  • Starts in well under a second and costs almost no memory beyond the application's own
  • One host kernel means high density: hundreds of idle containers fit where a few VMs would
  • The image is the deployment unit: the same bytes run on a laptop, CI and production
  • Resource limits and process visibility are plain kernel interfaces that standard tools (`ps`, `perf`, `strace`) can use from the host

Cons

  • A kernel vulnerability or misconfiguration affects every container on the host
  • Containers cannot run a different kernel or kernel modules, and depend on the host's cgroup and filesystem features
  • `/proc` and `/sys` values inside the container do not reflect its limits, which misleads runtimes that auto-size
  • Weaker isolation makes hostile multi-tenancy a poor fit without an additional sandbox layer

How an image is a stack of read-only layer diffs, how overlayfs merges them with one writable upper layer, and what copy-up and whiteouts do to size and speed.

The problem

An image shrinks by 400 MB after you delete a directory in a later `RUN`, or so you think: `docker image ls` is unchanged because the delete only added a marker on top of the layer that already holds the files. In production, a container that appends one line to a 2 GB database file makes the engine copy all 2 GB into its writable layer first, and the first write takes seconds. Both come from the same design: layers are immutable diffs merged by overlayfs.

How it works

  • An image is an ordered list of layers plus a config blob. Each layer is a tar of file additions, changes and deletions relative to the one below, identified by a content digest. `docker image inspect --format '{{json .RootFS.Layers}}'` lists the uncompressed digests: the verified three-step Dockerfile on `busybox:1.37` produced 4 layers (1 base + `RUN`, `RUN`, `COPY`).
  • Only filesystem-changing instructions add layers with content. In the verified build, `docker image history` shows each `RUN` and `COPY` with its own size (20.5kB, 8.19kB, 8.19kB); `ENV`, `WORKDIR`, `CMD` and `LABEL` only change the config and add no content.
  • Layers are shared by digest. Two images built `FROM` the same base keep one copy of the base on disk and pull it once, and `docker image ls` sums sizes per image so the totals overcount. Identical layers across tags, or across unrelated images, are also deduplicated by digest.
  • A container is the layer stack mounted by overlayfs plus one writable layer: `lowerdir` is the read-only layers (colon-separated, top first), `upperdir` is the container's own directory, and `workdir` is scratch space. The verified mount line shows `overlay on /var/lib/docker/rootfs/overlayfs/<id> type overlay (rw,relatime,lowerdir=...,upperdir=...,workdir=...)`.
  • Reading walks the stack from the top and returns the first hit. Writing to a file that lives in a lower layer first copies the whole file into the upper layer (copy-up) and then modifies the copy: after appending 2 bytes to a 4-byte `/big.txt`, the upper directory held a full 6-byte file. Large files that change a little therefore cost their full size once and a slow first write.
  • Deletion writes a whiteout: a character device with major and minor 0:0 named like the deleted file (verified: `c--------- b.txt`, `stat` says `character special file 0:0`). `docker diff` reports it as `D /b.txt`, and image layers use the same marker, so a file removed in a later `RUN` still occupies space in the earlier layer and in every pull.
  • `docker diff` compares the upper layer to the image: `A` added, `C` changed, `D` deleted (verified `C /big.txt`, `A /c.txt`, `D /b.txt`). Volumes and bind mounts are outside the overlay and never appear. `docker ps -s` and `docker inspect --size` report `SizeRw` (the upper layer, 4096 in the check) and `SizeRootFs`.
  • Where the layers live depends on the image store. On a fresh Engine 29 install the containerd image store with the overlayfs snapshotter is used (`docker info` shows `Storage Driver: overlayfs`, and `docker inspect` has `.Storage` instead of `.GraphDriver`); layers sit under `/var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.overlayfs/snapshots/<n>/fs`. Hosts upgraded from older versions typically keep the classic `overlay2` driver with data under `/var/lib/docker/overlay2/<id>/diff` (not run here).

Examples

Three filesystem instructions on a one-layer base give four layers; `rm /a.txt` does not remove it from the earlier layer.

bash
mkdir -p /tmp/demo-layers && cd /tmp/demo-layers
cat > Dockerfile <<'EOF'
FROM busybox:1.37
RUN echo one > /a.txt && echo big > /big.txt
RUN echo two > /b.txt && rm /a.txt
COPY note.txt /note.txt
EOF
echo hi > note.txt

docker build -q -t demo-layers .
docker image history demo-layers
docker image inspect demo-layers --format '{{json .RootFS.Layers}}'

Copy-up (`big.txt` is a full file in the upper layer), a new file, and a whiteout character device for the deleted `b.txt`.

bash
docker run -d --name demo-ov demo-layers sleep 300
docker exec demo-ov sh -c 'echo x >> /big.txt; rm /b.txt; echo new > /c.txt'

docker diff demo-ov
docker ps -s --filter name=demo-ov --format '{{.Size}}'

# The writable layer as seen from the host (containerd snapshotter path; overlay2 differs)
ID=$(docker inspect -f '{{.Id}}' demo-ov)
UPPER=$(mount | grep "$ID" | grep -o 'upperdir=[^,]*' | cut -d= -f2)
ls -l "$UPPER"
stat -c '%n %F %t:%T' "$UPPER/b.txt"

docker rm -f demo-ov

In practice

  • Docker Engine 29 (containerd image store) — Default image store on fresh installs, overlayfs snapshotter; classic `overlay2` remains on upgraded hosts.
  • Kubernetes (containerd, CRI-O) — Use the same overlayfs snapshots and layer digests, which is why nodes cache and share base layers across pods.
  • OCI image spec — Defines the layer media types (tar, tar+gzip, tar+zstd) and the whiteout conventions (`.wh.<name>`, `.wh..wh..opq`) used inside layer tarballs.
  • BuildKit — Builds one layer per filesystem-changing step and caches by content, so layer order directly decides cache hits.

Trade-offs

Pros

  • Shared layers mean one pull and one on-disk copy of a base image for any number of images and containers
  • Starting a container creates only an empty upper directory, so there is no filesystem copy at start
  • Immutable layers identified by digest make caching, verification and rollback simple
  • `docker diff` and the upper directory give a precise list of what a container changed

Cons

  • Deleting in a later layer never shrinks the image; files must be removed in the step that created them or in a separate build stage
  • Copy-up copies whole files, so writing to large files from the image is slow the first time and doubles their space
  • Heavy write workloads on the overlay are slower than on a volume or bind mount and are lost on `docker rm`
  • Many layers and many small files increase mount, pull and unpack time

Which process does what between `docker run` and your PID 1, what the OCI image and runtime specs standardise, and why restarting dockerd need not kill containers.

The problem

`docker run` is one command, yet `dockerd` is restarted for an upgrade and containers either survive or do not, `docker ps` hangs while the containers keep serving traffic, and a bug report mentions `containerd-shim` and `runc` as if they were obvious. Knowing which process owns which job tells you what a daemon restart will do, where a failure message such as `OCI runtime create failed` comes from, and why Kubernetes can drive the same containers without Docker at all.

How it works

  • The chain is `docker` CLI -> `dockerd` (REST API on `/var/run/docker.sock`) -> `containerd` (gRPC) -> `containerd-shim-runc-v2` -> `runc` -> your process. The CLI is a pure client; `dockerd` owns networks, volumes, build, the image API and container metadata under `/var/lib/docker`; `containerd` owns image content, snapshots and the task lifecycle.
  • Engine 29 starts its own containerd with a private socket, not the system one: the verified process list shows `containerd --config /var/run/docker/containerd/containerd.toml` as a child of `dockerd`, and `ctr -a /var/run/docker/containerd/containerd.sock -n moby containers ls` lists Docker's containers in the `moby` namespace. A separate containerd installed for Kubernetes uses `/run/containerd/containerd.sock` and the `k8s.io` namespace.
  • `runc` is the reference OCI runtime. It reads a bundle (a `config.json` plus a `rootfs`), calls `clone`/`unshare` for the namespaces, joins the cgroup, sets up mounts, capabilities and seccomp, then `exec`s your command and exits. It is not a daemon: the verified `ps` output contains no `runc` process while the container runs, and `runc --root /run/docker/runtime-runc/moby list` shows state kept on disk.
  • The shim (`containerd-shim-runc-v2`, one per container in this version) stays running as the container's parent: it holds stdio, collects the exit status and reports it back. It is reparented to PID 1 of the host rather than being a child of containerd (verified: shim PPID 1, container process PPID = shim), which is why containerd can restart without killing the container.
  • The OCI runtime spec is the bundle contract. The verified `config.json` had `ociVersion` 1.3.0, `process.args` `["sleep","300"]`, `root.path` pointing at the overlay mount, 6 namespaces (mount, network, time, uts, pid, ipc; the private time namespace is a default since Engine 29.5.0), 14 bounding capabilities, `linux.cgroupsPath` `/docker/<id>`, and `linux.resources` with `blockIO`, `devices`, `memory`. The bundle lives under `/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/<id>/`.
  • The OCI image spec is the other contract: a manifest, a config and layer blobs addressed by digest, with an index for multi-platform images. `docker save` on Engine 29 writes exactly that layout (verified: `index.json`, `oci-layout`, `manifest.json`, `blobs/sha256/`), which is why images move between Docker, containerd, Podman and Kubernetes unchanged.
  • Daemon restart behaviour depends on `live-restore`. It is off by default (`docker info --format '{{.LiveRestoreEnabled}}'` printed `false`), so stopping `dockerd` stops running containers; with `"live-restore": true` in `daemon.json` the shims keep containers running and `dockerd` reattaches on start. Not exercised here because the daemon was shared; the host-side evidence is that the shim is not a child of the daemon.
  • Errors name their layer. `docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: exec: "nosuchcmd": executable file not found in $PATH` came from runc during container init, and the CLI exit code was 127 (verified). Messages without `OCI runtime` come from `dockerd` or the CLI.

Examples

Walk the stack from the host: the shim, containerd's `moby` namespace, runc's on-disk state, and the OCI `config.json`.

bash
docker run -d --name demo-stack busybox:1.37 sleep 300

# Process tree: dockerd -> containerd; shim is reparented to PID 1; no runc process
ps -eo pid,ppid,cmd | grep -E 'dockerd|containerd|[s]leep 300' | grep -v grep

# Docker's private containerd, namespace "moby"
SOCK=/var/run/docker/containerd/containerd.sock
ctr -a "$SOCK" -n moby containers ls
ctr -a "$SOCK" -n moby tasks ls

# runc state and the OCI bundle
runc --root /run/docker/runtime-runc/moby list
ID=$(docker inspect -f '{{.Id}}' demo-stack)
B=/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/$ID
ls "$B"
python3 -c 'import json,sys; c=json.load(open(sys.argv[1])); print(c["ociVersion"], c["process"]["args"], [n["type"] for n in c["linux"]["namespaces"]])' "$B/config.json"

docker rm -f demo-stack

Which components and versions are running, and what an OCI-layer error looks like.

bash
docker info --format 'runtime={{.DefaultRuntime}} live-restore={{.LiveRestoreEnabled}} storage={{.Driver}}'
docker version --format '{{range .Server.Components}}{{.Name}}={{.Version}} {{end}}'

# A failure from the runc layer, with exit code 127
docker run --rm busybox:1.37 nosuchcmd; echo "exit=$?"

In practice

  • containerd — CNCF graduated container runtime; embedded by Docker Engine and used directly by Kubernetes through the CRI plugin.
  • runc, crun, youki — Interchangeable OCI runtimes: crun (C) and youki (Rust) implement the same bundle contract and are selected per container with `--runtime` or per `RuntimeClass` in Kubernetes.
  • Podman, CRI-O — Podman has no central daemon and uses conmon as the monitoring process in the shim role; CRI-O is a Kubernetes-only runtime built on the same OCI pieces.
  • Kata Containers, gVisor — Replace runc with an OCI-compatible runtime (`io.containerd.kata.v2`, `runsc`) that gives the container a micro-VM or user-space kernel, selected through containerd.
  • Open Container Initiative — Linux Foundation project that maintains the runtime, image and distribution specs, which is why images and bundles are portable across tools.

Trade-offs

Pros

  • Each layer has one job, so runtimes, image stores and orchestrators can be swapped without rewriting the others
  • Per-container shims keep containers independent of daemon restarts and crashes
  • Standard bundle and image formats mean images and runtimes from different vendors interoperate
  • Components can be inspected directly (`ctr`, `runc list`, `config.json`) when `docker` itself is unresponsive

Cons

  • More processes and sockets to know when debugging: a failure can sit in the CLI, dockerd, containerd, the shim or runc
  • Defaults are surprising: `live-restore` is off, so a daemon restart interrupts containers unless configured
  • Docker's private containerd and a system containerd are separate, with different sockets and namespaces, which confuses `ctr` usage
  • The version matrix (Engine, containerd, runc, kernel) matters for features such as cgroup v2 and newer OCI fields

The state machine from created to removed, what stop, kill, pause and rm actually send or freeze, the event sequence each produces, and how exit codes and restart policies attach to it.

The problem

A deploy takes exactly 10 seconds longer per container than expected, and the exit code of the old container is 137 instead of 0. The service ignores SIGTERM because it runs as PID 1 without a handler, so `docker stop` waits out its grace period and then kills it. Elsewhere, `docker ps` shows nothing while disk fills up with exited containers, or a container sits in `Created` and never ran. Each of these is a specific transition in the lifecycle, and `docker events` records which one happened.

How it works

  • The status values are `created`, `running`, `paused`, `restarting`, `removing`, `exited` and `dead` (verified from the error text of `docker ps --filter status=bogus`). `docker create` makes a container in `created` with `State.Pid` 0; `docker start` runs it; `docker run` is create plus start (plus attach unless `-d`).
  • `docker stop` sends SIGTERM (or the image's `STOPSIGNAL`), waits the grace period (10 s default, `-t` / `--stop-timeout` to change), then sends SIGKILL. Verified: `sleep 300` as PID 1 ignored SIGTERM, `docker stop` took 10.167 s and the exit code was 137 (128 + 9); with `-t 2` it took 2.19 s.
  • The kernel gives PID 1 of a PID namespace no default signal handlers, so SIGTERM is ignored unless the process installs a handler, which is why programs that exit instantly on Ctrl-C on the host can hang here. A shell script with a `trap` or `exec` into the real binary, or `--init` (docker-init as PID 1, verified as `/sbin/docker-init -- ps`), fixes it.
  • `docker kill` sends SIGKILL immediately, or the signal given with `-s`. The event stream for one stop shows the sequence (verified): `create`, `start`, `kill` (signal 15), `kill` (signal 9), `stop`, `die` (`exitCode` 137), `destroy` after `docker rm`. The relative order of `stop` and `die` is an observation from this run and may differ.
  • `docker pause` freezes the container's cgroup rather than sending a signal: the `freezer.state` file for the container read `FROZEN` (cgroup v1, verified) and `State.Status` was `paused`. Processes cannot even handle signals until `docker unpause`; this is not the same as `docker stop`. With cgroup v2, the freezer is `cgroup.freeze`.
  • On exit the container stays: `docker inspect` still shows `State.Status` `exited`, `ExitCode`, `FinishedAt` and `OOMKilled`, and `docker logs` still works until `docker rm`. The writable layer, logs and network config occupy disk until then, so leaving exited containers around is a slow disk leak. `docker run --rm` removes the container when it exits.
  • `docker rm` refuses a running container (`cannot remove container ...: container is running: stop the container before removing or force remove`, verified); `-f` kills then removes it. `dead` and `removing` appear when a removal fails or is in progress (for example a busy mount); these were not reproduced here.
  • A restart policy re-enters the lifecycle after exit: `--restart on-failure:2` on `sh -c 'exit 3'` ended as `exited`, `ExitCode` 3, `RestartCount` 2 (verified); status shows `restarting` during the backoff, which starts at 100 ms, doubles each time and is capped at one minute. `docker stop` or `docker kill` by the user does not trigger a restart.

Examples

States and timings: created with PID 0, paused via the cgroup freezer (cgroup v1 path), and a 10 s stop ending in exit 137.

bash
docker create --name demo-lc busybox:1.37 sleep 300
docker inspect -f '{{.State.Status}} pid={{.State.Pid}}' demo-lc
docker start demo-lc

docker pause demo-lc
docker inspect -f '{{.State.Status}} paused={{.State.Paused}}' demo-lc
cat /sys/fs/cgroup/freezer/docker/$(docker inspect -f '{{.Id}}' demo-lc)/freezer.state
docker unpause demo-lc

# sleep as PID 1 ignores SIGTERM, so stop waits the full 10 s and ends with SIGKILL
time docker stop demo-lc
docker inspect -f '{{.State.Status}} exit={{.State.ExitCode}}' demo-lc

# shorter grace period
docker start demo-lc
time docker stop -t 2 demo-lc
docker rm demo-lc

The `docker events` stream for create, start, stop and rm, and `on-failure:2` ending at `RestartCount` 2.

bash
# Event sequence for one container (background watcher, then the container's whole life)
timeout 20 docker events --filter container=demo-ev \
  --format '{{.Action}} {{.Actor.Attributes.signal}} {{.Actor.Attributes.exitCode}}' &
sleep 1
docker run -d --name demo-ev busybox:1.37 sleep 300
docker stop -t 1 demo-ev
docker rm demo-ev
sleep 1
kill %1 2>/dev/null

# Restart policy: exit code 3, two restarts, then it stays exited
docker run -d --name demo-rp --restart on-failure:2 busybox:1.37 sh -c 'exit 3'
sleep 6
docker inspect -f '{{.State.Status}} exit={{.State.ExitCode}} restarts={{.RestartCount}}' demo-rp
docker rm demo-rp

In practice

  • Kubernetes — Pod termination sends SIGTERM and waits `terminationGracePeriodSeconds` (default 30 s) before SIGKILL, the same two-step contract as `docker stop`.
  • Docker Compose — `docker compose down` and `stop` use each service's `stop_grace_period` (default 10 s), so a PID 1 that ignores SIGTERM makes every redeploy wait 10 s per service.
  • systemd — Services that wrap `docker run` need `docker stop` semantics (or `--rm`) to avoid leaving exited containers behind between restarts.
  • cAdvisor, `docker events` consumers — Container lifecycle events (`die`, `oom`, `health_status`) are the usual trigger for alerting on restarts and kills.

Trade-offs

Pros

  • A small, explicit state machine with events makes it possible to audit what happened to every container
  • Keeping exited containers allows `docker logs`, `docker inspect` and `docker cp` after the failure, which is often the only evidence
  • Separate stop and kill let well-behaved services drain connections before exit
  • `pause` freezes a process tree without signals or state loss, useful for consistent snapshots

Cons

  • Exited containers, their logs and writable layers accumulate until removed
  • A PID 1 that ignores SIGTERM turns every stop into a 10-second wait and an exit code of 137
  • `paused` containers keep their memory and cannot respond to health checks or signals
  • Restart policies mask crash loops: the container looks `running` while it is repeatedly failing

Building images

What each instruction does to the filesystem and to the image config, which ones add a layer, and the shell-versus-exec, ARG-versus-ENV and ENTRYPOINT-versus-CMD rules that cause most surprises.

The problem

A container prints nothing because the CMD ran through `/bin/sh -c`, `docker stop` takes ten seconds and ends with exit 137, an API token passed as ARG is readable in `docker history`, and EXPOSE 8080 publishes nothing. Each of these comes from a Dockerfile instruction doing something slightly different from what its name suggests. Knowing which instructions touch the filesystem and which only write image config removes most of the guessing.

How it works

  • Only instructions that change the filesystem produce a layer with bytes: RUN, COPY, ADD, and WORKDIR when it has to create the directory. ENV, ARG, LABEL, EXPOSE, USER, CMD, ENTRYPOINT, HEALTHCHECK, SHELL, STOPSIGNAL and VOLUME only write image config and appear as `0B` in `docker history`. In the snippet below `WORKDIR /srv/app` shows 12.3kB because it created the directory.
  • Shell form (`CMD sleep 300 && echo done`) is wrapped as `/bin/sh -c '...'`; exec form (`CMD ["sleep", "300"]`) runs the binary directly. With a compound command the shell stays PID 1 and the program is its child (observed: PID 1 `/bin/sh -c sleep 300 && echo done`, PID 7 `sleep 300`). A PID 1 without a SIGTERM handler ignores the signal, so `docker stop` waits the default 10 seconds and then sends SIGKILL, exit code 137. BusyBox `sh` execs a lone command, so the difference only shows with `&&`, `;` or pipes.
  • ENTRYPOINT fixes the executable and CMD supplies default arguments. `docker run img other args` replaces CMD only (observed `entry other args`); `--entrypoint` replaces ENTRYPOINT and also resets the image's CMD to empty (observed: `--entrypoint echo` with no arguments printed an empty line). A shell-form ENTRYPOINT is stored as `[/bin/sh -c echo entry]` and the CMD is ignored, so the container printed just `entry`. Only the last CMD in a Dockerfile counts.
  • ARG exists only while building; ENV is written to the image config and every container gets it. In the example `BUILD_MODE` is `dev` inside the RUN but empty at runtime, while `APP_ENV=prod` appears in `Config.Env`. An ARG declared before FROM is visible only in FROM lines unless it is declared again inside the stage (observed: empty, then `3.20` after redeclaring).
  • ARG is not a safe place for secrets. BuildKit records the values of ARGs in the history line of the RUN that follows (`RUN |1 BUILD_MODE=dev /bin/sh -c ...`), and `docker build --check` flags names like API_TOKEN with SecretsUsedInArgOrEnv. Use a secret mount instead (see the BuildKit entry).
  • COPY copies files as they are; ADD also unpacks a local tar archive and can fetch a URL. Observed: `ADD a.tar /extracted/` produced `/extracted/a.txt`, while `COPY a.tar ./plain.tar` kept the archive. Prefer COPY unless you need extraction. Files copied after `USER nobody` are still owned by root unless you write `COPY --chown=nobody` (observed `root root` without it).
  • EXPOSE is documentation plus a hint for `-P`: with no `-p` or `-P`, `docker port` printed nothing; with `-P` it showed `8080/tcp -> 0.0.0.0:32768`. VOLUME makes the path an anonymous volume at run time, and the image content there is copied into the new volume. With BuildKit, files written by a RUN after the VOLUME line stayed in the image in the run below (the classic builder was documented to discard such changes).
  • HEALTHCHECK, STOPSIGNAL, SHELL and USER are stored in the config: `Config.Healthcheck` (intervals in nanoseconds, e.g. 30000000000), `Config.StopSignal`, `Config.Shell`, `Config.User`. `docker build --check` runs the Dockerfile linter without building; it reported FromAsCasing, LegacyKeyValueFormat, SecretsUsedInArgOrEnv and JSONArgsRecommended for four short lines.

Examples

One Dockerfile with most instruction types; the history shows which ones carry bytes, the runs show ENTRYPOINT/CMD and ARG/ENV behavior.

bash
cat > a.txt <<'EOF'
hi
EOF
cat > Dockerfile <<'EOF'
FROM alpine:3.20
ARG BUILD_MODE=dev
ENV APP_ENV=prod
WORKDIR /srv/app
RUN echo "mode=$BUILD_MODE" > /mode.txt
COPY --chown=nobody a.txt .
EXPOSE 8080/tcp
USER nobody
STOPSIGNAL SIGINT
HEALTHCHECK --interval=30s --timeout=3s --retries=2 CMD wget -q -O /dev/null http://127.0.0.1:8080/ || exit 1
ENTRYPOINT ["echo", "entry"]
CMD ["default-arg"]
EOF
docker build -q -t demo-ins .

# metadata instructions show 0B; RUN, COPY and WORKDIR (it created the directory) carry data
docker history --format '{{.Size}}\t{{.CreatedBy}}' demo-ins | head -12
docker inspect demo-ins --format '{{.Config.Entrypoint}} {{.Config.Cmd}} {{.Config.Env}} {{.Config.StopSignal}}'

docker run --rm demo-ins                 # entry default-arg
docker run --rm demo-ins other args      # entry other args (CMD replaced)
docker run --rm --entrypoint sh demo-ins -c 'echo "[$BUILD_MODE]"; cat /mode.txt'

Shell form with a compound command keeps sh as PID 1; docker stop then waits out the 10 second timeout and the container dies with 137.

bash
cat > Dockerfile <<'EOF'
FROM busybox:1.37
CMD sleep 300 && echo done
EOF
docker build -q -t demo-form .
docker run -d --name demo-form demo-form
docker exec demo-form ps -o pid,ppid,args     # PID 1 is /bin/sh -c ..., sleep is its child
time docker stop demo-form                    # about 10s: sh as PID 1 ignores SIGTERM
docker inspect demo-form --format '{{.State.ExitCode}}'   # 137
docker rm demo-form

Lint without building: the rule names to grep for in CI.

bash
cat > Dockerfile <<'EOF'
FROM alpine:3.20 as base
ENV APP_SECRET abc
ARG API_TOKEN=xyz
CMD echo $UNDEFINED_VAR
EOF
docker build --check . 2>&1 | grep '^WARNING'

In practice

  • Official Docker Hub images (postgres, nginx, redis) — Use an exec-form ENTRYPOINT script that ends with `exec "$@"` so the real server becomes PID 1 and receives SIGTERM, with CMD holding the default server command.
  • docker build --check — Dockerfile linting built into BuildKit (rules such as FromAsCasing, JSONArgsRecommended, SecretsUsedInArgOrEnv); runs without building the image.
  • docker history — Lists every instruction with its size and the ARG values captured by RUN lines, which is why secrets passed as ARG leak.
  • docker compose — `entrypoint:` and `command:` in the Compose file override ENTRYPOINT and CMD from the image config; as with `docker run --entrypoint`, setting `entrypoint:` also discards the image's default CMD (the Compose spec: "Compose ignores any default command from the image").

Trade-offs

Pros

  • Exec form gives the program PID 1 and direct signal delivery with no extra shell process
  • Metadata instructions cost no bytes, so ENV, LABEL, USER and friends can be used freely for size
  • ENTRYPOINT plus CMD makes an image behave like a command with overridable default arguments
  • docker build --check catches common mistakes before a build runs

Cons

  • Exec form does no variable expansion or shell features; you must write `["sh", "-c", "..."]` explicitly when you need them
  • Instruction names mislead: EXPOSE publishes nothing, VOLUME has run-time side effects, ADD extracts silently
  • ENV is baked into every container and readable with `docker inspect`, so it is the wrong place for secrets as well
  • Behavior differences between BuildKit and the classic builder (such as VOLUME) mean old advice may not apply

Several FROM stages in one Dockerfile: only the last stage ships, unused stages are skipped, independent stages run in parallel, and scratch images need static binaries.

The problem

The image needs a compiler, headers and a package cache to build, but only the resulting binary to run. Shipping the build image drags hundreds of megabytes and a large attack surface into production, and cleaning up with `rm` in later layers does not shrink it. Splitting build and runtime into stages in one Dockerfile keeps a single source of truth without shipping the toolchain.

How it works

  • Every `FROM` starts a new stage with its own filesystem; the image you get is the last stage, or the one named by `--target`. Layers from earlier stages stay in the build cache but not in the image. Observed: the builder stage held a 100MB random file and `--target builder` produced a 222MB image (disk usage), while the default image was 6.78MB.
  • `COPY --from=<stage|index|image>` copies files from a named stage (`AS builder`), a stage number, or any other image (`COPY --from=busybox:1.37 /bin/busybox /x` also works). Only the copied files travel; ENV, USER and WORKDIR of the source stage do not.
  • BuildKit builds a dependency graph from the stages: a stage that the target does not depend on is never executed. In the example the `debugtools` stage printed nothing during the normal build and ran only with `--target debugtools`. Use this to keep test, debug and production images in one file.
  • Independent stages run in parallel. Two stages that each did `sleep 4` finished in 4.2s each and the whole `--no-cache` build took about 5.5s wall time on this 4-core VM, instead of the 8s a serial run would need. Order the Dockerfile so unrelated work lives in separate stages.
  • Caching is per step as usual: editing one stage invalidates only the stages that copy from it. Put slow, rarely changing work (toolchain install, dependency download) in its own stage so source edits do not touch it.
  • `FROM scratch` is an empty filesystem: no shell, no libc, no certificates, no `/etc/passwd`. A 132-byte static ELF copied into scratch ran and exited 42 (image 9.4kB), while `--entrypoint sh` failed with `executable file not found in $PATH` and exit 127. Anything you need at run time (CA bundle, tzdata, a user entry for `USER`) must be copied in from another stage.
  • A dynamically linked binary in scratch fails with a misleading message: busybox from `busybox:1.37` copied into scratch printed `exec /busybox: no such file or directory` (exit 255 here) because the missing file is the loader `/lib/ld-linux-x86-64.so.2`. Build static (for example Go with `CGO_ENABLED=0`) or choose a base that ships libc.
  • Secrets and tools used only in a build stage never reach the final image's layers or its `docker history`, which lists only the last stage's steps. That is no substitute for secret mounts, because the build cache on the builder host still holds the earlier stages.

Examples

Three stages: the default build skips debugtools, --target builder shows what would have shipped without a final stage.

bash
cat > Dockerfile <<'EOF'
# syntax=docker/dockerfile:1
FROM alpine:3.20 AS builder
RUN dd if=/dev/urandom of=/big.bin bs=1M count=100 2>/dev/null \
 && echo "result-from-builder" > /result.txt

FROM alpine:3.20 AS debugtools
RUN echo "only built with --target debugtools"

FROM busybox:1.37 AS final
COPY --from=builder /result.txt /result.txt
CMD ["cat", "/result.txt"]
EOF
docker build --progress=plain -t demo-ms . 2>&1 | grep -E '^#[0-9]+ \[(builder|debugtools|final)'
docker build -q --target builder -t demo-ms-builder .
docker images 'demo-ms*'      # compare the size of the final image and the builder target
docker run --rm demo-ms

A scratch final stage: a tiny static x86-64 binary is all it needs; there is no shell to exec into.

bash
cat > Dockerfile <<'EOF'
# syntax=docker/dockerfile:1
FROM alpine:3.20 AS build
# stand-in for "compile": write a 132-byte static x86-64 ELF that exits with 42
RUN { \
  printf '\x7f\x45\x4c\x46\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x3e\x00\x01\x00\x00\x00\x78\x00\x40\x00\x00\x00\x00\x00'; \
  printf '\x40\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x40\x00\x38\x00\x01\x00\x00\x00\x00\x00\x00\x00'; \
  printf '\x01\x00\x00\x00\x05\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x40\x00\x00\x00\x00\x00\x00\x00\x40\x00\x00\x00\x00\x00'; \
  printf '\x84\x00\x00\x00\x00\x00\x00\x00\x84\x00\x00\x00\x00\x00\x00\x00\x00\x10\x00\x00\x00\x00\x00\x00\xbf\x2a\x00\x00\x00\xb8\x3c\x00'; \
  printf '\x00\x00\x0f\x05'; \
} > /app && chmod 755 /app

FROM scratch
COPY --from=build /app /app
ENTRYPOINT ["/app"]
EOF
docker build -q -t demo-elf .
docker run --rm demo-elf; echo "exit=$?"                  # exit=42: no shell, no libc needed
docker run --rm --entrypoint sh demo-elf; echo "exit=$?"   # "sh": executable file not found, exit=127
docker images demo-elf

Why a dynamically linked binary fails in scratch: the error names the binary, but the loader is what is missing.

bash
cat > Dockerfile <<'EOF'
FROM busybox:1.37 AS src
FROM scratch
COPY --from=src /bin/busybox /busybox
ENTRYPOINT ["/busybox"]
EOF
docker build -q -t demo-dyn .
docker run --rm demo-dyn echo hi; echo "exit=$?"   # exec /busybox: no such file or directory

In practice

  • Go services — Typically compile with `CGO_ENABLED=0` in a `golang` stage and copy one binary into scratch or a distroless base, plus CA certificates if the program makes TLS calls.
  • Node and Python web apps — Install dev dependencies and build assets in one stage, then copy only `dist/` or the virtualenv into a slim runtime stage.
  • docker buildx bake and CI pipelines — Name stages `test`, `lint`, `release` and select them with `--target`, so CI runs tests from the exact Dockerfile that builds the release image.
  • Docker Hub official images — Many publish `-alpine` and `-slim` variants whose Dockerfiles are good references for what a minimal runtime stage needs.

Trade-offs

Pros

  • One Dockerfile holds build, test and runtime without shipping the toolchain
  • Unused stages are skipped and independent stages run in parallel
  • Final image shrinks by the size of everything that only the build needed
  • Smaller runtime contents mean fewer packages to patch and scan

Cons

  • scratch and distroless images have no shell, so debugging needs an ephemeral debug container or a separate debug target
  • Static linking and copying certificates, timezone data and users is your responsibility
  • Builder stages remain in the build cache and can consume disk on the build host
  • More stages mean more places for COPY --from paths to go stale unnoticed

How BuildKit decides a step is CACHED, what cache mounts and secret mounts add, and how to export the cache or find where the disk went.

The problem

Every CI run re-downloads the same packages, a rebuild after a one-line change takes ten minutes, and a private token has to reach `npm install` without ending up in an image layer. Meanwhile `docker image prune` frees nothing and the disk is still full because the build cache lives outside the image list. BuildKit has a precise rule for what counts as CACHED and three tools (cache mounts, secret mounts, exported caches) that cover these cases.

How it works

  • A step is CACHED when its cache key matches: for RUN the key is the parent layer plus the command text (plus ARG/ENV values visible to it); for COPY and ADD it is a checksum of the file contents. Observed: `touch f.txt` without changing the content left COPY and the following steps CACHED, and `RUN date +%s%N > /stamp` stayed CACHED forever because the command string never changed. BuildKit does not know a command is non-deterministic.
  • Invalidation cascades: a miss at step N reruns N and everything after it. Skip the cache with `--no-cache`, or per stage with `--no-cache-filter <stage>` (observed: only the named stage's RUN reran, and a later COPY that consumes its changed output reran as well). `--pull` re-resolves the base image tag.
  • `RUN --mount=type=cache,target=/var/cache/dl` mounts a directory that persists between builds and is not part of any layer. Observed: the counter file went from 0 to 1 across two builds even though the RUN had to rerun (BUST changed), and `ls /var/cache/dl` in the finished image failed with No such file or directory. Use it for package-manager download caches, compiler caches and the like.
  • A cache mount is identified by `id` (default: the target path), so two RUN steps with different targets can share one cache by giving the same `id` (observed: a file written at `/c` was visible at `/other`). `sharing=shared` (default), `private` or `locked` controls concurrent builds; `uid`, `gid` and `mode` set ownership (observed `owner=1000`). Treat the contents as disposable: the cache can be garbage-collected, and a fresh CI runner starts empty.
  • `RUN --mount=type=secret,id=tok` exposes the file `/run/secrets/tok` for that RUN only; supply it with `docker build --secret id=tok,src=tok.txt`. Observed: the RUN saw the file, the finished image had no `/run/secrets`. Unlike ARG, the value does not appear in the history line.
  • Cache can be exported and imported for CI: `--cache-to type=local,dest=DIR[,mode=max]` and `--cache-from type=local,src=DIR`, or `type=registry`, `type=gha`, `type=inline`. On this engine (Docker 29 with the containerd image store) `--cache-to type=local` worked on the default `docker` driver and wrote `blobs`, `index.json`, `oci-layout`. With the classic image store the `docker` driver typically rejects cache export and you need a builder with `--driver docker-container` (not tested here, as was importing the cache on a second builder). `mode=max` also exports intermediate stages; `mode=min` only the final ones.
  • The cache belongs to the builder, not to images. `docker image prune` and `docker rmi` do not touch it. `docker buildx du` lists the records (type such as `exec.cachemount`, size, usage count, last used); `docker builder prune` (optionally with `--filter until=24h`) or `docker buildx prune --max-used-space <size>` (older buildx: `--keep-storage`) trims it. They were not run here because this daemon is shared; the commands act on the whole builder.
  • `--progress=plain` prints every step with its output and a `CACHED` line for hits, and it is the quickest way to see where a cache miss starts. Combine with the next entry: the order of instructions decides how much of the cache survives a change.

Examples

A cache mount survives a rerun of its RUN and never lands in the image.

bash
cat > Dockerfile <<'EOF'
# syntax=docker/dockerfile:1
FROM alpine:3.20
ARG BUST=0
RUN --mount=type=cache,id=demo-dl,target=/var/cache/dl,sharing=locked \
    n=$(cat /var/cache/dl/count 2>/dev/null || echo 0); \
    echo $((n+1)) > /var/cache/dl/count; \
    echo "builds that used this cache before: $n"; \
    echo "$BUST" > /out
EOF
docker build --progress=plain --build-arg BUST=1 -t demo-bk . 2>&1 | grep 'builds that used'
docker build --progress=plain --build-arg BUST=2 -t demo-bk . 2>&1 | grep 'builds that used'   # RUN reruns, cache dir survives
docker run --rm demo-bk ls /var/cache/dl    # No such file or directory: the mount is not in the image

A secret mount is readable during one RUN and absent from the result.

bash
echo 's3cret' > tok.txt
cat > Dockerfile <<'EOF'
FROM alpine:3.20
RUN --mount=type=secret,id=tok test -s /run/secrets/tok && echo "secret visible during this RUN only"
EOF
docker build --progress=plain --secret id=tok,src=tok.txt -t demo-sec . 2>&1 | grep 'secret visible'
docker run --rm demo-sec ls /run/secrets      # No such file or directory

Bypass the cache for one stage, export the cache, and list the build cache records.

bash
cat > Dockerfile <<'EOF'
FROM alpine:3.20 AS deps
RUN date +%s > /stamp
FROM alpine:3.20
COPY --from=deps /stamp /stamp
EOF
docker build -q -t demo-nc .
sleep 2
docker build --progress=plain --no-cache-filter deps -t demo-nc . 2>&1 | grep -E 'deps 2/2|CACHED'   # the deps RUN reruns, and the COPY that consumes its new output reruns too

# export the build cache to a directory (works on this engine; see mechanisms for the driver caveat)
docker build -q --cache-to type=local,dest=/tmp/demo-cache,mode=max -t demo-nc .
ls /tmp/demo-cache                  # blobs  index.json  oci-layout (and a temporary ingest dir)
docker buildx du | head -5             # build cache records, by ID, size and last use

In practice

  • BuildKit — The default builder in Docker Engine; a `# syntax=docker/dockerfile:1` first line pins the Dockerfile frontend version, which decides which `RUN --mount` options and `--check` rules exist.
  • docker/build-push-action with GitHub Actions cache — Typically uses `cache-from: type=gha` and `cache-to: type=gha,mode=max` so a fresh runner can reuse layers from earlier runs.
  • pip, npm, Go modules, apt — Common cache mount targets such as `/root/.cache/pip`, `/root/.npm`, `/go/pkg/mod` and `/var/cache/apt`, so a changed manifest downloads only the new packages.
  • docker buildx — The CLI plugin that exposes `du`, `prune`, `bake`, `imagetools` and builder management for BuildKit.

Trade-offs

Pros

  • Cache mounts make dependency changes download only what is new, without bloating layers
  • Secret mounts keep credentials out of layers, history and cache keys
  • Exported caches let ephemeral CI runners start warm
  • `CACHED` and `--check` output make build behavior inspectable

Cons

  • Builds can depend on state that is not in the Dockerfile, so a cold cache may behave differently from a warm one
  • Cache grows silently outside the image list and needs explicit pruning
  • Exporting with `mode=max` can be large and slow to upload, and may need a non-default builder driver
  • Commands with side effects (apt-get update, date, curl) are cached by text, so stale results persist

A changed step invalidates every step after it, so order instructions from least to most volatile: manifests before install, source last, ARG late, and a .dockerignore.

The problem

A one-line change in `app.py` triggers a full dependency install because `COPY . .` sits above `RUN pip install`. Another build reruns everything after a version label was added near the top. The cache works exactly as specified; the Dockerfile just orders volatile inputs before expensive steps. Reordering costs nothing and turns a multi-minute rebuild into a few CACHED lines.

How it works

  • The rule: a changed or missed step invalidates itself and every step after it. Steps before it stay CACHED. Order instructions from least to most frequently changing: base image, system packages, dependency manifests, dependency install, source code, build step, metadata.
  • Observed with the manifest-first order (`COPY requirements.txt .`, `RUN install`, `COPY . .`): after editing only `app.py`, the install step printed `CACHED`; after editing `requirements.txt`, the install ran again (`installing deps` appeared) while `WORKDIR` stayed CACHED. With `COPY . .` above the install, the same `app.py` edit reran the install.
  • COPY invalidation is by content checksum, not by modification time: `touch` alone does not invalidate. But `COPY . .` includes every file in the context, so a README or test edit invalidates the layer. Copy only the files an expensive step needs before running it.
  • A changed ARG invalidates every RUN after its declaration, even when the command never mentions it. Observed: with `ARG GIT_SHA` at the top and three RUN lines, changing the value reran all three; with the ARG declared after the first RUN, the first stayed CACHED. Declare build-time stamps (commit SHA, build date) as late as possible, or after the expensive steps.
  • The same applies to LABEL and ENV placed early with changing values: they change the config, and every later step's parent. Put per-build metadata at the end, or attach it as an OCI annotation at build time instead of baking it into early layers.
  • The build context matters too. BuildKit sends only the files that COPY/ADD actually reference: with `COPY Dockerfile /x` the transfer was 73 bytes beside a 50MB file, but with `COPY . /app` it transferred 50.01MB. Adding `blob.bin` to `.dockerignore` cut that to about 100 bytes and also kept the file out of `COPY . /app`.
  • Non-deterministic commands cache by text: `RUN apt-get update` alone is cached indefinitely and goes stale, so combine `apt-get update && apt-get install -y ...` in one RUN, and bust deliberately (`--no-cache-filter`, a version pin changed in the command) when you want fresh data.
  • Conditional work belongs in separate stages (see multi-stage builds) so that an edit to one part cannot invalidate unrelated steps, and so that independent stages can run in parallel.

Examples

Manifest first: a source edit leaves the install CACHED; a manifest edit reruns it.

bash
printf 'flask==3.1.0\n' > requirements.txt
echo 'print(1)' > app.py
cat > Dockerfile <<'EOF'
FROM alpine:3.20
WORKDIR /app
COPY requirements.txt .
RUN echo installing deps && sleep 2 && cat requirements.txt > /deps.txt
COPY . .
CMD ["cat", "/deps.txt"]
EOF
docker build -q -t demo-lo .
echo 'print(2)' > app.py                 # edit source only
docker build --progress=plain -t demo-lo . 2>&1 | grep -E 'CACHED|installing'   # the install step is CACHED
echo 'flask==3.1.1' > requirements.txt       # edit the manifest
docker build --progress=plain -t demo-lo . 2>&1 | grep -E 'CACHED|installing'   # install runs again

A changed ARG invalidates only the RUN steps after its declaration.

bash
cat > Dockerfile <<'EOF'
FROM alpine:3.20
RUN echo step-unrelated > /a
ARG GIT_SHA=none
RUN echo "sha=$GIT_SHA" > /b
EOF
docker build -q --build-arg GIT_SHA=aaa -t demo-arg .
docker build --progress=plain --build-arg GIT_SHA=bbb -t demo-arg . 2>&1 | grep -E 'CACHED|RUN'
# step 2 is CACHED because it comes before the ARG; move the ARG to the top and both RUN lines rerun

Context transfer with and without .dockerignore.

bash
head -c 50000000 /dev/urandom > blob.bin
printf 'FROM alpine:3.20\nCOPY . /app\n' > Dockerfile
docker build --progress=plain --no-cache -t demo-ctx . 2>&1 | grep 'transferring context'    # ~50MB
echo blob.bin > .dockerignore
docker build --progress=plain --no-cache -t demo-ctx . 2>&1 | grep 'transferring context'    # about 100 bytes

In practice

  • Python (requirements.txt, poetry.lock) — Copy only the lock or requirements file, install, then copy the source; the install layer survives code-only commits.
  • Node (package.json and lockfile) — Copy `package.json` and the lockfile, run `npm ci`, then copy the source; adding a cache mount for `/root/.npm` speeds the misses too.
  • Go (go.mod, go.sum) — Copy the two module files and run `go mod download` before copying packages, so dependency download is cached separately from compilation.
  • CI image builds — Typically pass the commit SHA as the last ARG or as an annotation so every commit does not rebuild the full image.

Trade-offs

Pros

  • Costs nothing: reordering instructions reuses the existing cache machinery
  • Turns dependency-install time into a one-off per manifest change
  • A tight .dockerignore also shrinks the context sent to the builder
  • Late metadata keeps per-commit builds fast

Cons

  • Splitting COPY into manifest and source steps adds lines and duplicates paths that can drift apart
  • Some tools need the whole tree during install (editable installs, monorepo workspaces), which defeats the pattern
  • A stale cache can hide changes in system packages or upstream base images
  • The relevant files differ per language, so each project needs its own ordering

Layers only add: deleting a file in a later layer hides it but keeps its bytes. Measure with docker history, clean inside the same RUN, use COPY --chown, and ship from a small final stage.

The problem

The image is 1.2GB, pushes take minutes, nodes pull it slowly, and `RUN rm -rf /tmp/build` in the next line changed nothing. Image layers are stacked, read-only diffs: deleting a file in a later layer adds a whiteout marker, and the bytes remain in the earlier layer. Slimming means not putting the bytes into a layer in the first place.

How it works

  • Measure first: `docker history` shows the size of each layer, `docker images` shows two numbers on Docker 29 with the containerd image store. DISK USAGE counts the unpacked layers plus the compressed blobs, so it is roughly double for incompressible data; CONTENT SIZE is the compressed size that a registry push transfers. Use CONTENT SIZE for transfer cost.
  • Deleting in a later layer does not shrink the image. Observed with a 50MB random file: `COPY blob.bin /tmp/` then `RUN rm /tmp/blob.bin` left CONTENT SIZE at 53.6MB (the history shows a 50MB COPY layer and an 8.19kB RUN layer). Doing the work in one RUN, or never copying the file (`RUN --mount=type=bind,source=blob.bin,...`), gave 3.63MB, the size of plain alpine.
  • Clean up in the same RUN that created the mess: install and delete the package cache in one command. Where possible use cache mounts or bind mounts so the temporary data never becomes a layer. Package managers have flags for it (`apt-get install --no-install-recommends` plus removing `/var/lib/apt/lists` in the same RUN; `pip install --no-cache-dir`), which could not be run here without network access to package repositories.
  • `chown` and `chmod` duplicate data. `COPY blob.bin /opt/` followed by `RUN chown -R nobody /opt` stored the 50MB file twice (CONTENT SIZE 104MB), while `COPY --chown=nobody` (and `--chmod`) gave 53.6MB. The same applies to anything that rewrites files in place in a later layer.
  • Multi-stage builds are the structural fix: compilers, headers and caches live in a builder stage and only the output is copied into a small final stage (a 6.78MB image beside a 222MB builder in the earlier experiment). The base image matters as well: `python:3.13-slim` showed an 87.7MB Debian base layer plus a 40.4MB interpreter layer, so the CONTENT SIZE was 45.3MB.
  • Exclude files from the context with `.dockerignore` (`.git`, `node_modules`, local build output, test data), since `COPY . .` copies everything it sees and a later `rm` cannot remove it from the layer.
  • Flattening with `docker export | docker import` produced a single layer (12.1MB instead of 112MB for the earlier image with a deleted file), but it also dropped CMD and ENV: `Cmd=[]` and `Env=[]` (even PATH is gone), and you lose layer sharing. BuildKit has no `--squash`; a final stage of `FROM scratch` plus `COPY --from=builder / /` gave one layer with the deleted data gone (observed `layers=1`), and CMD can be set explicitly in that stage.
  • Smaller does not always mean better. Alpine uses musl instead of glibc, which can change behavior or performance of binaries and Python wheels. Distroless and scratch bases remove the shell, making debugging harder. Verify that the slim image still runs your workload and has the CA certificates, timezone data and user entries it needs.

Examples

Deleting after COPY keeps the bytes (53.6MB); never storing them gives 3.63MB.

bash
head -c 50000000 /dev/urandom > blob.bin
cat > Dockerfile.a <<'EOF'
FROM alpine:3.20
COPY blob.bin /tmp/blob.bin
RUN rm /tmp/blob.bin
EOF
cat > Dockerfile.b <<'EOF'
FROM alpine:3.20
RUN --mount=type=bind,source=blob.bin,target=/src/blob.bin cp /src/blob.bin /tmp/b && rm /tmp/b
EOF
docker build -q -f Dockerfile.a -t demo-slim-a .
docker build -q -f Dockerfile.b -t demo-slim-b .
docker images 'demo-slim-*'    # CONTENT SIZE is the compressed size a push transfers
docker history --format '{{.Size}}\t{{.CreatedBy}}' demo-slim-a | head -3

chown in a separate layer doubles the file; COPY --chown does not.

bash
cat > Dockerfile.c <<'EOF'
FROM alpine:3.20
COPY blob.bin /opt/blob.bin
RUN chown -R nobody /opt
EOF
cat > Dockerfile.d <<'EOF'
FROM alpine:3.20
COPY --chown=nobody blob.bin /opt/blob.bin
EOF
docker build -q -f Dockerfile.c -t demo-slim-c .
docker build -q -f Dockerfile.d -t demo-slim-d .
docker images 'demo-slim-[cd]'    # c stores the 50MB file twice, d once

Flattening works, but the config (CMD, ENV, PATH) is lost with it.

bash
docker create --name demo-flat-src demo-slim-a
docker export demo-flat-src | docker import - demo-flat:1
docker rm demo-flat-src
docker image inspect demo-slim-a --format 'layers={{len .RootFS.Layers}} cmd={{.Config.Cmd}}'
docker image inspect demo-flat:1 --format 'layers={{len .RootFS.Layers}} cmd={{.Config.Cmd}} env={{.Config.Env}}'

In practice

  • alpine, debian:bookworm-slim, python:3.13-slim — Common small base images; compare their CONTENT SIZE and libc (musl or glibc) before choosing.
  • Distroless and scratch bases — Typically used as multi-stage final stages for static binaries; no shell or package manager.
  • docker history, dive — `docker history` is built in; third-party tools such as dive visualize layer contents but are not required.
  • docker buildx build --output type=local — Exports the final filesystem to a directory so you can inspect exactly what is in the image.

Trade-offs

Pros

  • Faster pulls and pushes, which shortens deploys and autoscaling cold starts
  • Fewer packages means fewer CVEs to track and a smaller attack surface
  • Lower registry and node disk usage
  • Measuring with docker history exposes the real cause instead of guessing

Cons

  • Minimal bases remove tools you use while debugging
  • musl or missing libc can break binaries and wheels in surprising ways
  • Flattening gives up layer sharing between images and drops config unless re-applied
  • Over-merging RUN steps hurts readability and cache granularity

A tag is a movable name, a digest is the content hash: what each one pins, how multi-arch indexes add a second digest level, and how to pin a base image reproducibly.

The problem

A build that worked yesterday fails today with no code change, because `FROM python:3.13-slim` now resolves to a different image. Or two nodes run different bytes under the same tag. A tag is a name the registry owner can repoint at any time; a digest is a hash of the content. Reproducible builds and safe rollouts depend on knowing which of the two a reference actually uses.

How it works

  • A reference is `[registry/]repo[:tag][@sha256:digest]`. The tag defaults to `latest` when omitted, and `latest` is only a naming convention; it is not 'newest'. A tag is a mutable pointer stored by the registry (or locally). A digest is the SHA-256 of the manifest bytes and cannot be repointed.
  • Building the same tag twice moves it. Observed: `docker build -t demo-tag:1` after changing the Dockerfile left `demo-tag:1` pointing at a new image with a new digest and image ID, and the old image no longer carried the name. `docker tag SRC NEW` adds another name for the same image without copying data.
  • When a reference has both a tag and a digest, the digest wins and the tag is ignored. Observed: `FROM alpine:3.99@sha256:d9e8...` built successfully although `alpine:3.99` does not exist, and the log line read `FROM docker.io/library/alpine:3.99@sha256:d9e8...`. For readability write the real tag next to the digest, but know only the digest is enforced.
  • Multi-arch images have two levels. `alpine:3.20` resolves to an image index (`application/vnd.oci.image.index.v1+json`) with digest `sha256:d9e853e8...`; the index lists one manifest per platform, and `linux/amd64` has its own digest `sha256:c64c687c...`. `docker buildx imagetools inspect` shows the index and every entry. Pinning the index digest keeps every platform pinned; pinning a platform digest locks you to that architecture.
  • Where to read the digest: `docker image inspect IMG --format '{{index .RepoDigests 0}}'` for an image pulled from a registry, or `docker images --digests`. On Docker 29 with the containerd image store, the `Id` of the pulled alpine image was the same value as its index digest, whereas the classic store typically reports a separate image ID computed from the image config. Do not assume the two are interchangeable across engines.
  • On the containerd image store even a locally built image listed a RepoDigest (observed `demo-td@sha256:...`) without ever being pushed, so treat RepoDigests as meaningful only for images that came from or went to a registry. The digest that `docker push` prints is what deploy manifests should reference, because the same tag can be pushed again with different content.
  • Pinning has a cost: nothing updates the base image unless you do it. Pin by digest in production Dockerfiles and deploy manifests, and let an update tool or a scheduled job open pull requests that bump the digest, so security fixes arrive deliberately and are reviewable. Floating tags like `3.20` still receive rebuilt patch releases; `latest` can jump major versions.
  • BuildKit resolves the tag to a digest at build time and prints it (`FROM docker.io/library/alpine:3.20@sha256:...`) in the plain progress log, so the build log tells you what a floating tag meant on that day. The check happens against the registry on each build unless the reference is already pinned (hedge: the exact revalidation rules depend on builder settings and were not examined here).

Examples

Read the repo digest and pin it in FROM; the tag next to it is ignored, even a nonexistent one.

bash
docker pull -q alpine:3.20
docker image inspect alpine:3.20 --format '{{index .RepoDigests 0}}'
SHA=$(docker image inspect alpine:3.20 --format '{{index .RepoDigests 0}}' | cut -d@ -f2)
cat > Dockerfile <<EOF
FROM alpine:3.99@$SHA
RUN echo pinned
EOF
cat Dockerfile
docker build --progress=plain -t demo-td . 2>&1 | grep -E '\] FROM|ERROR'   # the nonexistent tag 3.99 is ignored

Index digest on top, one manifest per platform below it.

bash
docker buildx imagetools inspect alpine:3.20 | head -9   # index digest, then one manifest per platform
docker image inspect alpine:3.20 --format '{{.Descriptor.mediaType}} {{.Descriptor.digest}}'

A tag moves when you rebuild it; docker tag only adds a name.

bash
printf 'FROM alpine:3.20\nRUN echo one > /v\n' > Dockerfile
docker build -q -t demo-tag:1 .
printf 'RUN echo two > /v\n' >> Dockerfile
docker build -q -t demo-tag:1 .         # same tag now points at a different image
docker images --digests demo-tag
docker tag demo-tag:1 demo-tag:stable   # a second name for the same image, no copy

In practice

  • Docker Hub official images — Tags like `3.20` receive rebuilt images; each tag is an index with digests per platform.
  • Kubernetes pod specs — An `image: repo@sha256:...` field pins the content, so every node runs the same bytes regardless of tag moves; a typo in tag or digest ends in ImagePullBackOff.
  • OCI image spec — Defines manifests, image indexes and content-addressable digests that tags merely point to.
  • Dependency-update bots (Renovate, Dependabot) — Can bump a `tag@digest` reference so pins stay current through reviewable pull requests.

Trade-offs

Pros

  • Digest pins give byte-identical pulls on every host and CI run
  • Rollbacks are exact: redeploy the old digest
  • Tags stay human-readable names for humans and dashboards
  • Build logs record which digest a floating tag resolved to

Cons

  • Pinned bases never get security fixes unless an update process bumps them
  • Digests are unreadable and are a pain to compare by eye in review
  • Index digest versus platform digest confusion causes failures on other architectures
  • Tag semantics (latest, minor, patch) are a per-project convention, not enforced by Docker

Runtime & limits

A container is an ordinary process with its own namespaces (what it can see) and a cgroup (what it can use); here is where to read both on the host.

The problem

Someone says a container is a lightweight VM, then a process inside it cannot see the host's processes but a limit set on the container still throttles it, and nobody can explain why. The model is simpler and more useful: a container is a normal Linux process whose view is cut down by namespaces and whose consumption is capped by a cgroup. Once you know where both live on the host, you can inspect a container without Docker, enter just its network namespace with host tools, and explain every limit you set.

How it works

  • A namespace changes what a process can see. Docker creates a new set for each container: `mnt` (filesystem tree), `pid` (process IDs), `net` (interfaces, routes, ports), `ipc`, `uts` (hostname) and `cgroup`; on kernels with time namespaces there is also `time`. The `user` namespace is not created by default, so root in the container is root (UID 0) on the host unless `userns-remap` or rootless mode is on.
  • Every process exposes its namespaces as symlinks in `/proc/<pid>/ns/`. Two processes share a namespace exactly when the inode numbers in the link targets are equal, for example `pid:[4026532265]`. Comparing the host shell's `/proc/self/ns/net` with the container's `/proc/<pid>/ns/net` is the definitive test of whether `--network host` took effect.
  • The container's PID on the host is `docker inspect -f '{{.State.Pid}}' <name>`. Its first process is PID 1 inside the new PID namespace but an ordinary PID such as 1555 on the host, visible in host `ps`. `nsenter -t <pid> -n ip addr` runs a host binary inside only that net namespace, which is the standard way to debug a container image that ships no tools.
  • A cgroup changes what a process can consume. Docker creates one per container (cgroupfs driver: `/sys/fs/cgroup/<controller>/docker/<container-id>/` on cgroup v1) and writes your flags into its files: `memory.limit_in_bytes`, `cpu.cfs_quota_us`, `cpu.shares`, `pids.max`. The effective values can be read back there, and `docker inspect` shows the request while the cgroup file shows what the kernel enforces.
  • On cgroup v1 each controller (memory, cpu, pids, blkio, cpuset...) has its own tree under `/sys/fs/cgroup/<controller>/`. On cgroup v2 there is a single unified tree and the files are renamed: `memory.max`, `cpu.max`, `cpu.weight`, `pids.max`; with the systemd cgroup driver the path is typically `/sys/fs/cgroup/system.slice/docker-<id>.scope/` (cgroup v2). Check which one you run with `docker info -f '{{.CgroupVersion}}'`.
  • Inside the container, `/proc/<pid>/cgroup` shows the path of the cgroup, and `/sys/fs/cgroup` is mounted read-only with the container's own cgroup as its root, which is why the container can cat its own `memory.limit_in_bytes`. With `--cgroupns private` (the default on cgroup v2) the container gets its own cgroup namespace and sees `/` as its root; on cgroup v1 the default is `host`.
  • Namespaces are shared on purpose with flags: `--pid=host` (see all host processes), `--network=host`, `--ipc=host`, `--uts=host`, and `--pid=container:<name>` or `--network=container:<name>` to join another container's namespace. This is how sidecars and debug containers attach to a target without a new image. Every `host` setting removes one isolation boundary.
  • What namespaces and cgroups do not isolate: the kernel itself (containers share one kernel, so a kernel bug or a missing syscall filter affects all of them), `/proc/meminfo` and `/proc/cpuinfo` (they show the host unless a runtime like lxcfs fakes them), the clock (except time namespaces), and kernel-wide limits such as `fs.file-max`. Capabilities, seccomp and LSMs add the rest of the sandbox.

Examples

Read a container's namespaces and cgroup limits straight from the host, without docker exec.

bash
docker run -d --name demo-ns --memory 256m --cpus 1.5 --pids-limit 100 --cpu-shares 512 busybox sleep 600
PID=$(docker inspect -f '{{.State.Pid}}' demo-ns)
CID=$(docker inspect -f '{{.Id}}' demo-ns)

# What it can see: one symlink per namespace; compare inode numbers with the host
ls -l /proc/$PID/ns
readlink /proc/self/ns/pid /proc/$PID/ns/pid

# Enter only its network namespace, using the host's own ip binary
nsenter -t $PID -n ip -br addr

# What it can use (cgroup v1 paths; v2 uses memory.max, cpu.max, cpu.weight)
cat /sys/fs/cgroup/memory/docker/$CID/memory.limit_in_bytes   # 268435456
cat /sys/fs/cgroup/cpu/docker/$CID/cpu.cfs_quota_us           # 150000
cat /sys/fs/cgroup/cpu/docker/$CID/cpu.shares                 # 512
cat /sys/fs/cgroup/pids/docker/$CID/pids.max                  # 100
docker rm -f demo-ns

Namespaces are a kernel feature; Docker only picks which ones to share.

bash
# A PID namespace without Docker: this ps sees only itself
unshare --pid --fork --mount-proc sh -c 'ps; readlink /proc/self/ns/pid'

# Join another container's PID namespace
docker run -d --name demo-a busybox sleep 600
docker run --rm --pid=container:demo-a busybox ps     # demo-a's sleep is PID 1
docker run --rm --network host busybox readlink /proc/self/ns/net
readlink /proc/self/ns/net                            # same inode as above
docker rm -f demo-a

In practice

  • runc — The OCI runtime reads the container's config.json, calls clone/unshare with the requested namespace flags and writes the limits into the cgroup before exec-ing the entrypoint.
  • Kubernetes pods — Containers in a pod share the net and IPC namespaces through a pause container; `shareProcessNamespace: true` adds a shared PID namespace, the same mechanism as `--pid=container:`.
  • Debug containers — `docker run --pid=container:<name> --network=container:<name>` with a tools image is how you attach strace or tcpdump to an image that has none.
  • Rootless Docker and userns-remap — Add the user namespace so that UID 0 in the container maps to an unprivileged UID on the host, reducing what a container escape can do.

Trade-offs

Pros

  • Start-up is a few syscalls rather than booting a kernel, so containers start in tens to hundreds of milliseconds
  • Everything is plain files and symlinks under /proc and /sys, so a container can be inspected and debugged with ordinary host tools
  • Isolation is composable: share exactly one namespace (net, pid) with another container instead of all or nothing
  • Limits can be changed on a live container (`docker update`) because they are just cgroup file writes

Cons

  • One shared kernel means a kernel exploit or an unfiltered syscall crosses every container boundary; this is weaker than a VM boundary
  • Not everything is namespaced or accounted: /proc/meminfo, /proc/cpuinfo and kernel-wide limits show host values
  • cgroup v1 and v2 differ in file names and in swap and OOM semantics, so tooling and runbooks written for one break on the other
  • Sharing host namespaces (--pid=host, --network=host) quietly removes the isolation the rest of your hardening assumes

Why docker stop takes 10 seconds and ends in exit 137, why shell-form CMD never sees SIGTERM, and what --init adds.

The problem

docker stop hangs for exactly ten seconds and the container exits with 137 instead of shutting down cleanly, so in-flight requests are cut and buffers are lost. Or a CI container is stopped with Ctrl-C and ignores it. Or after a week the PID limit is hit because thousands of defunct processes pile up. All three trace back to the same fact: the first process in a container is PID 1, and the kernel and Docker treat PID 1 differently from every other process.

How it works

  • The kernel only delivers a signal to PID 1 of a PID namespace if PID 1 installed a handler for it; the default action (terminate) is not applied to it. A SIGTERM or SIGINT sent to a script or binary that never calls `signal()` or `trap` is therefore ignored. Even `kill -9 1` run inside the container does not kill it (checked: the container kept running); SIGKILL only works from the parent namespace, which is what Docker uses.
  • `docker stop` sends the image's STOPSIGNAL (or `--stop-signal`, default SIGTERM) to PID 1, waits 10 seconds (`-t`, `--stop-timeout`, Compose `stop_grace_period`), then sends SIGKILL. A PID 1 that ignores SIGTERM therefore costs the full timeout and ends with exit code 137 (128 + 9); measured here at 10.2 s with `docker stop` and 3.2 s with `-t 3`.
  • Shell form (`CMD /app/run`, `ENTRYPOINT /app/run`) runs `/bin/sh -c '/app/run'`, so the shell is PID 1 and your program is its child. A shell does not forward SIGTERM to children, and as PID 1 it has no handler, so the signal is swallowed. Checked with `CMD /loop.sh && echo done`: `ps` showed `/bin/sh -c ...` as PID 1 and stop ended with 137. Some shells exec a lone final command and hide the problem; compound commands expose it.
  • Exec form (`CMD ["/app/run"]`) makes your program PID 1. A wrapper entrypoint must hand over with `exec "$@"`; without `exec` the script stays PID 1 and the app is a child. With `exec`, `ps` showed the real program as PID 1 (checked), and now the app itself must handle SIGTERM: a shell script with `trap 'cleanup; exit 0' TERM` stopped in 0.16 s with exit 0.
  • Runtimes differ in what they do with SIGTERM when they are PID 1: a Go program needs `signal.Notify`, a Node.js process typically needs a `process.on("SIGTERM")` handler, and a Python process typically needs `signal.signal`. Without one, the default action is skipped for PID 1 and the process keeps running until the SIGKILL. Servers such as nginx or PostgreSQL handle signals themselves, and their images pick the right one (STOPSIGNAL SIGQUIT and SIGINT respectively).
  • `--init` (Compose `init: true`, daemon `init: true`) mounts Docker's bundled init binary (a tini build, `/sbin/docker-init`) as PID 1 and runs your command as its child. It forwards signals to the child and reaps zombies. Checked: `ps` showed `/sbin/docker-init -- /loop.sh`, and `docker stop` took 0.155 s with exit 143 (128 + 15), because the child's default SIGTERM action now applies.
  • A zombie is a child that exited but whose parent has not called `wait()`; it keeps a PID table entry. Orphaned processes are re-parented to PID 1, so if PID 1 never reaps, every orphan stays `Z` forever. Checked: a backgrounded `sleep` in `docker exec` showed up as `Z` with PPID 1 under a plain `sleep 600` PID 1, and was gone with `--init`. Zombies consume PIDs against `--pids-limit`.
  • Exit codes tell you who ended the process: 143 means it died of SIGTERM (nobody handled it, or it re-raised), 137 means SIGKILL (stop timeout, `docker kill`, or the OOM killer: check `State.OOMKilled`), 0 or a small number means the app handled SIGTERM and chose its own code. Set `STOPSIGNAL` or `--stop-signal` when the app's graceful signal is not SIGTERM, and raise the timeout for slow drains.

Examples

A shell loop as PID 1 swallows SIGTERM (10 s, exit 137); with --init it dies in about 0.15 s (exit 143).

bash
cat > loop.sh <<'EOF'
#!/bin/sh
while true; do sleep 1; done
EOF
chmod +x loop.sh
cat > Dockerfile <<'EOF'
FROM busybox
COPY loop.sh /loop.sh
CMD ["/loop.sh"]
EOF
docker build -q -t demo-loop .

# PID 1 is the script, which has no SIGTERM handler: waits the full 10 s
docker run -d --name demo-p1 demo-loop
time docker stop demo-p1
docker inspect -f '{{.State.ExitCode}}' demo-p1       # 137
docker rm demo-p1

# docker-init is PID 1 and forwards SIGTERM: stops at once
docker run -d --init --name demo-p1 demo-loop
docker exec demo-p1 ps                                # /sbin/docker-init -- /loop.sh
time docker stop demo-p1
docker inspect -f '{{.State.ExitCode}}' demo-p1       # 143
docker rm demo-p1

Reproduce a zombie with a plain PID 1, then see --init reap it; the entrypoint ends with exec.

bash
# Orphans are re-parented to PID 1; a PID 1 that never wait()s leaves zombies
docker run -d --name demo-z busybox sleep 600
docker exec demo-z sh -c '(sleep 0.2 &); sleep 1; ps -o pid,ppid,stat,args'
# ... a row with STAT Z and PPID 1 remains
docker rm -f demo-z

docker run -d --init --name demo-z busybox sleep 600
docker exec demo-z sh -c '(sleep 0.2 &); sleep 1; ps -o pid,ppid,stat,args'
# ... no Z row: docker-init reaped it
docker rm -f demo-z

# Wrapper entrypoints must exec, or the wrapper stays PID 1
cat > entry.sh <<'EOF'
#!/bin/sh
echo "preparing"
exec "$@"
EOF

Compose equivalents: --init, --stop-signal and --stop-timeout.

yaml
services:
  api:
    image: busybox
    command: ["sleep", "600"]
    init: true
    stop_signal: SIGINT
    stop_grace_period: 30s

In practice

  • tini / docker-init — A tiny init that forwards signals and reaps zombies; Docker bundles it behind `--init`, and many images install tini as ENTRYPOINT to get the same on Podman or Kubernetes.
  • Official images (nginx, postgres) — Set STOPSIGNAL (SIGQUIT for nginx graceful stop, SIGINT for PostgreSQL fast shutdown) and end docker-entrypoint.sh with `exec "$@"` so the server becomes PID 1.
  • Kubernetes — terminationGracePeriodSeconds plays the role of `docker stop -t`; the kubelet sends SIGTERM to PID 1 of each container, so the same PID 1 rules apply, and there is no init unless the image has one.
  • Process managers (supervisord, s6-overlay) — Used as PID 1 when one container must run several processes; they reap children and decide how to propagate the stop signal.

Trade-offs

Pros

  • `--init` fixes signal forwarding and zombie reaping without changing the image or the application
  • Exec form plus a real signal handler gives a clean shutdown with the app as PID 1 and nothing extra in the image
  • `STOPSIGNAL` and the stop timeout let each service choose its own graceful-shutdown contract
  • Exit codes 137 and 143 are a quick fingerprint of how the process ended

Cons

  • `--init` is a runtime flag, not part of the image, so a container started without it (a Compose file that forgot `init: true`) silently reverts to the old behavior
  • init forwards the signal but does not wait for the whole process tree; a double-forking app can still be left running until SIGKILL
  • Long stop timeouts delay deploys and rollouts; short ones cut off in-flight work
  • Shell wrappers are convenient (env substitution, setup) and break signal delivery unless every one ends in `exec`

What --memory, --memory-swap, --cpus and --cpu-shares write into the cgroup, how an OOM kill shows up in State.OOMKilled, and why the container still sees the host's CPUs.

The problem

A service is killed every few hours with exit 137 and no error in its own logs. Another one runs at half the speed it did on a laptop, and its CPU graph is suspiciously flat at 100%. A third reads 16 GB from /proc/meminfo and sizes its heap for a host it does not own. All of these come from limits that Docker writes into cgroup files, and from what the kernel does when the container reaches them.

How it works

  • `--memory 64m` writes `memory.limit_in_bytes` (cgroup v1) or `memory.max` (cgroup v2). The limit covers anonymous memory, page cache and kernel memory charged to the cgroup; when usage reaches it the kernel first reclaims cache, and only if that fails does the OOM killer run inside that cgroup. The minimum is 6m (`Minimum memory limit allowed is 6MB`).
  • `--memory-swap` is the total of memory plus swap, not the swap alone. Unset, Docker allows twice `--memory` (checked: 64m gives `memory.memsw.limit_in_bytes` of 134217728); equal to `--memory` means no swap; `-1` means unlimited swap. Setting it without `--memory` is rejected (`You should always set the Memory limit when using Memoryswap limit`), and a value below `--memory` is rejected too. In cgroup v2 it maps to `memory.swap.max` as the difference (cgroup v2). Hosts with no swap make this moot.
  • When the cgroup runs out of memory the kernel kills one process with the highest OOM score, logs `Memory cgroup out of memory: Killed process <pid> (<comm>)` to `dmesg`, and Docker records an `oom` event and sets `State.OOMKilled=true`. If the victim was PID 1, the container exits with 137 (checked). If the victim was a child, the container can live on and even exit 0 while `OOMKilled` is still true (checked), so the flag means 'an OOM kill happened', not 'the container died of it'.
  • `--memory-reservation` is a soft limit (`memory.soft_limit_in_bytes`; `memory.low` on cgroup v2, which is a different mechanism) that the kernel only enforces when the host is short of memory; it must be lower than `--memory` (`Minimum memory limit can not be less than memory reservation limit`). `--oom-kill-disable` needs a `--memory` limit (otherwise Docker warns), and when the cgroup is full the allocating process is frozen instead of killed: checked as `State.Status=running`, `OOMKilled=true`, cgroup `under_oom 1`, and the process never completes. It is typically unavailable on cgroup v2.
  • `--cpus 1.5` is a CFS bandwidth quota: `cpu.cfs_quota_us=150000` per `cpu.cfs_period_us=100000` on v1, or `cpu.max` = `150000 100000` on v2. It limits CPU time per 100 ms window, not which cores run. A multithreaded process can burn the quota early in the window and then stall until the next one; `cpu.stat` shows this as `nr_throttled` and `throttled_time` (checked: a busy loop with `--cpus 0.5` throttled in 50 of 51 periods and `docker stats` showed 49.95%). The value cannot exceed the host's CPUs, and `--cpu-quota` cannot be combined with `--cpus`.
  • `--cpu-shares` writes `cpu.shares` (default 1024; cpu.weight on cgroup v2, converted) and is only a relative weight under contention: two containers at 512 and 1024 split a fully busy CPU 1:2, but a lone container with 512 can use every idle CPU. It is a priority, not a limit. `--cpuset-cpus 0` pins the container to CPU 0 through `cpuset.cpus` and is the only CPU flag that changes what the process sees via affinity (`nproc` printed 1).
  • `nproc`, `/proc/cpuinfo` and `/proc/meminfo` inside the container keep showing the host (checked: `--cpus 0.5 --memory 64m` still gave `nproc` 4 and the host's MemTotal), while the real limit is in the cgroup files. Software that sizes thread pools, worker counts or heaps from those numbers over-allocates: JVMs have read the cgroup for years, but older Go runtimes (before 1.25) set GOMAXPROCS from the host CPU count, and many other runtimes need an explicit setting.
  • `--pids-limit` writes `pids.max` (default `max`, checked): once reached, `fork` fails with `Resource temporarily unavailable`. `--ulimit nofile=256:512` sets soft:hard rlimits for processes in the container, not a cgroup, and the default is inherited from the daemon (20000 on this host). `docker update` changes memory, CPU and pids limits of a running container; a new `--memory` below current usage can trigger an immediate OOM kill.

Examples

Reproduce an OOM kill. The second run shows OOMKilled=true with exit code 0.

bash
# OOM kill of PID 1's pipeline: exit 137 and OOMKilled=true
docker run --name demo-oom --memory 64m --memory-swap 64m busybox \
  sh -c 'head -c 200m /dev/zero | tail'
echo $?                                               # 137
docker inspect -f 'exit={{.State.ExitCode}} oom={{.State.OOMKilled}}' demo-oom
dmesg | grep -i "memory cgroup out of memory" | tail -1
docker rm demo-oom

# Victim is a child: the shell survives, exit 0, OOMKilled is still true
docker run --name demo-oom2 --memory 64m busybox \
  sh -c 'head -c 200m /dev/zero | tail > /dev/null; echo "shell survived rc=$?"'
docker inspect -f 'exit={{.State.ExitCode}} oom={{.State.OOMKilled}}' demo-oom2
docker rm demo-oom2

Throttling, the host-sized nproc, live updates and the validation errors (cgroup v1 paths).

bash
docker run -d --name demo-c --cpus 0.5 --memory 64m busybox sh -c 'while :; do :; done'
sleep 5
CID=$(docker inspect -f '{{.Id}}' demo-c)
cat /sys/fs/cgroup/cpu/docker/$CID/cpu.cfs_quota_us    # 50000
cat /sys/fs/cgroup/cpu/docker/$CID/cpu.cfs_period_us   # 100000
cat /sys/fs/cgroup/cpu/docker/$CID/cpu.stat            # nr_throttled grows
docker stats --no-stream --format '{{.Name}} {{.CPUPerc}} {{.MemUsage}}' demo-c
docker exec demo-c nproc                               # host CPU count, not 0.5
docker update --cpus 1 demo-c
cat /sys/fs/cgroup/cpu/docker/$CID/cpu.cfs_quota_us    # 100000
docker rm -f demo-c

docker run --rm --memory-swap 256m busybox true        # rejected: needs --memory
docker run --rm --memory 64m --memory-reservation 128m busybox true   # rejected

The same limits in Compose. `deploy.resources.limits` is also applied by docker compose, outside Swarm.

yaml
services:
  worker:
    image: busybox
    command: ["sleep", "600"]
    mem_limit: 128m
    memswap_limit: 128m
    mem_reservation: 64m
    cpus: 0.5
    pids_limit: 100
    ulimits:
      nofile:
        soft: 1024
        hard: 2048

In practice

  • Kubernetes — `resources.limits.memory` becomes the same cgroup memory limit and `limits.cpu` the CFS quota, so a pod throttled at its CPU limit and an OOMKilled pod are the same mechanics as above; requests map to shares/weight.
  • JVM and Go services — Container-aware JVMs size the heap from the cgroup limit (`-XX:MaxRAMPercentage`); Go before 1.25 needs GOMAXPROCS set (or automaxprocs) to match `--cpus`.
  • cAdvisor / docker stats — Report usage against the cgroup limit; `docker stats` memory usage excludes page cache (checked: 3.5 MiB shown while `memory.usage_in_bytes` was about 104 MB after writing a 100 MB file), yet that cache still counts toward the limit.
  • systemd-oomd and earlyoom — Host-level killers that act before the kernel's cgroup OOM and can end a container whose own limit was never reached; the cause then is host memory, not `--memory`.

Trade-offs

Pros

  • A memory limit turns a leaking container into one dead container instead of a starved host
  • `--cpus` gives a predictable ceiling per container, which makes noisy neighbours bounded
  • `--cpu-shares` and `--memory-reservation` allow overcommit while keeping a priority order under pressure
  • All limits are plain cgroup files, so they are observable (`cpu.stat`, `memory.failcnt`) and changeable at runtime with `docker update`

Cons

  • A hard memory limit kills without warning and without a stack trace; the only evidence is exit 137, `OOMKilled` and a dmesg line
  • CFS quota throttling hurts tail latency of bursty or multithreaded services even when average CPU is far below the limit
  • The container still sees host CPUs and memory, so unaware runtimes size themselves wrongly
  • Page cache counts toward `--memory`, so a file-heavy service can approach the limit with little real heap

What --restart actually reacts to, the backoff between restarts, and why an unhealthy status never restarts a plain Docker container.

The problem

A container is marked unhealthy for hours and nothing happens, because plain Docker does not act on health. Another one with `--restart always` crash-loops in a tight cycle, floods the logs and restarts again after you stopped it or rebooted. A deploy script declares success because the container is `running` while the app inside is still migrating. Restart policy and healthcheck are two separate mechanisms, and most surprises come from assuming one drives the other.

How it works

  • `--restart` takes `no` (default), `on-failure[:max-retries]`, `always` and `unless-stopped`. The Docker daemon implements it, not containerd or runc: it watches for the container's main process to exit and starts it again in the same container, keeping the filesystem, ID and name. `docker inspect` shows the policy under `HostConfig.RestartPolicy` and the counter as `RestartCount`.
  • `on-failure` restarts only on a non-zero exit; exit 0 ends it. The optional `:N` caps restarts, and the first start is not counted. Checked with `--restart on-failure:3` and a command that always exits 3: the container ran 4 times (4 log lines), `RestartCount` was 3, and it ended `exited` with `ExitCode 3`.
  • Between restarts the daemon waits with an exponentially increasing delay, starting at 100 ms and doubling each time; the delay resets after the container ran for about 10 seconds (documented behavior, delay reset not tested). Checked with `--restart always` on a command that exits at once: 7 restarts in 12 seconds with gaps growing from about 0 to 4 s, and the status shown as `restarting`. Without the cap on retries, a crash loop never stops.
  • `docker stop` and `docker kill` both count as a manual stop: the container stays down under `always` and `unless-stopped` (checked, `RestartCount` stayed 0 after `docker kill`). Documented difference: after the daemon restarts, `always` starts the container again, while `unless-stopped` does not if you had stopped it (not tested here, as it needs a daemon restart). `docker update --restart on-failure:2 <name>` changes the policy of an existing container (checked), and `--restart` cannot be combined with `--rm`.
  • A healthcheck is a command that the engine runs with the equivalent of `docker exec` inside the container: `--health-cmd`, `--health-interval`, `--health-timeout`, `--health-retries`, `--health-start-period` (Dockerfile `HEALTHCHECK`, Compose `healthcheck`). Documented defaults are interval 30 s, timeout 30 s, retries 3, start period 0 s (defaults not re-verified). Exit 0 is success, 1 is failure, and 2 is reserved; exit 2 was also recorded as failing (checked).
  • Status flow: `starting` -> `healthy` after a success, `unhealthy` after `retries` consecutive failures (`State.Health.FailingStreak`). Failures during `--health-start-period` do not count toward the retries, but a success during it already makes the container healthy. Checked: a server that needed 5 s with a 2 s start period went `starting` -> `unhealthy` after 4 s -> `healthy` after 6 s. `docker ps` shows `(health: starting)`, `(healthy)` or `(unhealthy)`.
  • A check that exceeds the timeout is recorded with `ExitCode -1` and the output `Health check exceeded timeout (1s)` (checked). `State.Health.Log` keeps the last 5 results with start time, end time, exit code and up to 4 KB of output, and `docker events` gets a `health_status` event on each change plus an `exec_create`/`exec_die` pair for every probe, which is noisy.
  • Docker Engine itself never acts on `unhealthy`: checked, a container stayed `running` with `Health.Status=unhealthy`. Health matters to whoever reads it: `docker compose up --wait`, `depends_on: condition: service_healthy`, load balancers and monitoring, or an autoheal sidecar. The probe needs its binary (curl, wget) inside the image, so distroless images need an in-app probe command, and the probe's process counts against the container's cgroup.

Examples

Restart policy behavior: retry cap, exit code kept, manual kill not restarted, --rm conflict.

bash
# on-failure:3 -> 4 runs, RestartCount 3, ends exited with the last exit code
docker run -d --name demo-rs --restart on-failure:3 busybox sh -c 'echo run; exit 3'
sleep 8
docker inspect -f 'status={{.State.Status}} restarts={{.RestartCount}} exit={{.State.ExitCode}}' demo-rs
docker logs demo-rs | wc -l                           # 4
docker rm -f demo-rs

# A manual kill is not restarted, even with --restart always
docker run -d --name demo-rs --restart always busybox sleep 600
docker kill demo-rs; sleep 2
docker inspect -f '{{.State.Status}} restarts={{.RestartCount}}' demo-rs   # exited restarts=0
docker rm -f demo-rs

docker run --rm --restart always busybox true         # error: cannot specify both --restart and --rm

Observe the healthcheck states, then confirm that unhealthy leaves the container running.

bash
cat > Dockerfile <<'EOF'
FROM busybox
HEALTHCHECK --interval=2s --timeout=1s --start-period=2s --retries=2 \
  CMD wget -q -O /dev/null http://127.0.0.1:8080/ || exit 1
CMD ["sh", "-c", "mkdir -p /www && echo ok > /www/index.html; sleep 5; exec httpd -f -p 8080 -h /www"]
EOF
docker build -q -t demo-health .
docker run -d --name demo-h demo-health
for i in 1 2 3 4; do sleep 2; docker ps --filter name=demo-h --format '{{.Status}}'; done
# (health: starting) -> (unhealthy) -> (healthy): the 5 s boot outlasts the 2 s start period

docker inspect -f '{{json .Config.Healthcheck}}' demo-h
docker inspect -f '{{.State.Health.Status}} streak={{.State.Health.FailingStreak}}' demo-h
docker rm -f demo-h

# Unhealthy does not restart anything: the container stays running
docker run -d --name demo-h --health-cmd 'test -f /tmp/ok' --health-interval 1s --health-retries 2 busybox sleep 600
sleep 4
docker inspect -f '{{.State.Status}} {{.State.Health.Status}}' demo-h   # running unhealthy
docker rm -f demo-h

Compose: restart policy, a healthcheck, and a dependent service that waits for service_healthy.

yaml
services:
  web:
    image: busybox
    command: ["sh", "-c", "mkdir -p /www && echo ok > /www/index.html && exec httpd -f -p 8080 -h /www"]
    restart: unless-stopped
    healthcheck:
      test: ["CMD", "wget", "-q", "-O", "/dev/null", "http://127.0.0.1:8080/"]
      interval: 5s
      timeout: 2s
      retries: 3
      start_period: 5s
  client:
    image: busybox
    command: ["sleep", "600"]
    depends_on:
      web:
        condition: service_healthy

In practice

  • Docker Compose — `up --wait` and `depends_on: condition: service_healthy` are the main consumers of the health status on a single host.
  • Kubernetes probes — A liveness probe failure makes the kubelet restart the container, which plain Docker never does for `unhealthy`; Kubernetes ignores the image's HEALTHCHECK and uses its own probes.
  • Official images (postgres, mysql, redis) — Usually ship no HEALTHCHECK; teams add `pg_isready`, `mysqladmin ping` or `redis-cli ping` in Compose.
  • Autoheal sidecars — Small containers that watch `docker events` for `health_status: unhealthy` and call `docker restart`, filling the gap that Docker leaves open.

Trade-offs

Pros

  • `--restart` recovers from crashes and host reboots with one flag and no external supervisor
  • The exponential backoff keeps a crash loop from pinning a CPU at full speed
  • A healthcheck separates 'process is running' from 'service is answering', which `docker ps` and Compose can use
  • `State.Health.Log` stores the last probe outputs, so the reason for a failure is one inspect away

Cons

  • `always` and `on-failure` hide a crash loop behind a `running`/`restarting` status; the real signal is `RestartCount` rising
  • A restart reuses the same container, so a corrupted writable layer or a leaked file survives it
  • Unhealthy changes nothing by itself, and teams often discover that only during an incident
  • A healthcheck costs a process every interval and tests only what you wrote; a shallow `GET /` can report healthy while the database is down

How stdout and stderr become files or network streams, why json-file grows without limit by default, and what blocking versus non-blocking delivery does to the app.

The problem

The host disk fills up and `du` points at /var/lib/docker/containers: one chatty container wrote a 40 GB json log because Docker's default driver never rotates. After a log shipper is added, a slow remote endpoint makes the application itself stall on its own print statements. And the file the application writes to /var/log/app.log is invisible to `docker logs`. Log handling in Docker is a driver choice made per container, with defaults that are unsafe on a long-running host.

How it works

  • Docker captures only the container's stdout and stderr pipes (or the TTY) and passes every line to a logging driver. Writes to files inside the container go to the writable layer and are invisible to `docker logs`; the usual fix is to log to stdout, or to symlink the log file to `/dev/stdout` (checked: a write to the symlink appeared in `docker logs`).
  • The default driver `json-file` appends one JSON object per line, `{"log":"...\n","stream":"stdout","time":"..."}`, to `/var/lib/docker/containers/<id>/<id>-json.log`; `docker inspect -f '{{.LogPath}}'` prints the path. By default there is no rotation: `HostConfig.LogConfig.Config` is `{}` (checked), so the file grows until the container is removed.
  • `--log-opt max-size=10m --log-opt max-file=3` rotates by size and keeps 3 files (`<id>-json.log`, `.log.1`, `.log.2`). Checked with 10k and 3: after 2000 lines only about 225 remained, in files of about 10 KB each, and `docker logs` returned just those, so rotation also deletes history. `compress=true` gzips rotated files (`<id>-json.log.1.gz`, checked). Options apply to containers created after they are set; an existing container's log config cannot be changed.
  • Daemon-wide defaults live in `/etc/docker/daemon.json` as `"log-driver"` and `"log-opts"` and need a daemon restart; they affect only containers created afterwards, and a container keeps the config it was created with. `docker info -f '{{.LoggingDriver}}'` shows the daemon default and `docker inspect` the per-container value. Compose sets it per service under `logging:`.
  • The `local` driver stores logs in a compact binary format (`container.log` under `local-logs/`) and is meant as a safer default: the documentation gives it a default of 20m and 5 files with compression, not re-verified here. `docker logs` works and `LogPath` is empty (checked), but the file is not plain JSON, so `tail` and `jq` on the host no longer work.
  • `docker logs` needs a driver that can read back. `none` fails with `configured logging driver does not support reading` (checked). For remote drivers such as syslog, fluentd, gelf, splunk or awslogs, Docker typically keeps a local copy so that `docker logs` still works (dual logging, since 20.10): checked with syslog sent to a UDP port nobody listens on, where `docker logs` still printed the line.
  • Remote drivers connect when the container starts: `--log-driver syslog --log-opt syslog-address=tcp://127.0.0.1:5514` and the same for fluentd fail `docker run` with `failed to initialize logging driver: dial tcp ...: connection refused` (checked). For fluentd, `fluentd-async=true` lets the container start without the endpoint (checked). UDP destinations such as the one above do not fail, since nothing is connected, and silently lose data.
  • Delivery is `mode=blocking` by default: when the driver is slow, the application's write to stdout blocks and the app stalls. `--log-opt mode=non-blocking --log-opt max-buffer-size=4m` puts a ring buffer in between and drops messages when it is full; the options are accepted (checked), the drop behavior is as documented and was not forced here. `labels`, `env` and `tag` options add attributes to each record: checked, a `--label app=web` plus `labels=app` produced `"attrs":{"app":"web",...}` in the JSON line.

Examples

json-file rotation keeps only max-file x max-size of history; the default keeps everything.

bash
docker run -d --name demo-log --log-opt max-size=10k --log-opt max-file=3 busybox sh -c '
  i=0; while [ $i -lt 2000 ]; do echo "line $i padding padding padding padding padding"; i=$((i+1)); done
  sleep 600'
sleep 3
docker inspect -f '{{json .HostConfig.LogConfig}}' demo-log
LP=$(docker inspect -f '{{.LogPath}}' demo-log)
ls -l $(dirname $LP) | grep -- -json.log                    # -json.log, .log.1, .log.2 (~10 KB each)
docker logs demo-log | wc -l                          # a few hundred, not 2000
docker rm -f demo-log

# Default config has no rotation at all
docker run -d --name demo-log busybox sleep 60
docker inspect -f '{{json .HostConfig.LogConfig}}' demo-log   # {"Type":"json-file","Config":{}}
docker rm -f demo-log

Remote drivers fail at container start; none cannot be read back; daemon.json sets the default.

bash
# Driver errors at start vs. with fluentd-async
docker run --rm --log-driver syslog --log-opt syslog-address=tcp://127.0.0.1:5514 busybox echo hi
# docker: ... failed to initialize logging driver: dial tcp 127.0.0.1:5514: connection refused
docker run --rm --log-driver fluentd --log-opt fluentd-address=127.0.0.1:24224 \
  --log-opt fluentd-async=true busybox echo hi

# No reader: docker logs refuses
docker run -d --name demo-log --log-driver none busybox sleep 60
docker logs demo-log                                  # configured logging driver does not support reading
docker rm -f demo-log

# Daemon-wide default (restart dockerd; only affects new containers)
cat > daemon.json <<'EOF'
{
  "log-driver": "local",
  "log-opts": { "max-size": "20m", "max-file": "5" }
}
EOF

Per-service rotation, compression and extra attributes in Compose.

yaml
services:
  app:
    image: busybox
    command: ["sh", "-c", "while true; do echo tick; sleep 5; done"]
    labels:
      app: web
    logging:
      driver: json-file
      options:
        max-size: "10m"
        max-file: "3"
        compress: "true"
        labels: "app"
        tag: "{{.Name}}"

In practice

  • Docker Engine defaults — `json-file` without rotation is the out-of-the-box setting, which is why a missing `max-size` is a classic cause of full /var/lib/docker on long-lived hosts.
  • Fluent Bit / Fluentd / Vector — Either run as a sidecar or host agent that tails the `*-json.log` files, or receive records through the fluentd/gelf driver; tailing keeps the app independent of the collector's availability.
  • journald driver — Sends records to systemd-journald so `journalctl CONTAINER_NAME=<name>` and the journal's own rotation apply; useful where journald is already the log pipeline.
  • Kubernetes — The runtime writes container logs as files that the kubelet rotates by `containerLogMaxSize` and `containerLogMaxFiles`; the idea (rotate by size and count) is the same as json-file options.

Trade-offs

Pros

  • Logging to stdout needs no log files, rotation scripts or volume mounts in the application
  • `json-file` with `max-size` and `max-file` bounds disk use per container with two options
  • `docker logs` works across restarts of the container and, with dual logging, also with remote drivers
  • Labels, env and tag options attach container identity to each record without app changes

Cons

  • Without options, `json-file` grows unbounded, and deleting the file while dockerd holds it open does not free the space until it closes it
  • Rotation discards history by design: a short window of logs is all `docker logs` can show after a burst
  • A blocking driver ties application latency to the log endpoint; non-blocking trades that for dropped lines
  • Driver and options are fixed at container creation, so changing them means recreating the container

Networking & storage

What bridge, user-defined bridge, host, none, container:<id>, macvlan and overlay each attach to the container's network namespace, and which one gives name resolution and isolation.

The problem

One container cannot reach another by name, a service listening on 127.0.0.1 in one container is invisible to its sidecar, and someone switched to --network host to make it work and now two services fight over port 80. Every one of these is a question of what the container's network namespace is attached to. The mode chosen at docker run decides addressing, isolation, name resolution and whether -p means anything.

How it works

  • Every container gets its own network namespace that starts with only a loopback interface; --network decides what is plugged into it. Values: `bridge` (default), a user-defined network name, `host`, `none`, `container:<name|id>`, or a driver such as macvlan, ipvlan or overlay.
  • Default bridge: the `docker0` Linux bridge with subnet 172.17.0.0/16 and a veth pair per container. Outbound traffic leaves through `-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE`. There is no DNS by container name here: `nslookup` of a neighbour's name returns NXDOMAIN, and `--link` only works with a deprecation warning.
  • User-defined bridge (`docker network create`): a separate `br-<first 12 chars of the network id>` bridge on the next free subnet (172.18.0.0/16 on a fresh host), plus the embedded DNS server at 127.0.0.11, so containers resolve each other by name and alias. This is what Compose creates per project (`<project>_default`). Isolation between bridges is an iptables rule, not just a different subnet: `-A DOCKER ! -i br-xxx -o br-xxx -j DROP`. A container on the default bridge pinging the IP of a container on a user-defined network gets 100% packet loss. Add a container to several networks with `docker network connect`; it gets one interface per network.
  • `--network host` skips namespace creation: the process sees the host's interfaces (lo, eth0, docker0, br-*), binds host ports directly and has the host's hostname. There is no NAT hop, so it is the lowest-overhead mode, but ports collide with host services, `-p` is discarded (`WARNING: Published ports are discarded when using host network mode`) and `docker inspect` shows no IP.
  • `--network none` leaves only `lo`: no route, and `wget http://1.1.1.1` fails with `Network is unreachable`. Use it for batch jobs that must not touch the network, or when you attach interfaces yourself. An internal network (`docker network create --internal`) is the softer version: containers talk to each other but there is no default route out.
  • `--network container:<name>` joins another container's namespace: same eth0 and IP, and `127.0.0.1:8080` reaches the other container's listener. Because the namespace belongs to the target, `--hostname`, `-p` and `--dns` are rejected with `conflicting options: ... and the container type network mode`. This is the sidecar pattern, and what a Kubernetes pod does with its pause container.
  • macvlan gives each container its own MAC and an address on the physical LAN (`-d macvlan --subnet ... --gateway ... -o parent=eth0`); ipvlan shares the parent's MAC (`ipvlan_mode=l2` or `l3`; this kernel refused to create an ipvlan port, so it is untested here). Names resolve on a macvlan network through the usual 127.0.0.11 resolver. The host typically cannot reach its own macvlan containers through the parent NIC (a container ping to the host's IP got 100% loss here), and the switch may limit MACs per port. `overlay` spans hosts but needs Swarm: on a non-swarm engine `docker network create -d overlay` fails with `This node is not a swarm manager`.
  • Read the address from `docker inspect -f '{{index .NetworkSettings.Networks "mynet" "IPAddress"}}' <c>`. On Engine 29 (checked on 29.8.2) there is no top-level `.NetworkSettings.IPAddress`; the legacy top-level fields are deprecated since API 1.21 and dropped in recent API versions (the template errors with `map has no entry for key "IPAddress"`), and a network name with a hyphen needs `index` because `.Networks.my-net` is a template syntax error.

Examples

Name resolution exists on a user-defined bridge and not on the default one; the inspect path is per network.

bash
docker network create appnet
docker run -d --name db --network appnet busybox sleep 600
docker run -d --name solo busybox sleep 600            # default bridge

docker run --rm --network appnet busybox ping -c1 db   # resolves via 127.0.0.11
docker exec solo ping -c1 db                           # ping: bad address 'db'

# Engine 29: the address lives under the per-network entry
docker inspect -f '{{index .NetworkSettings.Networks "appnet" "IPAddress"}}' db
docker network inspect appnet -f '{{(index .IPAM.Config 0).Subnet}}'

container:<name> shares everything including 127.0.0.1; host shows the host's interfaces; none shows only lo.

bash
docker run -d --name svc busybox sh -c 'while true; do echo hi | nc -l -p 8080; done'

# Join svc's namespace: same IP, and svc's listener is on 127.0.0.1
docker run --rm --network container:svc busybox nc -w2 127.0.0.1 8080   # hi
docker run --rm --network container:svc --hostname x busybox true
# docker: Error response from daemon: conflicting options: hostname and the network mode

docker run --rm --network none busybox ip -o addr      # lo only
docker run --rm --network host busybox ip -o -4 addr   # the host's interfaces

In practice

  • Docker Compose — Creates a user-defined bridge `<project>_default` for every project, which is why service names resolve between services without any `links`.
  • Kubernetes pod networking — All containers of a pod share one namespace through a pause container, the same mechanism as `--network container:<id>`; hence localhost between sidecars.
  • Prometheus node_exporter — Its documentation recommends `--net=host` (plus a root bind mount) so it sees the host's real interfaces, not the container's veth.
  • Pi-hole, Home Assistant — Commonly run with macvlan or host networking because they need LAN-visible addresses, broadcast or mDNS that NAT does not forward.
  • Podman — Uses the same mode names but a different default network stack (netavark), so published-port and bridge details differ from Docker.

Trade-offs

Pros

  • User-defined bridges give name resolution, per-project isolation and no host port use for east-west traffic.
  • host mode removes the NAT and veth hop, which matters for packet-rate-heavy or many-port workloads.
  • container:<id> makes sidecars (proxies, debug shells, exporters) share localhost with the app without changing the app image.
  • none and --internal are real isolation primitives, enforced in the kernel rather than by convention.

Cons

  • The default bridge has no DNS by name and is shared by every container started without --network.
  • host mode gives up port isolation: two containers cannot both bind :80, and the container sees host-local services on 127.0.0.1.
  • macvlan/ipvlan depend on the switch and parent NIC, usually cannot talk to the host directly, and need manual IP planning.
  • container:<id> couples lifecycles: the joined container is tied to the target namespace and generally has to be recreated when the target is.

How -p becomes a DNAT rule, a FORWARD accept rule and a docker-proxy process, why ufw does not block published ports, and where DOCKER-USER fits.

The problem

The service answers on curl localhost:8081 but not from another machine, or the opposite: a database published with -p 5432:5432 on a server is reachable from the internet even though ufw says deny. A new container fails to start with port is already allocated. All of it comes from what docker run -p really installs: NAT and filter rules written straight into iptables, plus a helper process, none of which goes through the host firewall front-end you configured.

How it works

  • Syntax: `-p [host_ip:]host_port:container_port[/tcp|udp]`. The default host_ip is 0.0.0.0 (`docker port` prints `80/tcp -> 0.0.0.0:8081`). `-p 80` picks a random host port from the ephemeral range, 32769 in a test (range 32768-60999 here), and `-P` publishes every port declared with EXPOSE. EXPOSE alone publishes nothing: a container built with `EXPOSE 80/tcp` and started without -p has an empty `docker port`. It only fills `.Config.ExposedPorts` metadata and tells `-P` which ports to map.
  • The NAT rule goes in the nat table: `-A DOCKER ! -i docker0 -p tcp -m tcp --dport 8081 -j DNAT --to-destination 172.17.0.3:80`. Traffic reaches it through `PREROUTING -m addrtype --dst-type LOCAL -j DOCKER` for packets from outside, and through `OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER` for the host's own non-loopback connections. `! -i docker0` keeps traffic that arrives on the bridge itself from being DNATed.
  • A matching filter rule is also added: `-A DOCKER -d 172.17.0.3/32 ! -i docker0 -o docker0 -p tcp -m tcp --dport 80 -j ACCEPT`, and FORWARD has policy DROP. On Engine 29 FORWARD jumps to `DOCKER-USER` first and then `DOCKER-FORWARD` (which holds `DOCKER-CT`, `DOCKER-INTERNAL`, `DOCKER-BRIDGE`); older engines use `DOCKER-ISOLATION-STAGE-1/2`, so read your own `iptables -S` rather than a blog post.
  • The 127.0.0.0/8 exclusion in OUTPUT is why `docker-proxy` exists: it listens on 0.0.0.0:8081 (`ss -ltnp` shows `users:(("docker-proxy",...))`) and relays loopback connections, because DNAT does not apply to packets sent to 127.0.0.1. Requests to the host's own LAN IP take the DNAT path and bump the rule's packet counter (`iptables -t nat -L DOCKER -n -v -x`). `userland-proxy: false` in daemon.json (or `--userland-proxy=false` on dockerd) removes the helper process.
  • `-p 127.0.0.1:8082:80` produces a rule restricted with `-d 127.0.0.1/32`. A client on another network reaching the host's LAN IP on 8082 gets `Connection refused`. Bind databases and admin ports to 127.0.0.1 (or a private interface IP) unless something external really needs them.
  • Why ufw does not protect published ports: the packet is DNATed in PREROUTING and then forwarded, so it never visits the INPUT chain where ufw's allow/deny rules for local services live, and Docker's FORWARD accept rules are evaluated ahead of ufw's forward chains. UNVERIFIED here (no ufw on this host), but widely documented. Put filters in `DOCKER-USER`, which is evaluated first and is never touched by dockerd.
  • DOCKER-USER sees packets after DNAT, so match the container IP and container port, not the published ones: with `iptables -I DOCKER-USER -d 172.17.0.3 -p tcp --dport 80 -j DROP`, a client from another bridge timed out on host:8081. It does not affect the host's own `curl host-ip:8081`, because locally generated packets go through OUTPUT, not FORWARD.
  • Related rules and limits: a second `-p 8081:80` fails at start with `Bind for 0.0.0.0:8081 failed: port is already allocated`; `--network host` discards -p; `--internal` networks get no DNAT rule at all; container-to-container traffic on the same network uses the container port directly and needs no -p.

Examples

What -p installs, read straight from the kernel; run as root on the Docker host.

bash
docker run -d --name web -p 8081:80 busybox httpd -f -p 80
docker port web                  # 80/tcp -> 0.0.0.0:8081

iptables -t nat -S DOCKER        # the DNAT rule
iptables -S DOCKER               # the per-container ACCEPT rule
iptables -S FORWARD              # policy DROP, DOCKER-USER first
ss -ltnp | grep 8081             # docker-proxy holds the host port

docker run -d --name web-local -p 127.0.0.1:8082:80 busybox httpd -f -p 80
iptables -t nat -S DOCKER | grep 8082   # note the -d 127.0.0.1/32 match

Filtering published ports in DOCKER-USER; replace HOST_LAN_IP with the host's address. The host's own curl is unaffected.

bash
# DOCKER-USER sees post-DNAT packets: match the container IP and port 80, not 8081
CIP=$(docker inspect -f '{{index .NetworkSettings.Networks "bridge" "IPAddress"}}' web)
iptables -I DOCKER-USER -d "$CIP" -p tcp --dport 80 -j DROP

# From a container on another network the published port now times out
docker network create probe
docker run --rm --network probe busybox wget -T2 -qO- http://HOST_LAN_IP:8081

iptables -D DOCKER-USER -d "$CIP" -p tcp --dport 80 -j DROP   # undo

In practice

  • ufw / firewalld on a Docker host — The classic surprise: `ufw deny 5432` does not close a port published with -p. The documented fix is a rule in DOCKER-USER or binding to 127.0.0.1.
  • docker-proxy — The userland proxy process started per published port and protocol; visible in `ps` and `ss -ltnp`, and optional via `userland-proxy: false` in daemon.json.
  • Docker Engine firewall backend — Engine 29 reports `Firewall Backend: iptables` in `docker info`; an experimental nftables backend (`firewall-backend` option in daemon.json) is opt-in and was not verified here. On iptables-nft hosts `nft list ruleset` shows the same rules.
  • Traefik, nginx-proxy — Reverse proxies publish only 80/443 and reach app containers over a shared user-defined network, so the apps need no -p at all.

Trade-offs

Pros

  • One flag exposes a service with kernel-level NAT, no application changes and no reverse proxy needed for simple cases.
  • Binding to 127.0.0.1 or a specific interface IP gives a cheap, verifiable exposure boundary.
  • DOCKER-USER is a stable, documented hook that survives dockerd restarts, unlike hand-edited DOCKER chains.

Cons

  • Published ports bypass the host firewall front-end by default, so a default `-p 5432:5432` is world-reachable.
  • NAT hides the client address from the container when traffic goes through docker-proxy, and DNAT adds conntrack entries that can fill under load.
  • Rules are generated per container start, so they are hard to audit statically and differ between Engine versions.
  • Host port space is global: a fixed -p collides across stacks, while a random one needs `docker port` to discover.

The embedded resolver at 127.0.0.11, what names it answers, how /etc/resolv.conf is generated, and why the default bridge has no name resolution.

The problem

A service calls http://db:5432 and gets could not resolve host: db, but only when the containers were started with plain docker run, not under Compose. Or name resolution works for a minute and then a restarted container is unreachable by its old name. DNS inside a container is not the host's resolver: it is an embedded server in dockerd with its own rules about who is visible and who is forwarded.

How it works

  • On a user-defined network the container's /etc/resolv.conf is generated by Docker (header `# Generated by Docker Engine.`) and contains `nameserver 127.0.0.11` and `options timeout:2 attempts:3 ndots:0`, plus comment lines like `# ExtServers: [host(8.8.8.8) host(8.8.4.4)]` listing the upstreams. Editing the file stops Docker from regenerating it.
  • 127.0.0.11 is a socket created inside the container's own network namespace by dockerd. `ss -lun` in the namespace shows `127.0.0.11:<random udp port>` and `ss -ltn` a TCP twin; `DOCKER_OUTPUT`/`DOCKER_POSTROUTING` iptables rules inside the namespace DNAT port 53 to those ports and SNAT the replies back. That is why `docker exec ... ss` can see it and why a policy that blocks loopback inside the container breaks DNS.
  • It answers for: the container name, every `--network-alias` (Compose adds the service name and `aliases:`), the hostname and the 12-character short ID, all scoped to the networks the asking container is attached to. A partial name (`w` for `w1`) is NXDOMAIN, and a name that exists only on a network the container is not on is NXDOMAIN too.
  • Only running containers resolve: after `docker stop`, `nslookup <name>` returns NXDOMAIN, and after `docker start` it resolves again (the address may be different). Several containers with the same alias get several A records in one answer (172.18.0.5 and 172.18.0.6 in a test): crude round-robin chosen by the client, not health-aware load balancing.
  • Names it does not own are forwarded to the upstream servers taken from the host's /etc/resolv.conf when the container started. `--dns 1.1.1.1` on a user-defined network changes only the `ExtServers` comment: resolv.conf still says 127.0.0.11. On the default bridge, `--dns` is written straight into resolv.conf. `--dns-search` and `--dns-option ndots:2` land as `search` and `options` lines.
  • The default bridge has no embedded server: resolv.conf is a copy of the host's (`nameserver 8.8.8.8` on this host), and other containers' names return NXDOMAIN. `--link name:alias` writes an /etc/hosts line (`172.17.0.2 alpha <short-id> <name>`) and prints `Links on the default bridge network are deprecated`. Prefer a user-defined network.
  • A container already running on the default bridge switches to the embedded resolver as soon as it is connected to a user-defined network with `docker network connect`; its resolv.conf then shows `nameserver 127.0.0.11`. Disconnecting it again does not restore the old file: it still said `127.0.0.11` afterwards, so a name that stops resolving may need the container recreated.
  • /etc/hosts is separate: `--add-host db.local:10.0.0.9` adds a line there, which `ping` and `getent hosts` honour but `nslookup` ignores (it queries DNS only and returned NXDOMAIN). Loopback nameservers are dropped because they are not valid inside the container: when the host resolv.conf lists only 127.0.0.53, Docker reads /run/systemd/resolve/resolv.conf instead, and with no non-loopback server left it falls back to 8.8.8.8 and 8.8.4.4 (source: moby libnetwork resolvconf; not reproduced here).

Examples

Aliases return one A record per container; partial names and stopped containers do not resolve.

bash
docker network create appnet
docker run -d --name w1 --network appnet --network-alias web busybox sleep 600
docker run -d --name w2 --network appnet --network-alias web busybox sleep 600

docker run --rm --network appnet busybox sh -c 'grep -v "^#" /etc/resolv.conf; nslookup web; nslookup w'
# nameserver 127.0.0.11
# options timeout:2 attempts:3 ndots:0
# web -> two A records; w -> ** server can't find w: NXDOMAIN

docker stop -t1 w2
docker run --rm --network appnet busybox nslookup w2   # NXDOMAIN while stopped

Inspecting the embedded DNS plumbing; on a user-defined network --dns only changes the upstream.

bash
# See the resolver socket and the DNAT rules from the host side (root)
PID=$(docker inspect -f '{{.State.Pid}}' w1)
nsenter -t "$PID" -n iptables-save -t nat | grep 127.0.0.11
nsenter -t "$PID" -n ss -lun

# Per-container overrides
docker run --rm --network appnet --dns 1.1.1.1 --dns-search corp.test \
  --dns-option ndots:2 busybox cat /etc/resolv.conf

In practice

  • Docker Compose — Each service name becomes a network alias, so `db` resolves between services; `aliases:` under a network adds more names for one service.
  • Kubernetes CoreDNS — Cluster DNS is a different design: a real service in the cluster, `ndots:5` and search domains by default, versus Docker's `ndots:0` embedded resolver.
  • systemd-resolved hosts — When the host resolv.conf lists only 127.0.0.53, Docker reads systemd-resolved's /run/systemd/resolve/resolv.conf instead; if no non-loopback nameserver is left, containers get a substituted public resolver (8.8.8.8, 8.8.4.4). Check what was copied with `docker run --rm busybox cat /etc/resolv.conf`.
  • nginx upstream blocks — nginx resolves upstream hostnames once at start; behind a recreated container the cached IP goes stale unless you set `resolver 127.0.0.11` and use a variable in proxy_pass.

Trade-offs

Pros

  • Names stay stable while container IPs change on every recreate, so configuration can use `db` instead of 172.18.0.x.
  • Aliases give a cheap multi-instance name and let one container answer to several names.
  • Scoping by network means a container cannot resolve names on networks it is not part of, a mild form of service isolation.

Cons

  • No such thing on the default bridge, and the failure looks like a generic resolve error in the application.
  • Only running containers resolve, so start-order races surface as NXDOMAIN rather than connection refused.
  • Round-robin A records are not health-checked and clients that cache the first answer pin to one container.
  • A per-container --dns on user-defined networks changes only the upstream, which surprises people who read resolv.conf.

Where each storage type lives, who creates the path, when image content is copied into a volume, what survives docker rm, and why anonymous volumes pile up.

The problem

A database container is rebuilt and its data is gone, or docker volume ls shows forty hex-named volumes nobody remembers creating, or a bind mount over /app hides the files the image shipped. The storage type chosen at run time decides who creates the path, whether image content shows through, what survives docker rm, and how fast writes are.

How it works

  • Named volume: a directory Docker owns at `/var/lib/docker/volumes/<name>/_data` (`docker volume inspect` shows Driver `local`, Scope `local`). `docker volume create` is optional: `-v name:/path` creates it on first use. It survives `docker rm` and `docker compose down`, and is only deleted by `docker volume rm`, `docker compose down -v` or a prune.
  • Bind mount: a host path mounted as-is, with the host's ownership and modes. With `-v /host/path:/c` a missing source is silently created as `root:root` mode 755; with `--mount type=bind,src=...,dst=...` it fails with `bind source path does not exist`. Prefer `--mount` when a typo should be an error.
  • Copy-up: when a volume is empty at mount time, Docker copies the image's content at that path into it. In a test, a fresh volume at `/app/data` showed the image's `seed.txt`; after the file was deleted and the container restarted with the same empty volume, it reappeared. A non-empty volume is left alone. A bind mount never copies: the image's files at that path are hidden (0 files seen). `:nocopy` (or `--mount ...,volume-nocopy`) disables copy-up.
  • Anonymous volume: created by `-v /path` or a Dockerfile `VOLUME` line (visible in `.Config.Volumes`), named with 64 hex characters, `Driver local`. `docker rm` leaves it behind; `docker rm -v` and `docker run --rm` remove it with the container. `docker volume rm` on an in-use volume fails with `volume is in use`. These are the usual source of unexplained disk usage.
  • tmpfs: `--tmpfs /scratch:size=16m,mode=1777` or `--mount type=tmpfs,dst=/t,tmpfs-size=8m`; mountinfo shows `tmpfs rw,size=16384k`. It lives in RAM and is gone when the container stops (a restart showed an empty /scratch), and writes beyond the size fail (a 20 MB `dd` into an 8 MB mount stopped at 8 MB). Memory used is typically charged to the container's cgroup, so count it against `--memory`.
  • Container writable layer: everything not on a mount lands in the overlayfs upper layer. It survives `docker stop`/`start`/`restart` (a file written to `/layer` was still there after restart) but is deleted with the container (`docker rm`). It is copy-on-write, slower for write-heavy random IO, and gone on `docker compose up --force-recreate`.
  • Mount options: `:ro` (or `readonly` in `--mount`) gives `Read-only file system` on any write. Compose long syntax (`type: bind`, `source: ./conf`, `read_only: true`) is equivalent, and relative sources resolve against the Compose file's directory. SELinux `:z`/`:Z` relabelling and mount propagation (`rshared`) exist for Linux bind mounts but were not exercised here.
  • Volume drivers: the built-in `local` driver passes options to mount(8), e.g. `docker volume create --driver local --opt type=tmpfs --opt device=tmpfs --opt o=size=8m,uid=1000 v` (verified, showed `uid 1000` and an 8 MiB tmpfs) or `type=nfs` with `o=addr=...,rw` and `device=:/export` (NFS not tested). Third-party plugins implement the same API for network or cloud block storage.

Examples

Copy-up, bind mount hiding, :nocopy, and the anonymous volume leak versus rm -v.

bash
cat > Dockerfile <<'EOF'
FROM busybox:1.37
RUN mkdir -p /app/data && echo seed > /app/data/seed.txt
EOF
docker build -q -t seeded .

docker run --rm -v appdata:/app/data seeded ls /app/data              # seed.txt (copy-up)
docker run --rm -v "$PWD/hostdir:/app/data" seeded ls /app/data       # empty: bind hides it
docker run --rm -v nocp:/app/data:nocopy seeded ls /app/data          # empty: copy-up off

docker run -d --name anon -v /app/data seeded sleep 100
docker rm -f anon                  # anonymous volume stays
docker volume ls -q                # 64-hex name
docker run -d --name anon2 -v /app/data seeded sleep 100
docker rm -f -v anon2              # removed together with the container

Named volume, read-only bind mount and tmpfs in Compose; docker compose down keeps dbdata, down -v removes it.

yaml
name: stores
services:
  db:
    image: busybox:1.37
    command: ["sleep", "600"]
    volumes:
      - dbdata:/var/lib/db
      - type: tmpfs
        target: /run/scratch
        tmpfs:
          size: 16777216
  api:
    image: busybox:1.37
    command: ["sleep", "600"]
    volumes:
      - ./conf:/etc/app:ro
      - ./data:/var/data
volumes:
  dbdata: {}

In practice

  • Official postgres and mysql images — Declare `VOLUME` for the data directory (`/var/lib/postgresql` in the 18+ postgres images), so a plain `docker run` without -v silently creates an anonymous volume that outlives `docker rm`.
  • Docker Compose — Prefixes named volumes with the project name (`stores_dbdata`); `down` keeps them and `down -v` deletes them, a frequent cause of both surprise data loss and leftover disks.
  • Kubernetes emptyDir with medium: Memory — The same idea as tmpfs mounts, including that the contents count against the memory limit.
  • CI runners — Typically mount the workspace as a bind mount and caches as named volumes, so cache survives container recreation while the source tree stays visible on the host.

Trade-offs

Pros

  • Named volumes are managed by Docker, portable across hosts' directory layouts, and support drivers and copy-up of initial content.
  • Bind mounts give direct host access to files, which is right for source code, config and anything an editor or host tool also touches.
  • tmpfs keeps secrets and scratch data off disk and is fast, with a hard size cap.
  • The writable layer needs no setup and is fine for throwaway state.

Cons

  • Volumes are opaque from the host's point of view: you need docker commands or root to look inside `/var/lib/docker/volumes`.
  • Bind mounts couple the container to host paths and ownership, and an empty or wrong source silently hides image content.
  • Anonymous volumes accumulate invisibly and are not cleaned by `docker rm`.
  • tmpfs and the writable layer lose data on stop or rm, and overlayfs writes are slower than a volume on the same disk.

Why a non-root container gets Permission denied on a bind mount, how numeric UIDs cross the container boundary, and the ordered list of fixes.

The problem

The container runs as a non-root user and gets Permission denied on a bind-mounted directory that looks writable on the host. Or the reverse: the container runs as root and leaves root-owned files in the project directory that your editor cannot delete. The kernel compares numeric UIDs and GIDs, and Docker passes them through unchanged, so a name match between host and container means nothing.

How it works

  • Files carry numeric owners. The name shown by `ls -l` comes from the /etc/passwd of whichever side is looking, so user `app` (10001) in an image and an unrelated host user 10001 are the same identity to the kernel. Compare `ls -ln` output, not names.
  • Without user namespace remapping, UID 0 in the container is UID 0 on the host. A root container writing to a bind mount creates `0:0` files on the host (verified: `-rw-r--r-- 1 0 0 by-root` inside a directory owned by 1000). Daemon-level `userns-remap` and rootless mode shift the range through /etc/subuid; both change this and were not tested here.
  • Bind mounts keep the host's ownership and mode. A host directory owned by 1000:1000 mode 755, mounted into a container running as UID 10001, fails with `touch: /out/by-app: Permission denied`; `--user 1000:1000` makes it work and the new files are owned by 1000 on the host.
  • A named volume takes its initial ownership from the image: copy-up also copies the owner and mode of the image's directory onto the volume root. With `RUN mkdir /srv/owned && chown app:app /srv/owned` in the image, a new volume at that path was `10001:10001` and writable by `app`. If the path does not exist in the image, Docker creates it `0:0` mode 755, and a non-root user gets Permission denied.
  • `--user 4242` without a group sets gid 0 (`uid=4242 gid=0(root) groups=0(root)`), and with no passwd entry `$HOME` is `/`. Always pass both, `--user 1000:1000`, or declare a real user with a fixed numeric UID in the Dockerfile (`USER 10001`) so bind-mount permissions can be planned.
  • tmpfs mounts default to `0:0` mode 1777 (`drwxrwxrwt`), so any user can write; `--tmpfs /t:uid=10001,mode=700` set owner and mode. Use the `uid=`, `gid=` and `mode=` options instead of chmod in an entrypoint.
  • Fix in this order: (1) run with the UID that owns the data (`--user "$(id -u):$(id -g)"` for dev bind mounts); (2) chown in the image before mounting so volumes inherit it; (3) share through a group, `chmod 2770` on the directory plus `--group-add <gid>` (verified: with the setgid bit new files got group 2000); (4) a root entrypoint that chowns and then drops privileges with gosu or su-exec. Avoid `chmod 777`, and remember that a recursive chown on a big bind mount is slow and changes host files.
  • Docker Engine has no `:U` volume option (Podman does): `-v ./u:/u:U` fails with `invalid mode: U`. SELinux hosts need `:z` or `:Z` for labels, and file sharing from Docker Desktop VMs follows its own rules; neither was available here.

Examples

The host directory's numeric owner decides access; --user must match it.

bash
mkdir hostdir && chown 1000:1000 hostdir

# root writes: files are root-owned on the host
docker run --rm -v "$PWD/hostdir:/out" busybox touch /out/by-root

# UID 10001 is not the owner: denied
docker run --rm --user 10001:10001 -v "$PWD/hostdir:/out" busybox touch /out/x
# touch: /out/x: Permission denied

# match the owner, or share through a group
docker run --rm --user 1000:1000 -v "$PWD/hostdir:/out" busybox touch /out/by-1000
ls -ln hostdir        # by-root is 0 0, by-1000 is 1000 1000

A volume inherits the image directory's owner at copy-up; a path missing from the image becomes root-owned.

bash
cat > Dockerfile <<'EOF'
FROM busybox:1.37
RUN adduser -D -u 10001 app && mkdir -p /srv/owned && chown app:app /srv/owned
USER app
EOF
docker build -q -t owned .

docker run --rm -v ownedvol:/srv/owned owned sh -c 'ls -ldn /srv/owned; touch /srv/owned/x && echo ok'
# drwxr-xr-x 2 10001 10001 ... /srv/owned   then ok  (volume inherited the image owner)
docker run --rm -v newvol:/srv/new owned sh -c 'ls -ldn /srv/new; touch /srv/new/x'
# drwxr-xr-x 2 0 0 ... /srv/new   then Permission denied  (path missing in image: root-owned)

# group sharing with a setgid directory
mkdir shared && chown 0:2000 shared && chmod 2770 shared
docker run --rm --user 10001 --group-add 2000 -v "$PWD/shared:/s" busybox sh -c 'id; touch /s/a'

In practice

  • Official postgres image — Its entrypoint starts as root, chowns the data directory to the `postgres` user and then re-executes itself with gosu, the usual fix when a volume may arrive with the wrong owner.
  • Rootless Docker and Podman — Run the daemon without root and map container UID 0 to your own UID through /etc/subuid and /etc/subgid, which changes the ownership picture on bind mounts. Not tested here.
  • Kubernetes fsGroup — A pod-level `securityContext.fsGroup` makes the kubelet chgrp volumes on mount, the cluster-side equivalent of `--group-add` plus a setgid directory.
  • Dev containers and Compose dev stacks — Commonly pass `user: "${UID}:${GID}"` or build an image user with the host's UID so edited files stay owned by the developer.

Trade-offs

Pros

  • Pass-through numeric ownership is simple and has no hidden translation layer to debug.
  • Fixed numeric UIDs in the image make bind-mount and volume permissions predictable and reviewable.
  • Group sharing with a setgid directory lets several UIDs write without chmod 777.

Cons

  • Root containers leave root-owned files on the host, which then need sudo to delete.
  • The same image can behave differently depending on the host UID that owns the bind mount.
  • Entry-point chown fixes cost startup time on large trees and change host file ownership permanently.
  • Names in `ls -l` can mislead, since host and container /etc/passwd are unrelated.

Security & Compose

What USER and --user actually change, why a numeric uid beats a name, and the ownership traps (gid 0, WORKDIR, volumes) that appear once you stop being root.

The problem

The image has no USER line, so the app runs as uid 0 with 14 capabilities. A path-traversal bug, a bind-mounted host directory or a mounted socket now starts from root, and every file the app writes into a bind mount is owned by root on the host. You add USER app, and the next deploy dies with Permission denied on /app, or on a volume, or on port 80, and the quick fix is to put root back.

How it works

  • Default: with no USER and no --user, the process runs as uid=0 gid=0 with CapEff 00000000a80425fb (the 14 default capabilities). Without user-namespace remapping or rootless mode, uid 0 in the container is uid 0 on the host, restricted only by capabilities, seccomp and the mount namespace, which is why files created on a bind mount show up as root-owned on the host.
  • USER in a Dockerfile sets `Config.User` in the image and accepts a name, a uid, or `uid:gid`. `docker run --user` overrides it. A name is looked up in the image's /etc/passwd when the container starts, so `--user nosuch` fails with `unable to find user nosuch: no matching entries in passwd file` and docker exits 125; a numeric uid needs no passwd entry at all.
  • Name the gid too. `--user 12345` with no matching passwd entry runs as `uid=12345 gid=0(root)`, so the process is in group root and can write anything that is group-writable by root. Use `uid:gid`, for example `USER 10001:10001`. A numeric USER is also what policy checks can verify; a name cannot be proven non-root without reading the image's passwd file.
  • USER changes who runs the process, not who owns files. `COPY` creates root-owned files unless you pass `--chown=uid:gid`. `WORKDIR /app` creates the directory owned by whichever user is current at that instruction: before `USER` it is root-owned and the app gets `touch: /app/new: Permission denied`; after `USER 10001` it is owned by 10001. Create and chown writable directories in a `RUN` before switching.
  • Volumes: an empty named volume is initialised from the image directory it covers, including ownership (a volume mounted over /data with `chown 10001` in the image comes up owned by 10001). A volume mounted at a path that does not exist in the image is created root-owned, and a bind mount keeps the host directory's owner; in both cases a non-root app gets Permission denied until you chown or match uids.
  • Low ports: Docker sets `net.ipv4.ip_unprivileged_port_start=0` inside the container's network namespace, so a non-root process can bind port 80 even with `--cap-drop ALL` (verified, while the host value was 1024). With `--network host` the host's value applies and the same bind fails with `Permission denied`. NET_BIND_SERVICE is therefore usually unnecessary on a bridge network.
  • Non-root processes get no effective capabilities. With `--user 1000` the bounding set still shows the defaults but `CapEff` is 0, and `--cap-drop ALL --cap-add NET_BIND_SERVICE` shrinks the bounding set to 0x400 while `CapEff` stays 0, because Docker does not set ambient capabilities (a plain `--cap-add NET_BIND_SERVICE` changes nothing, since it is already a default). A capability you add helps only processes that start as root, or binaries with file capabilities, and does not make a non-root app privileged.
  • User-namespace remapping (`userns-remap` in daemon.json) and rootless mode map container uid 0 to an unprivileged host uid, so even a root process in the container has no host privileges. They are daemon-level choices with their own limits on volumes, networking and ports, and they complement `USER`; they do not replace it.

Examples

Numeric uid:gid, files chowned at COPY time, and a writable directory created before the switch (WORKDIR after USER is owned by that user).

bash
echo hi > hello.txt
cat > Dockerfile <<'EOF'
FROM busybox:1.37
RUN addgroup -g 10001 app && adduser -D -u 10001 -G app -h /home/app app
RUN mkdir /data && chown 10001:10001 /data
COPY --chown=10001:10001 hello.txt /home/app/hello.txt
USER 10001:10001
WORKDIR /w
CMD ["sh", "-c", "id; ls -ldn /w /data /home/app/hello.txt; touch /data/f && echo data-ok"]
EOF
docker build -q -t app-nonroot .
docker run --rm app-nonroot
docker inspect app-nonroot --format 'User={{.Config.User}}'

What the container sees under --user; the last command shows a non-root process listening on port 80.

bash
# non-root has CapEff 0 even though the bounding set keeps the defaults
docker run --rm --user 1000 busybox:1.37 sh -c 'id; grep -E "^(CapEff|CapBnd)" /proc/self/status'
# uid without a passwd entry: gid falls back to 0
docker run --rm --user 12345 busybox:1.37 id
# unknown name fails before the process starts
docker run --rm --user nosuch busybox:1.37 id; echo "exit=$?"
# bind 80 as uid 1000 with every capability dropped: allowed (nc times out waiting)
docker run --rm --user 1000 --cap-drop ALL busybox:1.37 sh -c 'nc -l -p 80 -w1; echo rc=$?'

In practice

  • nginxinc/nginx-unprivileged, bitnami images — Variants that run as a fixed non-root uid and listen on a high port (8080), so they also work where the platform refuses root.
  • Official postgres and redis images — Start as root only to fix ownership of the data directory, then drop to an unprivileged user with gosu or su-exec; that entrypoint needs CHOWN, SETUID and SETGID, which matters if you also use --cap-drop ALL.
  • Distroless nonroot tags — Ship a ready-made nonroot user with uid 65532, so USER can be numeric without adding useradd to the image.
  • Kubernetes runAsNonRoot — The kubelet rejects a container whose image USER is non-numeric or root, which is why a numeric USER is the portable form.
  • Rootless Docker and Podman — Run the daemon (or engine) itself as an unprivileged user, so the host side of container uid 0 is not root.

Trade-offs

Pros

  • A compromised app process cannot write to root-owned image paths, bind mounts it should not own, or use root-only kernel interfaces
  • A numeric uid:gid is checkable by admission policy without reading the image
  • Costs one Dockerfile line and a few chown calls; no runtime overhead

Cons

  • Every writable path needs explicit ownership; volumes and bind mounts are the usual breakage
  • Images that expect root at start (package installs, chown in the entrypoint) need rework
  • Does not remove the shared kernel; it narrows what a bug in the app can do, not what a kernel bug can do

The 14 default capabilities, --cap-drop ALL, the default seccomp filter, no-new-privileges and --read-only with --tmpfs, each checked in /proc/self/status.

The problem

The container runs fine as root with defaults, so nobody asked what it can do. Then a vulnerable dependency lets an attacker open a raw socket, drop a setuid helper, overwrite a script in the image or unpack a binary into /tmp and run it. Each of those is a separate kernel-enforced switch: capabilities, seccomp, no-new-privileges and the read-only rootfs. They are cheap to turn on and expensive to discover missing during an incident.

How it works

  • Default capability set: root in a container keeps 14 capabilities (`CapEff 00000000a80425fb`, decoded by `capsh --decode`: CHOWN, DAC_OVERRIDE, FOWNER, FSETID, KILL, SETGID, SETUID, SETPCAP, NET_BIND_SERVICE, NET_RAW, SYS_CHROOT, MKNOD, AUDIT_WRITE, SETFCAP). Capabilities such as SYS_ADMIN, NET_ADMIN, SYS_TIME and SYS_PTRACE are absent: in a default container `mount` fails with `permission denied`, `date -s` with `Operation not permitted` and `ip link add` with `RTNETLINK answers: Operation not permitted`.
  • `--cap-drop ALL --cap-add X` sets both `CapEff` and `CapBnd` to exactly X (`NET_BIND_SERVICE` gives `0000000000000400`). `docker inspect` shows it as `HostConfig.CapDrop=[ALL]` and `HostConfig.CapAdd=[CAP_NET_BIND_SERVICE]`; the same fields are `cap_drop` and `cap_add` in Compose. Drop one capability on its own when that is all you need: `--cap-drop NET_RAW` makes `ping` fail with `permission denied (are you root?)`.
  • Dropping ALL is the right start, but an entrypoint that does chown, su-exec or gosu will fail with `Operation not permitted` (a `chown` without CHOWN did), so add back only what the error names. Capabilities apply to root; a non-root process has an empty effective set anyway, so the combination `--user` plus `--cap-drop ALL` is belt and braces rather than redundant, because it also removes the bounding set that setuid binaries could use.
  • `--privileged` is not a capability tweak: it gave `CapEff 000001fffeffffff`, `Seccomp: 0` (no filter) and every host device (111 entries in /dev versus 14). `--cap-add SYS_ADMIN` alone is close to root on the host and was enough for `unshare -m` to succeed. Also treat `-v /var/run/docker.sock:...`, `--pid host`, `--device` and `--network host` as privileges: whoever reaches the socket can ask the daemon to start a privileged container.
  • Default seccomp: the engine applies a built-in profile, visible as `Seccomp: 2` and `Seccomp_filters: 1` in /proc/self/status; `--security-opt seccomp=unconfined` shows `Seccomp: 0`. The filter blocks namespace creation, so `unshare -U`, `unshare -p` and `unshare -m` return `Operation not permitted` in a default container (the first succeeds with `seccomp=unconfined`, and `-m` succeeds with `--cap-add SYS_ADMIN`). A custom profile is passed with `--security-opt seccomp=profile.json` and replaces the default, it does not extend it.
  • `--security-opt no-new-privileges` (Compose: `security_opt: ["no-new-privileges:true"]`) sets `NoNewPrivs: 1`; the kernel then ignores setuid bits and file capabilities on exec. With a setuid-root test binary run as uid 1000, the default container printed `uid=1000 euid=0` and the flagged one `uid=1000 euid=1000`. Inspect shows `HostConfig.SecurityOpt=[no-new-privileges]`.
  • `--read-only` mounts the root as `overlay / overlay ro`, and writes fail with `Read-only file system` (EROFS). Mounted volumes and `--tmpfs` stay writable, `/dev/shm` stays a writable 64 MB tmpfs, but `/etc/hosts` and `/etc/resolv.conf` become read-only too, so an entrypoint that appends to them breaks. `--tmpfs /tmp` defaults to `rw,nosuid,nodev,noexec`: a binary copied there fails with `Permission denied` (exit 126) unless you add `exec`.
  • Put it together as defaults you remove per service: read-only root with only the needed tmpfs and volumes, `cap_drop: [ALL]`, `no-new-privileges`, a numeric non-root user and a `pids_limit`. Image scanning, SBOMs and signatures help you audit what is inside an image and where it came from, but they say nothing about these runtime switches, which have to be set at `docker run` or in Compose.

Examples

A hardened docker run; the checks print CapEff, NoNewPrivs and Seccomp from inside the container.

bash
docker run --rm --read-only --tmpfs /tmp:rw,noexec,nosuid,size=16m \
  --cap-drop ALL --cap-add NET_BIND_SERVICE \
  --security-opt no-new-privileges --pids-limit 100 \
  --user 10001:10001 busybox:1.37 sh -c '
    grep -E "^(CapEff|NoNewPrivs|Seccomp):" /proc/self/status
    touch /tmp/ok && echo tmp-writable
    touch /etc/x || echo rootfs-read-only'

The same switches in Compose; docker compose config accepts it as written.

yaml
name: demo-hard
services:
  app:
    image: busybox:1.37
    user: "10001:10001"
    read_only: true
    tmpfs:
      - /tmp:size=16m,mode=1777
    cap_drop: [ALL]
    cap_add: [NET_BIND_SERVICE]
    security_opt:
      - no-new-privileges:true
    pids_limit: 100
    command:
      - sh
      - -c
      - |
        id
        grep -E '^(CapEff|NoNewPrivs|Seccomp):' /proc/self/status
        touch /tmp/ok && echo tmp-writable
        touch /etc/x 2>&1 || true

Seeing the default filter and capability set, and a syscall the default seccomp profile refuses.

bash
docker run --rm busybox:1.37 grep -E '^(CapEff|Seccomp|Seccomp_filters):' /proc/self/status
docker run --rm --security-opt seccomp=unconfined busybox:1.37 grep -E '^Seccomp:' /proc/self/status
docker run --rm busybox:1.37 unshare -m true; echo "exit=$?"
docker run --rm --cap-add SYS_ADMIN busybox:1.37 unshare -m true; echo "exit=$?"
capsh --decode=00000000a80425fb   # host side, from libcap tools

In practice

  • Docker default seccomp profile — A built-in JSON profile in the engine that denies syscalls such as namespace creation, keyring access, reboot and module loading unless a capability unlocks them.
  • CIS Docker Benchmark — Includes checks for no-new-privileges, dropping capabilities, avoiding --privileged and mounting the root filesystem read-only.
  • Kubernetes securityContext — readOnlyRootFilesystem, capabilities.drop: [ALL] and allowPrivilegeEscalation: false map to the same kernel features as --read-only, --cap-drop ALL and no-new-privileges.
  • gVisor and Kata Containers — Alternative runtimes that put a user-space kernel or a VM between the container and the host kernel when these switches are not enough.
  • Podman — Uses the same flag names (--cap-drop, --read-only, --security-opt) and a similar default capability set.

Trade-offs

Pros

  • Each switch removes a whole class of post-exploit actions at the kernel level, with no agent or sidecar
  • Read-only root makes the image contents immutable at runtime and turns unexpected writes into loud errors
  • The same four settings port directly to Kubernetes securityContext

Cons

  • Apps that write next to their binaries, to /var/run or to /etc need tmpfs mounts or volumes found by trial and error
  • Dropping ALL breaks root-start entrypoints (chown, gosu) until the right capability is added back
  • A shared kernel remains: these narrow the attack surface but are not a VM boundary

Where ENV, ARG, deleted files and -e leak a secret (docker history, inspect, layer tarballs), and how RUN --mount=type=secret and Compose secrets avoid it.

The problem

A build needs a private package token and the app needs a database password. The token goes in as ARG, the password as ENV, a credentials file is COPYed and later deleted with rm. The image is pushed, and `docker history`, `docker inspect` or an unpacked layer tarball hands every one of them to anyone who can pull it. Rotating the secrets is then the only fix.

How it works

  • Runtime `-e TOKEN=...` and Compose `environment:` live in the container config: `docker inspect` shows `Config.Env=[TOKEN=abc123 ...]` to anyone who can call the Docker API, and `/proc/1/environ` inside the container contains it. Child processes inherit it, and crash reporters and debug endpoints often dump the whole environment.
  • `ENV DB_PASSWORD=hunter2` in a Dockerfile is baked into the image: `docker inspect <image>` shows it in `Config.Env` and `docker history --no-trunc` shows an `ENV DB_PASSWORD=hunter2` line. `ARG` is not safer: `--build-arg API_TOKEN=ARGVALUE999` appeared in `docker history --no-trunc` as `ARG API_TOKEN=ARGVALUE999` and again in every later line as `RUN |1 API_TOKEN=ARGVALUE999 /bin/sh -c ...`, although it was not in `Config.Env`.
  • Deleting does not remove: after `COPY token.txt /tmp/token.txt` and `RUN rm /tmp/token.txt`, `docker save` produced a layer tarball that still contained `tmp/token.txt` with the value, next to a later layer holding only the whiteout. A layer is immutable, so a secret must never be written to the image filesystem in any step, not even a temporary one. A multi-stage build only helps if the secret was used in a stage that is not shipped.
  • BuildKit secrets: `RUN --mount=type=secret,id=api_token ...` mounts the value at `/run/secrets/api_token` (mode 0400, root, 16 bytes for a 16-byte value) for that one instruction only. The final image had no `/run/secrets`, `docker history` showed just the command text, and grepping the saved layers found nothing. `env=API_TOKEN` exposes it as an environment variable for that instruction instead. Supply it with `--secret id=api_token,src=token.txt`, or `--secret id=API_TOKEN` to read the host environment variable of that name.
  • A missing build secret is silent by default: the mount is absent and the command fails later with `No such file or directory`. Add `required=true` to fail at once with `secret nothere: not found`. The mount protects the value from the image, not from your own command: `echo $(cat /run/secrets/x) > /out` writes it into a layer.
  • Runtime secrets as files: Compose `secrets:` with `file: ./db_password.txt` bind-mounts the host file read-only at `/run/secrets/db_password` (the `mount` output showed a ro bind), and the value never appears in `Config.Env`. Permissions are those of the host file: a 0600 root-owned file with `user: "1000:1000"` gave `cat: can't open ... Permission denied`, so chmod/chown the source or run as the owner. A secret sourced from `environment: NAME` is written as a file in the container.
  • Compose warns that `mode` and `uid` on secrets are not supported outside Swarm, yet the Compose file reference says they take effect for `environment:` sources and are ignored for `file:` sources (a bind mount). In the test an environment-sourced secret with `uid: 1000` and `mode: 0400` set came out as `-r-------- 1000` despite the warning, one without them as `-r--r--r-- 0`, and the file-sourced one kept the host file's mode; check `ls -ln /run/secrets` instead of trusting the warning. Point the app at the path with the `_FILE` convention (`DB_PASSWORD_FILE=/run/secrets/db_password`), which many official images such as postgres and mysql support; check the image's documentation.
  • Compose interpolates `$VAR` in the file before the container sees it: a `command` using `$DB_PASSWORD_FILE` produced `The "DB_PASSWORD_FILE" variable is not set. Defaulting to a blank string` and an empty value, so write `$$DB_PASSWORD_FILE`. Scanners, SBOMs and image signatures can catch a secret already baked in or prove an image's origin, but they are detection after the fact; the fix is never putting the value in a layer.

Examples

A build secret as a file and as an env var; the history lines contain only the commands, never the value.

bash
printf 'S3CR3T-token-123' > token.txt
cat > Dockerfile.secret <<'EOF'
# syntax=docker/dockerfile:1
FROM busybox:1.37
RUN --mount=type=secret,id=api_token,required=true \
    echo "token bytes: $(wc -c < /run/secrets/api_token)"
RUN --mount=type=secret,id=api_token,env=API_TOKEN \
    echo "env length: ${#API_TOKEN}"
RUN ls /run/secrets 2>&1 || echo "no /run/secrets in final image"
EOF
docker build --no-cache --progress=plain \
  --secret id=api_token,src=token.txt -t app-secret -f Dockerfile.secret .
docker history --no-trunc --format '{{.CreatedBy}}' app-secret

What a puller sees: ARG and ENV values in history and inspect, and the deleted file still readable from its layer.

bash
cat > Dockerfile.leak <<'EOF'
FROM busybox:1.37
ARG API_TOKEN
ENV DB_PASSWORD=hunter2
RUN echo building
COPY token.txt /tmp/token.txt
RUN rm /tmp/token.txt
EOF
docker build -q -t app-leak --build-arg API_TOKEN=ARGVALUE999 -f Dockerfile.leak .
docker history --no-trunc --format '{{.CreatedBy}}' app-leak
docker inspect app-leak --format '{{.Config.Env}}'
docker save app-leak -o leak.tar && mkdir -p leak && tar -xf leak.tar -C leak
for f in $(find leak/blobs -type f); do
  tar -tf "$f" 2>/dev/null | grep -q 'tmp/token.txt' && tar -xOf "$f" tmp/token.txt 2>/dev/null
done; echo

Compose secrets: one from a host file (chmod it readable by the container user), one from a host environment variable; mounted under /run/secrets.

yaml
name: demo-rs
services:
  app:
    image: busybox:1.37
    user: "1000:1000"
    command: ["sh", "-c", "cat /run/secrets/db_password; echo; ls -ln /run/secrets"]
    environment:
      DB_PASSWORD_FILE: /run/secrets/db_password
    secrets:
      - db_password
      - source: api_key
        target: key.txt
secrets:
  db_password:
    file: ./db_password.txt
  api_key:
    environment: API_KEY_VALUE

In practice

  • Docker BuildKit — Secret mounts (`--mount=type=secret`) and SSH mounts (`--mount=type=ssh`) exist specifically so build-time credentials never enter a layer.
  • Official postgres and mysql images — Read `POSTGRES_PASSWORD_FILE` and `MYSQL_ROOT_PASSWORD_FILE` style variables so the value can come from a file under /run/secrets.
  • Docker Compose secrets — Compose Specification feature that gives services named secrets as files under /run/secrets without Swarm.
  • Kubernetes Secret volumes — Same pattern at cluster scale: a secret is mounted as a file, and environment variables are the less safe alternative.
  • CI systems such as GitHub Actions and GitLab CI — Pass masked CI variables to docker build with --secret instead of --build-arg, because build args are printed in docker history.

Trade-offs

Pros

  • Build secrets leave no trace in layers, history or the saved tarball
  • File-based runtime secrets stay out of docker inspect and the process environment
  • Rotating a file-mounted secret needs no image rebuild

Cons

  • The app must read a file path (or an entrypoint must translate it), which not every program supports
  • File permissions on the host source decide whether a non-root container can read the secret
  • Compose secrets are a bind mount of a plain file: anyone with host or Docker API access can still read it

service_started, service_healthy and service_completed_successfully with real healthchecks, what each waits for, and how a failed dependency fails docker compose up.

The problem

The web service starts, tries to connect, and crashes because the database container is up but not yet accepting connections. Or a migration runs in parallel with the app, or the app starts against an unmigrated schema. `depends_on` with no condition looks like it solves this and does not: it orders container starts, not readiness.

How it works

  • Short form `depends_on: [db]` means `condition: service_started` with `required: true` (`docker compose config` prints exactly that). Compose waits until the dependency's container is started, not until its process listens: a db that runs `sleep 5; nc -l -p 5432` was reported started while the dependent's `nc -z db 5432` printed `db-not-listening-yet`.
  • `service_healthy` waits until the dependency's health status is `healthy`, so the dependency needs a `healthcheck:` (or one inherited from the image). In the test, db created `/tmp/ready` after 6 s with a 1 s interval; the log showed `Container db Waiting`, then `Healthy` about 8 to 9 s later, and only then did the dependent start. `docker compose ps` shows `Up 9 seconds (healthy)`.
  • `service_completed_successfully` waits for the dependency to exit with code 0, which fits one-shot jobs such as migrations and seeding. The chain in the example (db healthy, then migrate exits 0, then web) started web only after migrate showed `Exited`. If migrate exits non-zero, `up` stops with `service "migrate" didn't complete successfully: exit 3` and web stays in `Created` state, so a failed migration cannot be missed.
  • A dependency that never turns healthy fails the whole `up`: after the healthcheck's retries (`retries: 2`, `interval: 1s`) the output was `dependency failed to start: container ... is unhealthy` and the dependent was never started. The wait is bounded only by the healthcheck settings (`start_period`, `interval`, `retries`, `timeout`), so size those for the slowest realistic start, not for the happy path.
  • `restart: true` under a dependency makes Compose restart the dependent when it restarts the dependency: after `docker compose restart db`, the web container logged `start` twice. It follows restarts that Compose performs; do not assume it fires when the engine's restart policy revives a crashed dependency (version-dependent, not tested).
  • `required: false` allows a dependency that may not be enabled, typically a service in a profile. Without it, a dependency on a service excluded by a profile gives `service "web" depends on undefined service "cache": invalid compose project`. With the profile enabled, `cache` starts before `web`. A dependency cycle is rejected up front: `dependency cycle detected: a -> b -> a`.
  • Scope: `docker compose up web` also starts web's dependencies (db was created and started), while `docker compose run --no-deps web ...` skips them. On `down`, Compose typically stops dependents before the services they depend on.
  • Dependencies gate only the start. A db can restart or become unhealthy after web is running, and `depends_on` does nothing about it, so the application still needs connection retries with backoff. Use the condition for ordering and clear failure on startup, not as the only resilience mechanism.

Examples

db becomes healthy after about 8 s, migrate runs once and must exit 0, then web starts; $$ stops Compose from interpolating the date command.

yaml
name: demo-dep
services:
  db:
    image: busybox:1.37
    command: ["sh", "-c", "sleep 6; touch /tmp/ready; sleep 300"]
    healthcheck:
      test: ["CMD", "test", "-f", "/tmp/ready"]
      interval: 1s
      timeout: 1s
      retries: 20
      start_period: 1s
  migrate:
    image: busybox:1.37
    command: ["sh", "-c", "echo migrating; sleep 2; echo done"]
    depends_on:
      db:
        condition: service_healthy
  web:
    image: busybox:1.37
    command: ["sh", "-c", "echo web-started at $$(date +%T); sleep 300"]
    depends_on:
      db:
        condition: service_healthy
        restart: true
      migrate:
        condition: service_completed_successfully

Run in the directory with the file above; ps -a shows migrate as exited (0) and db as healthy.

bash
docker compose config | grep -B1 -A3 depends_on | head -20
docker compose up -d        # returns once web has started, or fails with the reason
docker compose ps -a --format '{{.Service}} {{.State}} {{.Status}}'
docker compose logs --no-log-prefix web migrate
docker compose restart db   # restart: true also restarts web
docker compose down -v

In practice

  • Compose Specification — Defines the three conditions plus `required` and `restart` under long-syntax depends_on; Compose v2 implements them.
  • Postgres pg_isready healthchecks — The common db healthcheck is `pg_isready -U user -d db`, so service_healthy means the server accepts connections, not just that the process exists.
  • Flyway, Liquibase, Alembic migration containers — Run as a one-shot service that the app waits on with service_completed_successfully.
  • wait-for-it and dockerize scripts — The older workaround for readiness before healthcheck conditions existed; still useful where the dependency is outside Compose.
  • Kubernetes init containers and readiness probes — The orchestrator equivalent: init containers run to completion first, probes decide when traffic is sent.

Trade-offs

Pros

  • Declarative ordering replaces sleep loops and wait-for scripts in entrypoints
  • A failed migration or unhealthy db fails docker compose up with a named reason
  • restart: true keeps dependents consistent when a dependency is recreated by Compose

Cons

  • Only start-up is gated; later dependency failures still need application retries
  • service_healthy is only as good as the healthcheck; a check that passes too early gives false readiness
  • A long wait or an unhealthy dependency blocks the whole up, including unrelated services that depend on it

How profiles switch services on and off, how compose.override.yaml and repeated -f merge (append versus replace), and !reset / !override, checked with docker compose config.

The problem

One Compose file has to serve laptops, CI and production. Debug tools and seed jobs must not start on a plain `docker compose up`, the dev bind mounts and extra ports must not leak into the prod file, and nobody remembers which of three `-f` files wins. Teams copy the file three times and the copies drift, or they find out which merge rule applies only after two published ports end up on one service.

How it works

  • `profiles: [debug]` on a service excludes it by default: `docker compose config --services` printed only `web`. A service without `profiles` always runs. Enable profiles with `--profile debug` (repeatable), `COMPOSE_PROFILES=tools,debug`, or `--profile '*'` for all of them (all three printed the expected lists).
  • A service in several profiles starts if any of them is active: `seed` with `profiles: [tools, debug]` appeared with only `--profile debug`.
  • Naming a profiled service on the command line activates it without `--profile`: `docker compose up -d debug` started it, and `docker compose run --rm seed` ran the one-off. This is why profiles suit tools you run on demand.
  • Teardown gotcha: `docker compose down` only handles services of the active profiles. After starting `debug` by name, plain `down` left `demo-pr-debug-1` running and printed `Network ... Resource is still in use`; `--remove-orphans` did not help. Run `docker compose --profile debug down` (or set `COMPOSE_PROFILES`) with the same profiles you started with.
  • File selection: with no `-f`, Compose reads `compose.yaml` and then `compose.override.yaml` automatically. Any explicit `-f` disables that automatic override: `docker compose -f compose.yaml config` showed `LOG_LEVEL: info`, the single `18080` port and the original `sleep 300`. With several `-f`, later files win, and the order matters (swapping the files in the test restored `FEATURE: off` and `sleep 300`).
  • Merge rules, as `config` printed them: scalars and `command` are replaced (`sleep 999` replaced `sleep 300`); mappings merge key by key (`environment` kept `FEATURE` and changed `LOG_LEVEL`); label lists were normalised to a map and merged; `ports` and `volumes` are appended (both `18080` and `18081` were published, and both the named volume and the bind mount were kept). Because lists append, an override file cannot remove an entry on its own.
  • `!reset` and `!override` change that: `ports: !reset []` removed every inherited port, `labels: !reset null` removed the labels, and `volumes: !override` plus `command: !override [...]` replaced the inherited value instead of merging. Put them in the later file; the tags need a recent Compose v2 (the Compose file reference says `!override` requires 2.24.4 or later; `!reset` arrived in the same 2.24 line, so check yours), and older versions fail to parse them.
  • Always render before deploying: `docker compose config` prints the merged, interpolated project, `config -q` only validates, and `config --services` lists the active services. `${TAG:-1.37}` supplies a default and `${MSG:?MSG is required}` fails with `required variable MSG is missing a value: MSG is required`, which is the cheap way to make a missing environment variable a hard error.

Examples

compose.yaml: web always runs; debug and seed only with a matching profile.

yaml
name: demo-pr
services:
  web:
    image: busybox:1.37
    command: ["sleep", "300"]
    environment:
      LOG_LEVEL: info
      FEATURE: "off"
    ports:
      - "18080:80"
    volumes:
      - data:/data
  debug:
    image: busybox:1.37
    command: ["sleep", "300"]
    profiles: [debug]
  seed:
    image: busybox:1.37
    command: ["sh", "-c", "echo seeding"]
    profiles: [tools, debug]
volumes:
  data:

Profiles on and off, the automatic override versus explicit -f, and !reset / !override; down needs the profile the container was started with.

bash
cat > compose.override.yaml <<'EOF'
services:
  web:
    command: ["sleep", "999"]
    environment:
      LOG_LEVEL: debug
    ports:
      - "18081:80"
    volumes:
      - ./src:/app
EOF
cat > prod.yaml <<'EOF'
services:
  web:
    command: !override ["sleep", "42"]
    ports: !reset []
    volumes: !override
      - data:/data
    restart: unless-stopped
EOF
docker compose config --services                      # web
docker compose --profile debug config --services      # web debug seed
COMPOSE_PROFILES=tools docker compose config --services
docker compose config web                             # base + override.yaml merged
docker compose -f compose.yaml config web             # explicit -f: no auto override
docker compose -f compose.yaml -f prod.yaml config web
docker compose up -d debug && docker compose --profile debug down

In practice

  • Compose Specification: profiles — Standardises profiles and the COMPOSE_PROFILES variable; used for debug containers, test runners and seed jobs.
  • compose.yaml + compose.override.yaml — The default convention: committed base file, and an override with dev-only mounts and ports that applies automatically.
  • CI pipelines running -f compose.yaml -f compose.ci.yaml — Pass explicit files so the developer-only override is never picked up by accident.
  • include and extends in Compose — Alternatives when sharing service definitions across projects, instead of stacking more -f files.
  • Kustomize and Helm values — The Kubernetes counterparts of per-environment overlays, with their own merge semantics.

Trade-offs

Pros

  • One base file with small overlays avoids three drifting copies
  • Profiles keep tools and seed jobs out of the default up without removing them from the file
  • docker compose config makes the merge result reviewable and diffable in CI

Cons

  • List fields append, so an overlay cannot remove inherited ports or volumes without !reset or !override
  • Implicit compose.override.yaml makes a bare docker compose up differ between machines
  • down and ps only see the active profiles, which leaves containers behind if you forget the flag

Troubleshooting

docker ps shows nothing or a Restarting loop: read State.Status, ExitCode and State.Error to tell a command that never started from an app that ran and returned, then fix the entrypoint, the foreground process or the config.

The problem

You run `docker run -d app` and `docker ps` shows nothing, or the status flips between `Up 1 second` and `Restarting (1)` forever. A container lives exactly as long as its PID 1, so every case of an instant exit means one of two things: the command never started (a path, permission or interpreter problem that runc reports), or the program started, did its work or hit an error, and returned. The first job is to find out which, because the fixes have nothing in common.

How it works

  • Symptom: `docker ps` is empty, `docker ps -a` shows `Exited (N) 2 seconds ago`, or the container is stuck in `Restarting (N)` or `Created`. A container that is `Up` for a few seconds and then gone is the same problem seen with `-d`: nothing keeps PID 1 alive.
  • First look, always the same three reads: `docker ps -a`, then `docker inspect -f 'status={{.State.Status}} exit={{.State.ExitCode}} oom={{.State.OOMKilled}} err={{.State.Error}}'`, then `docker logs`. `status=created` with a non-empty `err` means the process never started (checked: `/nonexistent-cmd` gave `created`, exit 127 and an `exec: ... no such file or directory` error). `status=exited` with empty `err` means the app ran and returned, and the reason is in the logs.
  • Exit 0 within a second is a command that simply finished. A shell with no stdin and no TTY exits at once (checked: `docker run -d alpine:3.20 sh` was `Exited (0)` after 0.13 s, while `-dit` stayed `Up`). A one-shot script, a migration or a `--version` check also ends this way. The container is working as designed; the image needs a long-running foreground process.
  • A daemonizing wrapper is the classic exit 0: `service nginx start`, a script that ends with `app &`, or a binary that forks and exits the parent. PID 1 returns, the container stops, and the background child is killed with it (checked: a wrapper that backgrounded `sleep 300` gave `Exited (0)` and the child was gone). Run the server in the foreground (`nginx -g 'daemon off;'`) or `exec` it as the last line of the entrypoint script.
  • Exit 1 to 3 with a message in `docker logs` is the application refusing to start: a missing environment variable, an unreachable database, a bad config path, a Python `ModuleNotFoundError` (exit 1) or `can't open file` (exit 2). Under `--restart=always` this becomes a loop (checked: `Restarting (3) 1 second ago`, `RestartCount` climbing, a `die` event with `exitCode=3` every few hundred milliseconds), and `docker logs` keeps the output of all runs, so the same line repeats.
  • Never started, exit 126, 127 or 255: 126 is a file without the execute bit or a directory given as the command (checked: `permission denied`). 127 is a wrong path or `--entrypoint`. 255 with a log line `exec /x.sh: no such file or directory` while the file exists means the shebang interpreter is missing or has a carriage return: `#!/bin/sh\r` from Windows line endings (checked), or `#!/bin/bash` on Alpine (checked). `exec /x.sh: exec format error` is a script without a shebang, or a binary built for another architecture (checked for the script).
  • Look inside without guessing: `docker run --rm -it --entrypoint sh image` starts the image with a shell instead of the broken entrypoint, and `docker cp <stopped-container>:/path -` reads files from a container that already exited (checked). To stop a restart loop from hiding the evidence, run `docker update --restart=no <name>`, which worked on a looping container (checked), then read the logs once.
  • Fix by cause, then verify: set the missing config, make the process stay in the foreground, `chmod +x` before COPY, convert to LF (`.gitattributes` with `eol=lf` for `*.sh`), install the interpreter or use `#!/bin/sh`. Verify with `docker run -d`, wait longer than the restart backoff, and confirm `State.Status=running` with `RestartCount` unchanged. Prevent it with a CI step that runs the image and asserts `.State.ExitCode`.

Examples

The three reads that split 'never started' from 'ran and returned'.

bash
docker ps -a --filter name=myapp --format '{{.Names}} | {{.Status}} | {{.Command}}'
# myapp | Exited (3) 1 second ago | "sh -c 'echo ...'"
docker inspect -f 'status={{.State.Status}} exit={{.State.ExitCode}} oom={{.State.OOMKilled}} err={{.State.Error}}' myapp
# status=exited exit=3 oom=false err=
# never started instead looks like:
# status=created exit=127 oom=false err=failed to create task for container: ... exec: "/nonexistent-cmd": stat /nonexistent-cmd: no such file or directory
docker logs --tail 20 myapp
docker events --since 30s --filter container=myapp --filter event=die

Reproduce the CRLF shebang trap: the file exists, the log says it does not.

bash
printf '#!/bin/sh\r\necho hi\r\n' > entry.sh
chmod +x entry.sh
cat > Dockerfile <<'EOF'
FROM busybox
COPY entry.sh /entry.sh
ENTRYPOINT ["/entry.sh"]
EOF
docker build -q -t crlf-demo .
docker run --rm crlf-demo; echo "rc=$?"
# exec /entry.sh: no such file or directory
# rc=255
docker run --rm --entrypoint sh crlf-demo -c "head -1 /entry.sh | od -c | head -1"
# 0000000   #   !   /   b   i   n   /   s   h  \r  \n

Break a restart loop long enough to read the evidence, then look inside the stopped container.

bash
docker update --restart=no myapp
docker logs --tail 5 myapp
docker cp myapp:/tmp/err.txt - | tar -xO
docker run --rm -it --entrypoint sh myapp-image

In practice

  • Docker Engine — Only the foreground process of PID 1 keeps a container alive; there is no init system that adopts a daemonized child, and when PID 1 returns, the runtime tears down the whole PID namespace.
  • runc — Reports `exec: ...: no such file or directory` or `permission denied` from the container-init stage, before the application runs; Docker surfaces it in `State.Error` and as exit 126 or 127.
  • Official nginx image — Ships `CMD ["nginx", "-g", "daemon off;"]` for exactly this reason: without `daemon off;` nginx forks and the container exits 0.
  • Kubernetes — The same exit shows up as `CrashLoopBackOff` with the code in `lastState.terminated.exitCode`; the diagnosis is identical, with the pod restart policy playing the role of `--restart`.

Trade-offs

Pros

  • Reading `State.Status`, `ExitCode` and `Error` first splits the problem in two in under a minute and avoids rebuilding the image on a guess.
  • `--entrypoint sh` and `docker cp` need no changes to the image and work on the exact artifact that fails.
  • Exit codes and `State.Error` are stable fields that can be asserted in CI.

Cons

  • A restart policy hides the first failure behind a loop; `docker logs` then mixes every run, so you must read it after `--restart=no`.
  • Sleep-forever tricks (`tail -f /dev/null`) keep the container up but hide a service that never started.
  • With `docker run --rm`, a failed container disappears with its State and its logs, so debug runs should drop `--rm`.

Decode a container exit code by origin: the application, the shell, Docker and runc before the app ran (125, 126, 127), a signal (128+n) or runc init after start (255), and confirm it with State.Error and OOMKilled.

The problem

A pipeline reports `exited with code 137`, a dashboard shows `Exited (255)`, or `docker run` returns 125 before any log line exists. The number alone is ambiguous: 127 can come from runc or from a shell inside the container, 137 can be the OOM killer, a manual kill or a slow shutdown, and 0 can hide a failure. The way to decode it is to ask who set the number, then confirm with the fields Docker records next to it.

How it works

  • Symptom and where to read it: `docker run` returns the container's code as its own exit status; for a running or finished container use `docker wait <name>` or `docker inspect -f '{{.State.ExitCode}}'`; `docker events --filter event=die` prints `exitCode=` per death. Statuses are 8 bits: `exit 300` in a shell returned 44 (checked).
  • Four origins explain every code. The application or its shell sets 0 to 2 and any custom value. Docker and runc set 125, 126 and 127 when they fail before the app runs. The kernel kills by signal and the runtime reports 128 plus the signal number. runc's init process exits 255 when the final exec fails after the container started.
  • 125 means the docker CLI or daemon itself failed and no container process ran: an unknown flag or a pull of a missing image both returned 125 (checked), as do name conflicts and `port is already allocated`. The message is on the stderr of `docker run`; there may be no container to inspect.
  • 126 and 127 come from runc's exec step. `permission denied` (no execute bit, or a directory given as the command) returned 126 and `no such file or directory` for a command or `--entrypoint` returned 127 (both checked). The container is left in `State.Status=created` with `State.Error` filled in. The same 127 from `sh -c nonexistent_cmd` has `Status=exited` and an empty `Error`, because that shell is the application (checked).
  • 255 is what runc's init process exits with (it crashes with 255 when `StartInitialization` fails) when the exec fails after start, with the message on the container's output, not in `State.Error` (checked: `status=exited exit=255 err=[]`). Typical causes: a script with a CRLF shebang or a missing interpreter (`exec /x.sh: no such file or directory`), a script with no shebang or a wrong-architecture binary (`exec format error`), and a dynamic binary in an image with no loader. An application doing `exit(-1)` also gives 255.
  • 128+n is a signal: 130 is SIGINT, 137 SIGKILL, 139 SIGSEGV and 143 SIGTERM; `kill -l n` names the rest (134 is typically SIGABRT from an assert, not run here). 139 appeared only from a real fault (a null read via ctypes), because a signal sent from inside to PID 1 is ignored unless PID 1 installed a handler: `kill -SEGV $$` as PID 1 ended with 0 (checked).
  • 137 has three different causes. If `State.OOMKilled` is true the memory limit killed it (checked: `--memory 32m` with a 200 MB allocation gave 137 and `OOMKilled=true`). If it is false, either `docker stop` timed out on a PID 1 that ignores SIGTERM (checked: 10.18 s, then 137) or someone ran `docker kill` (checked: 137). The OOM killer can also take only a child of PID 1, leaving exit 0 with `OOMKilled=true`.
  • 143 means SIGTERM killed a process that did not handle it: with `--init`, `docker stop` took 0.2 s and returned 143 (checked), against 10 s and 137 without it. That is often acceptable; a deliberate clean shutdown needs a handler that exits 0. Exit 0 is not proof of success either, because a daemonizing wrapper or a shell that lost its stdin returns 0, and `--restart on-failure` does not restart on 0.

Examples

Reproduce the codes on throwaway containers; every line below was run on Engine 29.8.2.

bash
docker run --rm busybox sh -c 'exit 2'; echo "app: $?"
# app: 2
docker run --rm --badflag busybox 2>/dev/null; echo "docker: $?"
# docker: 125
docker run --rm busybox /etc/passwd 2>/dev/null; echo "no exec bit: $?"
# no exec bit: 126
docker run --rm busybox /nonexistent-cmd 2>/dev/null; echo "runc not found: $?"
# runc not found: 127
docker run --rm busybox sh -c 'nonexistent_cmd' 2>/dev/null; echo "shell not found: $?"
# shell not found: 127
docker run --rm busybox sh -c 'kill -INT $$'; echo "sigint: $?"
# sigint: 130
docker run --rm python:3.13-slim python -c 'import ctypes; ctypes.string_at(0)'; echo "segv: $?"
# segv: 139
docker run -d --name demo-s busybox sleep 1000 >/dev/null
time docker stop demo-s; docker inspect -f 'exit={{.State.ExitCode}}' demo-s
# real 0m10.18s   exit=137
docker rm demo-s
docker run -d --init --name demo-t busybox sleep 1000 >/dev/null
docker stop demo-t; docker inspect -f 'exit={{.State.ExitCode}}' demo-t
# exit=143
docker rm demo-t

Tell the three kinds of 137 apart, and the two kinds of 127.

bash
docker inspect -f 'status={{.State.Status}} exit={{.State.ExitCode}} oom={{.State.OOMKilled}} err={{.State.Error}}' myapp
# runc 127:  status=created exit=127 ... err=failed to create task ... exec: "/x": stat /x: no such file or directory
# shell 127: status=exited  exit=127 oom=false err=
# OOM 137:   status=exited  exit=137 oom=true  err=
# kill 137:  status=exited  exit=137 oom=false err=
docker events --since 10m --filter event=die --filter event=oom --format '{{.Time}} {{.Type}} {{.Action}} {{.Actor.Attributes.name}} exit={{.Actor.Attributes.exitCode}}'
docker ps -a --filter status=exited --format '{{.Names}} | {{.Status}}'

In practice

  • runc and containerd — 126 and 127 follow the shell convention for 'cannot execute' and 'not found', and runc applies it when its exec step fails; Docker keeps the message in `State.Error`.
  • Kubernetes — Reports the same numbers in `lastState.terminated.exitCode`, with `reason: OOMKilled` set for memory kills; 137 plus that reason is the pod-level version of `State.OOMKilled=true`.
  • Docker Compose — `docker compose up --abort-on-container-exit` and `--exit-code-from <service>` pass a container's exit code through as the command's exit status, which is how CI sees it.
  • CI runners — A job failing with exit 137 on a small runner is usually the memory limit or a stop timeout; the `OOMKilled` flag settles it in one inspect.

Trade-offs

Pros

  • Asking who set the number resolves most ambiguity in one `docker inspect`.
  • `State.Error` and `OOMKilled` are recorded by Docker itself and survive after the process is gone.
  • The mapping is stable across images and languages, so the same reading works for any container.

Cons

  • Codes carry little detail; 1 only says that the app failed, and the cause is still in the logs.
  • Exit 0 and a custom code can mean anything, since the application decides them.
  • `docker rm` or `--rm` deletes the State, so capture the code (`docker wait`, `die` events) before removal.

no space left on device on a Docker host: split the usage with docker system df -v, docker ps -s and du on the data root, find unrotated json logs, dangling volumes, stopped containers and build cache, and clean only what you can name.

The problem

A Docker host starts failing with `no space left on device`: `docker pull` and `docker build` abort, containers cannot write, or dockerd itself becomes unstable. `df` shows the filesystem holding `/var/lib/docker` at 100%, but nothing says whether the weight is images, the writable layers of containers, volumes, build cache or one container's log. Cleaning blindly with `docker system prune -a --volumes` or `docker volume prune -a` can delete data you still need, so the aim is to attribute the bytes first and remove only what you can name.

How it works

  • Symptom: `no space left on device` from pull, build or a container write; `df -h $(docker info -f '{{.DockerRootDir}}')` near 100%. Check inodes too with `df -i` (see the Linux disk and inode entry). Then `docker system df` splits usage into Images, Containers, Local Volumes and Build Cache with a RECLAIMABLE column, and `docker system df -v` lists every image, container, volume and cache record. On the test host it showed containers at 209.7MB and volumes at 146.8MB after the scenario below, matching the files written.
  • Then the host view: `du -xh --max-depth=1 /var/lib/docker | sort -h`. On Engine 29.8.2 with the containerd image store most bytes sit under `containerd/` (2.6G of 2.8G here), with `volumes/`, `containers/` (config and logs) and `buildkit/` beside it. A container's layer lives at `/var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.overlayfs/snapshots/<n>/fs` (checked: a 150 MB file found there); with the classic overlay2 store the path is typically `/var/lib/docker/overlay2/<id>/diff`.
  • Writable layers: `docker ps -a -s` adds SIZE and virtual size; a running container that wrote 150 MB showed `157MB (virtual 162MB)` and a stopped one `52.4MB`. Stopped containers keep their layer until `docker rm`, and `docker diff <name>` lists the files. Large data written inside a container is the symptom of a missing volume.
  • Logs are the usual surprise. With json-file and no rotation (the default) a chatty container grows `/var/lib/docker/containers/<id>/<id>-json.log` without bound: 400,000 lines became 56 MB, and `docker system df` still showed that container at 4.1kB, so the log is invisible to it. Find them with `du -sh /var/lib/docker/containers/*/*-json.log | sort -h`.
  • Free a log in place with `truncate -s 0 "$(docker inspect -f '{{.LogPath}}' name)"`: the container kept logging afterwards (checked). Do not `rm` the file: dockerd keeps it open, so the space is not returned and `/proc/<pid>/fd` shows `(deleted)`, while `docker logs` fails with `no such file or directory` (checked). Fix the cause with `--log-opt max-size=10m --log-opt max-file=3` or `daemon.json`, which only affects containers created afterwards.
  • Volumes: `docker volume ls -f dangling=true` lists volumes no container uses. An anonymous volume (`-v /path` or an image `VOLUME`) outlives `docker rm` unless the volume is removed with it (checked: 62.9 MB stayed after `docker rm`, under a hash name). A named volume with data is never garbage by itself, so check the contents (`docker volume inspect`, `du` on the Mountpoint) before removing it.
  • Images and build cache: `docker images -f dangling=true` lists untagged `<none>` images (a retag may leave the old image listed or not depending on the image store, so go by the list). `docker buildx du` shows the cache, which was 1.5 GB here, and `docker builder prune --filter until=24h` trims by age. Clean in order of evidence: logs, stopped containers, dangling volumes, unused images, cache.
  • Clean only what you can name. Label filters scope the prune commands: `docker container prune -f --filter label=x=1`, `docker volume prune -a -f --filter label=x=1` and `docker image prune -f --filter label=x=1` each removed only labelled objects (checked). `docker system prune -a --volumes` deletes every unused image plus unused anonymous volumes (the CLI help says `--volumes` prunes anonymous volumes; on Engine 23.0 and later named volumes need `docker volume prune -a`, older engines removed them too), so treat it as a last resort. Verify with `docker system df` and `df -h`; if `df` does not drop, look for deleted-but-open files with `lsof +L1`.

Examples

Attribute the bytes before deleting anything: Docker's own view first, then the host view.

bash
docker system df
# TYPE            TOTAL   ACTIVE   SIZE      RECLAIMABLE
# Images          18      0        646.7MB   621MB (96%)
# Containers      3       2        209.7MB   52.43MB (25%)
# Local Volumes   3       0        146.8MB   146.8MB (100%)
# Build Cache     255     0        1.507GB   1.415GB
docker system df -v | head -40
docker ps -a -s --format 'table {{.Names}}\t{{.Status}}\t{{.Size}}'
du -xh --max-depth=1 /var/lib/docker | sort -h | tail -6
du -sh /var/lib/docker/containers/*/*-json.log | sort -h | tail -3
docker volume ls -f dangling=true
docker buildx du | tail -3

A labelled disk-eater and a clean-up that can only touch it. Replace the label with your own.

bash
docker run -d --name eat1 --label demo=disk busybox sh -c 'dd if=/dev/zero of=/big bs=1M count=150 2>/dev/null; sleep 600'
docker run -d --name eat2 --label demo=disk busybox sh -c 'i=0; while [ $i -lt 400000 ]; do echo "line $i padding padding padding padding padding"; i=$((i+1)); done; sleep 600'
docker volume create --label demo=disk eat-vol
sleep 5
docker ps -a -s --filter label=demo=disk --format 'table {{.Names}}\t{{.Size}}'
ls -lh "$(docker inspect -f '{{.LogPath}}' eat2)"
truncate -s 0 "$(docker inspect -f '{{.LogPath}}' eat2)"
docker rm -f eat1 eat2
docker container prune -f --filter label=demo=disk
docker volume prune -a -f --filter label=demo=disk
docker image prune -f --filter label=demo=disk

Rotate json logs per service so the next incident cannot happen. Applies to containers created after the change.

yaml
services:
  app:
    image: busybox
    command: ["sh", "-c", "while true; do echo tick; sleep 1; done"]
    logging:
      driver: json-file
      options:
        max-size: "10m"
        max-file: "3"
        compress: "true"

In practice

  • Docker Engine — The default json-file driver has no rotation unless `max-size` is set, which is why `/var/lib/docker/containers` is the first place to look when only one container grows.
  • containerd image store — Since the containerd store became the default for fresh installs (Engine 29.0), layers and snapshots live under `/var/lib/docker/containerd/...` rather than `overlay2`, so old runbooks that `du` the overlay2 directory find nothing.
  • CI build hosts — Build cache is the usual grower on a shared builder; a scheduled `docker builder prune --filter until=...` bounds it without touching images or volumes.
  • dockerd, df, lsof — Deleted-but-open files make `df` stay full after a clean-up; this is the same mechanism as in the Linux deleted-but-open-files entry, with dockerd as the holder.

Trade-offs

Pros

  • Attributing first with `docker system df -v`, `docker ps -s` and `du` makes every deletion defensible and often frees more than a blanket prune.
  • Truncating a log is instant, needs no restart and keeps the container running.
  • Label-scoped prune commands make clean-up safe on shared daemons.

Cons

  • `docker system df` does not count container logs, and its reclaimable figure ignores shared layers, so the numbers are a guide.
  • Truncating drops log history; if the logs matter, ship them first.
  • Anonymous volumes have no label to filter on, so they must be removed by name after you check them.

A published port resets, a service name does not resolve, or a container cannot reach out: check the listening address, the -p mapping, the network type for DNS, then DNS servers, host forwarding and DOCKER-USER rules, in that order.

The problem

A service works inside its container but `curl localhost:8080` from the host gets `Connection reset by peer`, a client container cannot resolve `db`, or a container cannot reach the internet. Docker networking has four layers that fail in different ways, and the same symptom can come from two of them. The reliable method goes from the inside out: what is listening and on which address, what the port mapping points at, which network type gives name resolution, then DNS servers and the host's forwarding rules.

How it works

  • Symptom to layer: 'reset by peer' or 'connection refused' on a published port is the listener or the mapping; 'bad address' or NXDOMAIN is name resolution; a timeout is usually a firewall or forwarding drop. Do not change anything until you know the layer.
  • First look inside: list listeners with `docker exec <c> netstat -tln` or `ss -ltn`. If the image has no tools, enter its network namespace from the host: `nsenter -t $(docker inspect -f '{{.State.Pid}}' <c>) -n ss -ltn` (checked). The address column is the answer to half of all cases.
  • Listening on 127.0.0.1 inside the container is the top cause. `-p 18083:8000` still creates the DNAT rule, but the app only accepts on the container's loopback, so the host got `Connection reset by peer` (curl exit 56) and the container's own IP got `Connection refused` (curl exit 7), while `wget http://127.0.0.1:8000` inside the container worked (all checked). Bind to `0.0.0.0` (`--host 0.0.0.0`, `HOST=0.0.0.0`, `httpd -p 8000`).
  • A wrong container port in `-p` fails the same way. `-p 18082:80` with the app on 8000 gave `Connection reset by peer`, and `docker port <c>` printed `80/tcp -> 0.0.0.0:18082`, which looks fine; the rule is `-A DOCKER ! -i docker0 -p tcp -m tcp --dport 18082 -j DNAT --to-destination <ip>:80` in `iptables -t nat -S DOCKER`. Docker does not check what listens, and `EXPOSE` does not publish. Compare the right-hand port with the listener's port.
  • An unpublished port is reachable by container IP but not by `localhost`. The container's IP worked from the host (`172.17.0.5:8000`) while `localhost:8000` gave exit 7. Take the IP from `.NetworkSettings.Networks.<net>.IPAddress`: on Engine 29.8.2 the top-level `.NetworkSettings.IPAddress` is gone and the template errors with `map has no entry for key` (checked). A publish without an address binds `0.0.0.0`; `-p 127.0.0.1:18085:8000` keeps it on the host's loopback (checked with `ss`).
  • Names need a user-defined network. On the default `bridge`, `ping <other-container-name>` gave `bad address` and `/etc/resolv.conf` simply copied the host's servers; `--link` printed a deprecation warning. On a user-defined bridge the embedded resolver `127.0.0.11` answered `nslookup web` with the container's address and NXDOMAIN for unknown names. Containers on different networks cannot resolve each other (checked), and `localhost` inside a container is the container itself, so a client must use the service name and the container port, not the host's published port.
  • Container to host and container to outside: `host.docker.internal` is not defined on Linux by default (`bad address`); `--add-host=host.docker.internal:host-gateway` maps it to the bridge gateway `172.17.0.1` (checked), and a host service bound to `127.0.0.1` is then `Connection refused` while one bound to `0.0.0.0` answers (checked). With a dead resolver (`--dns 192.0.2.53`), `nslookup` printed `connection timed out; no servers could be reached` and `wget` printed `bad address` (checked), so an unreachable DNS server looks like a missing name.
  • Forwarding and firewall, the last layer: traffic leaving a container crosses the host's FORWARD chain, whose policy Docker sets to DROP, with `DOCKER-USER` evaluated first (`iptables -S FORWARD`, `iptables -S DOCKER-USER`). A DROP for the network's subnet in DOCKER-USER made `wget` time out, and `net.ipv4.ip_forward=0` did the same for outbound traffic (both checked). In that state `curl localhost:<published>` and container-to-container traffic on one bridge still worked, so 'works locally, fails for remote or outbound' points here. A remote client reaching a published port with `ip_forward=0`, and the interplay with ufw or firewalld, typically fail the same way but were not run here. Restore the setting and remove only your own test rule.

Examples

Inside out: listener, mapping, container IP, then the same request from a peer container.

bash
C=web
docker exec "$C" netstat -tln
# tcp 0 0 127.0.0.1:8000 0.0.0.0:* LISTEN    <- loopback only: the host sees a reset
nsenter -t "$(docker inspect -f '{{.State.Pid}}' "$C")" -n ss -ltn
docker port "$C"
# 8000/tcp -> 0.0.0.0:18083
docker inspect -f '{{json .NetworkSettings.Ports}}' "$C"
IP=$(docker inspect -f '{{.NetworkSettings.Networks.bridge.IPAddress}}' "$C")
curl -sS -m 3 "http://$IP:8000"; echo "rc=$?"
curl -sS -m 3 "http://localhost:18083"; echo "rc=$?"
# rc=7 refused on the container IP, rc=56 reset on the published port
iptables -t nat -S DOCKER | grep -- '--dport 18083'

Services reach each other by name and container port on the Compose default network; only the host side is published, and only on loopback.

yaml
services:
  web:
    image: busybox
    command: ["httpd", "-f", "-p", "8000", "-h", "/www"]
    volumes:
      - ./www:/www:ro
    ports:
      - "127.0.0.1:18090:8000"
  client:
    image: busybox
    command: ["sh", "-c", "wget -q -O- http://web:8000"]
    depends_on:
      - web

The forwarding layer: read-only checks first. Only add or remove a DOCKER-USER rule that you created yourself.

bash
cat /proc/sys/net/ipv4/ip_forward
# 1
iptables -S FORWARD
# -P FORWARD DROP
# -A FORWARD -j DOCKER-USER
# -A FORWARD -j DOCKER-FORWARD
iptables -S DOCKER-USER
iptables -t nat -S POSTROUTING | grep MASQUERADE
docker run --rm busybox wget -T3 -q -O /dev/null http://1.1.1.1; echo "egress rc=$?"
docker run --rm --dns 192.0.2.53 busybox nslookup example.com 2>&1 | grep -i 'timed out'
# ;; connection timed out; no servers could be reached
docker run --rm --add-host=host.docker.internal:host-gateway busybox grep host.docker /etc/hosts
# 172.17.0.1 host.docker.internal

In practice

  • Docker Engine — Publishing is DNAT plus a userland proxy: it forwards to a container IP and port without checking that anything listens there, which is why a wrong port and a loopback-only listener look identical from outside.
  • Docker Compose — Creates a user-defined bridge per project, so services resolve each other by service name without `links`; `ports:` is only needed for traffic from the host or outside.
  • DOCKER-USER chain — The documented place for site policy: rules there run before Docker's own FORWARD rules and survive Docker rewriting its chains, so host firewall rules belong there, not in `DOCKER-FORWARD`.
  • Kubernetes — A container bound to 127.0.0.1 is invisible to a Service for the same reason, and the same listen-address check applies before touching the Service definition.

Trade-offs

Pros

  • Working from the listener outward isolates the layer with a handful of read-only commands, and `nsenter` works on images with no tools.
  • Each failure leaves a distinct signature (reset, refused, `bad address`, timeout), so the symptom narrows the layer before any change.
  • Compose service names and a user-defined network remove most hard-coded IP and `links` problems.

Cons

  • Reset by peer is ambiguous: a loopback listener and a wrong container port look the same, so both checks are needed.
  • iptables output differs across backends (nf_tables here) and with ufw or firewalld, so rule listings are not portable.
  • Toggling `ip_forward` or adding DROP rules to prove a theory affects every container on the host; do it on a test network.