Linux Troubleshooting Cheatsheet

How to find out what a slow, full or unreachable Linux host is doing: the USE method and a one-minute checklist, then CPU, memory, disk and network triage, and step-by-step playbooks for CPU saturation, memory pressure, slow disks, connection exhaustion and unresponsive hosts. Each entry gives the problem, how to read the tools, runnable commands, real-world notes and trade-offs.

Updated: 2026-10-09

29 entries

Quick tables

Reference and comparison tables for quick scanning.

USE method: tool by resource

One command or file per cell. `sar` needs the `sysstat` package; `perf`-style tools and reading `dmesg` may need root. Treat a blank answer as 'no easy Linux metric', not 'fine'.

USE method: tool by resource
ResourceUtilizationSaturationErrors
CPU`mpstat -P ALL 1`, `pidstat 1`, `top` (per core and per process)`vmstat 1` column `r` above core count, `/proc/pressure/cpu`, cgroup `cpu.stat` `nr_throttled``dmesg` machine-check or thermal messages (rare)
Memory capacity`free -m` (`available`), `/proc/meminfo` (`MemAvailable`)`vmstat 1` `si`/`so`, `/proc/pressure/memory`, `sar -B` (page scan and steal rates)OOM lines in `dmesg`, failed allocations, cgroup `memory.events` `oom_kill`
Storage device`iostat -xz 1` `%util` (rough on NVMe and RAID), `sar -d``iostat -xz 1` `aqu-sz` and `await`, `/proc/pressure/io`, `vmstat` `b``dmesg` `I/O error`, `smartctl -a`, filesystem remount read-only
Storage capacity`df -h`, `df -i` (inodes)Little direct signal; writes fail at 100% (`ENOSPC`)`No space left on device` in application and kernel logs
Network interface`sar -n DEV 1`, `ip -s link` against link speed`ip -s link` dropped and missed, `nstat -az` (`TcpExtTCPBacklogDrop`), `ss -ti` `retrans``ip -s link` errors, `ethtool -S` counters, `sar -n EDEV 1`
TCP stack`ss -s` (socket totals), `/proc/net/sockstat``ss -ltn` `Recv-Q` on LISTEN, `nstat -az TcpExtListenOverflows`, `sar -n ETCP 1` `retrans/s``nstat -az` (`TcpAttemptFails`, `TcpExtTCPTimeouts`), `/proc/net/snmp`
Kernel mutex and scheduler`pidstat -w 1` context switches, `perf` lock profiling`vmstat 1` `cs`, `r` and `b` with low CPU use, `/proc/<pid>/stack` (root)Hung task messages in `dmesg`
File descriptors and tasks`cat /proc/sys/fs/file-nr`, `ls /proc/<pid>/fd | wc -l`, `ps -eLf | wc -l`Nothing queues: allocation fails at the limit`Too many open files` (`EMFILE`/`ENFILE`), `fork: Resource temporarily unavailable` (`EAGAIN`)
conntrack table`conntrack -C` against `nf_conntrack_max`None before the limit; then packets drop`dmesg` `nf_conntrack: table full, dropping packet`, `conntrack -S` `drop` and `insert_failed`
cgroup (container)`memory.current`, `cpu.stat` `usage_usec` (cgroup v2)`cpu.stat` `nr_throttled`, `memory.pressure`, `memory.events` `high``memory.events` `oom` and `oom_kill`

The method comes from Brendan Gregg; the cells here are a condensed, independent mapping to Linux commands. Names such as `nr_throttled` assume cgroup v2.

/proc and /sys files worth knowing

These are the plain-text kernel interfaces that the usual tools read, so you can query them directly on a minimal host. `<pid>` is a process ID. Some files need root, and the `pressure` files exist only when the kernel has PSI enabled.

/proc and /sys files worth knowing
FileWhat it holdsRead it for
`/proc/loadavg`1/5/15 min load, runnable/total tasks, last PIDSame source as `uptime`; cheap to poll
`/proc/meminfo`System memory counters in kB`MemAvailable`, `Slab`/`SReclaimable`, `Dirty`, `Committed_AS`, `AnonPages`
`/proc/pressure/{cpu,memory,io}`PSI: `some`/`full` share of time stalled, `avg10/60/300`Kernel-measured saturation; `memory full` rising means real thrashing
`/proc/<pid>/status`Process state, `VmRSS`, `VmSwap`, `Threads`, context-switch countsQuick memory and thread count; `State: D` check
`/proc/<pid>/stat`One line of raw counters (CPU ticks, state, RSS pages)What `ps` and `top` parse; use it for scripts
`/proc/<pid>/smaps_rollup`Summed memory map: `Rss`, `Pss`, `Anonymous`, `Swap`True per-process memory (`Pss`) without parsing the full `smaps`
`/proc/<pid>/limits`Effective resource limits of that process`Max open files`, `Max processes`; the limit actually applied after systemd
`/proc/<pid>/fd/`One symlink per open file descriptorCounting fds, spotting `(deleted)` files and leaked sockets
`/proc/<pid>/wchan` and `stack`Kernel function a sleeping task waits in; full kernel stack (root)Why a task is in `D` state
`/proc/net/sockstat`Socket totals per protocol, TCP `tw` and `orphan` countsTime-wait pile-up, orphaned sockets, socket memory
`/proc/net/snmp` and `/proc/net/netstat`Cumulative IP, TCP and UDP counters (what `nstat` prints)Retransmits, resets, listen overflows
`/proc/net/softnet_stat`Per-CPU packet processing stats, hexSecond column dropped packets, third `time_squeeze`
`/proc/diskstats`Cumulative per-device I/O countersSource of `iostat`; compute rates from two samples
`/proc/vmstat`Virtual-memory event counters`pgmajfault`, `pswpin`/`pswpout`, `oom_kill`, `allocstall_*`
`/proc/sys/`, i.e. `sysctl`Tunable kernel parameters (`vm.*`, `net.*`, `fs.*`, `kernel.*`)Read with `sysctl -a`; persist changes in `/etc/sysctl.d/`
`/sys/fs/cgroup/<group>/`cgroup v2 files: `memory.max`, `memory.current`, `memory.events`, `cpu.stat`, `cpu.max`Container limits, throttling and OOM counts
`/sys/class/net/<if>/statistics/`Per-interface byte, packet, error and drop countersWhat `ip -s link` shows, as one file per counter

Most counters are cumulative since boot: sample twice and subtract. Field layouts are documented in the kernel's `Documentation/filesystems/proc.rst` and `Documentation/admin-guide/`.

Signals and exit codes

Signal numbers shown are for x86 and ARM Linux; a few (SIGBUS, SIGUSR1, SIGSTOP and others) differ on MIPS, Alpha and SPARC. A process killed by signal N is reported by the shell as exit status 128+N.

Signals and exit codes
Signal or codeNumberDefault action / meaningTypical source
`SIGHUP`1Terminate; daemons often reload config insteadTerminal closed, `kill -HUP`, `systemctl reload` units
`SIGINT`2TerminateCtrl-C in a terminal
`SIGQUIT`3Terminate and dump coreCtrl-\ in a terminal; the JVM prints a thread dump instead
`SIGABRT`6Terminate and dump core`abort()`, failed `assert`, glibc heap corruption detection
`SIGBUS`7Terminate and dump coreBad mmap access, truncated mapped file, misaligned access
`SIGKILL`9Terminate immediately; cannot be caught, blocked or ignored`kill -9`, the OOM killer, a cgroup memory limit, `docker kill`
`SIGUSR1` / `SIGUSR2`10 / 12Terminate unless the program handles themApplication-defined (log reopen, debug dump)
`SIGSEGV`11Terminate and dump coreInvalid memory access, NULL dereference, stack overflow
`SIGPIPE`13TerminateWriting to a pipe or socket whose reader has gone
`SIGALRM`14Terminate`alarm()` or interval timer expired
`SIGTERM`15Terminate; can be caught for graceful shutdown`kill` default, `systemctl stop`, Kubernetes pod termination
`SIGCHLD`17Ignored by defaultA child exited or stopped; the parent should `wait()`
`SIGCONT`18Resume a stopped process`kill -CONT`, `fg`
`SIGSTOP`19Stop; cannot be caught or ignored`kill -STOP`, debuggers
`SIGTSTP`20Stop; can be caughtCtrl-Z in a terminal
Exit 00SuccessNormal completion
Exit 1 / 21 / 2General error / misuse of a shell builtin or bad arguments (convention only)Application-defined; many tools use 2 for usage errors
Exit 126126Command found but not executableMissing execute bit, wrong architecture, a directory
Exit 127127Command not foundTypo, `PATH` missing in cron/systemd, absent interpreter in a shebang or container image
Exit 130128+2Killed by `SIGINT`Ctrl-C
Exit 137128+9Killed by `SIGKILL`OOM killer, cgroup limit, `kill -9`, or the orchestrator's kill after the grace period
Exit 139128+11Killed by `SIGSEGV`Segmentation fault; look for a core dump or `dmesg` line
Exit 143128+15Killed by `SIGTERM`Normal stop request where the process did not handle the signal

`kill -l` lists the numbers on the current machine, and `man 7 signal` has the full table. Exit 137 alone does not say who sent the SIGKILL: check `dmesg -T | grep -i -E 'killed process|oom'` and the container's OOM status first.

vmstat and iostat columns: what they mean and when to worry

Column names are from procps-ng `vmstat` and sysstat 12.x `iostat -x`. Ignore the first line of both tools (average since boot). Thresholds are rules of thumb: compare with the same host on a quiet day.

vmstat and iostat columns: what they mean and when to worry
ToolColumnMeaningRed flag
`vmstat``r`Runnable tasks (running or waiting for a CPU)Consistently above the CPU core count
`vmstat``b`Tasks in uninterruptible sleep (usually waiting on IO)Persistently above 0 together with high `wa`
`vmstat``swpd` / `free` / `buff` / `cache`Swap used, idle memory, buffer cache, page cache (kB)Low `free` alone is normal; use `free -m` `available` instead
`vmstat``si` / `so`Memory swapped in from / out to disk (kB/s)Sustained non-zero `si` and `so` together: swap thrashing
`vmstat``bi` / `bo`Blocks received from / sent to block devices (blocks/s, 1024 bytes)Large `bo` bursts with `b` and `wa` rising: writeback stall
`vmstat``in` / `cs`Interrupts and context switches per second`cs` far above its baseline with high `sy`: lock contention or too many threads
`vmstat``us` / `sy` / `id` / `wa` / `st`CPU time: user, kernel, idle, waiting for IO, stolen by hypervisor`wa` high with idle CPUs: IO bound; `st` above a few percent: noisy neighbour
`iostat -x``r/s` / `w/s`Read / write requests completed per second (IOPS)Plateau at a known device or volume IOPS cap
`iostat -x``rkB/s` / `wkB/s`Read / write bandwidth in kB per secondFlat at the throughput cap while `await` climbs
`iostat -x``rareq-sz` / `wareq-sz`Average read / write request size in kBTiny sizes at high IOPS: random small-IO workload
`iostat -x``r_await` / `w_await`Average ms from submission to completion, including queue timeRising steadily under constant load, or far above this device's baseline
`iostat -x``aqu-sz`Average queue length (`avgqu-sz` in older sysstat)Above the device's useful parallelism with rising `await`
`iostat -x``rrqm/s` / `wrqm/s`Requests merged by the block layer per secondInformational: high merges mean sequential access
`iostat -x``%util`Share of time the device had IO outstanding100% on a single HDD means saturated; on SSD/NVMe/RAID it only means never idle
`iostat -x``f_await`Average ms for flush requests (sysstat 12.3.1+, kernel 5.5+)Large values: fsync-heavy writers or a slow write cache

`vmstat -d` prints per-disk totals and `vmstat -D` a summary; `bi`/`bo` count whole-system block IO, while `iostat -x` is per device. Details: `iostat-reading`.

The 11 TCP states

`ss` prints these with hyphens (`TIME-WAIT`, `CLOSE-WAIT`, and `ESTAB` for ESTABLISHED); RFC 793 and `netstat` use underscores. Only a few of them are ever worth worrying about in numbers.

The 11 TCP states
StateMeaningWho is in itNormal?
`LISTEN`Waiting for incoming connections; the accept queue lives here.A server socket after `listen()`.Yes. Check `Recv-Q` against `Send-Q` for overflow.
`SYN-SENT`SYN sent, waiting for SYN-ACK.The connecting client.Briefly. Many stuck here means the peer is unreachable, filtered, or dropping SYNs.
`SYN-RECV`SYN received, SYN-ACK sent, waiting for the final ACK.The server, for half-open connections.Briefly. A large count suggests a SYN flood or a full backlog.
`ESTABLISHED`Handshake complete, data flows both ways.Both ends.Yes. Growth without traffic hints at leaks or idle connections never closed.
`FIN-WAIT-1`Sent FIN, waiting for its ACK (or the peer's FIN).The side that closed first.Briefly. Many lingering means the peer is not answering.
`FIN-WAIT-2`FIN acknowledged, waiting for the peer's FIN.The side that closed first.Briefly. Lingers while the peer sits in CLOSE-WAIT; orphaned sockets are bounded by `tcp_fin_timeout`.
`CLOSE-WAIT`Peer sent FIN and the kernel ACKed it; waiting for the local application to call `close()`.The side that was closed on, i.e. your application.No, if it persists. It is an application bug (leaked connection).
`CLOSING`Both sides sent FIN at the same time; waiting for the ACK.Either end, in a simultaneous close.Rare and short-lived.
`LAST-ACK`Sent our FIN after CLOSE-WAIT, waiting for the final ACK.The side that closed second.Briefly. Many stuck means the final ACK is lost or filtered.
`TIME-WAIT`Waiting 60 s (fixed in Linux) so stray segments die and the last ACK can be resent.The side that closed first.Yes. Thousands are harmless unless you run out of ephemeral ports.
`CLOSED`No connection; a conceptual state with no socket.Nobody. `ss` does not list it for real sockets.n/a

State counts at a glance: `ss -tan | awk 'NR>1 {c[$1]++} END {for (s in c) print s, c[s]}'`.

sysctl essentials for troubleshooting

Read with `sysctl <key>` or under `/proc/sys/`. Defaults vary by kernel version, distribution and RAM size; values marked 'this box' were read on an Ubuntu 24.04 VM (kernel 6.18), the rest are documented kernel defaults. Change one key at a time and measure.

sysctl essentials for troubleshooting
KeyDefault / typicalWhat it doesWhen to change
`vm.swappiness`60 (this box)Relative preference for swapping anonymous pages versus reclaiming page cache.Lower (for example 10) on latency-sensitive database hosts; it is a bias, not a switch, and 0 does not disable swap.
`vm.overcommit_memory`0 (heuristic)0 refuses obviously excessive allocations, 1 always allows, 2 enforces a commit limit (`overcommit_ratio`).1 for software that forks large processes (Redis snapshots); 2 only with deliberate capacity planning.
`vm.dirty_ratio`20 (this box)Percent of available memory that may be dirty before writers are forced to write back synchronously.Lower it on hosts with fast bursts and slow disks to shorten stalls; see the dirty-page notes in the memory entry.
`vm.dirty_background_ratio`10 (this box)Dirty percentage at which background writeback starts.Lower to start flushing earlier and smooth write bursts; keep it below `dirty_ratio`.
`vm.max_map_count`65530Maximum number of memory mappings per process.Raise for Elasticsearch, some JVM and mmap-heavy databases that document a required value (commonly 262144).
`vm.panic_on_oom`0 (this box)0 lets the OOM killer pick a victim; 1 or 2 panics the kernel instead.Only on clustered hosts that must fail over cleanly, together with `kernel.panic` for auto reboot.
`net.core.somaxconn`4096 (this box; 128 on kernels before 5.4)Upper bound on the `listen()` backlog of any socket.Raise for busy listeners when `ListenOverflows` grows; the application must also request a larger backlog.
`net.ipv4.tcp_max_syn_backlog`1024 (this box; scales with memory)Maximum queued half-open (SYN_RECV) connections.Raise together with `somaxconn` under SYN bursts; keep `tcp_syncookies` enabled.
`net.ipv4.ip_local_port_range`32768-60999 (this box)Ephemeral port range for outgoing connections.Widen on clients or proxies that open many connections to one destination; fix connection reuse first.
`net.ipv4.tcp_tw_reuse`2 (loopback only; this box)Allows reuse of TIME_WAIT sockets for new outgoing connections when timestamps make it safe (1 = all, 2 = loopback only).Set to 1 on clients that exhaust ports; it does not affect incoming connections.
`net.ipv4.tcp_keepalive_time`7200 s (this box)Idle seconds before the first TCP keepalive probe on sockets with `SO_KEEPALIVE`.Lower (for example 300) when NATs or load balancers drop idle flows or dead peers must be detected sooner.
`net.ipv4.tcp_keepalive_intvl` and `tcp_keepalive_probes`75 s and 9 (this box)Interval between probes, and unanswered probes before the connection is dropped.Tune with `tcp_keepalive_time`: dead-peer detection is about time + intvl x probes.
`net.netfilter.nf_conntrack_max`262144 (this box; scales with memory)Maximum tracked connections; beyond it new packets are dropped.Raise on NAT gateways and Kubernetes nodes when `conntrack -C` nears the maximum; each entry costs memory.
`net.core.netdev_max_backlog`1000 (this box)Per-CPU queue of packets received by the NIC and not yet processed.Raise if `/proc/net/softnet_stat` shows drops in the second column on fast NICs.
`fs.file-max`1645244 (this box; scales with memory)System-wide limit on open file handles.Rarely the real limit; the per-process `LimitNOFILE` or `ulimit -n` usually hits first.
`fs.inotify.max_user_watches`130054 (this box; scales with memory)Maximum inotify watches per user.Raise for file watchers, IDEs and node tools failing with ENOSPC on watch creation.
`kernel.pid_max`32768 (this box)Highest pid value; also bounds the number of tasks.Raise on hosts with very many processes or threads; also check `kernel.threads-max` and cgroup `pids.max`.
`kernel.panic`0 by default (1 on this box)Seconds to wait after a panic before rebooting; 0 means stay halted.Set to a small positive value on unattended servers so they come back by themselves.
`kernel.hung_task_timeout_secs`120 (this box)Seconds a task may stay in D state before the kernel logs a hung-task warning.Rarely changed; use the warning as evidence of blocked I/O.
`kernel.sysrq`distro-dependentBitmask of allowed Magic SysRq functions; 0 disables, 1 enables all.Allow the diagnostic keys you need before an incident; remember the console or the trigger file must be reachable.

Persist a change in a file under `/etc/sysctl.d/` and apply with `sysctl --system`; `sysctl -w` lasts only until reboot. Namespaced network keys may be per-container.

Method & overview

For every resource (CPU, memory, disk, network, plus software limits) check Utilization, Saturation and Errors in a fixed order, so you find the bottleneck instead of guessing.

The problem

The box is slow and there are fifty metrics on screen. Most people stare at whichever graph they know best, or tune something that was never the bottleneck. The USE method replaces that with a short, exhaustive loop: list every resource, and for each one ask three questions in order: how busy is it, is work queueing behind it, and is it throwing errors.

How it works

  • Definitions: Utilization is the fraction of time (or capacity) the resource was busy; Saturation is extra work it could not service yet (a queue); Errors are counts of failed operations. A resource at 100% utilization is usually the bottleneck, and any sustained saturation is a performance problem even at lower utilization.
  • Check errors first when they are cheap: they are quick to read (`dmesg`, `ip -s link`, `sar -n EDEV`, `smartctl`) and a nonzero count often explains everything. Then utilization, then saturation. Use the table `tool-by-resource` below for the exact command per cell.
  • CPU: utilization is `mpstat -P ALL 1` (watch one hot core hiding in a low average); saturation is the run queue (`vmstat 1` column `r` above the core count, or `/proc/pressure/cpu`); errors are rare (machine-check or thermal messages in `dmesg`).
  • Memory: utilization is `free -m` `available`, not `free`; saturation is swapping (`vmstat` `si`/`so`), allocation stalls, `/proc/pressure/memory` and OOM kills; errors are `dmesg` OOM reports and failed allocations, plus ECC errors where the hardware reports them.
  • Disk: utilization is `iostat -xz 1` `%util` (misleading on parallel devices such as NVMe and RAID, see `iostat-reading`); saturation is `aqu-sz` and rising `await`, or `/proc/pressure/io`; errors are `I/O error` lines in `dmesg` and SMART data.
  • Network: utilization is throughput against link speed (`sar -n DEV 1`, `ip -s link`); saturation is drops, overruns and retransmits (`nstat`, `sar -n ETCP 1`, `ss -ti`); errors are the `errors`/`dropped` counters in `ip -s link` and `ethtool -S`.
  • Do not forget software resources, which hit limits long before hardware does: file descriptors (`/proc/<pid>/limits`, `fs.file-nr`), threads and `kernel.pid_max`, the conntrack table, listen backlog, ephemeral ports, and cgroup limits (`cpu.stat` throttling, `memory.events`). Treat each as a resource with a usage, a queue and an error such as `EMFILE`.
  • Walk the list once, write down anything abnormal, then drill into the worst resource with per-process tools (`pidstat`, `perf`, `strace`). The method finds the resource, not the root cause; it tells you which playbook to open next.

Examples

A USE walk for the hardware resources; each line is tagged with the letter it answers.

bash
# One pass over the four hardware resources: Utilization / Saturation / Errors
mpstat -P ALL 1 3            # U: CPU busy per core
vmstat 1 3                   # S: r > cores, si/so > 0
cat /proc/pressure/cpu       # S: "some avg10=..." = share of time tasks waited
free -m                      # U: look at "available"
iostat -xz 1 3               # U: %util   S: aqu-sz, await
sar -n DEV 1 3               # U: rx/tx kB/s versus link speed
ip -s link show              # E: errors, dropped, missed per interface
dmesg -T | tail -n 50        # E: I/O errors, OOM, NIC resets (may need root)

Software resources have the same three questions: how full, is anything queued, what error.

bash
# Software resources: the limits that bite first
cat /proc/sys/fs/file-nr          # allocated, unused, max file handles
cat /proc/sys/kernel/pid_max      # ceiling for PIDs and thread IDs
cat /proc/sys/net/netfilter/nf_conntrack_count /proc/sys/net/netfilter/nf_conntrack_max
ss -s                             # socket totals, incl. timewait
grep -E "Max open files|Max processes" "/proc/<pid>/limits"
cat "/sys/fs/cgroup/<group>/cpu.stat"   # nr_throttled, throttled_usec (cgroup v2)

In practice

  • Brendan Gregg's USE method — The method comes from Brendan Gregg: build a checklist of every resource and read utilization, saturation and errors for each. His published checklists map the idea to Linux commands; this entry and the table below are a condensed, independent version.
  • sysstat and procps — `mpstat`, `iostat`, `sar` (sysstat) and `vmstat`, `free`, `top` (procps-ng) cover most cells of the grid without installing anything exotic, which is why they are the default toolbox.
  • Pressure Stall Information (PSI) — Since kernel 4.20, `/proc/pressure/{cpu,memory,io}` reports directly how long tasks stalled waiting for a resource, which is a saturation metric measured by the kernel instead of inferred from queue lengths. It is absent if the kernel is built without PSI or booted with `psi=0`.
  • In practice — A common pitfall is checking only utilization. Averages hide a single saturated core, and a disk at 30% `%util` can still have a deep queue; always pair a utilization number with a saturation number.

Trade-offs

Pros

  • Exhaustive and cheap: a fixed checklist stops you from fixating on the wrong resource.
  • Works with tools already on every box, and gives a vocabulary teams can share in incident channels.

Cons

  • Finds the constrained resource, not the cause; you still need profiling and tracing afterwards.
  • Utilization and saturation are hard to define for some resources (a mutex, a connection pool), and cloud VMs hide host-level contention such as noisy neighbours except through `%steal`.

The three load numbers count runnable plus uninterruptible (D state) tasks, so read them against core count and trend, and never as CPU percent.

The problem

`uptime` says load average is 12 and someone pages you. Is the box on fire, or is that fine? On Linux the number mixes CPU demand with tasks blocked on I/O, it is not a percentage, and what it means depends on the core count and on whether it is rising or falling.

How it works

  • The three numbers are exponentially damped moving averages of the number of tasks in the run queue (state `R`, running or runnable) plus tasks in uninterruptible sleep (state `D`), over roughly 1, 5 and 15 minutes. The kernel samples every few seconds; they are not exact counts of any instant.
  • Linux counts `D` tasks, unlike traditional Unix. So load can be high with idle CPUs when many threads wait on a slow disk, an NFS server or a stuck lock in the kernel. Always check `%wait`/`wa` and the list of `D` tasks before concluding it is a CPU problem.
  • Load is a count of threads, not processes, and it is not divided by core count. Rough reading: load near the number of cores (`nproc`) means the CPUs are about fully used; load well above the core count means tasks are waiting for CPU or for I/O. Load 4 is idle-ish on 64 cores and saturated on 4.
  • Read the three numbers as a trend: `1 > 5 > 15` (for example 9.1, 4.2, 1.3) means load is climbing; `1 < 5 < 15` means the episode is already ending. A high 15-minute value with a low 1-minute value is an incident that has passed, which is worth knowing before you start digging.
  • Load is not CPU percent. A CPU-bound job gives load 1 per busy core with `us` near 100; a pile of threads blocked on disk gives high load with low `us`. Compare with `vmstat 1`: `r` is the runnable count, `b` the uninterruptible count, which are the two ingredients of load.
  • To see what is in the number, list tasks in `R` and `D` state (see the snippet). Kernel threads such as `kworker` can show up in `D` for short bursts; a user process that stays in `D` for many seconds is the signal to look at storage, NFS or a hung device (`d-state-and-zombies`).
  • In containers `/proc/loadavg` shows the host's load, not the container's, because it is not namespaced (unless a tool such as lxcfs rewrites it). Use `cpu.stat` and PSI in the cgroup for per-container questions.
  • Fork-heavy workloads (CI runners, shell-script loops) can make load spike in bursts shorter than the averaging windows; PSI (`/proc/pressure/cpu`) and `vmstat 1` give a more timely picture than the 1-minute average.

Examples

Where the numbers come from; the fourth field of /proc/loadavg is runnable over total tasks.

bash
uptime
# 01:28:50 up 4 min,  0 user,  load average: 0.03, 0.02, 0.00
cat /proc/loadavg
# 0.03 0.02 0.00 2/113 1706
#  1m   5m   15m  runnable/total scheduling entities, last PID created
nproc                          # cores to compare against

Break load into runnable and uninterruptible tasks; wchan names the kernel function a D task sleeps in.

bash
# What is inside the load number? Count threads by state, then show the D ones.
ps -eLo state | sort | uniq -c | sort -rn
# R and D count towards load; S (sleeping) and I (idle kernel thread) do not.
ps -eLo state,pid,tid,wchan:24,comm | awk '$1 == "D"'
vmstat 1 5      # r = runnable, b = uninterruptible

In practice

  • Linux kernel (kernel/sched/loadavg.c) — Load is computed from the count of running plus uninterruptible tasks and updated every five seconds with fixed-point exponential decay; that is why short bursts blur out.
  • Brendan Gregg, Linux Load Averages: Solving the Mystery — Documents why Linux includes `D` state in the number (a 1993 change) and how to decompose it with tracing; a good read if you need to explain it to a team.
  • htop and top — Both print the three values in their header next to a task summary; `top` also shows `wa` and `st`, which hint at whether load is I/O or hypervisor related.
  • In practice — Alert on PSI or run-queue length per core rather than raw load: a fixed threshold like `load > 10` is wrong on every other host size.

Trade-offs

Pros

  • One glance at `uptime` gives a free, always-available summary of demand plus trend.
  • Including `D` tasks makes it a rough proxy for any resource contention, not only CPU.

Cons

  • It mixes causes (CPU wait and I/O wait) so a high value does not tell you which resource is short.
  • Not normalized by core count, host-wide inside containers, and smoothed enough to lag fast changes.

Ten standard commands, run in order in the first minute on a slow host, that tell you which resource to dig into next.

The problem

You just logged in to a host that someone says is slow. You have a minute to decide whether to look at CPU, memory, disk or network. Ten ordinary commands, run in a fixed order, answer that without any special tooling; the goal is not to solve anything but to pick the right next step.

How it works

  • `uptime`: the three load averages (see `load-average`). It tells you whether the problem is rising, steady or over, and whether the box was recently rebooted (a short uptime is itself a finding).
  • `dmesg -T | tail`: the last kernel messages. OOM kills, `blocked for more than 120 seconds`, `I/O error`, `nf_conntrack: table full` and NIC resets all show up here (see `kernel-logs`); it may need root when `kernel.dmesg_restrict=1`.
  • `vmstat 1`: `r` runnable (compare with cores), `b` blocked, `si`/`so` swapping, `us sy id wa st` CPU split. The first line is an average since boot, so ignore it and read the lines after.
  • `mpstat -P ALL 1`: per-core CPU. One core at 100% with the rest idle means a single-threaded bottleneck or interrupt pinning that the overall average hides.
  • `pidstat 1`: per-process CPU in a rolling table (unlike `top` it does not clear the screen), so you can see which process burns CPU and keep the history in your terminal.
  • `iostat -xz 1`: per-device `r/s w/s`, throughput, `r_await`/`w_await` latency, `aqu-sz` queue and `%util`; `-z` hides idle devices. Latency and queue depth matter more than `%util` (see `iostat-reading`).
  • `free -m`: look at `available`, not `free`; `buff/cache` is reclaimable and not a problem by itself (see `free-and-available`). Nonzero `Swap used` only matters if `vmstat` shows `si`/`so` activity.
  • `sar -n DEV 1`: per-interface `rxkB/s` and `txkB/s` against the link speed. Needs the `sysstat` package; it is the quickest way to see whether a NIC is near saturation.
  • `sar -n TCP,ETCP 1`: `active/s`, `passive/s` connection rates and `retrans/s` retransmits. A rising `retrans/s` means loss or congestion; very high `active/s` with short flows suggests missing connection reuse.
  • `top`: finish with the overview that ties it together (load, task states, per-process CPU and memory). Press `1` for per-core, `H` for threads, `M` or `P` to sort by memory or CPU. If anything above was abnormal, jump to the matching topical entry or playbook.

Examples

The ten commands in order; the counts (1 5) keep each run to five seconds, and top -b makes the last one scriptable.

bash
uptime
dmesg -T | tail
vmstat 1 5
mpstat -P ALL 1 5
pidstat 1 5
iostat -xz 1 5
free -m
sar -n DEV 1 5
sar -n TCP,ETCP 1 5
top -bn1 | head -n 20

How to read vmstat: the first data line is the average since boot, the rest are per interval.

bash
# Reading the lines: values below are illustrative of an idle 4-core host
vmstat 1 2
# procs -----------memory---------- ---swap-- -----io---- -system-- -------cpu-------
#  r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st gu
#  0  0      0 15470200  32880 620828    0    0    32    16 1365 1456  3  2 95  0  0  0
# Worry when: r stays above the core count, b > 0 for long, si/so nonzero, wa or st high.

In practice

  • Linux Performance Analysis in 60,000 Milliseconds — The checklist originates from a Netflix tech blog post written by Brendan Gregg and the Netflix performance team; this entry paraphrases the idea with its own notes.
  • sysstat — `mpstat`, `pidstat`, `iostat` and `sar` ship in the `sysstat` package; on many distributions the periodic data collector is not enabled by default, so `sar` without a live interval has no history.
  • atop, dstat, glances — All-in-one monitors that show several of these panels together; `atop` can also record history, which helps for incidents that are over by the time you log in.
  • In practice — Paste the output into the incident channel before changing anything: it timestamps the state of the host and makes a later postmortem much easier.

Trade-offs

Pros

  • Fast, repeatable and tool-light; a new on-call engineer can follow it without prior context.
  • Covers every resource once, which protects against tunnel vision.

Cons

  • Shows only the present: intermittent stalls and incidents that ended earlier need recorded history (`sar` collection, `atop`, metrics).
  • A triage pass, not a diagnosis; per-second sampling also misses sub-second bursts.

Read the kernel ring buffer with dmesg and journalctl -k to spot OOM kills, hung tasks, NIC resets and filesystem errors, and know when the evidence has already rotated away.

The problem

Many host-level failures never show up in application logs: the OOM killer took your process, a disk started returning I/O errors, a NIC reset, or a task was stuck for two minutes. The kernel writes these to a fixed-size ring buffer, and if you look late or without the right flags you either see nothing or the wrong boot.

How it works

  • `dmesg` reads the kernel ring buffer of the current boot. `dmesg -T` converts timestamps to wall-clock time (approximate: it is derived from boot time, so it can drift after suspend or long uptimes), `dmesg -H` is a pager with relative times, `dmesg -w` follows new messages.
  • Filter by severity with `dmesg --level=err,warn` (levels: `emerg`, `alert`, `crit`, `err`, `warn`, `notice`, `info`, `debug`). Messages are prefixed with the subsystem, such as `EXT4-fs (vda):`, `nvme0n1:`, `Out of memory:`, which makes `grep` effective.
  • The buffer has a fixed size (set by `CONFIG_LOG_BUF_SHIFT`, raisable with the `log_buf_len=` boot parameter), and old lines are overwritten. On a busy host a flood of repeated messages (for example a failing disk) can push out the OOM report you wanted; read `dmesg` early, and keep a persistent copy.
  • On systemd hosts `journalctl -k` shows the same kernel messages through the journal, and `journalctl -k -b -1` gives the previous boot, but only if the journal is persistent (`/var/log/journal` exists or `Storage=persistent`). Without it a reboot erases the evidence, which is the usual reason a crash leaves no trace.
  • Narrow the journal with `-p err` (priority and above), `--since "1 hour ago"` / `--until`, `-u <unit>` for one service, and `-o short-precise` for microsecond stamps. `--list-boots` shows which boots exist; `-f` follows. `journalctl` needs root or membership in `systemd-journal` or `adm`.
  • `kernel.dmesg_restrict=1` (the default on many current distributions, including recent Ubuntu releases) makes `dmesg` fail for non-root users with `Operation not permitted`; use `sudo`, or read `journalctl -k` as a member of the right group. `kernel.printk_ratelimit` and per-message ratelimiting also suppress repeats, so a missing duplicate is not proof it did not happen.
  • OOM: `Out of memory: Killed process 1234 (name) total-vm:... anon-rss:...` and, for cgroup limits, `oom-kill:constraint=CONSTRAINT_MEMCG`; the `Memory cgroup out of memory` line names the cgroup. See `oom-killer` for reading the full report.
  • Hung task: `INFO: task <comm>:<pid> blocked for more than 120 seconds` followed by a stack, meaning a task was in `D` state beyond `kernel.hung_task_timeout_secs`. Storage: `blk_update_request: I/O error, dev sda, sector ...`, `EXT4-fs error`, `Remounting filesystem read-only`, `XFS ... Metadata corruption detected`. Network: `NETDEV WATCHDOG: eth0 ... transmit queue 0 timed out`, `Link is Down`, and `nf_conntrack: table full, dropping packet`.

Examples

dmesg: recent lines, severity filter, a grep for the usual suspects, and follow mode. Needs root if dmesg_restrict is 1.

bash
dmesg -T | tail -n 50                  # recent kernel messages, wall-clock time
dmesg -T --level=err,warn              # only errors and warnings
dmesg -T | grep -iE "out of memory|oom-kill|blocked for more|I/O error|NETDEV WATCHDOG|table full"
dmesg -wT                              # follow while you reproduce the problem
# [Fri Oct  9 01:24:10 2026] EXT4-fs (vda): warning: mounting unchecked fs, running e2fsck is recommended

journalctl equivalents, including the previous boot; replace nginx.service with the unit you care about.

bash
journalctl -k -b                        # kernel messages, this boot
journalctl -k -b -1 -p err              # previous boot, errors and worse (needs a persistent journal)
journalctl -k --since "1 hour ago"      # time window
journalctl -u nginx.service --since "10 min ago" -o short-precise
journalctl --list-boots                 # which boots are available
sysctl kernel.dmesg_restrict            # 1 means dmesg needs root

Make the journal persistent so the previous boot's kernel log is still there after a crash.

bash
# Keep the journal across reboots: with the default Storage=auto, journald
# writes to disk once /var/log/journal exists.
sudo mkdir -p /var/log/journal
sudo systemctl restart systemd-journald
# Or set Storage=persistent in /etc/systemd/journald.conf

In practice

  • util-linux dmesg and systemd journald — `dmesg` reads `/dev/kmsg` (or `syslog(2)`); `systemd-journald` also collects it, so `journalctl -k` can outlive the ring buffer if storage is persistent. The kernel documentation for `printk` describes levels and ratelimits.
  • kdump and netconsole — For panics that leave nothing on disk, `kdump` captures a vmcore and `netconsole` streams kernel messages to another host over UDP; set them up before you need them.
  • rsyslog and log shippers — Shipping kernel messages to a central system (via journald or rsyslog's `imklog`) is what makes OOM and disk errors searchable fleet-wide after the host is gone.
  • In practice — A common pitfall is reading `dmesg -T` timestamps as exact; compare against `journalctl -k -o short-precise` or the monotonic times when you need to line events up with application logs.

Trade-offs

Pros

  • The kernel is the only witness for OOM kills, hardware errors and hung tasks; reading it takes seconds and needs no extra tooling.
  • Persistent journald storage and forwarding give you evidence that survives reboots.

Cons

  • The ring buffer is small and shared, so floods overwrite old lines; ratelimits drop repeats.
  • Access is often restricted (`dmesg_restrict`, journal group), and `-T` times are approximate.

CPU & processes

Split CPU time into user, system, iowait and steal, find the busy process and thread, and check cgroup throttling before blaming the code.

The problem

An alert says CPU is at 90%, or a service got slow and the load average is high. "High CPU" is not one problem: time spent in user code, in the kernel, waiting for disk, or stolen by a hypervisor each point to a different fix. Split the number first, then find the process and thread, and in containers check whether the cgroup is throttling you even though the host looks idle.

How it works

  • Split the total: `mpstat -P ALL 1` (sysstat) or the `%Cpu(s)` line of `top` shows `us` (user code), `sy` (kernel/syscalls), `wa` (idle while waiting on IO), `si`/`hi` (interrupts), `st` (stolen by the hypervisor). High `us` means application compute; high `sy` means syscalls, page faults, network or lock activity; high `wa` is really a disk question (see `iostat-reading`).
  • One core pinned versus all cores busy: `mpstat -P ALL` shows per-CPU rows. A single CPU at 100% with the rest idle usually means a single-threaded hot loop, one busy interrupt queue, or a process pinned by affinity; load on all cores means real capacity pressure.
  • Find the process: `top` (press `P` to sort by CPU, `1` for per-core view) or `ps -eo pid,ppid,pcpu,etime,comm --sort=-pcpu | head`. Note that `ps` `%CPU` is lifetime average (CPU time divided by elapsed time), while `top` and `pidstat -u 1` show the current interval, so use those for "what is hot right now". Values above 100% mean multiple threads.
  • Find the thread: `top -H -p "<pid>"` or `pidstat -t -p "<pid>" 1` list threads with their TIDs. Convert a TID to hex (`printf '%x\n' "<tid>"`) to match a JVM `jstack` `nid=0x...` entry, or use `perf` (see `perf-and-flamegraphs`) to see which functions burn the time.
  • Steal: `st` above a few percent on a VM means the hypervisor gave your CPU time to someone else (noisy neighbour, oversold host, burstable instance out of credits). The guest cannot fix it: move the workload, resize, or raise it with the provider. Stolen time also inflates run-queue length without any process looking busy.
  • cgroup throttling: a container with a CPU limit can be throttled while the host is idle. On cgroup v2 read `cpu.stat` in the container's cgroup: rising `nr_throttled` and `throttled_usec` against `nr_periods` mean the quota in `cpu.max` is too small for the bursts. cgroup v1 uses `cpu.cfs_quota_us` and a `cpu.stat` with `throttled_time` in nanoseconds.
  • Pressure Stall Information: `cat /proc/pressure/cpu` (kernel 4.20+, needs PSI enabled) reports the share of time tasks were runnable but waiting for a CPU (`some` line). It measures saturation directly and is more honest than load average; the same file exists per cgroup as `cpu.pressure`.
  • Decide the branch: high `us` goes to profiling (`perf`), high `sy` to `strace -c` or `perf` with kernel symbols, high `wa` to disk tools, high `st` to the provider, throttled to the CPU limit, many runnable threads with high involuntary switches to `context-switches`.

Examples

From box to process to thread. Replace "<pid>" with the busy process id.

bash
# Whole-box split, then per-process, then per-thread (1 s samples)
mpstat -P ALL 1 3
pidstat -u 1 5
pidstat -t -p "<pid>" 1 5
top -H -b -n 1 -p "<pid>" | head -20
# mpstat columns: %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle

Inside a container the cgroup files are usually at the cgroup root. The comment lines are illustrative.

bash
# cgroup v2: is this cgroup being throttled? (path depends on the runtime)
cat /sys/fs/cgroup/cpu.max
cat /sys/fs/cgroup/cpu.stat
# nr_periods 12000
# nr_throttled 3400   <- a large share of periods throttled
cat /proc/pressure/cpu

In practice

  • sysstat (mpstat, pidstat) — The standard way to get per-CPU and per-process rates over an interval, and part of the usual 60-second checklist.
  • Kernel docs: cgroup-v2 and PSI — `cpu.max`, `cpu.stat` and `cpu.pressure` are documented in the kernel admin-guide (cgroup-v2.rst and the PSI document).
  • In practice — A frequent cause of unexplained latency in containers is a CPU limit set too close to average usage: the average looks fine while short bursts get throttled.

Trade-offs

Pros

  • Sampling tools like `mpstat` and `pidstat` are cheap enough to run on a production host during an incident.
  • Splitting by us/sy/wa/st narrows the cause in under a minute before any profiling.

Cons

  • Interval tools miss spikes shorter than the sample period; use a longer history (`sar`) or PSI for those.
  • Percentages inside VMs and containers are relative to what the guest sees, not necessarily what the host really delivered.

Processes stuck in uninterruptible sleep (D) or left as zombies (Z): how to find what they wait on, and what can and cannot be fixed.

The problem

The load average is 40 but CPU is idle, `kill -9` does nothing, or `ps` is full of `<defunct>` entries. Both cases are about processes that look stuck but for different reasons: a D-state task is waiting inside the kernel (usually for IO) and cannot be interrupted, while a zombie has already exited and only its exit status remains. They need opposite handling.

How it works

  • State codes in `ps`/`top`: `R` running or runnable, `S` interruptible sleep, `D` uninterruptible sleep, `T` stopped, `t` traced, `Z` zombie, `I` idle kernel thread. Extra characters follow: `<` high priority, `N` low priority, `s` session leader, `l` multi-threaded, `+` foreground process group.
  • Why D exists: a task in a non-interruptible wait is inside a kernel code path that must complete (disk or NFS IO, a lock held across IO, a driver). Signals, including `SIGKILL`, are only handled when it returns. Many D tasks are normal for a moment; tasks that stay in D for many seconds are the problem.
  • D counts toward load: Linux includes D-state tasks in the load average, which is why a stalled disk or NFS server shows up as high load with idle CPUs. List them with `ps -eo state,pid,ppid,wchan:32,comm | awk '$1=="D"'`.
  • What it waits on: `cat /proc/<pid>/wchan` names the kernel function it sleeps in (for example `nfs_...`, `io_schedule`, `jbd2_log_wait_commit`). As root, `cat /proc/<pid>/stack` shows the kernel stack when the kernel exposes it. Then correlate with `iostat -xz 1`, `dmesg -T` and `mount` for the device or server involved.
  • Typical causes: a hung NFS/CIFS mount (hard mounts retry forever), a failing disk or controller with long timeouts, a stuck multipath or iSCSI path, a filesystem journal commit stalled by a slow device, FUSE servers that stopped answering, and heavy memory reclaim. The kernel's hung-task detector logs `blocked for more than 120 seconds` (`kernel.hung_task_timeout_secs`), so check `dmesg`.
  • Fixing D: fix the thing it waits for: restore the NFS server, replace the disk, remove the dead path, or kill the FUSE server so the waiting calls fail. A lazy unmount (`umount -l`) or choosing a `soft` NFS mount (at the risk of I/O errors and data loss) can limit future hangs. If the device never responds, a reboot is sometimes the only way out.
  • Zombies (`Z`): the process has exited, its memory and files are gone, but its entry stays until the parent calls `wait()` to read the exit status. You cannot kill a zombie, since it is already dead. Find the parent with `ps -eo pid,ppid,stat,comm | awk '$3 ~ /^Z/'` and fix or restart the parent; once the parent exits the zombies are reparented to PID 1 (or a subreaper) and reaped.
  • Why zombies matter: each holds a PID and a process-table slot, so thousands of them can approach `kernel.pid_max` and make `fork` fail with `EAGAIN` ("Resource temporarily unavailable"). In containers, an entrypoint that is not an init never reaps orphans; use an init such as `tini` or `docker run --init`.

Examples

Find D-state tasks and what they wait on. The hung-task line only appears after the detector timeout (120 s by default).

bash
# D-state tasks and the kernel function they sleep in
ps -eo state,pid,ppid,wchan:32,comm | awk '$1=="D"'

# One task in detail (stack needs root and kernel support)
cat /proc/"<pid>"/wchan; echo
cat /proc/"<pid>"/stack

# Recent hung-task reports
dmesg -T | grep -i "blocked for more than"

Zombies cannot be killed: fix, signal or restart the parent that never calls wait().

bash
# Zombies and their parents
ps -eo pid,ppid,stat,comm | awk '$3 ~ /^Z/'

# Who is the parent, and how many zombies does it hold?
ps -o pid,ppid,stat,etime,cmd -p "<ppid>"
ps -eo ppid,stat | awk '$2 ~ /^Z/ {n[$1]++} END {for (p in n) print p, n[p]}'

In practice

  • Linux kernel hung-task detector — Logs `blocked for more than N seconds` with a stack when a task stays in D; the timeout is `kernel.hung_task_timeout_secs`.
  • tini and docker run --init — Minimal init processes whose main job is to forward signals and reap orphaned zombies inside containers.
  • In practice — NFS hard mounts are a common reason for unkillable D processes during a storage outage; the processes recover on their own when the server returns.

Trade-offs

Pros

  • `wchan` and the hung-task log pinpoint the failing subsystem without special tools.
  • Reaping zombies by fixing the parent addresses the cause instead of hiding it.

Cons

  • A D-state process cannot be forced to exit; the only options may be restoring the device or rebooting.
  • `/proc/<pid>/stack` needs root and kernel support, and a single sample may miss the long wait.

Read voluntary versus involuntary switches, run-queue length and the cs rate to tell lock contention or CPU oversubscription from healthy blocking.

The problem

`vmstat` shows `cs` in the hundreds of thousands, a service has high system CPU for no obvious reason, or latency is jittery under load. A context switch is the kernel replacing one running task with another on a CPU. Some are healthy (a thread blocks on IO), some are a symptom (too many runnable threads, or contended locks that keep putting threads to sleep).

How it works

  • Two kinds per task: a voluntary switch happens when the task gives up the CPU itself (blocks on IO, a lock, a futex, `sleep`); an involuntary switch happens when the scheduler takes the CPU away (time slice used up or a higher-priority task became runnable). Counters live in `/proc/<pid>/status` as `voluntary_ctxt_switches` and `nonvoluntary_ctxt_switches`.
  • Per process rates: `pidstat -w 1` prints `cswch/s` (voluntary) and `nvcswch/s` (involuntary); add `-t` for threads. Many voluntary switches mean the task constantly waits (locks, IO, network, small writes); many involuntary switches mean it is competing for CPU with other runnable tasks.
  • System-wide: in `vmstat 1`, `cs` is switches per second and `in` is interrupts per second (including the timer). Look for a change against the baseline rather than an absolute number, since a busy but healthy box can do very high rates. `r` is the run queue: tasks running or runnable. A sustained `r` well above the CPU count (`nproc`) means CPU saturation.
  • Interpretation: high `cs` plus high `sy` and low `us` hints at lock or futex contention, tiny IO or syscalls, or a thundering herd of threads. High `nvcswch/s` with a long run queue points to oversubscription: more runnable threads than cores, often from oversized thread pools or CPU limits.
  • Find the contention: confirm with `strace -c -f -p <pid>` (many `futex` calls), `perf` with kernel stacks, or `perf stat -e context-switches,cpu-migrations -p <pid>` (needs perf; software events work in most VMs). Language runtimes expose their own lock profilers.
  • Fixes: shrink thread pools to roughly the core count for CPU-bound work, batch small operations, use fewer locks or sharded locks, replace spin-sleep polling with blocking waits or event loops, and size container CPU limits for the actual number of threads running at once.
  • Affinity and priority: `taskset -cp "<pid>"` shows the CPUs a process may use; `taskset -c 0-3 cmd` pins a new one. Pinning can reduce cache misses and migrations for latency-critical work, but can also leave other cores idle while a pinned task waits, so measure before and after. `nice`/`renice` change fair-share weight, `chrt` the scheduling policy.
  • Migrations: `cpu-migrations` counts a task moving between CPUs, which loses cache warmth. A very high rate is mostly relevant on NUMA machines or tightly latency-sensitive services; check `lscpu` and `numactl -H` before tuning.

Examples

Three views of context switching: system, process/thread, cumulative.

bash
# System-wide: r (run queue), in (interrupts/s), cs (switches/s)
vmstat 1 5
# Per process and per thread: voluntary vs involuntary
pidstat -w 1 5
pidstat -w -t -p "<pid>" 1 5
# Cumulative counters for one task
grep ctxt /proc/"<pid>"/status
# cswch/s = voluntary (blocked), nvcswch/s = involuntary (preempted)

Compare run-queue length with core count, check affinity, count switches and migrations for 10 s (needs perf).

bash
nproc
taskset -cp "<pid>"
perf stat -e context-switches,cpu-migrations -p "<pid>" -- sleep 10

In practice

  • sysstat pidstat -w — Reports `cswch/s` and `nvcswch/s` per task or thread; the quickest way to see which kind of switching a process does.
  • perf (perf stat, perf sched) — `perf stat` counts context-switch and migration events; `perf sched` can record and analyse scheduler latency in more depth.
  • In practice — Oversized thread pools (hundreds of threads on a handful of cores) are a frequent source of high involuntary switch rates and jittery tail latency.

Trade-offs

Pros

  • The counters are always collected by the kernel, so reading them is free and safe in production.
  • Voluntary versus involuntary immediately separates "waiting on something" from "short of CPU".

Cons

  • There is no universal threshold: a high `cs` is only meaningful relative to a known-good baseline.
  • Counts do not say which lock or resource is the cause; you still need `strace`, `perf` or a language-level profiler.

Sample where CPU time goes with perf top and perf record, turn the stacks into a flame graph, and know the symbol, permission and VM limits.

The problem

You know a process burns CPU but not which code. `perf` samples the CPU's instruction pointer and call stack at a fixed rate, so a profile shows where time is spent with low overhead. A flame graph turns thousands of stack samples into one picture: width is share of samples, height is call depth.

How it works

  • Live view: `perf top` (root) shows the hottest functions across the whole system right now; `perf top -p "<pid>"` limits it to one process. Press `q` to quit. It is a quick "what is the CPU doing" look, nothing is saved.
  • Record then report: `perf record -F 99 -g -p "<pid>" -- sleep 30` samples 99 times per second per CPU (an odd rate avoids lockstep with timers) with call stacks, for 30 s. `perf report --no-children` shows self time by symbol; `perf report --stdio` works over ssh. The data lands in `perf.data` in the current directory.
  • Flame graph: `perf script` dumps the stacks; Brendan Gregg's FlameGraph scripts (`stackcollapse-perf.pl`, `flamegraph.pl`) fold them into an SVG. Read it by looking for wide plateaus at the top: that is where samples were actually on-CPU. Click to zoom; compare two graphs from before and after a change.
  • Reading the stack needs unwinding: the default `-g` uses frame pointers, which are missing in many distro-built binaries and libraries compiled with `-fomit-frame-pointer`, giving broken stacks. Alternatives: `--call-graph dwarf` (larger files, higher overhead), or rebuild with frame pointers; some recent distros enable them by default, so check yours.
  • Symbols: you need symbol tables for names instead of hex addresses. Install the `-dbg`/debuginfo packages or build with `-g`; for JIT languages (JVM, Node) emit a `/tmp/perf-<pid>.map` using the runtime's perf support. Kernel symbols need `kernel.kptr_restrict` to allow them (root usually does).
  • Permissions: `kernel.perf_event_paranoid` controls who may profile. At 2 or lower unprivileged users can profile their own user-space; kernel profiling needs 1 or root, and `-1` removes restrictions. Some distro kernels default higher, in which case use root (or `CAP_PERFMON`, 5.8+).
  • VMs and containers: many VMs do not expose a hardware PMU, so the cycles event fails with "not supported". Fall back to the software clock: `perf record -e cpu-clock -F 99 -g ...`. In containers perf needs `CAP_PERFMON`/`CAP_SYS_ADMIN` and often an adjusted seccomp profile, and the perf binary must match the host kernel.
  • Scope: this is on-CPU profiling. A thread blocked on IO or a lock does not show up; use off-CPU analysis (bcc `offcputime`, `strace -T`, `perf trace`) for time spent waiting.
  • eBPF alternatives: `bpftrace -e 'profile:hz:99 /pid == 1234/ { @[ustack] = count(); }'` or bcc-tools `profile` aggregate stacks in-kernel with less data movement. They need root, a reasonably recent kernel (4.9+ for profile sampling) and the packages (`bpftrace`, `bcc-tools`).

Examples

Package: "perf" (Debian/Ubuntu: linux-tools matching the kernel). Needs root or a relaxed kernel.perf_event_paranoid.

bash
# Sample one process for 30 s at 99 Hz with call stacks
perf record -F 99 -g -p "<pid>" -- sleep 30
perf report --no-children --stdio | head -40

# No hardware PMU (many VMs): use the software clock
perf record -e cpu-clock -F 99 -g -p "<pid>" -- sleep 30

Fold the stacks and render an SVG. The scripts come from the FlameGraph repository.

bash
# Flame graph with Brendan Gregg's FlameGraph scripts (git clone first)
perf script > out.perf
./FlameGraph/stackcollapse-perf.pl out.perf > out.folded
./FlameGraph/flamegraph.pl out.folded > cpu.svg
# open cpu.svg in a browser; wide boxes at the top = where CPU time went

Check or relax the paranoid level, and the bpftrace one-liner for user stacks. Replace 1234 with the PID.

bash
cat /proc/sys/kernel/perf_event_paranoid
# Temporary change; persist via /etc/sysctl.d/ if you really want it
sysctl -w kernel.perf_event_paranoid=1

bpftrace -e 'profile:hz:99 /pid == 1234/ { @[ustack] = count(); }'

In practice

  • perf (linux-tools) — The kernel's own profiler, maintained in the kernel tree; covers hardware counters, software events, tracepoints and scheduler analysis.
  • FlameGraph (Brendan Gregg) — The original flame graph scripts; many tools (async-profiler, pprof, speedscope) produce or read the same style of graph.
  • bpftrace and bcc-tools — eBPF front ends with ready-made profilers (`profile`, `offcputime`) that avoid writing a perf.data file.

Trade-offs

Pros

  • Sampling at 99 Hz has low overhead and shows real production behaviour, including kernel time.
  • A flame graph makes the hottest code path obvious and diffs well before and after a fix.

Cons

  • Stacks are only as good as unwinding and symbols; missing frame pointers or debug info give unreadable output.
  • It shows on-CPU time only, and needs privileges, matching kernel tools, and in VMs often a software-clock fallback.

Attach to a live process, summarise or filter system calls, and spot slow calls, ENOENT loops and futex waits, while respecting its overhead.

The problem

A process hangs, spins, or fails with a vague error, and the logs say nothing. Every interaction with the outside world (files, sockets, time, memory) goes through a system call, and `strace` prints them with arguments, return values and optionally timing, so you can see what the process is actually doing without its source code.

How it works

  • Start or attach: `strace -f -o /tmp/out.txt cmd args` runs a command under trace; `strace -f -p "<pid>"` attaches to a live process (root, or same user with `kernel.yama.ptrace_scope` allowing it). `-f` follows threads and child processes, which you almost always want. Detach with Ctrl-C; the process keeps running.
  • Summary first: `strace -c -f -p "<pid>"`, then Ctrl-C after 10-30 s prints a table of calls, errors and time per syscall. Use it to spot what dominates (many `futex`, `epoll_wait`, `read`) before reading a raw trace. `-w` (with `-c`) reports wall-clock latency instead of system CPU time.
  • Timing: `-T` appends the time spent inside each call (`<0.000054>`), `-tt` prefixes wall-clock timestamps with microseconds, `-r` prints deltas between calls. Sort slow calls with `awk` on the `<...>` field to find one `read` or `connect` that took seconds.
  • Filter: `-e trace=openat,read,write` picks syscalls; classes like `-e trace=%file`, `%network`, `%process` group them (strace 4.17+ uses the `%` prefix; older versions use `file`, `network`). `-e status=failed` shows only failing calls (strace 5.2+). `-s 256` lengthens printed strings (default 32), `-y` or `-yy` decorate file descriptors with path or socket details.
  • What to look for: a loop of `openat(...) = -1 ENOENT` shows a missing file or wrong search path; `EACCES`/`EPERM` shows permissions or SELinux/AppArmor; `connect(...)` that never returns or returns `ETIMEDOUT`/`ECONNREFUSED` shows network issues; repeated `EAGAIN` with `poll` is normal for non-blocking IO.
  • Reading a hang: a process sitting in `futex(...)` is waiting on a lock or condition variable; in `epoll_wait`/`poll`/`select` it is idle waiting for events; in `read` on a socket it waits for the peer; in `wait4`/`waitid` it waits for a child. The last line before it stops printing is the answer.
  • Overhead: `strace` uses `ptrace` and stops the process twice per syscall, so a syscall-heavy service can slow down by an order of magnitude or more, and timing is distorted. Avoid it on a latency-critical production process under load: use `perf trace`, `bpftrace` or bcc `syscount`/`opensnoop`, or trace a single replica briefly.
  • Limits: it shows the kernel boundary only, not library calls or functions in the program (`ltrace -p "<pid>"` traces dynamic library calls, with the same ptrace costs). Another tracer or a debugger cannot be attached at the same time, and containers need `CAP_SYS_PTRACE` and a seccomp profile that permits `ptrace`.

Examples

Start with the summary, then a timed trace to a file. Attaching needs root or ptrace permission; expect slowdown.

bash
# 1) Where does the time go? Summary after Ctrl-C (or add: -- sleep 20)
strace -c -f -p "<pid>"

# 2) Slow calls and timestamps, written to a file
strace -f -tt -T -s 128 -yy -o /tmp/strace.out -p "<pid>"
grep -E '<[1-9]\.[0-9]+>' /tmp/strace.out | head   # calls that took 1 s or more

Filters narrow the noise: failed file calls reveal ENOENT/EACCES loops; %network shows connect/send/recv.

bash
# Failing file lookups only
strace -f -e trace=%file -e status=failed -p "<pid>"

# Network calls with socket details
strace -f -yy -e trace=%network -p "<pid>"

# Library calls (dynamic libraries), same ptrace overhead
ltrace -p "<pid>"

In practice

  • strace — The standard syscall tracer on Linux; the man page documents `-c`, `-T`, `-e trace=` classes and `-e status=`.
  • perf trace, bpftrace, bcc-tools — Lower-overhead alternatives for production: `perf trace`, `syscount`, `opensnoop` and `execsnoop` avoid stopping the process on every call.
  • In practice — "Works as root, fails as service user" and "cannot find config" problems are frequently solved in one trace by looking at the first `ENOENT` or `EACCES` before the error message.

Trade-offs

Pros

  • Works on any binary without source, symbols or restart, and shows exact arguments and errors.
  • Answers "what is it blocked on" for a hung process in seconds.

Cons

  • ptrace overhead can slow a busy process drastically and distort timing; risky on hot production paths.
  • Shows only syscalls, needs privilege, and cannot run alongside a debugger on the same process.

Memory

Read free and /proc/meminfo correctly: why low 'free' is normal, what 'available' estimates, and where slab, shared and page cache fit.

The problem

A dashboard says the box has 200 MB free out of 16 GB and someone wants to page the on-call. On Linux, unused RAM is wasted RAM, so the kernel fills it with page cache and gives it back on demand. The question is never how much is free, but how much can be handed to a new allocation without pushing something out, and that is what `available` estimates.

How it works

  • Read `available`, not `free`: `free` is memory nobody is using at all, `available` is the kernel's estimate (`MemAvailable` in `/proc/meminfo`, since kernel 3.14) of what a new workload could get without swapping. A box with 300 MB free and 9 GB available is healthy; a box with 3 GB free and 100 MB available is about to reclaim hard.
  • Columns of `free`: `total` is usable RAM, `free` is `MemFree`, `shared` is `Shmem` (tmpfs, `/dev/shm`, SysV shared memory), `buff/cache` is `Buffers` + `Cached` + `SReclaimable`, `available` is `MemAvailable`. The meaning of `used` depends on the version: procps-ng 4.0.4 (checked here) computes `total - available`, while older 3.3.x releases compute `total - free - buff/cache`. Check `free -V` before comparing numbers across hosts.
  • How `MemAvailable` is estimated: roughly `MemFree` minus the zone low watermarks, plus the part of page cache and reclaimable slab the kernel expects it can drop. It is an estimate, not a guarantee: dirty or heavily mapped cache, `Shmem` and pinned pages are not freely reclaimable, so treat a falling trend as the signal rather than a single number.
  • Fields worth knowing in `/proc/meminfo`: `AnonPages` (heap and stack, cannot be dropped without swap), `Cached` (page cache, includes `Shmem`), `Buffers` (block device metadata), `Slab` = `SReclaimable` + `SUnreclaim`, `Dirty` and `Writeback` (data not yet on disk), `Committed_AS` against `CommitLimit` (promised versus allowed address space), `HugePages_*` (reserved, not available to normal allocations).
  • When the numbers do not add up, the missing memory is usually kernel-side or reserved: `SUnreclaim` slab, page tables, pre-allocated huge pages, GPU or driver buffers. Summing process RSS double-counts shared pages and ignores the kernel, so use `Pss` from `/proc/<pid>/smaps_rollup` when you need a fair per-process share.
  • What to alert on: `available` as a percentage of `total` staying low for minutes and trending down, together with swap-in/out in `vmstat` or non-zero memory PSI. Do not alert on `free`, and do not alert on a single dip of `available` that recovers after a batch job.
  • Containers do not get their own view: `free` and `/proc/meminfo` inside a container normally show the host totals (unless the runtime fakes them, for example with lxcfs). Read the container's cgroup files instead, see the cgroup entry.
  • Triage order for 'memory looks full': `free -m`, then `vmstat 1` for `si`/`so` and `b`, then the top consumers by RSS, then `dmesg -T` for OOM reports. If `available` is fine and nothing swaps, stop: the cache is doing its job.

Examples

free on an idle 16 GB VM (procps-ng 4.0.4): almost everything is free, available is the number to read.

bash
free -m
# total used free shared buff/cache available
# Mem: 16094 575 15109 13 637 15519
# Swap: 0 0 0
free -V

The meminfo fields that explain most 'where did my RAM go' questions.

bash
grep -E '^(MemTotal|MemFree|MemAvailable|Buffers|Cached|AnonPages|Shmem|Slab|SReclaimable|SUnreclaim|Dirty|Writeback|Committed_AS|CommitLimit):' /proc/meminfo
# MemAvailable: 15888780 kB
# SReclaimable: 25632 kB
# CommitLimit: 8240476 kB

Top RSS consumers, and a fairer per-process number (Pss) for one PID.

bash
ps -eo pid,rss,vsz,comm --sort=-rss | head -n 6
grep -E '^(Rss|Pss|Pss_Anon|Pss_File|Pss_Shmem):' "/proc/<pid>/smaps_rollup"

In practice

  • procps-ng free and htop — Both read /proc/meminfo; htop's memory bar colours used, buffers and cache separately, which makes the 'cache is not pressure' point visible at a glance.
  • Linux kernel docs (filesystems/proc) — The authoritative description of every /proc/meminfo field, including how MemAvailable is defined; read it before building an alert on a field you have not verified.
  • In practice — A frequent false alarm is a monitoring agent graphing 'free' or 'used' from an old formula; switch the panel to MemAvailable and the incident usually disappears.

Trade-offs

Pros

  • `free` and `/proc/meminfo` need no privileges, no packages and add no overhead.
  • `MemAvailable` collapses cache, reclaimable slab and watermarks into one comparable number.

Cons

  • `MemAvailable` is a heuristic and knows nothing about cgroup limits, so a container can be starved while the host looks fine.
  • Definitions of `used` differ between procps versions, which makes cross-host comparisons misleading.

Find out who the kernel killed and why: global versus cgroup OOM, the dmesg report, oom_score_adj, overcommit settings and user-space killers.

The problem

A process vanished with no crash log, a shell printed `Killed`, a container exited with code 137, or the host went unresponsive and then recovered. When the kernel cannot reclaim enough memory it picks a victim and sends SIGKILL, and the only reliable record is in the kernel log. You need to tell which kind of OOM it was, who died, and whether the victim was the cause or just the biggest process.

How it works

  • Two flavours: a global OOM (the whole machine ran out and reclaim failed) and a cgroup OOM (a memory cgroup hit `memory.max` or the v1 limit and reclaim inside it failed). A cgroup OOM can happen while the host has gigabytes free; it kills only inside that cgroup. The `constraint=` field tells which: `CONSTRAINT_NONE` is global, `CONSTRAINT_MEMCG` is a cgroup.
  • Find the report with `dmesg -T | grep -iE 'out of memory|oom-kill|killed process'` or `journalctl -k`. The block starts with `<comm> invoked oom-killer: gfp_mask=..., order=..., oom_score_adj=...` (the process that triggered it, not necessarily the victim), then a memory summary and a task table, then `oom-kill:constraint=...` and the `Killed process <pid> (<name>) total-vm:... anon-rss:... file-rss:... shmem-rss:...` line. `anon-rss` is the number that matters.
  • Victim choice: each task gets `oom_score` (visible in `/proc/<pid>/oom_score`), roughly its share of memory (RSS, swap and page tables) adjusted by `oom_score_adj`. The highest score dies. `oom_score_adj` ranges from -1000 (never kill) to 1000 (kill first); kernel threads and PID 1 are exempt. Lowering it for one process pushes the kill onto its neighbours, so use it deliberately.
  • A shell `Killed` message or exit status 137 only means 'got SIGKILL' (128 + 9). Confirm with the kernel log before blaming OOM: a supervisor timeout, `kill -9` or a container runtime can send the same signal. In cgroup v2, `memory.events` has an `oom_kill` counter, and with `memory.oom.group=1` the kernel kills the whole cgroup together instead of a single task.
  • Overcommit decides whether the allocation fails early or the kill comes later. `vm.overcommit_memory=0` (default) refuses only obviously absurd requests, so `malloc` succeeds and the OOM hits when pages are touched. `1` never refuses. `2` enforces `CommitLimit` (swap + RAM x `vm.overcommit_ratio`/100), so allocations fail with `ENOMEM` instead of someone being killed; it needs applications that handle failure, and `Committed_AS` must stay under the limit.
  • Related knobs: `vm.panic_on_oom` (0 default; 1 or 2 panic the box instead of killing, useful with auto-reboot clusters), `vm.oom_kill_allocating_task` (kill the requester instead of scoring everyone), `vm.min_free_kbytes` (the reserve the kernel keeps for emergency allocations; setting it far too low or too high both cause trouble, so change it only with evidence).
  • Before the kernel acts, a user-space killer can act earlier: `systemd-oomd` (uses cgroup v2 and PSI, acts on units that opt in with `ManagedOOMMemoryPressure=`) and `earlyoom` (watches `MemAvailable` and swap). They exist because a box can thrash for minutes, unresponsive, before the kernel OOM finally fires.
  • Real cause versus victim: the killed process is the one with the highest score, not necessarily the leaker. Compare the task table before the kill (who holds the most `rss`), the trend of `available` beforehand, and what changed in the deploy. A well-behaved database killed because a runaway sidecar ate memory is a classic.

Examples

Find OOM reports; the comment lines are the kernel lines from a real cgroup OOM on a 50 MB limit.

bash
dmesg -T | grep -iE 'invoked oom-killer|oom-kill:|killed process'
# python3 invoked oom-killer: gfp_mask=0xcc0(GFP_KERNEL), order=0, oom_score_adj=0
# oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=/,mems_allowed=0,oom_memcg=/lt,task_memcg=/lt,task=python3,pid=2497,uid=0
# Memory cgroup out of memory: Killed process 2497 (python3) total-vm:63540kB, anon-rss:50944kB, file-rss:5792kB, shmem-rss:0kB, UID:0 pgtables:164kB oom_score_adj:0
journalctl -k --since "-1h" | grep -i oom

Inspect and adjust how likely a process is to be chosen (runtime only; use OOMScoreAdjust= in the unit file to persist).

bash
cat "/proc/<pid>/oom_score" "/proc/<pid>/oom_score_adj"
# make this process a less likely victim (lowering needs root); not persistent across restarts
echo -500 > "/proc/<pid>/oom_score_adj"

Overcommit settings and the commit accounting that mode 2 enforces.

bash
sysctl vm.overcommit_memory vm.overcommit_ratio vm.panic_on_oom vm.oom_kill_allocating_task
grep -E '^(CommitLimit|Committed_AS):' /proc/meminfo
# persist changes in /etc/sysctl.d/99-memory.conf, not with sysctl -w alone

In practice

  • Linux kernel docs (admin-guide/sysctl/vm, vm/overcommit-accounting) — Define panic_on_oom, oom_kill_allocating_task, overcommit modes and the CommitLimit formula; check them for your kernel version.
  • systemd-oomd — A user-space OOM killer that acts on cgroup v2 memory pressure (PSI) per unit, configured with ManagedOOMMemoryPressure= and ManagedOOMSwap=.
  • earlyoom — A small daemon that kills the largest process when MemAvailable and free swap fall below thresholds, before the box starts to thrash.
  • In practice — Exit code 137 on a container is a prompt to read the cgroup events and kernel log, not proof of an application bug; Kubernetes reports it as OOMKilled.

Trade-offs

Pros

  • The kernel OOM killer keeps the machine alive by sacrificing one process, and leaves a detailed report in the log.
  • cgroup limits confine the damage to one workload, and `oom_score_adj` lets you protect critical daemons.

Cons

  • The victim is chosen by memory size, not by importance or by who caused the problem.
  • Default overcommit hides the failure until pages are touched, and a box can thrash for minutes before the kernel finally acts.
  • Strict mode (`overcommit_memory=2`) avoids kills but breaks software that reserves large address space it never uses.

How memory.max, memory.high and memory.events work in cgroup v2 (and the v1 equivalents), and why a container can be OOM-killed while the host looks fine.

The problem

A container keeps getting killed at 512 MB while the node has 40 GB free, or `docker stats` and `free` disagree, or a service under `MemoryMax=` slows to a crawl without dying. The limit lives in a cgroup, not in the host's global counters, so you have to read the cgroup's own files to see how close it is and what it is spending memory on.

How it works

  • Find out which version you have: `stat -fc %T /sys/fs/cgroup` prints `cgroup2fs` on a v2 (unified) host and `tmpfs` on v1 or hybrid hosts. The cgroup of a process is in `/proc/<pid>/cgroup`; on v2 it is the path after `0::`, appended to `/sys/fs/cgroup`. Container runtimes create one cgroup per container.
  • cgroup v2 limits: `memory.max` is the hard limit; when usage reaches it the kernel reclaims inside the cgroup first and only then OOM-kills a task in it. `memory.high` is a throttle: above it the cgroup's allocations are slowed and reclaimed hard, but it is never OOM-killed for crossing it. `memory.min` and `memory.low` protect memory from reclaim. `memory.swap.max` limits swap separately.
  • What `memory.current` includes: anonymous memory, page cache (`file`), kernel memory such as slab and socket buffers, and `shmem`/tmpfs. A cache-heavy cgroup sitting at 100% of its limit is often fine because clean cache is reclaimable; dirty pages, tmpfs data and anonymous memory are not. Break it down with `memory.stat` (`anon`, `file`, `file_dirty`, `shmem`, `slab`, `pgmajfault`).
  • `memory.events` holds counters: `low`, `high` (times throttled), `max` (times the limit was hit), `oom` (limit reached and an allocation was about to fail), `oom_kill` (tasks killed). Read it as a story: rising `high` or `max` with no `oom_kill` means the workload is paying with latency at its limit (reclaim, major faults); rising `oom_kill` means tasks died. `memory.events.local` excludes children, and `memory.peak` (newer kernels) shows the high-water mark.
  • Why the host looks fine: the OOM is local to the cgroup, so global `MemFree` and `MemAvailable` stay high, and `free` inside the container reports host totals. The evidence is the kernel line `Memory cgroup out of memory: Killed process ...` with `constraint=CONSTRAINT_MEMCG` and `oom_memcg=` naming the cgroup, plus the cgroup's own `memory.events`.
  • cgroup v1 equivalents: `memory.limit_in_bytes` (hard limit), `memory.usage_in_bytes` (includes cache), `memory.max_usage_in_bytes`, `memory.failcnt` (times the limit was hit), `memory.soft_limit_in_bytes`, `memory.stat` with `total_` prefixed hierarchical counters, `memory.oom_control` with an `oom_kill` count, and `memory.memsw.*` for memory plus swap combined. All of these were observed on this (v1 memory controller) host.
  • Runtimes map their flags onto these files: Docker `--memory` and Kubernetes `limits.memory` become `memory.max` (or the v1 limit). Runtimes that read the cgroup size themselves behave better: modern JVMs derive their heap from the cgroup limit (`-XX:MaxRAMPercentage`), and Go supports `GOMEMLIMIT`; older runtimes that read `/proc/meminfo` size themselves to the host and get killed.
  • Sizing rule of thumb: the limit must cover the application's peak anonymous memory plus runtime overhead (thread stacks, metaspace or native memory, allocator arenas) plus a margin for page cache and kernel memory. A limit equal to the heap size is a common cause of repeated OOM kills.

Examples

cgroup v2: locate a process's cgroup and read limit, usage, events and the biggest consumers.

bash
stat -fc %T /sys/fs/cgroup
cg=$(sed -n 's/^0:://p' "/proc/<pid>/cgroup")
d="/sys/fs/cgroup$cg"
cat "$d/memory.max" "$d/memory.high" "$d/memory.current"
cat "$d/memory.events"
# oom_kill 3 means three tasks were killed inside this cgroup
grep -E '^(anon|file|file_dirty|shmem|slab|pgmajfault) ' "$d/memory.stat"

cgroup v1 (memory controller) equivalents; failcnt counts times the limit was hit.

bash
d="/sys/fs/cgroup/memory<cgroup-path>"
cat "$d/memory.limit_in_bytes" "$d/memory.usage_in_bytes" "$d/memory.max_usage_in_bytes"
cat "$d/memory.failcnt"
grep -E '^(total_rss|total_cache|total_dirty|total_shmem) ' "$d/memory.stat"
grep oom_kill "$d/memory.oom_control"

Memory pressure of one cgroup (v2, needs PSI); some/full above zero means tasks are stalled on memory.

bash
cat "$d/memory.pressure"
# some avg10=0.00 avg60=0.00 avg300=0.00 total=0
# full avg10=0.00 avg60=0.00 avg300=0.00 total=0

In practice

  • Linux kernel docs (admin-guide/cgroup-v2, admin-guide/cgroup-v1/memory) — The reference for memory.max, memory.high, memory.min/low, memory.events, memory.stat and the v1 files.
  • systemd resource control — MemoryMax=, MemoryHigh= and MemorySwapMax= in a unit or drop-in map directly to the cgroup v2 files; systemd-cgtop shows live per-cgroup usage.
  • Docker and Kubernetes — docker run --memory and Kubernetes limits.memory set the cgroup limit; pod-level triage of OOMKilled is covered on the Kubernetes sheet.
  • In practice — Monitoring tools often chart a 'working set' (usage minus inactive file cache) rather than raw usage so that reclaimable cache does not trigger alerts.

Trade-offs

Pros

  • A per-workload limit isolates a leaking or greedy service from its neighbours and from the host.
  • `memory.high` gives a soft ceiling that applies back-pressure instead of killing.

Cons

  • Cache, kernel memory and tmpfs count against the limit, so the number you must size for is larger than the heap.
  • Hitting `memory.max` after reclaim fails is abrupt, and pressure just below it shows up as latency rather than an error.
  • cgroup v1 and v2 expose different files and semantics, so scripts must handle both.

Tell harmless swap from thrashing using vmstat si/so and PSI, and know what swappiness, dirty-page sysctls and drop_caches really do.

The problem

Swap shows 3 GB used and someone wants it disabled, or the box crawls with plenty of 'free' memory, or a runbook says to drop the caches. Swap use alone is not a problem and the page cache is not waste. What matters is whether pages are moving in and out right now and whether tasks are stalling on memory.

How it works

  • The page cache holds file data that was read or written; clean pages are dropped instantly when memory is needed, so it is the cheapest memory to reclaim. It appears as `buff/cache` and is the reason 'free' trends to zero on a busy file server. Only when the working set no longer fits does the cache stop helping and reads go back to disk.
  • Swap holds anonymous pages (heap, stack) that the kernel decided are cold. A few GB of used swap with `si` and `so` at zero in `vmstat` just means idle memory was parked to make room for cache, which is harmless. Check per-process swap with `grep VmSwap /proc/<pid>/status`.
  • Thrashing looks different: sustained non-zero `si` and `so` in `vmstat 1` (KiB/s), growing `b` (blocked tasks), high `wa`, rising major faults (`majflt/s` in `sar -B`, `pswpin/s` and `pswpout/s` in `sar -W`), load climbing while CPU is idle. Hot code and data keep being evicted and re-read, so everything gets slow even though 'nothing is using CPU'.
  • Pressure Stall Information gives the direct measure: `/proc/pressure/memory` (kernel 4.20+ with PSI enabled) reports `some` (at least one task stalled) and `full` (all non-idle tasks stalled) as percent of time over 10, 60 and 300 seconds. Sustained `some` of several percent or any meaningful `full` means the box is paying for lack of memory; cgroups have their own `memory.pressure`.
  • `vm.swappiness` (default 60) biases reclaim between anonymous pages and file cache: lower favours dropping cache, higher favours swapping. The range is 0-100 (up to 200 on kernel 5.8+). `0` does not disable swapping, it only makes it a last resort. Databases commonly run lower, but it is a bias, not a switch; to really forbid swap use `swapoff` or the cgroup's `memory.swap.max`.
  • Dirty pages are written back by flusher threads: background writeback starts at `vm.dirty_background_ratio` (default 10) and writers are throttled and blocked at `vm.dirty_ratio` (default 20), percentages of reclaimable memory. On large-RAM hosts those percentages can be many GB, producing long write stalls; `vm.dirty_background_bytes` and `vm.dirty_bytes` set absolute values, and `vm.dirty_expire_centisecs` (3000) is how old dirty data may get. Watch `Dirty` and `Writeback` in `/proc/meminfo`.
  • Do not use `drop_caches` as a fix. It only discards clean cache, so the box instantly re-reads everything from disk and gets slower, and it does nothing about anonymous memory. It is legitimate for cold-cache benchmarks, after `sync`, and never on a loaded production host without a reason.
  • With no swap configured (common on containers and Kubernetes nodes) anonymous memory cannot be evicted, so the page cache absorbs all the pressure and OOM comes sooner. Compressed swap (zram or zswap) is the middle option. Note that tmpfs and `/dev/shm` data count as memory and can only leave through swap.

Examples

Is memory moving right now? Watch si/so (KiB/s), b and wa for five seconds; zeros mean swap is idle.

bash
vmstat 1 5
# r b swpd free buff cache si so bi bo in cs us sy id wa st
# 0 0 0 15468384 32880 620716 0 0 228 16 1689 1867 7 3 90 0 0
sar -W 1 3
cat /proc/pressure/memory
# some avg10=0.00 avg60=0.00 avg300=0.00 total=0

Who is in swap, and the writeback settings that decide when writers stall.

bash
grep -H VmSwap /proc/[0-9]*/status 2>/dev/null | awk '$2 > 0' | sort -k2 -n -r | head -n 5
sysctl vm.swappiness vm.dirty_background_ratio vm.dirty_ratio vm.dirty_expire_centisecs
# on this VM: 60, 10, 20, 3000
grep -E '^(Dirty|Writeback):' /proc/meminfo

Lower swappiness at runtime, and the drop_caches command you should know but rarely run.

bash
# runtime only; persist in /etc/sysctl.d/99-memory.conf
sysctl -w vm.swappiness=10
# WARNING: discards clean cache and causes a burst of disk reads; benchmarking only
sync
echo 1 > /proc/sys/vm/drop_caches

In practice

  • Linux kernel docs (accounting/psi, admin-guide/sysctl/vm) — PSI semantics for some and full, and the exact meaning of swappiness and the dirty_* parameters for your kernel.
  • sysstat (sar -B, sar -W) — Historical paging and swap activity, useful for answering whether the box was thrashing at 03:00 when nobody was watching.
  • systemd-oomd and Facebook's oomd — Both act on PSI memory pressure rather than free-memory thresholds, which is what makes PSI a better signal than free or swap use.

Trade-offs

Pros

  • Swap lets the kernel park cold anonymous pages and keep a bigger, hotter page cache, and buys time before an OOM.
  • PSI and `vmstat` give a direct, cheap answer to 'is memory the bottleneck right now'.

Cons

  • Latency-sensitive services can suffer badly when hot pages land on slow swap, and a swapping box may stay half-alive for a long time.
  • Lower `swappiness` protects anonymous memory but makes the page cache smaller, which can hurt IO-heavy workloads.
  • Large dirty-page thresholds absorb bursts but turn into long, visible write stalls.

Separate a real leak from cache, fragmentation or kernel slab growth by tracking RSS over time, then pick the right profiler.

The problem

A service's memory climbs for hours until it is restarted or killed. Before reaching for a profiler you must establish that the growth is real, that it is in the process rather than the page cache or the kernel, and what kind of memory it is. Only then does the choice of tool become obvious.

How it works

  • Measure the trend, not a snapshot: sample RSS at a fixed interval for hours, for example `pidstat -r -p <pid> 60` or `ps -o rss= -p <pid>` in a loop, and compare against load. A sawtooth that returns to the same floor after GC or traffic dips is normal; a rising floor across cycles is a leak or unbounded cache. VSZ is not evidence, since reserved address space is cheap.
  • Split the RSS by type with `/proc/<pid>/status`: `RssAnon` (heap, stack, anonymous mappings), `RssFile` (mapped files and libraries) and `RssShmem` (shared memory), available on kernel 4.5+. Growing `RssFile` is mapped data or cache (usually benign), growing `RssAnon` is the allocation problem you are hunting, growing `RssShmem` points at tmpfs or shared segments.
  • Add detail with `/proc/<pid>/smaps_rollup` (kernel 4.14+: summed `Rss`, `Pss`, `Anonymous`, `Private_Dirty`) and `pmap -x <pid>` (per mapping). Take two snapshots an hour apart and diff them: growth in `[heap]` suggests malloc'd objects, many new anonymous regions suggests arenas, thread stacks or mmap leaks.
  • Leak versus fragmentation versus cache: if the application's own accounting (object counts, GC heap size) is flat while RSS climbs, suspect the allocator. glibc with many threads creates multiple arenas and can hold freed memory (`MALLOC_ARENA_MAX=2` and `malloc_trim` are common experiments, jemalloc or tcmalloc another); a managed runtime that never shrinks its heap also looks like a leak. If the application's counts climb too, it is a real retention bug.
  • Rule out the things that are not leaks: page cache growth (shows in `buff/cache`, not in process RSS), tmpfs files, `SReclaimable` dentry and inode caches after a big directory scan, and many threads each reserving stack. If the whole box grows but no process does, look at the kernel: `grep -E 'Slab|SUnreclaim' /proc/meminfo` and `slabtop -o -s c` (root).
  • Pick the tool by what you can change: `valgrind --leak-check=full` or `massif` for test runs (large slowdown), `heaptrack` for allocation call stacks with lower overhead, AddressSanitizer/LeakSanitizer when you can rebuild, jemalloc heap profiling (`MALLOC_CONF`, needs a build with profiling) and bcc `memleak` for live processes (root, eBPF). Always measure the overhead before attaching to production.
  • Language-level shortcuts often beat native tools: Go `pprof` heap profile, JVM heap dump with `jcmd <pid> GC.heap_dump` plus Native Memory Tracking for off-heap, Python `tracemalloc`, Node `--heapsnapshot-signal`. Capture the profile before the restart, because the evidence vanishes with the process.
  • Buy time while you investigate: a cgroup `memory.high` or an alert on `RssAnon` growth rate, a scheduled restart as an explicit stopgap, and a heap dump or profile taken near the peak. Record the growth rate (MB per hour per request volume) so you can tell whether a fix worked.

Examples

Log RSS once a minute for a process (leave it running in tmux or a unit, then plot or eyeball it).

bash
pid="<pid>"
while kill -0 "$pid" 2>/dev/null; do
    echo "$(date +%FT%T) $(awk '/^(RssAnon|RssFile|RssShmem):/ {printf "%s=%s ", $1, $2}' "/proc/$pid/status")"
    sleep 60
done
# 2026-10-09T01:29:10 RssAnon:=384 RssFile:=3024 RssShmem:=0

Per-process view with pidstat and a mapping-level view you can diff between two snapshots.

bash
pidstat -r -p "<pid>" 60 3
# UID PID minflt/s majflt/s VSZ RSS %MEM Command
pmap -x "<pid>" | sort -k3 -n -r | head -n 8
cp "/proc/<pid>/smaps_rollup" "/tmp/rollup.$(date +%s)"

If no process explains the growth, look at kernel slab (needs root); the top rows on this VM were ext4_inode_cache, dentry and buffer_head.

bash
grep -E '^(Slab|SReclaimable|SUnreclaim):' /proc/meminfo
slabtop -o -s c | head -n 12
# OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME

In practice

  • valgrind (memcheck, massif) and heaptrack — Established native heap analysis; valgrind is slow but thorough, heaptrack records allocation stacks with a GUI for exploring them.
  • jemalloc and tcmalloc profilers — Allocator-level heap profiling that can be enabled on a running service with modest overhead if the build includes it.
  • bcc-tools memleak — An eBPF tool that traces allocations and frees of a live process and prints outstanding allocation stacks; needs root and a recent kernel.
  • In practice — Many reported leaks turn out to be unbounded caches, queues or per-connection buffers, so check application metrics for sizes before reaching for native tooling.

Trade-offs

Pros

  • RSS trend plus `status`/`smaps_rollup` is free, safe in production and often enough to classify the problem.
  • Allocator and language profilers point at the code that retains memory, not just the process.

Cons

  • Native profilers (valgrind, ASAN) add heavy overhead or need a rebuild, so they are usually run in staging.
  • RSS includes shared and file-backed pages, so a single number can mislead without the anon/file split.
  • A leak that grows slowly needs hours of data before the trend is trustworthy.

Disk & files

Read iostat -xz field by field (r/s, w/s, await, aqu-sz, %util) and know why %util lies on SSD, NVMe and RAID.

The problem

The app is slow, `top` shows high `%wa`, and someone says 'the disk is busy'. `iostat -xz` is the standard way to confirm it, but its columns are easy to misread: a device at 100% `%util` may be fine, and a device at 40% may already be the bottleneck. You need to know which columns carry latency, which carry queueing, and which only describe how long the device was non-idle.

How it works

  • Run `iostat -xz 1` (package `sysstat`; `-x` extended stats, `-z` hides idle devices). Ignore the first report: it is the average since boot, and every later report covers the last interval. Add `-p ALL` for partitions or `-N` for device-mapper names.
  • Throughput and IOPS: `r/s` and `w/s` are completed requests per second, `rkB/s` and `wkB/s` are bandwidth, `rareq-sz` and `wareq-sz` are the average request size in kB. Small requests at high IOPS stress a device differently from large sequential ones, so read the pair: 40k IOPS at 4 kB is about 160 MB/s, 200 IOPS at 1 MB is about 200 MB/s.
  • Latency: `r_await` and `w_await` are the average milliseconds a request takes from submission to completion, including time queued in the kernel. This is the number applications feel. As a rough guide, a few ms is normal on spinning disks, well under 1 ms on healthy NVMe, and a steady climb under constant load means the device is saturated. Averages hide tail latency, see `io-latency-tools` for distributions.
  • Queueing: `aqu-sz` (called `avgqu-sz` in older sysstat releases) is the average number of requests in flight or waiting. Values near the device's useful parallelism (1 for a single spinning disk, tens to hundreds for NVMe) with rising `await` mean saturation. `rrqm/s` and `wrqm/s` are requests merged by the block layer; a high merge rate with large `*areq-sz` is a sequential workload.
  • `%util` is the share of time the device had at least one request outstanding. For a single spinning disk, 100% really means saturated. For SSD, NVMe, RAID and cloud volumes that serve many requests in parallel, 100% only means 'never idle', not 'no headroom'; judge them by `await` and `aqu-sz` against a known baseline instead.
  • Removed columns: `svctm` was removed in sysstat 12.1.2 because it could not be computed correctly, so ignore advice that tells you to compare `await - svctm`. Newer kernels also report discard (`d/s`, `dkB/s`, `d_await`) and flush (`f/s`, `f_await`) columns; a large `f_await` points at fsync-heavy writers or a slow write cache.
  • Stacked devices: LVM, dm-crypt, md RAID and multipath appear as extra rows (`dm-0`, `md0`). The upper device shows what the filesystem asked for, the lower ones what the hardware did. RAID 5/6 turns one small write into several reads and writes, and RAID 1/10 writes each block twice, so physical IOPS exceed logical IOPS; compare the rows rather than adding them. In VMs and cloud, remember the volume may be throttled at a quota you cannot see from inside.
  • Combine with context: `vmstat 1` for `b` (blocked) and `wa`, `/proc/pressure/io` for how much time tasks actually stall on IO, and `pidstat -d 1` to find who is doing it. Read `iostat` as the device view, not the process view.

Examples

Extended device stats every second, idle devices hidden; the columns to read first are in the comments.

bash
iostat -xz 1
# Device   r/s   rkB/s  r_await rareq-sz   w/s   wkB/s  w_await wareq-sz  aqu-sz  %util
# vda    45.00 2096.00    19.31    46.58 151.00 153608.00   4.81  1017.27    1.60  84.00
# read: w_await 4.8 ms at 150 writes/s of ~1 MB each; aqu-sz 1.6; util 84% on a virtual disk

Cross-checks: blocked tasks and IO pressure stall time.

bash
vmstat 1 5
cat /proc/pressure/io
# some avg10=6.17 avg60=2.90 avg300=1.66 total=7904913

In practice

  • sysstat (iostat, sar -d) — `iostat` reads `/proc/diskstats`; `sar -d -p` keeps the same counters historically if the sysstat collector is enabled, which lets you look at yesterday's 03:00 spike.
  • Brendan Gregg's USE method — For disks: utilization is `%util` (with the parallel-device caveat), saturation is `aqu-sz` and wait time, errors are in `dmesg` and SMART. Treat latency (`await`) as the symptom to confirm and the other two as the explanation.
  • In practice — Compare against the same device on a quiet day. An `await` of 8 ms means different things on a cloud volume with a throttle, a local NVMe and a 7200 rpm disk, and no table of thresholds replaces your own baseline.

Trade-offs

Pros

  • Already installed on most servers, needs no root for the device view and costs almost nothing to run.
  • Shows latency, queueing and throughput in one line, which is enough to say whether the device or the workload is the problem.

Cons

  • Device-level only: it cannot say which process or file is responsible, and averages hide tail latency.
  • `%util` is misleading for parallel devices, and inside a VM it cannot see host-side throttling or noisy neighbours.

When writes fail with ENOSPC, tell block exhaustion from inode exhaustion with df -h and df -i, then find what grew with du.

The problem

Writes fail with `No space left on device`, a database refuses to start, or logs stop. The error is the same whether the filesystem ran out of blocks, ran out of inodes, hit a quota, or only looks full because of reserved space. The job is to tell which one it is in under a minute, then find what grew without making things worse.

How it works

  • Start with `df -hT` for blocks and `df -i` for inodes on the affected mount. `Use%` near 100 in `df -h` means blocks; `IUse%` near 100 in `df -i` means inodes, typically millions of tiny files (mail queues, session files, cache directories, an unbounded `/tmp`). Check the right mount point: `/var`, `/var/lib/docker` and `/tmp` are often separate filesystems.
  • Find the big directories without crossing filesystems: `du -xh --max-depth=1 /var | sort -h | tail`, then descend one level at a time. `-x` stops `du` from walking into other mounts, which otherwise makes the totals meaningless. On a huge tree `ncdu -x /var` (package `ncdu`) is faster to navigate, and `find /var -xdev -type f -size +500M` lists single big files.
  • Inode exhaustion: count files per directory with `for d in /var/*; do echo "$(find "$d" -xdev | wc -l) $d"; done | sort -n | tail`. On ext4 the inode count is fixed when the filesystem is created (`mkfs.ext4 -i` / `-N`), so the only fixes are deleting files or rebuilding the filesystem; XFS allocates inodes dynamically but can still hit its `imaxpct` cap.
  • Space that df counts but du does not see: a file deleted while a process still holds it open (see `deleted-but-open-files`), data hidden underneath a mount point (files written to `/data` before the disk was mounted over it; check by bind-mounting the parent filesystem elsewhere), and filesystem metadata or journal.
  • Reserved blocks: ext2/3/4 reserve 5% for root by default, so `Avail` reaches zero while `Used` is about 95%, and unprivileged services fail while root can still write. On a large data volume lower it with `tune2fs -m 1 <device>`; `tune2fs -l` shows the current reserved block count. Do not set it to 0 on the root filesystem, where it protects system services.
  • Common growers: application logs without rotation, `systemd-journald` (`journalctl --disk-usage`, then `journalctl --vacuum-size=500M`), core dumps in `/var/lib/systemd/coredump`, package caches, and for containers the json-file logs under `/var/lib/docker/containers/*/` and unused layers (`docker system df`). A huge file may also be sparse: compare `ls -ls` (allocated blocks) with `ls -l` (apparent size), and use `du --apparent-size` knowingly.
  • `tmpfs` (`/run`, `/dev/shm`, often `/tmp`) is RAM-backed, so a full tmpfs shows in `df` but also eats memory; it can be filled while the disk has room. Quotas (`quota`, `xfs_quota`) and container storage limits produce the same ENOSPC with free space still showing, so check them when `df` looks healthy.
  • Free space safely: truncate a live log instead of deleting it (`: > file`, so the writer keeps a valid descriptor), clean old rotated logs, vacuum the journal, prune container data you can prove is unused. Avoid `rm -rf` on directories you have not looked at, and never delete database files or WAL by hand.

Examples

Blocks versus inodes, then the biggest directories on that one filesystem only.

bash
df -hT /var
df -i /var
du -xh --max-depth=1 /var 2>/dev/null | sort -h | tail -8
# Filesystem  Inodes  IUsed   IFree IUse% Mounted on
# /dev/vda  16777216 322264 16454952   2% /

Common non-obvious growers: journal, big files, and container data.

bash
journalctl --disk-usage
# vacuum is destructive for old logs: keep only the newest 500 MB
journalctl --vacuum-size=500M
find /var -xdev -type f -size +500M -exec ls -lh {} + 2>/dev/null
docker system df

Count files per top-level directory to find the inode eater.

bash
for d in /var/*; do
    echo "$(find "$d" -xdev 2>/dev/null | wc -l) $d"
done | sort -n | tail -5

In practice

  • GNU coreutils (df, du) — `df` asks the filesystem for its totals, `du` walks the directory tree. They answer different questions, which is why they disagree when files are deleted-but-open, hidden under a mount, or sparse.
  • ncdu — An interactive `du` that scans once and lets you browse sizes; `-x` keeps it on one filesystem. Scanning a very large tree itself causes IO, so run it with `ionice -c3` when the host is already struggling.
  • In practice — A frequent cause of a full root filesystem is a single service with debug logging left on or a missing logrotate rule; fixing the space without fixing the writer brings the alert back within days.

Trade-offs

Pros

  • Two commands (`df -h`, `df -i`) classify the failure immediately, and `du -x` finds most culprits in a few steps.
  • All tools are in base packages and safe to run on a live host.

Cons

  • `du` on a huge tree is slow and generates IO on a host that may already be in trouble.
  • Cleanup is only a stopgap; the inode ceiling on ext4 cannot be raised without rebuilding the filesystem.

When df says the disk is full but du finds nothing, look for deleted files still held open by a process (lsof +L1) and release the space safely.

The problem

`df` says the filesystem is 100% full, yet `du` over the whole tree adds up to far less, and deleting more files changes nothing. A process still has the deleted file open, so the kernel cannot free its blocks until the last descriptor closes. This is classic after someone runs `rm` on a big log while the service is still writing to it.

How it works

  • Symptom: `df -h` shows high `Use%`, `du -xsh /mountpoint` shows much less, and the gap does not shrink after deleting. Check first that the gap is not a mount hiding data or a quota (see `disk-and-inode-full`).
  • First look: `lsof +L1` lists open files with a link count below 1, meaning deleted. Add the mount point to limit it (`lsof +L1 /var`), and sort by the SIZE column to find the big ones. Without lsof: `ls -l /proc/*/fd 2>/dev/null | grep '(deleted)'`. Both need root to see other users' processes.
  • Common causes: a log file removed by hand or by a cleanup script while the daemon keeps writing; logrotate that renames and recreates the file but the service is never told to reopen it (no `postrotate` reload or signal); long-running processes (JVMs, `tail -f`, shells) holding old temporary files or deleted jars.
  • Fix, preferred: make the owner release it. Reload or restart the service so it reopens its log (`systemctl reload <unit>`, `systemctl restart <unit>`, or the signal the daemon documents, such as `SIGHUP` or `SIGUSR1`). The blocks are freed the moment the last descriptor closes.
  • Fix, when you cannot restart: truncate through the descriptor with `: > /proc/<pid>/fd/<N>` (N from the `ls -l` or the lsof FD column, without the trailing `w`). The file is deleted, so no name remains, but the inode is still the same object; truncating it frees the blocks at once and the process keeps writing from its current offset, which can create a sparse file that grows again. It is lossy by design: the old content is gone.
  • logrotate trade-off: `copytruncate` copies the file and truncates the original so the service never needs a reload, but lines written between the copy and the truncate are lost, and the copy temporarily needs as much free space as the log. Prefer `postrotate` with a reload where the service supports it, and keep `copytruncate` for software that cannot reopen files.
  • Verify: `df -h` on the mount drops immediately after the descriptor closes or is truncated, and `lsof +L1` no longer lists the file. Quick reproduction of the whole effect: open a file in a script, `rm` it, and watch `df` stay flat until the script exits.
  • Prevent: give every long-lived writer a rotation story (reload hook or `copytruncate`), alert on `df` and on `lsof +L1` size totals, and avoid ad-hoc `rm` on log files that a service is writing; truncate them instead.

Examples

Find deleted-but-open files, biggest first, and see the same thing through /proc.

bash
lsof +L1 /var 2>/dev/null | sort -k7 -n -r | head
# COMMAND  PID USER FD  TYPE DEVICE   SIZE/OFF NLINK    NODE NAME
# python3 1933 root 3w  REG  254,0    5000000     0 1630974 /tmp/delme.log (deleted)
ls -l "/proc/<pid>/fd" | grep deleted

Release the space without restarting: truncate through the descriptor (data in the file is lost).

bash
# replace <pid> and <N> with the values from lsof / ls -l (here fd 3)
# WARNING: this destroys the deleted file's content; prefer a reload when possible
: > "/proc/<pid>/fd/<N>"
df -h /var
lsof +L1 /var

logrotate stanza: reload the service so it reopens the file (swap in your unit).

bash
cat > /etc/logrotate.d/myapp <<'EOF'
/var/log/myapp/*.log {
    daily
    rotate 7
    compress
    missingok
    postrotate
        systemctl reload myapp.service >/dev/null 2>&1 || true
    endscript
}
EOF

In practice

  • lsof — `+L1` selects files with fewer than one link; it is the standard one-liner for this problem and is available in the `lsof` package on every major distro.
  • logrotate — Its man page documents `copytruncate` and warns about the small window where data can be lost between the copy and the truncate; `postrotate` scripts are the alternative.
  • In practice — A deleted file stays visible to `lsof +L1` and `/proc/<pid>/fd` as `(deleted)` with its full size until the holder exits, and `: > /proc/<pid>/fd/<N>` drops that size to 0 immediately.

Trade-offs

Pros

  • A reload or restart frees space cleanly and fixes the root cause (the writer never reopened its log).
  • Truncating through `/proc/<pid>/fd/<N>` reclaims space instantly with no downtime.

Cons

  • Truncating discards the content and can leave a sparse file when the writer continues at its old offset.
  • A restart may be unacceptable for stateful services, and `copytruncate` can lose lines.

Diagnose 'Too many open files': per-process limit versus system-wide limit, how to find the process leaking descriptors, and how to raise limits for systemd services.

The problem

A service starts logging `Too many open files`, accepts no new connections, or fails to open config and data files, while CPU and memory look fine. A file descriptor covers regular files, pipes, epoll instances and every socket, so a connection leak exhausts descriptors just as a file leak does. You need to know which limit was hit and who is using the descriptors.

How it works

  • Two different errors: `EMFILE` ('Too many open files', errno 24) means this process hit its own `RLIMIT_NOFILE`; `ENFILE` ('Too many open files in system', errno 23) means the system-wide table is full. EMFILE is by far the more common one, and a limit of 1024 is the usual suspect.
  • Per-process limit: `ulimit -n` shows the soft limit of the current shell; for a running process read `cat /proc/<pid>/limits` (row `Max open files`, soft and hard). The soft limit is enforced, the hard limit is its ceiling. Change a live process with `prlimit --pid <pid> --nofile=65536:65536` (needs root or the right capability).
  • System-wide: `cat /proc/sys/fs/file-nr` prints three numbers: allocated handles, unused-but-allocated (0 on modern kernels), and the maximum, which is `fs.file-max`. Hitting it is rare on modern hosts (it scales with RAM) and shows as ENFILE or kernel messages `VFS: file-max limit ... reached`. `fs.nr_open` is the ceiling any per-process limit can be raised to.
  • systemd services do not read `/etc/security/limits.conf` (that is PAM, used for login sessions). Set `LimitNOFILE=` in the unit or a drop-in (`systemctl edit <unit>`), then restart the unit. systemd's default soft limit is 1024 with a high hard limit, so a daemon that never raises its own soft limit stays at 1024. Containers inherit the limit from the runtime or the daemon configuration.
  • Who is using them: count per process with `for p in /proc/[0-9]*; do echo "$(ls $p/fd 2>/dev/null | wc -l) ${p#/proc/} $(cat $p/comm 2>/dev/null)"; done | sort -rn | head`. Or `lsof -n -p <pid> | awk '{print $5}' | sort | uniq -c` to split by type (`REG`, `IPv4`, `unix`, `FIFO`, `a_inode`). Needs root for other users' processes.
  • Interpreting the mix: thousands of `IPv4`/`IPv6` entries mean connections, so look at `ss -tanp` and the socket states (`CLOSE_WAIT` piling up is the app not closing; see `time-wait-close-wait`). Thousands of `REG` entries pointing at the same path or directory mean files opened and never closed. Many `a_inode` (epoll, eventfd) entries can mean a leak of event loops or threads.
  • Growth versus size: sample the count every few seconds (`watch -n5 'ls /proc/<pid>/fd | wc -l'`). A steady climb to the limit is a leak and raising the limit only delays it; a plateau just under the limit is a limit that is too low for the real concurrency, and the right fix is a higher `LimitNOFILE` sized to expected connections plus files plus headroom.
  • Verify: `/proc/<pid>/limits` shows the new value after the restart, the fd count settles well under it, and the error disappears from the logs. Prevent: export `process_open_fds` against `process_max_fds` (Prometheus client metric names) and alert at about 80%.

Examples

Which limit applies to this process, how many fds it holds, and what the system-wide table looks like.

bash
grep 'Max open files' "/proc/<pid>/limits"
# Max open files            20000                20000                files
ls "/proc/<pid>/fd" | wc -l
cat /proc/sys/fs/file-nr
# 126     0       1645244

Top fd holders across the host, then the type breakdown of one process.

bash
for p in /proc/[0-9]*; do
    echo "$(ls "$p/fd" 2>/dev/null | wc -l) ${p#/proc/} $(cat "$p/comm" 2>/dev/null)"
done | sort -rn | head -5
lsof -n -p "<pid>" | awk 'NR>1 {print $5}' | sort | uniq -c | sort -rn

Raise the limit for a systemd service (persistent), or for one live process (until it restarts).

bash
# persistent: opens an editor for a drop-in; add the two lines below
systemctl edit "<unit>"
#   [Service]
#   LimitNOFILE=65536
systemctl restart "<unit>"
# temporary, live process only
prlimit --pid "<pid>" --nofile=65536:65536

In practice

  • systemd (systemd.exec) — `LimitNOFILE=` sets `RLIMIT_NOFILE` for a unit. `/etc/security/limits.conf` applies only to PAM sessions, which is why editing it does nothing for a daemon started by systemd.
  • Linux kernel docs and man7.org — `proc_sys_fs` documents `file-max`, `file-nr` and `nr_open`; `getrlimit(2)` describes `RLIMIT_NOFILE` and the EMFILE/ENFILE semantics.
  • In practice — High-concurrency proxies, databases and brokers commonly ship with documentation asking for a high `nofile` limit; the number should come from connection counts plus files plus headroom, not a copy-pasted value.

Trade-offs

Pros

  • `/proc/<pid>/limits` and `/proc/<pid>/fd` give exact answers without any extra tooling or overhead.
  • A higher limit costs only a little kernel memory per open descriptor and is an easy, safe change for a legitimate high-concurrency service.

Cons

  • Raising the limit on a leaking process only postpones the failure and makes the eventual outage larger.
  • Walking `/proc/*/fd` across thousands of processes is slow and needs root to see everything.

Attribute disk load to processes with pidstat -d and iotop, measure latency distributions with biolatency, baseline with fio, and spot writeback stalls.

The problem

`iostat` says the device is busy or slow, but not who is responsible or whether the pain is a few slow outliers or a steady shift. You need to attribute IO to processes, see the latency distribution rather than the average, and tell a real device limit from a stall caused by the page cache flushing dirty data.

How it works

  • Who does the IO: `pidstat -d 1` (sysstat) lists `kB_rd/s`, `kB_wr/s`, `kB_ccwr/s` (writes cancelled, such as a truncated file before writeback) and `iodelay` per process; add `-t` for threads. `iotop -oPa` (package `iotop` or `iotop-c`) shows only processes doing IO, accumulated since start. Both need root for other users' processes; `iotop` also needs kernel delay accounting (`kernel.task_delayacct` on newer kernels).
  • Which file: `pidstat -d` gives the process, then `lsof -p <pid>` or `ls -l /proc/<pid>/fd` and `/proc/<pid>/io` (cumulative `read_bytes` and `write_bytes`) point at files. For a quick sample, `cat /proc/<pid>/io` twice and subtract.
  • Latency distribution: `biolatency` and `biosnoop` from bcc-tools (packaged as `bcc-tools`; on Debian/Ubuntu the commands are suffixed `-bpfcc`, e.g. `biolatency-bpfcc`) use eBPF to histogram every block IO in microseconds, per disk with `-D`, or list individual IOs with PID and latency. Needs root, a kernel with BPF support and matching headers. A bimodal histogram, say a fast peak and a slow tail at tens of ms, tells you more than any average.
  • Deeper tracing: `blktrace`/`blkparse` record every request through the block layer; `perf trace` and `strace -T -e trace=fsync,fdatasync,write -p <pid>` show the slow syscalls from the application side. All of these have measurable overhead on a busy host, so bound them in time.
  • Baseline with fio: measure what the device can do, ideally before production load. Test a file, never a raw device holding data. Use `--direct=1` to bypass the page cache, a fixed `--runtime`, and read the `clat` percentiles (p99, p99.9) not just IOPS. Different `--bs`, `--rw` and `--iodepth` answer different questions: a database-like random 4 kB read profile versus a log-like sequential write profile.
  • Priority and schedulers: `ionice -c3 <cmd>` puts a batch job in the idle class and `-c2 -n7` in best-effort low priority, but they only take effect with schedulers that honour IO priority: BFQ (and historically CFQ), and `mq-deadline` only in a limited way (kernel 5.14+ dispatches higher classes first); `kyber` and `none` ignore them. See and set the scheduler in `/sys/block/<dev>/queue/scheduler`; NVMe devices typically run `none`. For cgroup v2 limits use `io.max` and `io.weight`.
  • Writeback stalls: writers dirty the page cache fast, and when `Dirty` in `/proc/meminfo` crosses `vm.dirty_background_ratio` background flushing starts, and past `vm.dirty_ratio` the writers themselves are throttled or blocked. The signature is periodic freezes of write-heavy processes, `b` and `wa` jumping in `vmstat`, and `Dirty`/`Writeback` swinging in `/proc/meminfo` while the application sees fsync spikes. Check `/proc/pressure/io` for how much time tasks stalled.
  • Order of use: `iostat -xz 1` to confirm the device, `pidstat -d 1` or `iotop` to name the process, `biolatency` to see tail behaviour, and `fio` only when you need a hardware baseline or to reproduce a profile on a spare file. Remember a VM's disk may be throttled by the platform.

Examples

Attribute IO to processes (needs root to see everyone), once as a rate and once as an accumulated total.

bash
pidstat -d 1 5
# UID  PID  kB_rd/s  kB_wr/s kB_ccwr/s iodelay  Command
iotop -oPa -b -n 3

Safe fio baseline on a temp file: 4 kB random reads, direct IO, 10 s. Look at the clat percentiles.

bash
fio --name=randread --filename=/tmp/fio.test --size=200M \
    --rw=randread --bs=4k --iodepth=16 --ioengine=libaio \
    --direct=1 --runtime=10 --time_based --group_reporting
rm -f /tmp/fio.test
# read: IOPS=124k, BW=483MiB/s
#  clat percentiles (usec): 99.00th=[265], 99.90th=[1188]

Latency histogram per disk for 10 seconds (bcc-tools, root), and who is writing right now.

bash
# Debian/Ubuntu names; elsewhere the tool is /usr/share/bcc/tools/biolatency
biolatency-bpfcc -D 10 1
biosnoop-bpfcc | head -20
cat /sys/block/vda/queue/scheduler
# none [mq-deadline] kyber bfq

In practice

  • bcc-tools / bpftrace — `biolatency` and `biosnoop` come from the iovisor bcc project (Brendan Gregg is a major contributor); they hook block-layer tracepoints, so overhead is low compared with `blktrace` on busy hosts, but not zero.
  • fio — The standard flexible IO tester; its `clat` percentile output is the number to compare between volume types, and its job files make a profile reproducible across hosts.
  • In practice — Cloud block volumes commonly enforce IOPS and throughput caps, so fio on the volume shows the cap, not the hardware. Test during a quiet window, and never point fio at a device or file holding live data.

Trade-offs

Pros

  • `pidstat`/`iotop` name the process in seconds with no setup; eBPF tools show the full latency distribution and per-IO detail.
  • `fio` gives a reproducible, comparable baseline instead of anecdotes.

Cons

  • eBPF tools need root, a recent kernel and often an extra package; `blktrace` and `strace` add noticeable overhead.
  • `fio` generates real load and can disturb production, and `ionice` has little or no effect with schedulers that ignore IO priority.

Network

Use ss to list listeners and connections, read Recv-Q/Send-Q for LISTEN versus ESTAB, filter by state and port, and spot accept-queue overflows.

The problem

A service will not accept connections, a port is "already in use", or you suspect a connection leak. `ss` reads socket state straight from the kernel and answers which process listens where, how many connections sit in each TCP state, and whether the accept queue is overflowing. It has replaced `netstat` on modern distros.

How it works

  • `ss -tulpn` is the default "what is listening" command: TCP and UDP, listening sockets only, numeric ports, owning process. Seeing the process name needs root (or being the same user).
  • `ss -s` prints a summary by protocol and TCP state, including `timewait` and `orphaned` counts. It is the cheapest first look when you only want to know whether something is piling up.
  • Without `-l`, `ss` shows non-listening sockets, and `-a` shows both. Add `-n` to skip name resolution (otherwise reverse DNS can make the command crawl) and `-p` for the process.
  • Recv-Q and Send-Q mean different things by state. On a `LISTEN` socket, Recv-Q is the number of connections waiting in the accept queue and Send-Q is its maximum size (the smaller of the application's backlog and `net.core.somaxconn`). On `ESTAB`, Recv-Q is bytes received but not yet read by the application and Send-Q is bytes sent but not yet acknowledged.
  • A persistently non-zero Recv-Q on `ESTAB` means the application is not reading fast enough (stuck thread, GC pause, blocked on a downstream call). A growing Send-Q means the peer or the network is not acknowledging.
  • Filters are part of the command: `ss -tan state established '( dport = :443 or sport = :443 )'`, `ss -tn state close-wait`, `ss -tn dst 10.0.0.0/8`. State names use hyphens (`time-wait`, `close-wait`); `state connected` and `state synchronized` are built-in groups.
  • `ss -ti` adds per-connection TCP internals: `rtt`, `rto`, `cwnd`, `retrans`, `bytes_sent`, `delivery_rate`. `ss -to` adds timers (keepalive, retransmit). Use them to answer "is this one connection slow, or all of them".
  • Accept-queue overflow shows up as counters, not in `ss`: `nstat -az TcpExtListenOverflows TcpExtListenDrops` rising over time means the application calls `accept()` too slowly or the backlog is too small. Raise the backlog in the application as well as `somaxconn`, since the effective value is the minimum of the two.
  • `ss` is a drop-in for most `netstat` uses (`netstat -tulpn` becomes `ss -tulpn`). `netstat` is in the legacy `net-tools` package and walks `/proc/net/*`, which is slow with hundreds of thousands of sockets; `ss` uses netlink.

Examples

Listeners, state summary, and the sockets of one port (the Send-Q on LISTEN is the accept-queue limit).

bash
ss -tulpn
ss -s
ss -tan '( sport = :8080 or dport = :8080 )'
# State      Recv-Q Send-Q Local Address:Port  Peer Address:Port
# LISTEN     0      5          127.0.0.1:18080      0.0.0.0:*

Count connections per state, and find who holds the CLOSE-WAIT sockets (needs root for the process).

bash
ss -tan | awk 'NR>1 {c[$1]++} END {for (s in c) print s, c[s]}'
ss -tnp state close-wait
# Recv-Q Send-Q Local Address:Port  Peer Address:Port Process
# 1      0          127.0.0.1:18080    127.0.0.1:39752 users:(("python3",pid=1868,fd=5))

Per-connection TCP internals, plus the listen-queue overflow counters (values rising over time are the signal).

bash
ss -tio state established '( dport = :443 )'
nstat -az TcpExtListenOverflows TcpExtListenDrops
# TcpExtListenOverflows           0                  0.0
# TcpExtListenDrops               0                  0.0

In practice

  • iproute2 (ss) — `ss` ships in the `iproute2` package and queries the kernel through the netlink sock_diag interface, which is why it stays fast on hosts with huge socket tables.
  • nstat / netstat -s — `nstat` (also in iproute2) reads the kernel's SNMP counters; `-a` shows absolute values and `-z` includes zeros. Counters are cumulative, so compare two samples.
  • In practice — "Address already in use" on startup is usually a still-running old process (check `ss -tulpn`) rather than TIME_WAIT; a listener normally sets `SO_REUSEADDR` so TIME_WAIT sockets do not block a restart.

Trade-offs

Pros

  • Fast, kernel-native view of every socket without extra packages.
  • Filter language pushes selection into the tool, so you do not need long grep pipelines.
  • Per-connection TCP metrics (`-i`) show RTT and retransmits without a packet capture.

Cons

  • A point-in-time snapshot: short-lived connections and bursts are easy to miss without repeated sampling.
  • Process attribution (`-p`) needs root and can miss sockets owned by other namespaces.
  • Queue sizes and counters need interpretation; Recv-Q means different things per state.

TIME_WAIT is normal bookkeeping by the side that closed first, CLOSE_WAIT means your application never closed the socket; know which one to fix and which to ignore.

The problem

`ss -s` shows tens of thousands of `timewait`, or a service slowly accumulates `CLOSE-WAIT` sockets until it stops responding. The two look alike in a socket count but have opposite meanings: one is normal bookkeeping, the other is an application bug.

How it works

  • TIME_WAIT belongs to the side that closed first (sent the first FIN). It keeps the 4-tuple for 60 seconds (a compile-time constant in Linux, not `tcp_fin_timeout`) so delayed segments from the old connection cannot be mistaken for a new one. A busy client or proxy that closes connections accumulates them by design.
  • CLOSE_WAIT belongs to the side that received the FIN. The kernel already ACKed it and now waits for the application to call `close()`. There is no timeout: if the code never closes the socket (leaked response body, missing `finally`, a pool that drops references without closing), the socket stays until the process exits.
  • Who is at fault: many `TIME-WAIT` is usually fine, and the question is only whether you run out of ephemeral ports. Any lasting `CLOSE-WAIT` count is a bug in the process that owns the sockets, found with `ss -tnp state close-wait`.
  • Port exhaustion is per destination 4-tuple: a client making many short connections to one `ip:port` can use at most the `net.ipv4.ip_local_port_range` ports (default `32768 60999`, about 28k) within the 60 s window, roughly 470 new connections per second. The symptom is `connect()` failing with `EADDRNOTAVAIL` ("Cannot assign requested address").
  • `net.ipv4.tcp_tw_reuse` lets an outgoing `connect()` reuse a TIME_WAIT slot when TCP timestamps (`tcp_timestamps`) make it safe. Values: `0` off, `1` on, `2` loopback only (the default on current kernels). It only helps the client side and never affects incoming connections.
  • `tcp_tw_recycle` was removed in Linux 4.12 because it broke clients behind NAT. Any old tuning guide that recommends it should be ignored; do not expect it to exist. Lowering `tcp_fin_timeout` does not shorten TIME_WAIT either; it only affects orphaned FIN-WAIT-2 sockets.
  • The real fix for TIME_WAIT pressure is fewer connections: HTTP keep-alive, connection pools, multiplexed protocols (HTTP/2, gRPC). Widening `ip_local_port_range` or connecting from more source IPs are stop-gaps. Having the server close first moves TIME_WAIT onto the server, so it is a trade-off, not a fix.
  • For CLOSE_WAIT, take a thread dump or goroutine dump of the owner, look at the connection's local port in `ss -tnp`, and grep the code for paths that skip close on error. Closing in `finally` or `defer` and setting client timeouts is usually the fix; restarting only resets the counter.

Examples

Which side is it: count states, then list who owns the CLOSE-WAIT sockets.

bash
ss -tan | awk 'NR>1 {c[$1]++} END {for (s in c) print s, c[s]}'
ss -tnp state close-wait | head -20
# top owners of CLOSE-WAIT sockets, by process name
ss -tnp state close-wait | grep -o 'users:(("[^"]*"' | sort | uniq -c | sort -rn

Ephemeral port headroom: the range, current usage toward one destination, and the reuse setting.

bash
sysctl net.ipv4.ip_local_port_range net.ipv4.tcp_tw_reuse
# net.ipv4.ip_local_port_range = 32768 60999  (shown with a tab between the numbers)
# connections from this host to one destination, by state
ss -tan dst "10.0.0.5:5432" | awk 'NR>1 {c[$1]++} END {for (s in c) print s, c[s]}'
# runtime change; persist it in /etc/sysctl.d/
sysctl -w net.ipv4.ip_local_port_range="20000 60999"

In practice

  • Kernel docs (ip-sysctl) — `Documentation/networking/ip-sysctl.rst` documents `tcp_tw_reuse` (0/1/2), `ip_local_port_range` and `tcp_fin_timeout`; the 60 s TIME_WAIT length is a kernel constant, not a sysctl.
  • PostgreSQL / database clients — Services that open a fresh connection per request create a TIME_WAIT per request; a pool such as PgBouncer removes both the latency and the socket churn.
  • In practice — A CLOSE_WAIT count that grows steadily and never shrinks, and that all belong to one process, is a classic leaked-connection pattern, commonly an HTTP client whose response body is not closed on error paths.

Trade-offs

Pros

  • Reusing connections (keep-alive, pools) cuts latency, CPU and socket churn together.
  • `tcp_tw_reuse` is safe for outgoing connections when timestamps are on, and costs nothing to enable.
  • Seeing CLOSE_WAIT early gives a clear signal of a code defect before descriptors run out.

Cons

  • Widening the port range only delays exhaustion; it does not remove the cause.
  • Server-side close moves TIME_WAIT to the server and can hurt it under high connection rates.
  • Long-lived pooled connections need health checks and timeouts to avoid half-dead sockets.

Capture only what you need with BPF filters, save a pcap with bounded size, and read handshakes, resets and retransmits.

The problem

Logs and counters say a call fails or is slow, and you need to see what actually crossed the wire: did the SYN get an answer, who sent the RST, are segments being retransmitted. `tcpdump` captures packets at the interface and is available almost everywhere, but an unfiltered capture on a busy host is both overwhelming and expensive.

How it works

  • Start with the habit flags: `-i <iface>` (or `-i any` for all interfaces), `-nn` (no name or port-name lookups, which otherwise slow output and add DNS traffic), `-s0` (full packets; recent versions already default to 262144 bytes), and `-c <n>` to stop after n packets.
  • Filters are BPF expressions applied in the kernel, so they are the cheap way to cut volume: `host 10.0.0.5`, `port 443`, `tcp port 5432 and host 10.0.0.5`, `src net 10.0.0.0/8`, `not port 22` (always exclude your own SSH session when capturing remotely).
  • TCP flags can be matched directly: `'tcp[tcpflags] & tcp-syn != 0'` for SYNs, `'tcp[tcpflags] & (tcp-rst) != 0'` for resets. In the output, `[S]` is SYN, `[S.]` SYN-ACK, `[.]` ACK, `[P.]` data push, `[F.]` FIN, `[R]` or `[R.]` reset.
  • Reading a healthy handshake: `[S]` then `[S.]` then `[.]`. Repeated `[S]` with no answer means the packet is dropped or filtered on the way. An immediate `[R.]` in reply to `[S]` means nothing listens on that port (or a firewall rejects). A reset in the middle of a conversation comes from whichever end sent it, which tells you who closed abruptly.
  • Retransmits show up as the same `seq` number sent again with growing gaps (the retransmission timeout backs off exponentially). Duplicate ACKs and a burst of retransmits point to loss on the path; steady retransmits of one segment with no ACKs point to a dead peer or a one-way path.
  • Save to a file with `-w out.pcap`, analyse it later with `tcpdump -nn -r out.pcap` or in Wireshark (`Statistics`, `Follow TCP Stream`, `Expert Information`). For long captures bound the disk use: `-C 100 -W 5` rotates five files of about 100 MB, `-G 60 -w 'cap-%H%M%S.pcap'` starts a new file every 60 seconds.
  • Capturing needs root or the `CAP_NET_RAW` and `CAP_NET_ADMIN` capabilities. Overhead is mostly copying packets to user space; on a busy 10G+ link a broad capture can drop packets (see "packets dropped by kernel" in the summary) or disturb the service, so narrow the filter and keep captures short.
  • TLS payloads are encrypted, so tcpdump shows only the handshake shape, sizes and timing, which is often enough to diagnose resets, timeouts and MTU problems. Never leave a capture of unencrypted traffic lying around: it can contain credentials and personal data; delete it when done.
  • Inside a container the capture sees the namespace you run it in. Use `-i any` in the container, or run tcpdump from the host or a debug container in the same network namespace; the Kubernetes sheet covers the debug-container route.

Examples

Short, filtered, readable capture on loopback; the same shape works on eth0 with a host filter.

bash
tcpdump -i lo -nn -s0 -c 6 'tcp port 18081'
# 01:29:05.980110 IP 127.0.0.1.42638 > 127.0.0.1.18081: Flags [S], seq 2645541097, ...
# 01:29:05.980198 IP 127.0.0.1.18081 > 127.0.0.1.42638: Flags [S.], seq 3106878874, ack 2645541098, ...
# 01:29:05.980209 IP 127.0.0.1.42638 > 127.0.0.1.18081: Flags [.], ack 1, ...

Only SYNs without ACK, or only resets, to see who is connecting and who is hanging up (exclude your SSH session).

bash
tcpdump -i any -nn -c 20 'tcp[tcpflags] & (tcp-syn|tcp-ack) = tcp-syn and not port 22'
tcpdump -i any -nn -c 20 'tcp[tcpflags] & tcp-rst != 0 and not port 22'

Bounded capture to disk (5 files of about 100 MB) and a later read; the capture can hold sensitive data, so delete it afterwards.

bash
tcpdump -i eth0 -nn -s0 -C 100 -W 5 -w /var/tmp/cap.pcap 'host 10.0.0.5 and tcp port 5432'
# read it back, or open it in Wireshark
tcpdump -nn -r /var/tmp/cap.pcap | head -50

In practice

  • tcpdump / libpcap — Filters use pcap-filter syntax; `man pcap-filter` lists every primitive (`host`, `net`, `portrange`, `tcp[tcpflags]`).
  • Wireshark / tshark — Open a saved pcap for stream following, TCP analysis flags (retransmission, zero window) and protocol dissectors; `tshark` does the same in the terminal.
  • In practice — Capture on both ends when possible: if a packet leaves one host but never arrives at the other, the loss is in between (firewall, security group, MTU), not in either application.

Trade-offs

Pros

  • Ground truth: shows what is on the wire regardless of what applications or logs claim.
  • BPF filtering in the kernel keeps overhead low when the filter is narrow.
  • A saved pcap can be shared and re-analysed without access to the host.

Cons

  • Needs root or capabilities, and may capture sensitive data.
  • Broad captures on busy links drop packets and cost CPU and disk.
  • Encrypted payloads are opaque; you see shape and timing, not content.

Work out which resolver path an application really uses, compare getent with dig, and find slow lookups caused by search domains, ndots or dead nameservers.

The problem

Requests hang for several seconds before they succeed, a hostname resolves in `dig` but not in the application, or a container resolves internal names slowly. DNS problems are hard to see because the application's resolver path (libc, `nsswitch.conf`, a local stub, `search` domains) is not the same thing as the one query you typed into `dig`.

How it works

  • Start with what the application's resolver uses: `cat /etc/resolv.conf` (`nameserver`, `search`, `options timeout:`, `attempts:`, `ndots:`) and `grep ^hosts /etc/nsswitch.conf` (order such as `files dns`, or `files resolve [!UNAVAIL=return] dns` with systemd-resolved).
  • `getent hosts <name>` goes through libc and `nsswitch.conf` exactly as most programs do, including `/etc/hosts`. `dig` and `nslookup` bypass `nsswitch.conf` and `/etc/hosts` and talk straight to a DNS server. When the two disagree, the cause is `/etc/hosts`, nsswitch order, a caching layer, or the search list; trust `getent` for "what will my program see".
  • On hosts running systemd-resolved, `/etc/resolv.conf` often points at the stub `127.0.0.53`. `resolvectl status` shows the real upstream servers per link, `resolvectl query <name>` resolves through it and reports which server and protocol answered, and `resolvectl statistics` shows cache hits.
  • `dig @<server> <name>` asks one server directly, so it separates "my resolver is broken" from "the authoritative data is wrong". `dig +short` prints just the answer, `dig +trace` walks from the root servers down, `dig +noall +answer +stats` shows the TTL and the query time. Compare answers from two servers to find stale caches.
  • Slow resolution is usually timeouts on a dead or unreachable first nameserver: the resolver waits `timeout` seconds (default 5) before trying the next one. `time getent hosts <name>` and `dig @<each nameserver> <name>` identify the bad server; reorder, fix, or remove it.
  • Search-domain amplification: with `options ndots:5` (the default in Kubernetes pods), any name with fewer than five dots is first tried with every `search` suffix, so `api.example.com` can cost several failed queries (each also asked for A and AAAA) before the real one. Use a fully qualified name with a trailing dot (`api.example.com.`), lower `ndots` in the pod's `dnsConfig`, or cache with a node-local DNS cache.
  • A frequent cause of exactly 5 second delays is the resolver sending the A and AAAA queries in parallel from the same socket, and one reply being lost, so the resolver waits for its 5 second timeout and retries. `options single-request` or `single-request-reopen` in `resolv.conf` (documented in `man resolv.conf`) serialise or reopen the socket as a workaround; this was widely reported with conntrack races in older kernels, so confirm on your kernel before blaming it.
  • Check the network path too: `dig +tcp` tests TCP/53 (large answers fall back to TCP), and a `tcpdump -nn -i any port 53` during a slow lookup shows whether queries leave, whether replies return, and how long each takes. Missing replies point at a firewall, a security group, or conntrack drops, not at DNS data.
  • Remember caching: a negative answer (NXDOMAIN) is also cached for the zone's negative TTL, applications such as the JVM cache successful lookups on their own, and `nscd` or `systemd-resolved` may hold stale data. `resolvectl flush-caches` clears the systemd-resolved cache; there is no system-wide libc cache by default.

Examples

What will the program see: config first, then libc path versus direct DNS (timing the slow one).

bash
cat /etc/resolv.conf
grep '^hosts' /etc/nsswitch.conf
time getent hosts "example.com"
dig +noall +answer +stats example.com

Ask each configured nameserver directly and trace from the root; resolvectl needs systemd-resolved.

bash
for ns in $(awk '/^nameserver/ {print $2}' /etc/resolv.conf); do
    echo "== $ns"
    dig +time=2 +tries=1 +short @"$ns" example.com
done
dig +trace +short example.com
resolvectl status
resolvectl query example.com

Watch DNS on the wire during a slow lookup to see queries, replies and gaps (short capture, numeric only).

bash
tcpdump -i any -nn -tttt -c 20 'port 53'
# resolv.conf workaround to test the parallel A/AAAA theory:
# options single-request-reopen timeout:2 attempts:2

In practice

  • systemd-resolved / resolvectl — Provides a caching stub at `127.0.0.53`, per-link DNS servers and split DNS; `resolvectl` is the supported way to inspect it.
  • BIND dig — `dig` is part of the BIND utilities (package `dnsutils` or `bind-utils`); `+trace`, `+short` and `+tcp` are its most useful flags for triage.
  • Kubernetes / CoreDNS — Pods get `ndots:5` and a three-entry search list by default, which is why short names are convenient but multiply queries; a node-local DNS cache is a common mitigation.

Trade-offs

Pros

  • Comparing `getent` with `dig` quickly separates application-path issues from DNS data issues.
  • Querying each nameserver directly finds the single bad server behind intermittent slowness.
  • A short port 53 capture settles arguments about whether queries and replies exist.

Cons

  • `dig` does not use `nsswitch.conf`, so a successful `dig` proves little about the application.
  • Resolver settings like `ndots` or `single-request` are workarounds that can hide the real cause.
  • Application-level caches (JVM, language runtimes) make behaviour differ from the OS resolver.

Find where packets die: a full conntrack table, NIC or softnet drops, TCP retransmits, rp_filter, or an MTU black hole.

The problem

Connections time out intermittently, retransmits climb, or new connections fail while established ones keep working. Packets can die in many places: the NIC ring, the softirq backlog, the firewall or connection tracker, the routing and reverse-path checks, or a path that silently drops large packets. Work from the host outward, using counters that name the place.

How it works

  • conntrack table full: the kernel log says `nf_conntrack: table full, dropping packet` (check `dmesg -T | grep conntrack`). New flows are dropped while existing ones work, which matches "new connections time out, old ones fine". It affects hosts with iptables/nftables NAT or stateful rules, including Kubernetes nodes.
  • Measure the headroom: `conntrack -C` (or `/proc/sys/net/netfilter/nf_conntrack_count`) against `sysctl net.netfilter.nf_conntrack_max`. `conntrack -S` shows per-CPU counters: `insert_failed` and `drop` rising mean entries could not be created; `early_drop` means the kernel evicted entries to make room.
  • Fixes for conntrack: raise `nf_conntrack_max` (memory cost per entry, roughly a few hundred bytes; the hash size `nf_conntrack_buckets` is sized separately), shorten timeouts such as `nf_conntrack_tcp_timeout_time_wait`/`_established` where safe, exempt traffic that needs no tracking with a `NOTRACK`/`raw` table rule, and reduce short-lived connections. Persist changes under `/etc/sysctl.d/`.
  • Interface level: `ip -s link show <iface>` lists RX/TX `errors`, `dropped`, `missed` (and `overrun` on some drivers); `ethtool -S <iface>` (package `ethtool`) gives driver counters such as ring or buffer drops (names vary per driver). Rising `missed` or rx fifo counters point to the NIC ring being too small or the CPU not draining it; `ethtool -g` shows ring sizes.
  • Kernel level: `/proc/net/softnet_stat` has one row per CPU with hexadecimal columns; column 1 is processed packets, column 2 is `dropped` (backlog full), column 3 is `time_squeeze` (softirq ran out of budget). Non-zero and rising `dropped` or `time_squeeze` means the CPU handling network softirqs is saturated; look at `mpstat -P ALL 1` for one CPU at high `%soft`.
  • TCP level: `nstat -az TcpRetransSegs TcpExtTCPTimeouts TcpExtListenDrops` or `netstat -s | grep -i retrans` (or `ss -ti` per connection) show retransmission. A retransmit rate of more than a percent or so of sent segments usually means real loss or congestion on the path; compare two samples rather than reading the cumulative number.
  • Reverse-path and filtering: with `net.ipv4.conf.<iface>.rp_filter=1` (strict) packets whose source is not routable back out the same interface are dropped silently, a classic cause with asymmetric routing or multi-homed hosts. the `nstat` counter `TcpExtIPReversePathFilter` (rising means rp_filter drops), the `rp_filter` value and `log_martians` help confirm; `2` (loose) is the usual remedy. Local firewall rules (`iptables -L -n -v`, `nft list ruleset`) show per-rule packet counters.
  • MTU and PMTUD black holes: small requests succeed while large responses stall, often over VPNs, tunnels or overlays. Test with `ping -M do -s <size> <host>` (do not fragment; 1472 payload fits a 1500 MTU) and bisect the size, or `tracepath <host>` to report the path MTU. If ICMP "fragmentation needed" is filtered, TCP never learns the smaller MTU; fix the filter, lower the interface MTU, or enable `net.ipv4.tcp_mtu_probing`.

Examples

Is the conntrack table the problem: kernel message, usage versus limit, and per-CPU failure counters.

bash
dmesg -T | grep -i 'nf_conntrack: table full'
conntrack -C
sysctl net.netfilter.nf_conntrack_max
conntrack -S | head -3
# cpu=0   found=0 invalid=0 insert=0 insert_failed=0 drop=0 early_drop=0 error=0 ...
# runtime change; persist it in /etc/sysctl.d/
sysctl -w net.netfilter.nf_conntrack_max=524288

Walk the drops from the NIC upward: interface counters, driver stats, softnet backlog, TCP retransmits.

bash
ip -s link show eth0
ethtool -S eth0 | grep -iE 'drop|miss|fifo|err'
# per-CPU rows, hex: processed, dropped, time_squeeze
cat /proc/net/softnet_stat
nstat -az TcpRetransSegs TcpExtTCPTimeouts TcpExtIPReversePathFilter

Find the path MTU: do-not-fragment pings (1472 + 28 header bytes = 1500) and tracepath.

bash
ping -M do -s 1472 -c 3 "10.0.0.5"
ping -M do -s 1372 -c 3 "10.0.0.5"
tracepath -n "10.0.0.5"
sysctl net.ipv4.tcp_mtu_probing

In practice

  • conntrack-tools — The `conntrack` command lists (`-L`), counts (`-C`) and deletes tracked flows and prints per-CPU statistics (`-S`); it needs root and the `conntrack` package.
  • Kernel docs (nf_conntrack-sysctl) — `Documentation/networking/nf_conntrack-sysctl.rst` describes `nf_conntrack_max`, `nf_conntrack_buckets` and the timeout tunables.
  • Kubernetes nodes — kube-proxy and CNI plugins rely on conntrack, so a busy node can fill the table with short-lived service connections; "table full" in `dmesg` is a common first lead for unexplained timeouts there.
  • In practice — Drops that do not appear in any counter often point to a device outside the host (switch, load balancer, cloud security group); a capture on both ends proves it.

Trade-offs

Pros

  • Counters at each layer (NIC, softnet, conntrack, TCP) tell you where loss happens without capturing packets.
  • Most checks are read-only and cheap, so they are safe on a production host.
  • Raising conntrack limits or fixing MTU often resolves the issue immediately.

Cons

  • Counters are cumulative and driver-specific; you must sample twice and know your NIC's naming.
  • Raising `nf_conntrack_max` costs kernel memory and only postpones the problem if connection churn is the cause.
  • Loss outside the host is invisible to host counters.

Troubleshooting

Load is high or latency is up: separate us/sy/wa/st, find the hot process and thread, check cgroup throttling and steal, then profile.

The problem

Load average is climbing, requests queue up, or a monitoring graph shows CPU pinned. Load alone does not say whether processes are computing, waiting on disk, or starved by a hypervisor or a cgroup quota, so the first job is to classify the time, then find who spends it, then decide between fixing the code and adding capacity.

How it works

  • Symptom: load average well above the core count (`nproc`), high `us`/`sy` in `top`, rising latency, or `nr_throttled` growing for a container. A high load with idle CPUs is not this playbook: it usually means tasks blocked in D state, so go to the disk or unresponsive-host playbook.
  • First look: `uptime` for the trend, then `vmstat 1 5` and `mpstat -P ALL 1 3`. Classify the time: high `us` is application code, high `sy` is kernel work (syscalls, networking, page faults), high `wa` is waiting on I/O, high `st` is a noisy neighbour on a VM, high `%soft` is network interrupt handling. One CPU at 100% while the rest idle points to a single-threaded hot spot or IRQ/affinity pinning.
  • First look: find who. `pidstat -u 1 3` for per-process CPU, `top -H` (or `pidstat -t`) for the hot thread inside a process, and `cat /proc/pressure/cpu` (PSI, kernel 4.20+ with PSI enabled) for how much time runnable tasks spent waiting for a CPU.
  • Common causes: a hot loop or runaway regex/serialization in one thread, a thread pool sized far above the core count, GC thrash in a managed runtime, busy polling, or a retry storm that multiplies work after an upstream slowdown.
  • Common causes: high `sy` with lots of small syscalls or lock contention (see context-switch and strace entries), and high `wa` or `st`: `wa` means the CPU is idle but tasks wait for disk, so switch to the disk playbook; `st` means the host is oversubscribed, so move the instance rather than tuning the guest.
  • Common causes: in containers, the quota (`cpu.max` on cgroup v2, `cfs_quota_us` on v1) throttles the group even when the host is idle. `nr_throttled` and `throttled_usec` in the group's `cpu.stat` growing is the proof; the process then looks slow while `top` on the host shows spare CPU.
  • Fix: stop the runaway change (roll back, disable the feature, shed or rate-limit load), lower thread-pool sizes to about the core count for CPU-bound work, raise or remove the quota if throttled, and profile the hot function with `perf` before optimizing. Do not renice or kill a process before you know what it is doing.
  • Verify: `mpstat` and `pidstat` show the expected drop, run-queue `r` in `vmstat` falls back toward the core count, `/proc/pressure/cpu` `avg10` returns near zero, `nr_throttled` stops increasing, and request latency recovers. Compare after the same interval, not one lucky sample.
  • Prevent: alert on CPU saturation (run queue or PSI) rather than only utilization, set container CPU requests from measurements, keep a baseline `perf` flame graph from a healthy day, and load-test with realistic concurrency before a release.

Examples

Classify first, then find the process and the thread.

bash
uptime
# 14:02:11 up 3 days, load average: 8.01, 4.20, 2.10
nproc
vmstat 1 5
mpstat -P ALL 1 3
pidstat -u 1 3
top -H -b -n 1 -o %CPU | head -15
cat /proc/pressure/cpu
# some avg10=1.39 avg60=0.52 avg300=0.14 total=1104522

Is the container being throttled? cgroup v2; the path depends on your runtime.

bash
CG="/sys/fs/cgroup/<path-to-group>"
cat "$CG/cpu.max"
# 200000 100000   (quota and period in microseconds: 2 CPUs)
grep -E 'nr_periods|nr_throttled|throttled_usec' "$CG/cpu.stat"

In practice

  • Brendan Gregg's USE method — For CPU the saturation signal is the run queue or scheduler latency, not utilization; that is why `vmstat` `r` and PSI matter more than a single `%CPU` figure.
  • perf and flame graphs — After you know which process burns CPU, `perf record -F 99 -g` plus a flame graph answers which function; see the perf entry.
  • In practice — A frequent surprise is a container that is slow with the host half idle: the cgroup quota, not the machine, is the limit, so check `cpu.stat` before blaming the code.

Trade-offs

Pros

  • Classifying us/sy/wa/st first avoids optimizing code when the real limit is I/O, a hypervisor or a quota.
  • All first-look commands are read-only, cheap and available from procps and sysstat.

Cons

  • Sampling at 1 s hides short spikes; PSI and sar history catch what a live look misses.
  • Per-thread and profiling views need privileges (root or perf_event_paranoid changes) and add overhead of their own.

Host is swapping, stalling or OOM-killing: read available memory and PSI, find the top consumer, decide leak vs cache vs limit, then fix.

The problem

The host is swapping, latency spikes come in waves, a process was killed, or `dmesg` says an OOM kill happened. Free memory is the wrong number to chase, because Linux uses spare RAM as cache on purpose. The question is whether the system can still reclaim memory cheaply, who holds the rest, and whether the cause is a leak, a legitimate peak, or a limit set too low.

How it works

  • Symptom: `MemAvailable` low and falling, `si`/`so` non-zero in `vmstat` for more than a moment, `kswapd` busy in `top`, processes disappearing, or `Out of memory: Killed process` in the kernel log. Slow everything with high `wa` and little free swap is memory thrash, not a disk problem.
  • First look: `free -m` and read the `available` column, then `vmstat 1 5` for `si`/`so` (sustained swap-in and swap-out is the thrash signal), and `cat /proc/pressure/memory` where PSI is available: `some` means tasks stalled, `full` means everything non-idle stalled.
  • First look: who holds it. `ps -eo pid,rss,comm --sort=-rss | head`, then `grep -E 'MemAvailable|Slab|SReclaimable|Shmem' /proc/meminfo`. If the process list does not add up to `used`, look at `Slab`/`SReclaimable` (kernel caches, `slabtop`), `Shmem` and tmpfs, and huge page reservations.
  • First look: was something killed? `dmesg -T | grep -iE 'out of memory|oom-kill|killed process'` (or `journalctl -k`). The report names the victim, whether the limit was global or a cgroup (`oom-kill:constraint=` shows `CONSTRAINT_MEMCG` for a cgroup), and shows the memory table at the time.
  • Common causes: a leak (RSS grows monotonically and never returns), a legitimate load increase, too many worker processes each with its own cache, an unbounded in-process cache or queue, and a container limit smaller than heap plus page cache plus off-heap memory.
  • Common causes: page cache is not the enemy, but dirty or pinned cache plus tmpfs can be; also overcommit settings (`vm.overcommit_memory`) change when allocations fail versus when the OOM killer fires later.
  • Fix: relieve pressure first by shedding load or restarting the leaking process in a controlled way, then fix the cause: cap caches, lower concurrency, raise a too-small cgroup `memory.max` (and `memory.high` for early throttling), or add RAM. Adding swap only buys time and can turn a clean kill into minutes of thrash.
  • Verify: `MemAvailable` stable or recovering, `si`/`so` back to near zero, `memory.pressure` `avg10` near zero, no new OOM lines in the kernel log, and RSS of the suspect flat over a full traffic cycle (watch with `pidstat -r 60`).
  • Prevent: alert on `MemAvailable` percentage and on PSI rather than `free`, set cgroup limits with headroom for page cache, run a user-space OOM daemon (`systemd-oomd`, `earlyoom`) where a fast, predictable kill beats a hung host, and graph RSS per service to catch slow leaks.

Examples

Read available, swap activity and pressure, then the top consumers.

bash
free -m
#                total        used        free      shared  buff/cache   available
# Mem:           16094         584       15085          13         653       15510
vmstat 1 5
cat /proc/pressure/memory
ps -eo pid,rss,comm --sort=-rss | head -5
grep -E 'MemAvailable|SReclaimable|Shmem:' /proc/meminfo

Find out whether and why something was killed (journalctl may need root or the systemd-journal group).

bash
dmesg -T | grep -iE 'out of memory|oom-kill|killed process' | tail -5
journalctl -k --since "-2h" | grep -i oom
# watch one suspect over time: RSS in kB, one line per minute
pidstat -r -p "<pid>" 60

In practice

  • Kernel PSI (Documentation/accounting/psi) — Pressure Stall Information reports the share of time tasks were stalled on memory; it is a better alarm than free memory because it measures the harm, not the level.
  • systemd-oomd and earlyoom — User-space killers that act on PSI or available memory before the kernel OOM killer, which often fires only after the host has already been thrashing for a while.
  • In practice — A container killed with exit code 137 while the host looks fine is usually a cgroup limit, not host exhaustion; compare `memory.current` and `memory.events` of the group.

Trade-offs

Pros

  • Reading MemAvailable, swap rates and PSI together separates harmless cache from real pressure.
  • Everything above is read-only and safe to run on a struggling host.

Cons

  • RSS double counts shared pages, so per-process sums can exceed physical memory; PSS from `smaps_rollup` is more accurate but costs more to read.
  • On a host already thrashing even running diagnostics is slow; grab the OOM report and sample counters first, analyze later.

Writes hang, iowait is high or the filesystem is full: check space and inodes, per-device await and queue, find the process doing the I/O.

The problem

Writes fail with `No space left on device`, the application stalls on fsync, or `iowait` is high and everything that touches disk is slow. The usual mistake is to start with I/O tuning when the disk is simply full of logs, out of inodes, or held by a deleted file. Check space first, then latency and queueing, then who is generating the I/O.

How it works

  • Symptom: `No space left on device` (ENOSPC), a database or logger that stops accepting writes, `%iowait` high in `mpstat`, tasks in D state, or `await` in `iostat` jumping from sub-millisecond to tens of milliseconds. Note that ENOSPC can also mean inodes are exhausted, or a quota or container storage limit was hit.
  • First look, space: `df -h` and `df -i` for the affected mount, and compare with `du -xh --max-depth=1 /var | sort -h | tail`. If `df` says full but `du` cannot account for it, suspect deleted-but-open files (`lsof +L1`) or space under a mount point.
  • First look, latency: `iostat -xz 1 3` and read `r_await`, `w_await`, `aqu-sz` (older sysstat: `await`, `avgqu-sz`) and `%util`. High await with a growing queue means the device cannot keep up; high `%util` alone does not prove saturation on SSD, NVMe or RAID. `vmstat 1` columns `b` and `wa` show how many tasks wait.
  • First look, who: `pidstat -d 1 3` for per-process read and write rates and `iodelay`, `iotop -o` (needs the iotop package and root) for a live view. `cat /proc/pressure/io` gives the stall share where PSI is enabled.
  • Common causes: logs or core dumps filling the volume, a runaway journal or container log, deleted files still held open, inode exhaustion from millions of tiny files (session or cache directories), a backup or compaction job saturating the device, and a noisy neighbour on shared cloud storage.
  • Common causes: writeback stalls when dirty pages hit `vm.dirty_ratio`, a failing disk (look for I/O errors and resets in the kernel log), a degraded RAID rebuild, and network storage (NFS, EBS-like volumes) with its own throttles. If the kernel log shows `blocked for more than 120 seconds` or `I/O error`, treat it as hardware or storage first.
  • Fix: free space safely (rotate or compress logs, prune old images with the tool that owns them, truncate with `: > file` rather than deleting a file a process holds), restart or signal the process holding a deleted file, throttle or reschedule the heavy job (`ionice -c3` only has an effect with a scheduler that honors I/O priorities, such as BFQ or mq-deadline; `systemd-run` with `IOWeight` only takes effect with BFQ or the io.cost controller), and for sustained saturation move to faster or more devices.
  • Verify: `df -h` and `df -i` show headroom, `iostat` await and queue fall to the baseline, `wa` and `b` drop in `vmstat`, `/proc/pressure/io` `avg10` returns near zero, and the application writes succeed under normal load.
  • Prevent: alert on filesystem usage and on inode usage at 80 to 85 percent, enforce log rotation and journal size limits, keep databases on their own volume, record an `fio` baseline per volume type, and plan the alert on queue depth and latency instead of `%util`.

Examples

Space and inodes first, then the biggest directories on that filesystem only.

bash
df -h /var
df -i /var
du -xh --max-depth=1 /var 2>/dev/null | sort -h | tail -8
# files deleted but still held open by a process:
lsof +L1 2>/dev/null | head

Is the device slow, and which process is doing the I/O?

bash
iostat -xz 1 3
# read: r_await, w_await (ms), aqu-sz (queue), %util (misleading on SSD/NVMe)
pidstat -d 1 3
cat /proc/pressure/io
dmesg -T | grep -iE 'i/o error|blocked for more than|reset' | tail

In practice

  • sysstat (iostat, pidstat) — `iostat -x` reports per-device latency and queue size; `pidstat -d` attributes bytes to processes. Together they answer whether the device is slow and who is using it.
  • bcc-tools and bpftrace — `biolatency` and `biosnoop` show block I/O latency histograms and individual requests when sysstat averages are not enough; they need root and a recent kernel.
  • In practice — A write stall that comes in regular bursts often matches writeback of a large dirty-page backlog or a periodic job; correlate the timestamps before changing sysctls.

Trade-offs

Pros

  • Checking space and inodes before latency resolves the most common cause in seconds and without any tooling.
  • Per-device and per-process views point to the right owner (application, backup job, storage team).

Cons

  • iostat averages hide short stalls; histograms (bcc) or `sar` history are needed for tail latency.
  • Cloud volumes may be throttled outside the guest, so the guest sees high await with no local explanation.

Connection refused, timeouts or Too many open files: tell fd limits, backlog overflow, port exhaustion and conntrack limits apart and fix the right one.

The problem

Clients see `connection refused`, timeouts, `Cannot assign requested address` or `Too many open files`, and the service may look healthy on CPU and memory. Several different limits produce similar symptoms: process file descriptors, the listen backlog, ephemeral ports, the conntrack table, and upstream connection caps. The job is to find out which limit is hit, on which side, before raising any number.

How it works

  • Symptom: errors on accept or connect (`EMFILE` Too many open files, `EADDRNOTAVAIL` Cannot assign requested address, `ECONNREFUSED`, connect timeouts), a listener that stops accepting while the process is alive, or `nf_conntrack: table full, dropping packet` in the kernel log.
  • First look: socket totals and states with `ss -s` and `ss -tan | awk 'NR>1{c[$1]++} END{for(s in c) print c[s], s}'`. Thousands of `TIME-WAIT` on a client point to port churn, many `CLOSE-WAIT` means the local application is not closing sockets, many `SYN-RECV` points to a full backlog or a flood.
  • First look, fds: `cat /proc/<pid>/limits | grep 'open files'` against `ls /proc/<pid>/fd | wc -l`, and system-wide `cat /proc/sys/fs/file-nr` (allocated, unused, max). The limit that matters is usually the per-process soft limit, which a systemd unit sets with `LimitNOFILE`, not `fs.file-max`.
  • First look, backlog: for a LISTEN socket `ss -ltn` shows the current accept-queue length in `Recv-Q` and the configured backlog in `Send-Q`; `nstat -az TcpExtListenOverflows TcpExtListenDrops` counts overflows. Growing counters mean the app accepts too slowly or the backlog is too small.
  • Common causes: a descriptor leak (fds grow without bound, often sockets or files never closed), no connection reuse so every request opens a new socket and fills the ephemeral port range (`net.ipv4.ip_local_port_range`) with TIME_WAIT toward a single destination, and CLOSE_WAIT piling up because the code forgets to close after the peer hangs up.
  • Common causes: a backlog capped by `net.core.somaxconn` or the application's own `listen()` value, a conntrack table too small for the connection churn of a NAT or Kubernetes node (`nf_conntrack_max`), and a downstream such as a database hitting `max_connections` while every app instance holds its own large pool.
  • Fix: raise limits where the evidence points (`LimitNOFILE=` in the unit plus restart; `somaxconn` plus the application's backlog setting; port range; `nf_conntrack_max` with memory in mind), but fix the cause as well: close leaked fds, enable keep-alive or a connection pool, and pool or cap connections toward the database instead of raising its maximum.
  • Verify: fd count stays flat well below the limit under load, `ListenOverflows` stops increasing, the state histogram shows a steady TIME-WAIT count and no growing CLOSE-WAIT, conntrack count (`conntrack -C`) stays below max, and client errors disappear.
  • Prevent: alert on fd usage as a share of the process limit, graph socket states, size limits at deploy time (unit files, sysctl.d), set sensible client timeouts and pool sizes, and avoid per-request connections to the same destination. Persist sysctls in `/etc/sysctl.d/`, never only with `sysctl -w`.

Examples

Which limit? Socket states, listen queues and overflow counters, system fd usage.

bash
ss -s
ss -tan | awk 'NR>1{c[$1]++} END{for(s in c) print c[s], s}'
# 30 ESTAB
# 4 LISTEN
ss -ltn
nstat -az TcpExtListenOverflows TcpExtListenDrops
cat /proc/sys/fs/file-nr
# 137  0  1645244   (allocated, unused, fs.file-max)

One process: its fd limit versus its open fds, and what kind they are (root for other users' processes).

bash
PID="<pid>"
grep 'open files' "/proc/$PID/limits"
ls "/proc/$PID/fd" | wc -l
ls -l "/proc/$PID/fd" | grep -c 'socket:'
ls -l "/proc/$PID/fd" | grep ' -> ' | grep -vc 'socket:'
# first number: sockets, second: files, pipes and the rest
# conntrack usage (needs the conntrack tool and root)
conntrack -C
sysctl net.netfilter.nf_conntrack_max

In practice

  • iproute2 ss and nstat — `ss` replaces `netstat` for sockets and queues; `nstat` reads the kernel SNMP and TcpExt counters such as `TcpExtListenOverflows` as deltas.
  • conntrack-tools — `conntrack -C` and `conntrack -S` show table usage and drop counters; the `table full, dropping packet` kernel message is the confirming sign.
  • In practice — Raising `LimitNOFILE` or `somaxconn` on a leaking service only delays the same failure; track whether the fd or socket count is flat or growing before choosing between tuning and a code fix.

Trade-offs

Pros

  • Separating the limits first prevents the common mistake of raising a sysctl that was never the bottleneck.
  • ss, nstat and /proc are always available and cost almost nothing to read.

Cons

  • Counters are cumulative since boot (or since `nstat` last ran), so a single reading can show old overflows; compare deltas over a window.
  • Some limits are enforced on the other side (a database, a load balancer, a NAT gateway) and cannot be seen from this host.

ssh hangs or the box looks dead: tell OOM thrash, hung I/O, full root, fd exhaustion and kernel panic apart, use console and safe sysrq, and keep evidence.

The problem

ssh hangs or is refused, the service stopped answering, yet the machine may still reply to ping. A host can look dead for very different reasons: memory thrash, blocked I/O, a full root filesystem, exhausted descriptors or pids, a runaway CPU hog, or a kernel panic. The goal is to get one working channel, capture evidence before a reboot erases it, and only then pick the branch.

How it works

  • Symptom: ssh connects but hangs at login or after the banner, commands freeze, the web service times out, monitoring agent stops reporting. Ping replying only proves the network stack and kernel interrupts work, not that user space can run. If ping also fails, check network, hypervisor and power state before blaming the OS.
  • First look: get any channel. Try a second ssh with a short timeout (`ssh -o ConnectTimeout=5`), an already-open session, a serial or cloud console, or an out-of-band controller (iDRAC, iLO, IPMI SOL, cloud serial console). The console often prints kernel messages (OOM, hung task, panic) that ssh never gets to show.
  • First look: once on the box, use the lightest commands first: `uptime`, `dmesg -T | tail -50`, `df -h /` and `df -i /`, `free -m`, `vmstat 1 3`. Shells hang on a full or stuck root filesystem, so prefer commands already in memory and avoid tab completion and heavy tools such as `ps aux` if the box is thrashing.
  • Common causes, resources: memory thrash (swap storm, OOM killer working late), root filesystem or `/var` full or out of inodes so logins and logging fail, fd or pid exhaustion (`fork: retry: Resource temporarily unavailable`, `Too many open files`), and CPU starvation by a runaway process or fork bomb.
  • Common causes, kernel and hardware: tasks stuck in D state on a dead disk, NFS or SAN (`INFO: task ... blocked for more than 120 seconds` in the kernel log), failing hardware or filesystem errors that remounted the root read-only, a soft lockup, or a kernel panic and reboot loop (`kernel.panic` controls the reboot delay).
  • Fix: free the resource that is exhausted, in the cheapest way that works: stop the offender, free space, kill a leaking process by pid, or restore the storage path. If shell access is impossible and the console is unresponsive, use the magic SysRq keys if enabled (`kernel.sysrq`): `w` dumps blocked tasks and `m` dumps memory info to the log, and `f` asks the kernel to run the OOM killer once. The sequence `s`, `u`, `b` (sync, remount read-only, reboot) is a last resort and `b` reboots at once without syncing.
  • Fix, evidence: before rebooting, save what you can (`dmesg -T`, `journalctl -b`, `vmstat`, `ps` output, the console screen). The kernel ring buffer and a volatile journal disappear on reboot; if the cause may be a crash, enable kdump and look at `journalctl -b -1` afterwards.
  • Verify: a fresh ssh login completes in about a second, `uptime` load is back to normal, `dmesg` shows no new errors, `df -h /` and `df -i /` have headroom, and the service answers. Check the kernel log for a clean boot sequence if you rebooted.
  • Prevent: make the journal persistent (`Storage=persistent` in `/etc/systemd/journald.conf`, or create `/var/log/journal`), keep root and `/var` on separate or quota-limited filesystems, set `kernel.panic` and consider `vm.panic_on_oom` plus kdump for hosts that must self-recover, run a watchdog (hardware or systemd), and alert on PSI, disk and fd usage before things freeze.

Examples

Lightest first look once you are on the box (everything here is read-only).

bash
uptime
dmesg -T | tail -30
df -h /
df -i /
free -m
vmstat 1 3
# tasks stuck in uninterruptible sleep and where they wait:
ps -eo stat,pid,wchan:24,comm | awk '$1 ~ /^D/'

Magic SysRq as evidence collection (root, kernel.sysrq must allow it; not available in every container or VM).

bash
cat /proc/sys/kernel/sysrq
# bitmask; 0 = disabled, 1 = all functions, other values enable subsets
echo w > /proc/sysrq-trigger
# w = dump blocked (D state) tasks to the kernel log
echo m > /proc/sysrq-trigger
# m = dump memory info to the kernel log
dmesg -T | tail -60

Keep the journal across reboots so the next hang leaves evidence.

bash
# create the directory; journald then stores logs persistently
mkdir -p /var/log/journal
systemctl restart systemd-journald
journalctl --list-boots
journalctl -b -1 -p err --no-pager | tail -30

In practice

  • Linux kernel Documentation/admin-guide/sysrq.rst — Describes the `kernel.sysrq` bitmask and the keys; SysRq works only if the kernel is still running interrupts and scheduling enough to handle it, and some distributions restrict it by default.
  • systemd-journald and kdump — A persistent journal and a kdump crash kernel are what turn a mysterious freeze into a readable post-mortem; both must be set up before the incident.
  • In practice — A cloud serial console and a snapshot of the instance taken before a hard reboot are often the only ways to see why a headless VM stopped answering.

Trade-offs

Pros

  • Starting from a working channel and the kernel log avoids guessing and keeps the evidence that a reboot would erase.
  • Persistent journal and kdump cost little disk and make the next incident much faster to diagnose.

Cons

  • Console and out-of-band access must exist before the incident; you cannot add them to a frozen box.
  • SysRq and panic-on-OOM settings trade availability of one host for faster recovery, and a wrong key (for example an immediate reboot) loses unsynced data.