For every resource (CPU, memory, disk, network, plus software limits) check Utilization, Saturation and Errors in a fixed order, so you find the bottleneck instead of guessing.
The problem
The box is slow and there are fifty metrics on screen. Most people stare at whichever graph they know best, or tune something that was never the bottleneck. The USE method replaces that with a short, exhaustive loop: list every resource, and for each one ask three questions in order: how busy is it, is work queueing behind it, and is it throwing errors.
How it works
- Definitions: Utilization is the fraction of time (or capacity) the resource was busy; Saturation is extra work it could not service yet (a queue); Errors are counts of failed operations. A resource at 100% utilization is usually the bottleneck, and any sustained saturation is a performance problem even at lower utilization.
- Check errors first when they are cheap: they are quick to read (`dmesg`, `ip -s link`, `sar -n EDEV`, `smartctl`) and a nonzero count often explains everything. Then utilization, then saturation. Use the table `tool-by-resource` below for the exact command per cell.
- CPU: utilization is `mpstat -P ALL 1` (watch one hot core hiding in a low average); saturation is the run queue (`vmstat 1` column `r` above the core count, or `/proc/pressure/cpu`); errors are rare (machine-check or thermal messages in `dmesg`).
- Memory: utilization is `free -m` `available`, not `free`; saturation is swapping (`vmstat` `si`/`so`), allocation stalls, `/proc/pressure/memory` and OOM kills; errors are `dmesg` OOM reports and failed allocations, plus ECC errors where the hardware reports them.
- Disk: utilization is `iostat -xz 1` `%util` (misleading on parallel devices such as NVMe and RAID, see `iostat-reading`); saturation is `aqu-sz` and rising `await`, or `/proc/pressure/io`; errors are `I/O error` lines in `dmesg` and SMART data.
- Network: utilization is throughput against link speed (`sar -n DEV 1`, `ip -s link`); saturation is drops, overruns and retransmits (`nstat`, `sar -n ETCP 1`, `ss -ti`); errors are the `errors`/`dropped` counters in `ip -s link` and `ethtool -S`.
- Do not forget software resources, which hit limits long before hardware does: file descriptors (`/proc/<pid>/limits`, `fs.file-nr`), threads and `kernel.pid_max`, the conntrack table, listen backlog, ephemeral ports, and cgroup limits (`cpu.stat` throttling, `memory.events`). Treat each as a resource with a usage, a queue and an error such as `EMFILE`.
- Walk the list once, write down anything abnormal, then drill into the worst resource with per-process tools (`pidstat`, `perf`, `strace`). The method finds the resource, not the root cause; it tells you which playbook to open next.
Examples
A USE walk for the hardware resources; each line is tagged with the letter it answers.
# One pass over the four hardware resources: Utilization / Saturation / Errors
mpstat -P ALL 1 3 # U: CPU busy per core
vmstat 1 3 # S: r > cores, si/so > 0
cat /proc/pressure/cpu # S: "some avg10=..." = share of time tasks waited
free -m # U: look at "available"
iostat -xz 1 3 # U: %util S: aqu-sz, await
sar -n DEV 1 3 # U: rx/tx kB/s versus link speed
ip -s link show # E: errors, dropped, missed per interface
dmesg -T | tail -n 50 # E: I/O errors, OOM, NIC resets (may need root)Software resources have the same three questions: how full, is anything queued, what error.
# Software resources: the limits that bite first
cat /proc/sys/fs/file-nr # allocated, unused, max file handles
cat /proc/sys/kernel/pid_max # ceiling for PIDs and thread IDs
cat /proc/sys/net/netfilter/nf_conntrack_count /proc/sys/net/netfilter/nf_conntrack_max
ss -s # socket totals, incl. timewait
grep -E "Max open files|Max processes" "/proc/<pid>/limits"
cat "/sys/fs/cgroup/<group>/cpu.stat" # nr_throttled, throttled_usec (cgroup v2)In practice
- Brendan Gregg's USE method — The method comes from Brendan Gregg: build a checklist of every resource and read utilization, saturation and errors for each. His published checklists map the idea to Linux commands; this entry and the table below are a condensed, independent version.
- sysstat and procps — `mpstat`, `iostat`, `sar` (sysstat) and `vmstat`, `free`, `top` (procps-ng) cover most cells of the grid without installing anything exotic, which is why they are the default toolbox.
- Pressure Stall Information (PSI) — Since kernel 4.20, `/proc/pressure/{cpu,memory,io}` reports directly how long tasks stalled waiting for a resource, which is a saturation metric measured by the kernel instead of inferred from queue lengths. It is absent if the kernel is built without PSI or booted with `psi=0`.
- In practice — A common pitfall is checking only utilization. Averages hide a single saturated core, and a disk at 30% `%util` can still have a deep queue; always pair a utilization number with a saturation number.
Trade-offs
Pros
- Exhaustive and cheap: a fixed checklist stops you from fixating on the wrong resource.
- Works with tools already on every box, and gives a vocabulary teams can share in incident channels.
Cons
- Finds the constrained resource, not the cause; you still need profiling and tracing afterwards.
- Utilization and saturation are hard to define for some resources (a mutex, a connection pool), and cloud VMs hide host-level contention such as noisy neighbours except through `%steal`.