Every tool you use to measure CPU — top, htop, vmstat, Prometheus node_exporter — reads from /proc/stat. Understanding that file makes everything else click.
What the file contains. Each line starting with "cpu" shows cumulative tick counts since boot, broken into: user, nice, system, idle, iowait, irq, softirq, steal, guest, guest_nice. All in units of USER_HZ (usually 1/100th of a second).
The critical thing everyone gets wrong. You can't read the file once and get a percentage. CPU% is a rate — the delta between two reads divided by the elapsed time. Most tools sample every 1–3 seconds. This means your "CPU usage" reading is always at minimum 1–3 seconds stale.
iowait is not CPU usage. Contrary to popular belief, iowait is idle time — time the CPU spent doing nothing while waiting for IO. A system with 60% iowait is 60% idle. The confusion arises because iowait inflates the appearance of "busyness" when you're IO-bottlenecked.
steal time is hypervisor debt. In virtualized environments, steal is CPU cycles your VM wanted but the hypervisor gave to a different VM. You can't fix this by optimizing your code — you need to move to a less contended host or a dedicated instance.
How to read it yourself. Use awk to read /proc/stat twice with a one-second sleep between reads, then compute the delta. The idle delta divided by the total delta gives your true idle percentage — no tool required.
The cpum.ai agent samples /proc/stat every 15 seconds and keeps a rolling window of 60 samples. This gives you a 15-minute CPU history with proper delta-based percentages — not the instantaneous snapshot that misleads most dashboards.