When you run top, the second line shows something like: %Cpu(s): 45.2 us, 38.1 sy, 0.0 ni, 12.4 id...
Most people look at the combined number and move on. The split between us (user) and sy (system/kernel) is one of the most useful diagnostics on the line.
What user time means. User time is CPU spent running your application code — the Python interpreter, the JVM, the Go runtime. High user time with a culprit process means the workload itself is computationally heavy. This is the "expected" scenario.
What system time means. System time is CPU spent in kernel code on behalf of your process. Every syscall — read(), write(), accept(), mmap(), fork() — contributes to system time. If sy is above 20-30% it means your processes are making an unusual number of syscalls, or expensive ones.
High sy with normal us — three common causes.
*1. Excessive syscalls.* A tight loop calling write() for every log line instead of buffering, or calling gettimeofday() millions of times per second, will push sy high. Use strace -c -p PID to count syscalls and find the expensive ones.
*2. Page faults.* Heavy memory allocation in a tight loop generates minor page faults — kernel work that shows up in sy. OOM pressure generates major page faults (disk reads), which show in iowait but also elevate sy.
*3. Lock contention.* futex() calls — the syscall underlying pthread mutexes, Go sync primitives, Java monitors — are kernel work. If threads are fighting over the same mutex, sy climbs while actual work stalls.
The ratio to watch. A healthy compute-heavy server typically shows us:sy of 10:1 or better. Ratios below 4:1 warrant investigation. A ratio approaching 1:1 means your processes are spending as much time in the kernel as in application code — something is wrong.