Diagnosing a slow machine: CPU, memory and I/O

Start with a method, not with commands

When something is slow, people often run tools at random. A short routine finds the cause faster:

1. Define the symptom: what is slow, since when, for whom? Latency, throughput or errors?
2. Look for a change: deployments, configuration, traffic, data growth, a failing dependency.
3. Check each resource with the USE method: for CPU, memory, disk and network, ask three questions:
   • Utilisation: how busy is it?
   • Saturation: is work waiting in a queue?
   • Errors: is anything failing?
4. Drill into the suspect with a more detailed tool (profiler, trace).
5. Fix, then verify that the metric you started with improved.

100%

Figure SDA-RWS-12-1 · Slow-system investigation flow · © forcsguy.com

Reading the signals

CPU

• top or htop: which processes use CPU? Is time spent in user code, system (kernel) or iowait (waiting for disk)?
• Load average counts runnable and waiting tasks. Load much higher than the number of cores means queueing, but remember that tasks waiting on I/O also count.
• A flame graph (from perf or a language profiler) shows which functions consume the time.

Memory

• free -m and vmstat: available memory, swap use. Swapping heavily makes everything slow.
• A process whose resident memory only grows has a leak (or an unbounded cache). Watch memory over time, then take a heap profile.
• The operating system uses spare memory as a file cache, so "used" memory is not always a problem. Check available.

Disk (I/O)

• iostat -x: utilisation, queue length and await (latency per request). High await with a busy disk means storage is the bottleneck.
• iotop shows which process does the I/O.
• Full disks and exhausted inodes cause odd failures.

Network

• ss -s, netstat, and interface counters: connections in odd states, retransmissions, dropped packets.
• Check DNS time, connection pool limits and ephemeral port exhaustion.

Application level

• Slow queries and lock waits in the database, thread pool or connection pool exhaustion, garbage-collection pauses, and calls to slow dependencies.
• Distributed traces show where each request spends its time. Logs with a correlation id fill in the detail.

100%

Figure SDA-RWS-12-2 · Which tool looks at which layer · © forcsguy.com

A few useful commands

Question	Tool
What is using CPU and memory?	top, htop, ps aux --sort=-%cpu
Is the machine swapping or waiting on disk?	vmstat 1, iostat -x 1
What is a process doing?	strace -p PID (system calls), lsof -p PID (open files)
Network state	ss -tan, ping, traceroute, curl -w timing
Where is the time in the code?	a profiler or perf, then a flame graph

Real-world reading