Lesson 10 of 12 · 11 min · Troubleshooting
Checking CPU, memory and disk
What you will learn
- A quick routine to see whether a server is limited by CPU, memory, disk space, disk speed or the network
- How to read load average,
top,freeandvmstat - Finding what fills a disk with
dfanddu, and the three classic "disk full" surprises - What the out-of-memory killer does and how to see it
In simple words
When a server is slow, something has run short: the brain (CPU), the desk space (memory) or the storage room (disk). It is like finding out why a kitchen is slow: too few cooks, too little counter space, or a full store room? This lesson gives you a quick routine to find which one it is.
Words to know
- CPU: the part that does the computing. When it is at 100%, work has to wait.
- Memory (RAM): the fast working space for running programs. It is emptied when the machine restarts.
- Load average: how many tasks are running or waiting for their turn. Compare it with the number of CPUs.
- OOM killer: the kernel's last resort when memory runs out: it stops a process to save the rest.
First minute on a slow server
When someone says "the server is slow", run these four commands before changing anything:
uptime # load average
free -h # memory and swap
df -h # disk space per filesystem
top # who is using CPU and memory (press q to quit)
Then decide which resource is the bottleneck. A good habit (the USE method): for each resource ask about Utilisation (how busy), Saturation (is work queueing), and Errors.
CPU: load average and top
uptime
12:37:10 up 8:29, 0 user, load average: 1.85, 2.20, 1.63
The three numbers are the load average over 1, 5 and 15 minutes: roughly the number of processes that were running or waiting to run (Linux also counts those in uninterruptible disk wait, state D). Compare it with the number of CPU cores (nproc):
- Load below the core count: the CPUs are keeping up.
- Load around the core count: fully used.
- Load well above the core count: work is queueing.
A load of 4 is idle on 12 cores and a crisis on 2. Falling numbers (1 min lower than 15 min) mean it is recovering; rising means it is getting worse.
top shows the cause. The summary lines:
%Cpu(s): 4.8 us, 1.6 sy, 0.0 ni, 93.6 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
| Field | Meaning |
|---|---|
us |
time in user programs |
sy |
time in the kernel (system calls, networking) |
id |
idle |
wa |
waiting for disk or network I/O. High wa means a storage bottleneck, not a CPU one |
st |
stolen by the hypervisor: your VM shares a busy host (common in the cloud) |
In top, press P to sort by CPU, M by memory, 1 to show each core, c for full command lines. In a script, top -bn1 | head -15 prints one snapshot. ps aux --sort=-%cpu | head lists the heaviest processes. If one process uses 100% of one core while the rest idle, it may be a single-threaded program or a busy loop.
Memory
free -h
total used free shared buff/cache available
Mem: 14Gi 11Gi 494Mi 3.3Gi 6.2Gi 2.9Gi
Swap: 4.0Gi 3.9Gi 103Mi
The most common beginner mistake is reading free as "only 494 MB left!". Linux uses spare memory as a cache for files (buff/cache) and gives it back whenever programs need it. The number that matters is available: how much can be given to new programs without swapping. Low available is the real warning.
Swap is disk space used as slow overflow memory. A little used swap that never changes is harmless. Memory that keeps moving in and out of swap makes everything slow ("thrashing"). Check with vmstat 1 5:
r b swpd free buff cache si so bi bo in cs us sy id wa st
1 0 4088772 503156 118316 6334100 155 395 3430 1958 5545 6 5 2 93 0 0
si and so are swap-in and swap-out per second. Sustained non-zero values mean the machine is short of RAM. r is runnable processes (compare with cores) and b is processes blocked on I/O.
Find the biggest consumers: ps aux --sort=-%mem | head. The RSS column is the memory a process really holds in RAM.
The out-of-memory (OOM) killer
When memory is exhausted, the kernel kills a process to survive and logs it:
dmesg -T | grep -i -E "out of memory|killed process"
journalctl -k | grep -i "out of memory"
Out of memory: Killed process 58226 (java) total-vm:... anon-rss:367256kB ... UID:1000 ...
The victim gets signal 9, so its parent sees exit code 137. In containers and Kubernetes, a pod showing OOMKilled or exit code 137 means the same thing: it went over its memory limit. The fix is a memory leak fix, a higher limit, or fewer processes, not "restart it".
Disk space
df -h
Filesystem Size Used Avail Use% Mounted on
overlay 468G 206G 239G 47% /
df reports per filesystem; du reports a directory's size. To find what is large, work downwards one level at a time:
du -sh /var/* 2>/dev/null | sort -h | tail # biggest entries in /var
du -xh / --max-depth=2 2>/dev/null | sort -h | tail -15 # -x stays on one filesystem
Typical culprits: /var/log (no rotation), /var/lib/docker (images and layers: docker system df), /var/cache, core dumps, old kernels, and uploads.
Three "disk full" surprises
1. The disk is full but you cannot find the files. A process may hold a deleted file open: the space is not released until the process closes it or exits. We tested it on a small filesystem: after deleting a 3 MB file that a shell still had open, df still counted it while du saw nothing.
tmpfs 5.0M 3.0M 2.0M 60% /mnt/small <- df: 3 MB used
0 /mnt/small <- du: nothing visible
lsof +L1 lists open files with no remaining name:
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NLINK NODE NAME
bash 676 root 3w REG 0,185 3145728 0 5 /mnt/small/held (deleted)
Restart the owning process (the classic cause: a big log deleted with rm while the app still writes to it; truncate with : > file instead of deleting).
2. "No space left on device" but df -h shows free space: inodes ran out. Every file uses one inode, and the number is fixed when the filesystem is made. Millions of tiny files (sessions, mail queues, caches) can use them all:
df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
tmpfs 20 20 0 100% /mnt/small
Writing then fails with No space left on device although there are free bytes. Find the directory with the most files and clean it.
3. Reserved blocks and mounts. Some filesystems (ext4) reserve a percentage for root, so Avail can be smaller than Size - Used. And if a disk is not mounted where you expect, files are written into the directory on the root disk beneath it. findmnt and lsblk show what is mounted where.
Disk speed
If CPU is idle but wa is high and commands hang, the disk is the limit. iostat -x 1 (package sysstat) shows per-device utilisation (%util), queue size and wait times (await). A device near 100% %util with long waits is saturated. Processes stuck in state D in ps or top confirm it.
Putting it together
| Symptom | Likely resource | Check |
|---|---|---|
High load, high us |
CPU-bound program | top, ps --sort=-%cpu |
High load, high wa, many D processes |
disk or network storage | iostat -x 1, vmstat |
Low available memory, si/so active |
memory shortage | free -h, vmstat, OOM lines in dmesg |
| "No space left on device" | blocks or inodes | df -h, df -i, lsof +L1 |
High st |
noisy neighbour | ask the cloud provider, change instance |
Common mistakes
- Reading
freeas "used = problem". Look atavailable. - Deleting a log that is open and wondering why space did not return.
- Treating load average as CPU percent. It is a queue length; compare with cores.
- Restarting first, investigating later. You lose the evidence. Capture
top,free,dmesgoutput before you restart.
Try it yourself
- Record
uptime,nproc,free -handdf -hon your machine. Say which resource is closest to its limit. - Create a 200 MB file with
dd if=/dev/zero of=big bs=1M count=200orfallocate -l 200M big, then find it again using onlydufrom/-level directories, level by level. - On a throwaway machine, start a process that holds a file open (
tail -f big &), delete the file, and comparedfanddu. Find it withlsof +L1. - Run
yes > /dev/null &and watch it intop. What happens to the load average after a minute? Stop it.
Check yourself
A server has 4 cores and load average 12. What does that say, and what do you check next?
Work is queueing: about three times as many runnable (or I/O-blocked) tasks as cores. Check top/vmstat: high us means CPU-bound programs, high wa or many D processes means a disk problem.
free shows only 300 MB free of 16 GB. Is the server out of memory?
Not necessarily. Linux uses free memory for the page cache. Look at the available column and at swap activity (si/so).
df says 100% used but du finds only half of that. Name two possible reasons.
A deleted file is still held open by a process (lsof +L1), or the files sit under a mount point that hides them, or reserved blocks. Another case: du was run without permission to read some directories.
What does exit code 137 mean for a container?
It was killed with SIGKILL (128 + 9), most often by the OOM killer after exceeding its memory limit.
Interview questions
A disk shows "No space left on device" but df -h shows 40% free. What is wrong?
Probably inode exhaustion. Confirm with df -i, then find the directory with huge numbers of small files and remove or archive them.
How do you find out which process is using the most memory and what it did before it died?
ps aux --sort=-%mem | head for the current users; for a dead process dmesg -T or journalctl -k show the OOM killer's line, including the process name and memory figures.
What does a high wa value in top mean?
The CPU is idle, waiting for I/O to finish. The bottleneck is storage (or network storage), not CPU, so adding cores will not help.
Summary
- Start with
uptime,free -h,df -h,top; judge load against the number of cores. availablememory and swap activity matter, not "free". The OOM killer logs todmesgand ends processes with signal 9 (exit 137).- Find disk hogs with
du -sh * | sort -h; remember inodes (df -i) and deleted-but-open files (lsof +L1). - High
wameans a disk problem, highstmeans a noisy neighbour.
Going deeper
- proc_meminfo(5): what each memory figure means.
- top(1): every field and key.
- Linux kernel documentation: OOM killer and memory management concepts: how the kernel handles memory.
- The USE Method (Brendan Gregg): a checklist for resource problems.
See how much of this you can explain
A free mock interview asks you new questions on Linux for DevOps for DevOps Engineer and scores every answer, with what to fix.