Lesson 12 of 12 · 11 min · Troubleshooting
Boot and service failures
What you will learn
- The stages of booting, so you know where a failure is
- How to diagnose a failed service: status, exit codes, journal, and the usual causes
- What to do when a machine boots into emergency mode
- A checklist for "it worked yesterday"
In simple words
Starting a computer is a relay race: each stage hands over to the next until your services are running. If one runner drops the baton, the machine or the service doesn't come up. This lesson teaches the stages, so you can tell where it stopped and read the message that says why.
Words to know
- Boot: everything that happens from pressing the power button until the system is ready.
- Boot loader: the small program that loads the kernel into memory.
- Failed service: a service that systemd tried to start but that stopped with an error.
- Emergency mode: a minimal mode Linux falls back to when it can't finish booting, so you can repair it.
How a Linux machine boots
- Firmware (BIOS or UEFI) tests hardware and finds the boot loader. In the cloud this happens before you can see anything.
- Boot loader (usually GRUB) loads the kernel and an initial RAM filesystem (initramfs) into memory, passing it options (
cat /proc/cmdlineshows them). - Kernel starts, finds the hardware, uses the initramfs to load drivers and find the real root filesystem, then mounts it.
- systemd runs as PID 1 and starts everything needed to reach the default target (
systemctl get-default, normallymulti-user.targeton a server): mounts the filesystems in/etc/fstab, brings up networking, starts services in dependency order.
A failure at each stage looks different:
| Stage | Typical symptom | Typical cause |
|---|---|---|
| Boot loader | "No bootable device", GRUB prompt | broken boot config, disk changed |
| Kernel / initramfs | stops at "Kernel panic" or "cannot mount root fs" | wrong root device, missing driver, damaged filesystem |
| systemd | "Welcome to emergency mode", or boots but services are missing | a mount in /etc/fstab failed, a critical service failed |
| Service | machine up, application down | configuration, port, permissions, dependencies |
On cloud machines, the serial console or "system log" in the provider's console is how you see boot messages when SSH is not working.
Diagnosing a failed service
Start with the facts systemd already has:
systemctl --failed # which units failed
systemctl status web2 # why, in one screen
journalctl -u web2 -b --no-pager # its full log since boot
Here is a real case: we started a second web server on port 80 while nginx already used it.
× web2.service - Second web app
Active: failed (Result: exit-code) since Thu 2026-11-12 12:43:31 UTC; 1s ago
Process: 915 ExecStart=/usr/bin/python3 -m http.server 80 (code=exited, status=1/FAILURE)
Main PID: 915 (code=exited, status=1/FAILURE)
The journal explains the exit code:
python3[902]: OSError: [Errno 98] Address already in use
web2.service: Failed with result 'exit-code'.
ss -tlnp 'sport = :80' then shows who holds the port (nginx). The fix is to choose another port or stop the other service. After fixing, clear the failed state and start again: sudo systemctl reset-failed web2 && sudo systemctl start web2. If the unit restarted several times quickly, systemd gives up with "start request repeated too quickly".
Reading the status codes
| What you see | Meaning | Check |
|---|---|---|
status=1/FAILURE (or other small number) |
the program itself exited with an error | the program's own error in the journal |
status=203/EXEC |
systemd could not run ExecStart |
path wrong, no execute permission, missing interpreter |
status=200/CHDIR, 217/USER, 226/NAMESPACE |
WorkingDirectory missing, User does not exist, sandbox setting failed |
the unit's settings and the account |
code=killed, signal=KILL |
killed, often the OOM killer | dmesg -T, memory limits |
code=killed, signal=TERM |
stopped by request | who or what stopped it |
Result: timeout |
start or stop took too long | TimeoutStartSec, slow dependency, hung process |
Result: start-limit-hit |
too many restarts in a short time | fix the cause, reset-failed |
The usual causes, in the order to check
- The error is in the program's output. Read the lines just before systemd's "Failed" line. Configuration syntax errors are the most common (test with the program's own check:
nginx -t,sshd -t,apachectl configtest,visudo -c). - Port already in use.
ss -tlnp. - Permissions and ownership. The service user cannot read its config, write its log or data directory, or traverse a parent directory (
namei -l). - A missing dependency: another service, a mounted disk, a file, an environment variable. Look at
systemctl list-dependencies myapp, and whetherAfter=/Requires=express the real order. - Resources: disk full, out of memory, too many open files (
LimitNOFILE). - Security layers: SELinux (RHEL family) or AppArmor (Ubuntu) can deny actions that normal permissions allow; the denial appears in the audit log or
journalctl. Do not disable the layer; read the denial and add the right allowance. - Something changed: a package update, a config edit, a certificate that expired, a changed DNS record. Ask "what changed since it last worked?" and check
/var/log/apt/history.logordnf history, and your deployment history.
When the whole machine fails to boot
If a mount listed in /etc/fstab fails, systemd drops into emergency mode (a root shell asking for the root password, with a minimal system). Common reasons are a typo in fstab or a missing disk. In emergency mode:
journalctl -xb # the boot log with explanations
systemctl --failed
mount -o remount,rw / # the root filesystem may be read-only here
nano /etc/fstab # fix or comment the line
systemctl default # continue booting (or reboot)
Prevent the problem: for non-essential disks add the nofail option in fstab, so the machine boots even if the disk is missing, and always run sudo mount -a after editing fstab to test the change before rebooting.
Other recovery options:
- Rescue mode (
systemd.unit=rescue.targetadded to the kernel line in GRUB) gives a single-user shell with local filesystems mounted. - A previous kernel from the GRUB "Advanced options" menu if a new kernel does not boot.
- Attach the disk to another machine (cloud: detach the volume, attach it to a helper instance, mount, fix, and move it back) when you cannot get a shell at all. This is the standard cloud rescue procedure.
- Never cut power repeatedly: filesystems can be damaged. A read-only root usually means the kernel found an error; run
fsckon an unmounted filesystem.
Look at slow boots with systemd-analyze (total time) and systemd-analyze blame (slowest units).
A checklist for "it worked yesterday"
- What exactly fails, and what is the exact error text?
systemctl statusandjournalctl -u ... -bfor the service, then-p err -bfor everything.- What changed? Deployments, package updates, config edits, certificates, DNS, disk usage, new firewall rules.
- Resources:
df -h,df -i,free -h,dmesg -T | tail. - Reproduce in the smallest way: run the command as the service user (
sudo -u appsvc /path/to/app) to see the same error on the terminal. - Change one thing at a time and write down what you changed so you can undo it.
- After the fix, find the cause, not only the trigger, and add a check so it is found earlier next time.
Step 5 is powerful: services run with a different user, environment and working directory than your shell, which explains "it works when I run it by hand".
Common mistakes
- Rebooting to see if it fixes it, losing the evidence (memory state,
dmesg) and sometimes making the situation worse. - Editing
/etc/fstaband rebooting withoutmount -a. - Disabling SELinux or AppArmor instead of reading the denial.
- Fixing the symptom (restarting every night) and never the cause.
- Not looking at the first error. A later error is usually a consequence.
Try it yourself
- Create a service whose
ExecStartpoints to a missing file. Readstatus, the code (203/EXEC) and fix it. - Make two services listen on the same port, read the "Address already in use" error, and find the owner with
ss. - Set
User=to a user that does not exist and read the resulting status. - In a disposable VM, add a wrong line to
/etc/fstabwithoutnofailand see emergency mode, then recover. Never do this on a machine you care about.
Check yourself
What are the four stages of a Linux boot?
Firmware, boot loader (GRUB), kernel with initramfs, and systemd bringing up the target (mounts, networking, services).
A service fails with status=203/EXEC. What does it mean?
systemd could not execute the ExecStart program: wrong or relative path, missing file, no execute permission, or a bad #! interpreter.
Why does a script work from your shell but fail as a service?
A service runs as a different user, in a different working directory, with a minimal environment and no terminal. Check paths, permissions and variables, and reproduce with sudo -u user and env -i.
Why add nofail to /etc/fstab entries for data disks?
Without it, a missing or broken disk stops the boot in emergency mode. With nofail the system boots and the application can report the problem.
Interview questions
A server is stuck in emergency mode after a reboot. What happened and how do you fix it?
Usually a failed mount from /etc/fstab (wrong UUID, missing disk). Log in at the console, read journalctl -xb, remount root read-write, fix or comment the line (or add nofail), test with mount -a, and continue the boot.
Walk me through debugging a service that will not start.
systemctl status for the exit code, journalctl -u for the program's error, then check configuration syntax, port conflicts, file permissions for the service user, dependencies and resources; reproduce as the service user; fix one thing at a time.
What does "start request repeated too quickly" mean?
The service failed and restarted more often than the start limit allows, so systemd stopped trying. Fix the underlying error, systemctl reset-failed, then start it.
Summary
- Boot goes firmware → boot loader → kernel/initramfs → systemd; the symptom tells you the stage.
- For a failed service:
systemctl status, the exit code, then the journal lines before the failure. - Usual causes: config error, port in use, permissions, dependencies, resources, SELinux/AppArmor, or a recent change.
- Test
fstabwithmount -a, usenofailfor non-essential disks, and know rescue options (emergency mode, GRUB older kernel, attach the disk elsewhere).
Going deeper
- systemd.exec(5): exit status codes: what 200 to 243 mean.
- bootup(7): the boot process in detail.
- fstab(5): the file's fields and the
nofailoption.
See how much of this you can explain
A free mock interview asks you new questions on Linux for DevOps for DevOps Engineer and scores every answer, with what to fix.