Lesson 12 of 12 · 11 min · Troubleshooting

Boot and service failures

Covers: Ubuntu 24.04 LTS, systemd 255. Updated 13 Nov 2026. Part of the DevOps Engineer roadmap.

What you will learn

  • The stages of booting, so you know where a failure is
  • How to diagnose a failed service: status, exit codes, journal, and the usual causes
  • What to do when a machine boots into emergency mode
  • A checklist for "it worked yesterday"

In simple words

Starting a computer is a relay race: each stage hands over to the next until your services are running. If one runner drops the baton, the machine or the service doesn't come up. This lesson teaches the stages, so you can tell where it stopped and read the message that says why.

Words to know

  • Boot: everything that happens from pressing the power button until the system is ready.
  • Boot loader: the small program that loads the kernel into memory.
  • Failed service: a service that systemd tried to start but that stopped with an error.
  • Emergency mode: a minimal mode Linux falls back to when it can't finish booting, so you can repair it.

How a Linux machine boots

  1. Firmware (BIOS or UEFI) tests hardware and finds the boot loader. In the cloud this happens before you can see anything.
  2. Boot loader (usually GRUB) loads the kernel and an initial RAM filesystem (initramfs) into memory, passing it options (cat /proc/cmdline shows them).
  3. Kernel starts, finds the hardware, uses the initramfs to load drivers and find the real root filesystem, then mounts it.
  4. systemd runs as PID 1 and starts everything needed to reach the default target (systemctl get-default, normally multi-user.target on a server): mounts the filesystems in /etc/fstab, brings up networking, starts services in dependency order.

A failure at each stage looks different:

Stage Typical symptom Typical cause
Boot loader "No bootable device", GRUB prompt broken boot config, disk changed
Kernel / initramfs stops at "Kernel panic" or "cannot mount root fs" wrong root device, missing driver, damaged filesystem
systemd "Welcome to emergency mode", or boots but services are missing a mount in /etc/fstab failed, a critical service failed
Service machine up, application down configuration, port, permissions, dependencies

On cloud machines, the serial console or "system log" in the provider's console is how you see boot messages when SSH is not working.

Diagnosing a failed service

Start with the facts systemd already has:

systemctl --failed                 # which units failed
systemctl status web2              # why, in one screen
journalctl -u web2 -b --no-pager   # its full log since boot

Here is a real case: we started a second web server on port 80 while nginx already used it.

× web2.service - Second web app
     Active: failed (Result: exit-code) since Thu 2026-11-12 12:43:31 UTC; 1s ago
    Process: 915 ExecStart=/usr/bin/python3 -m http.server 80 (code=exited, status=1/FAILURE)
   Main PID: 915 (code=exited, status=1/FAILURE)

The journal explains the exit code:

python3[902]: OSError: [Errno 98] Address already in use
web2.service: Failed with result 'exit-code'.

ss -tlnp 'sport = :80' then shows who holds the port (nginx). The fix is to choose another port or stop the other service. After fixing, clear the failed state and start again: sudo systemctl reset-failed web2 && sudo systemctl start web2. If the unit restarted several times quickly, systemd gives up with "start request repeated too quickly".

Reading the status codes

What you see Meaning Check
status=1/FAILURE (or other small number) the program itself exited with an error the program's own error in the journal
status=203/EXEC systemd could not run ExecStart path wrong, no execute permission, missing interpreter
status=200/CHDIR, 217/USER, 226/NAMESPACE WorkingDirectory missing, User does not exist, sandbox setting failed the unit's settings and the account
code=killed, signal=KILL killed, often the OOM killer dmesg -T, memory limits
code=killed, signal=TERM stopped by request who or what stopped it
Result: timeout start or stop took too long TimeoutStartSec, slow dependency, hung process
Result: start-limit-hit too many restarts in a short time fix the cause, reset-failed

The usual causes, in the order to check

  1. The error is in the program's output. Read the lines just before systemd's "Failed" line. Configuration syntax errors are the most common (test with the program's own check: nginx -t, sshd -t, apachectl configtest, visudo -c).
  2. Port already in use. ss -tlnp.
  3. Permissions and ownership. The service user cannot read its config, write its log or data directory, or traverse a parent directory (namei -l).
  4. A missing dependency: another service, a mounted disk, a file, an environment variable. Look at systemctl list-dependencies myapp, and whether After=/Requires= express the real order.
  5. Resources: disk full, out of memory, too many open files (LimitNOFILE).
  6. Security layers: SELinux (RHEL family) or AppArmor (Ubuntu) can deny actions that normal permissions allow; the denial appears in the audit log or journalctl. Do not disable the layer; read the denial and add the right allowance.
  7. Something changed: a package update, a config edit, a certificate that expired, a changed DNS record. Ask "what changed since it last worked?" and check /var/log/apt/history.log or dnf history, and your deployment history.

When the whole machine fails to boot

If a mount listed in /etc/fstab fails, systemd drops into emergency mode (a root shell asking for the root password, with a minimal system). Common reasons are a typo in fstab or a missing disk. In emergency mode:

journalctl -xb                 # the boot log with explanations
systemctl --failed
mount -o remount,rw /          # the root filesystem may be read-only here
nano /etc/fstab                # fix or comment the line
systemctl default              # continue booting (or reboot)

Prevent the problem: for non-essential disks add the nofail option in fstab, so the machine boots even if the disk is missing, and always run sudo mount -a after editing fstab to test the change before rebooting.

Other recovery options:

  • Rescue mode (systemd.unit=rescue.target added to the kernel line in GRUB) gives a single-user shell with local filesystems mounted.
  • A previous kernel from the GRUB "Advanced options" menu if a new kernel does not boot.
  • Attach the disk to another machine (cloud: detach the volume, attach it to a helper instance, mount, fix, and move it back) when you cannot get a shell at all. This is the standard cloud rescue procedure.
  • Never cut power repeatedly: filesystems can be damaged. A read-only root usually means the kernel found an error; run fsck on an unmounted filesystem.

Look at slow boots with systemd-analyze (total time) and systemd-analyze blame (slowest units).

A checklist for "it worked yesterday"

  1. What exactly fails, and what is the exact error text?
  2. systemctl status and journalctl -u ... -b for the service, then -p err -b for everything.
  3. What changed? Deployments, package updates, config edits, certificates, DNS, disk usage, new firewall rules.
  4. Resources: df -h, df -i, free -h, dmesg -T | tail.
  5. Reproduce in the smallest way: run the command as the service user (sudo -u appsvc /path/to/app) to see the same error on the terminal.
  6. Change one thing at a time and write down what you changed so you can undo it.
  7. After the fix, find the cause, not only the trigger, and add a check so it is found earlier next time.

Step 5 is powerful: services run with a different user, environment and working directory than your shell, which explains "it works when I run it by hand".

Common mistakes

  • Rebooting to see if it fixes it, losing the evidence (memory state, dmesg) and sometimes making the situation worse.
  • Editing /etc/fstab and rebooting without mount -a.
  • Disabling SELinux or AppArmor instead of reading the denial.
  • Fixing the symptom (restarting every night) and never the cause.
  • Not looking at the first error. A later error is usually a consequence.

Try it yourself

  1. Create a service whose ExecStart points to a missing file. Read status, the code (203/EXEC) and fix it.
  2. Make two services listen on the same port, read the "Address already in use" error, and find the owner with ss.
  3. Set User= to a user that does not exist and read the resulting status.
  4. In a disposable VM, add a wrong line to /etc/fstab without nofail and see emergency mode, then recover. Never do this on a machine you care about.

Check yourself

What are the four stages of a Linux boot?

Firmware, boot loader (GRUB), kernel with initramfs, and systemd bringing up the target (mounts, networking, services).

A service fails with status=203/EXEC. What does it mean?

systemd could not execute the ExecStart program: wrong or relative path, missing file, no execute permission, or a bad #! interpreter.

Why does a script work from your shell but fail as a service?

A service runs as a different user, in a different working directory, with a minimal environment and no terminal. Check paths, permissions and variables, and reproduce with sudo -u user and env -i.

Why add nofail to /etc/fstab entries for data disks?

Without it, a missing or broken disk stops the boot in emergency mode. With nofail the system boots and the application can report the problem.

Interview questions

A server is stuck in emergency mode after a reboot. What happened and how do you fix it?

Usually a failed mount from /etc/fstab (wrong UUID, missing disk). Log in at the console, read journalctl -xb, remount root read-write, fix or comment the line (or add nofail), test with mount -a, and continue the boot.

Walk me through debugging a service that will not start.

systemctl status for the exit code, journalctl -u for the program's error, then check configuration syntax, port conflicts, file permissions for the service user, dependencies and resources; reproduce as the service user; fix one thing at a time.

What does "start request repeated too quickly" mean?

The service failed and restarted more often than the start limit allows, so systemd stopped trying. Fix the underlying error, systemctl reset-failed, then start it.

Summary

  • Boot goes firmware → boot loader → kernel/initramfs → systemd; the symptom tells you the stage.
  • For a failed service: systemctl status, the exit code, then the journal lines before the failure.
  • Usual causes: config error, port in use, permissions, dependencies, resources, SELinux/AppArmor, or a recent change.
  • Test fstab with mount -a, use nofail for non-essential disks, and know rescue options (emergency mode, GRUB older kernel, attach the disk elsewhere).

Going deeper

See how much of this you can explain

A free mock interview asks you new questions on Linux for DevOps for DevOps Engineer and scores every answer, with what to fix.