Lesson 8 of 12 · 11 min · Processes and services

Running programs as services with systemd

Covers: Ubuntu 24.04 LTS, systemd 255. Updated 13 Nov 2026. Part of the DevOps Engineer roadmap.

What you will learn

  • What systemd does and what a unit is
  • systemctl commands you use daily
  • Writing a service unit for your own application, with a restart policy and a dedicated user
  • Overriding a unit safely, and what timers are

In simple words

Some programs must run all the time, such as a web server. You don't start those by hand. You tell the system: start this when the machine switches on, and start it again if it crashes. systemd is the manager that follows those instructions. This lesson shows how to control it and how to hand it your own program.

Words to know

  • Service: a program that runs in the background all the time, with nobody typing.
  • systemd: the manager on most Linux systems that starts, stops and watches services.
  • Unit file: a small text file that tells systemd how to run one service.
  • systemctl: the command you use to talk to systemd: start, stop, status and more.

Why services

Starting a server with ./app & is fragile: it dies when you log out, does not start after a reboot, and nobody restarts it after a crash. A service manager solves this. On almost every modern distribution it is systemd, which runs as PID 1. It starts the system at boot, starts and supervises services in the right order, restarts them if they fail, and collects their output into one log (the journal).

systemd works with units. The types you meet most:

Unit File ends in Purpose
service .service a daemon or a one-shot job
timer .timer run another unit on a schedule (like cron)
socket .socket start a service when a connection arrives
target .target a named group/state, for example multi-user.target (normal server) or graphical.target
mount .mount a filesystem mount

Unit files from packages live in /usr/lib/systemd/system/. Your own go in /etc/systemd/system/, which takes priority.

systemctl: the daily commands

systemctl status nginx            # state, recent log lines, process tree
sudo systemctl start nginx
sudo systemctl stop nginx
sudo systemctl restart nginx      # stop then start (drops connections)
sudo systemctl reload nginx       # ask it to reload config without stopping (if supported)
sudo systemctl enable nginx       # start automatically at boot
sudo systemctl disable nginx
sudo systemctl enable --now nginx # enable and start in one go
systemctl is-active nginx         # prints active / inactive / failed
systemctl is-enabled nginx
systemctl --failed                # list units that failed

Active (running now) and enabled (starts at boot) are independent. A service can be enabled and stopped, or running and disabled. Real output for a running service:

● nginx.service - A high performance web server and a reverse proxy server
     Loaded: loaded (/usr/lib/systemd/system/nginx.service; enabled; preset: enabled)
     Active: active (running) since Thu 2026-11-12 12:36:42 UTC; 18s ago
    Process: 81 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
   Main PID: 89 (nginx)
      Tasks: 13 (limit: 2722)
     Memory: 8.9M (peak: 9.9M)

Read it top to bottom: where the unit file is, whether it is enabled, whether it is running and for how long, the main PID, resource use, and the last log lines below.

Writing a service

Create a tiny app that prints a line every five seconds:

cat > /usr/local/bin/hello.sh <<'EOF'
#!/bin/bash
while true; do echo "hello from myapp, pid $$"; sleep 5; done
EOF
chmod +x /usr/local/bin/hello.sh
sudo useradd --system --no-create-home --shell /usr/sbin/nologin appsvc

Then /etc/systemd/system/myapp.service:

[Unit]
Description=My demo app
After=network.target

[Service]
User=appsvc
ExecStart=/usr/local/bin/hello.sh
Restart=on-failure
RestartSec=2

[Install]
WantedBy=multi-user.target

Three sections:

  • [Unit] describes the unit and its relationships. After=network.target means "start me after networking is up" (ordering only). Requires= or Wants= add dependencies.
  • [Service] says how to run it. ExecStart is the command, with a full path. User runs it as a non-root account. Restart=on-failure restarts after a crash or non-zero exit; RestartSec waits between attempts. Type=simple is the default: systemd considers it started as soon as the process is forked. Other types: forking (the program daemonises itself), oneshot (runs and exits, for tasks).
  • [Install] says what enabling does. WantedBy=multi-user.target links the unit into the normal boot sequence.

After creating or changing a file, tell systemd to re-read it, then start and enable:

sudo systemctl daemon-reload
sudo systemctl start myapp
sudo systemctl enable myapp
Created symlink /etc/systemd/system/multi-user.target.wants/myapp.service → /etc/systemd/system/myapp.service.

enable just creates that symlink. Prove the restart policy works by killing the process:

sudo pkill -9 -f hello.sh
systemctl show myapp -p NRestarts -p ActiveState
NRestarts=1
ActiveState=active

The journal records what systemd did:

myapp.service: Failed with result 'signal'.
myapp.service: Scheduled restart job, restart counter is at 1.
Started myapp.service - My demo app.

Restart limits

systemd stops retrying if a service fails too many times in a short period (by default 5 starts in 10 seconds) and marks it failed; this protects the machine from a restart storm. Look for "start request repeated too quickly" in the log. Fix the cause, then systemctl reset-failed myapp and start again.

Useful [Service] settings

Setting Use
WorkingDirectory=/opt/app directory the process starts in
Environment=KEY=value / EnvironmentFile=/etc/myapp.env variables, or a file of them (keep secrets in a file with mode 600)
ExecReload=/bin/kill -HUP $MAINPID what systemctl reload does
TimeoutStopSec=30 how long to wait after SIGTERM before SIGKILL
LimitNOFILE=65535 raise the open-files limit
ProtectSystem=full, NoNewPrivileges=true, PrivateTmp=true sandboxing options that reduce damage if the service is compromised

systemd stops a service by sending SIGTERM to its processes, which is why handling that signal matters.

Changing a unit without editing the package file

Never edit files in /usr/lib/systemd/system/; a package update would overwrite them. Create a drop-in:

sudo systemctl edit myapp       # opens an editor and saves /etc/systemd/system/myapp.service.d/override.conf

For example, [Service] and Environment=APP_MODE=prod in that file. Confirm what systemd actually uses with systemctl cat myapp (shows the file and all drop-ins) and systemctl show myapp -p Environment.

Check a unit before you start it

systemd-analyze verify /etc/systemd/system/myapp.service

If ExecStart points to a file that does not exist it says:

myapp.service: Command /usr/local/bin/nope.sh is not executable: No such file or directory

and a real start fails with status=203/EXEC (the program could not be executed: wrong path, missing execute bit or wrong interpreter). More about reading failures in the troubleshooting lessons.

Timers: cron's modern replacement

A timer unit triggers a service of the same name:

# /etc/systemd/system/backup.timer
[Unit]
Description=Nightly backup
[Timer]
OnCalendar=*-*-* 02:00:00
Persistent=true
[Install]
WantedBy=timers.target

Persistent=true runs a missed job after the machine was off. systemctl list-timers shows when each timer fires next. Cron still works and is covered in the scripting course.

Common mistakes

  • Forgetting daemon-reload after editing a unit; systemd keeps using the old copy and warns about it.
  • Relative paths or shell syntax in ExecStart. It is not a shell: pipes, && and $VAR do not work unless you run /bin/bash -c '...'.
  • Running as root by default. Set User=.
  • Type=forking mismatch. If your program stays in the foreground, use simple.
  • Enabled but not started (or the reverse) after a reboot surprise.

Try it yourself

  1. In a systemd-enabled machine (a VM or WSL with systemd enabled; a plain Docker container has no systemd), create the demo service above, enable it and read its output with journalctl -u myapp.
  2. Kill it with kill -9 and confirm that it comes back. Then change Restart= to no, reload, and repeat.
  3. Break ExecStart on purpose and use systemctl status and systemd-analyze verify to find the reason.
  4. Add a drop-in that sets an environment variable and confirm it with systemctl show.

Check yourself

What is the difference between systemctl enable and systemctl start?

start runs the service now; enable makes it start at boot by creating a symlink in the target's wants directory. They are independent; enable --now does both.

Why do you need systemctl daemon-reload?

systemd reads unit files into memory. After you create or change one, daemon-reload makes it re-read them. Without it the old definition is still used.

What does Restart=on-failure do, and what stops an endless restart loop?

It restarts the service when it exits with an error or is killed by a signal (not after a clean systemctl stop). Start limits (default five starts within ten seconds) put the unit into failed to stop restart storms.

Where do you put your own unit files, and how do you customise a packaged one?

Own units in /etc/systemd/system/. For a packaged unit, create a drop-in with systemctl edit, never edit /usr/lib/systemd/system/ directly.

Interview questions

Explain the difference between reload and restart.

restart stops and starts the process, so connections are dropped. reload asks the running process to re-read its configuration (via ExecReload, often SIGHUP) and keeps serving. Not every service supports reload.

How would you make an application start at boot, restart if it crashes, and run as a non-root user?

Write a unit with User=, ExecStart= and Restart=on-failure, put WantedBy=multi-user.target in [Install], then daemon-reload and enable --now.

What is a target?

A group of units that represents a system state, such as multi-user.target for a text-mode server or graphical.target with a desktop. It replaces the older "runlevels". systemctl get-default shows the default.

Summary

  • systemd (PID 1) starts, orders and supervises services; its objects are units, controlled with systemctl.
  • Own unit files go in /etc/systemd/system/; run daemon-reload after changing them; use drop-ins for packaged units.
  • A good service has a full-path ExecStart, a non-root User, Restart=on-failure and WantedBy=multi-user.target.
  • status, systemctl cat, systemd-analyze verify and the journal are your first tools when a unit fails.

Going deeper

See how much of this you can explain

A free mock interview asks you new questions on Linux for DevOps for DevOps Engineer and scores every answer, with what to fix.