Lesson 8 of 12 · 11 min · Processes and services
Running programs as services with systemd
What you will learn
- What systemd does and what a unit is
systemctlcommands you use daily- Writing a service unit for your own application, with a restart policy and a dedicated user
- Overriding a unit safely, and what timers are
In simple words
Some programs must run all the time, such as a web server. You don't start those by hand. You tell the system: start this when the machine switches on, and start it again if it crashes. systemd is the manager that follows those instructions. This lesson shows how to control it and how to hand it your own program.
Words to know
- Service: a program that runs in the background all the time, with nobody typing.
- systemd: the manager on most Linux systems that starts, stops and watches services.
- Unit file: a small text file that tells systemd how to run one service.
- systemctl: the command you use to talk to systemd: start, stop, status and more.
Why services
Starting a server with ./app & is fragile: it dies when you log out, does not start after a reboot, and nobody restarts it after a crash. A service manager solves this. On almost every modern distribution it is systemd, which runs as PID 1. It starts the system at boot, starts and supervises services in the right order, restarts them if they fail, and collects their output into one log (the journal).
systemd works with units. The types you meet most:
| Unit | File ends in | Purpose |
|---|---|---|
| service | .service |
a daemon or a one-shot job |
| timer | .timer |
run another unit on a schedule (like cron) |
| socket | .socket |
start a service when a connection arrives |
| target | .target |
a named group/state, for example multi-user.target (normal server) or graphical.target |
| mount | .mount |
a filesystem mount |
Unit files from packages live in /usr/lib/systemd/system/. Your own go in /etc/systemd/system/, which takes priority.
systemctl: the daily commands
systemctl status nginx # state, recent log lines, process tree
sudo systemctl start nginx
sudo systemctl stop nginx
sudo systemctl restart nginx # stop then start (drops connections)
sudo systemctl reload nginx # ask it to reload config without stopping (if supported)
sudo systemctl enable nginx # start automatically at boot
sudo systemctl disable nginx
sudo systemctl enable --now nginx # enable and start in one go
systemctl is-active nginx # prints active / inactive / failed
systemctl is-enabled nginx
systemctl --failed # list units that failed
Active (running now) and enabled (starts at boot) are independent. A service can be enabled and stopped, or running and disabled. Real output for a running service:
● nginx.service - A high performance web server and a reverse proxy server
Loaded: loaded (/usr/lib/systemd/system/nginx.service; enabled; preset: enabled)
Active: active (running) since Thu 2026-11-12 12:36:42 UTC; 18s ago
Process: 81 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
Main PID: 89 (nginx)
Tasks: 13 (limit: 2722)
Memory: 8.9M (peak: 9.9M)
Read it top to bottom: where the unit file is, whether it is enabled, whether it is running and for how long, the main PID, resource use, and the last log lines below.
Writing a service
Create a tiny app that prints a line every five seconds:
cat > /usr/local/bin/hello.sh <<'EOF'
#!/bin/bash
while true; do echo "hello from myapp, pid $$"; sleep 5; done
EOF
chmod +x /usr/local/bin/hello.sh
sudo useradd --system --no-create-home --shell /usr/sbin/nologin appsvc
Then /etc/systemd/system/myapp.service:
[Unit]
Description=My demo app
After=network.target
[Service]
User=appsvc
ExecStart=/usr/local/bin/hello.sh
Restart=on-failure
RestartSec=2
[Install]
WantedBy=multi-user.target
Three sections:
- [Unit] describes the unit and its relationships.
After=network.targetmeans "start me after networking is up" (ordering only).Requires=orWants=add dependencies. - [Service] says how to run it.
ExecStartis the command, with a full path.Userruns it as a non-root account.Restart=on-failurerestarts after a crash or non-zero exit;RestartSecwaits between attempts.Type=simpleis the default: systemd considers it started as soon as the process is forked. Other types:forking(the program daemonises itself),oneshot(runs and exits, for tasks). - [Install] says what enabling does.
WantedBy=multi-user.targetlinks the unit into the normal boot sequence.
After creating or changing a file, tell systemd to re-read it, then start and enable:
sudo systemctl daemon-reload
sudo systemctl start myapp
sudo systemctl enable myapp
Created symlink /etc/systemd/system/multi-user.target.wants/myapp.service → /etc/systemd/system/myapp.service.
enable just creates that symlink. Prove the restart policy works by killing the process:
sudo pkill -9 -f hello.sh
systemctl show myapp -p NRestarts -p ActiveState
NRestarts=1
ActiveState=active
The journal records what systemd did:
myapp.service: Failed with result 'signal'.
myapp.service: Scheduled restart job, restart counter is at 1.
Started myapp.service - My demo app.
Restart limits
systemd stops retrying if a service fails too many times in a short period (by default 5 starts in 10 seconds) and marks it failed; this protects the machine from a restart storm. Look for "start request repeated too quickly" in the log. Fix the cause, then systemctl reset-failed myapp and start again.
Useful [Service] settings
| Setting | Use |
|---|---|
WorkingDirectory=/opt/app |
directory the process starts in |
Environment=KEY=value / EnvironmentFile=/etc/myapp.env |
variables, or a file of them (keep secrets in a file with mode 600) |
ExecReload=/bin/kill -HUP $MAINPID |
what systemctl reload does |
TimeoutStopSec=30 |
how long to wait after SIGTERM before SIGKILL |
LimitNOFILE=65535 |
raise the open-files limit |
ProtectSystem=full, NoNewPrivileges=true, PrivateTmp=true |
sandboxing options that reduce damage if the service is compromised |
systemd stops a service by sending SIGTERM to its processes, which is why handling that signal matters.
Changing a unit without editing the package file
Never edit files in /usr/lib/systemd/system/; a package update would overwrite them. Create a drop-in:
sudo systemctl edit myapp # opens an editor and saves /etc/systemd/system/myapp.service.d/override.conf
For example, [Service] and Environment=APP_MODE=prod in that file. Confirm what systemd actually uses with systemctl cat myapp (shows the file and all drop-ins) and systemctl show myapp -p Environment.
Check a unit before you start it
systemd-analyze verify /etc/systemd/system/myapp.service
If ExecStart points to a file that does not exist it says:
myapp.service: Command /usr/local/bin/nope.sh is not executable: No such file or directory
and a real start fails with status=203/EXEC (the program could not be executed: wrong path, missing execute bit or wrong interpreter). More about reading failures in the troubleshooting lessons.
Timers: cron's modern replacement
A timer unit triggers a service of the same name:
# /etc/systemd/system/backup.timer
[Unit]
Description=Nightly backup
[Timer]
OnCalendar=*-*-* 02:00:00
Persistent=true
[Install]
WantedBy=timers.target
Persistent=true runs a missed job after the machine was off. systemctl list-timers shows when each timer fires next. Cron still works and is covered in the scripting course.
Common mistakes
- Forgetting
daemon-reloadafter editing a unit; systemd keeps using the old copy and warns about it. - Relative paths or shell syntax in
ExecStart. It is not a shell: pipes,&&and$VARdo not work unless you run/bin/bash -c '...'. - Running as root by default. Set
User=. Type=forkingmismatch. If your program stays in the foreground, usesimple.- Enabled but not started (or the reverse) after a reboot surprise.
Try it yourself
- In a systemd-enabled machine (a VM or WSL with systemd enabled; a plain Docker container has no systemd), create the demo service above, enable it and read its output with
journalctl -u myapp. - Kill it with
kill -9and confirm that it comes back. Then changeRestart=tono, reload, and repeat. - Break
ExecStarton purpose and usesystemctl statusandsystemd-analyze verifyto find the reason. - Add a drop-in that sets an environment variable and confirm it with
systemctl show.
Check yourself
What is the difference between systemctl enable and systemctl start?
start runs the service now; enable makes it start at boot by creating a symlink in the target's wants directory. They are independent; enable --now does both.
Why do you need systemctl daemon-reload?
systemd reads unit files into memory. After you create or change one, daemon-reload makes it re-read them. Without it the old definition is still used.
What does Restart=on-failure do, and what stops an endless restart loop?
It restarts the service when it exits with an error or is killed by a signal (not after a clean systemctl stop). Start limits (default five starts within ten seconds) put the unit into failed to stop restart storms.
Where do you put your own unit files, and how do you customise a packaged one?
Own units in /etc/systemd/system/. For a packaged unit, create a drop-in with systemctl edit, never edit /usr/lib/systemd/system/ directly.
Interview questions
Explain the difference between reload and restart.
restart stops and starts the process, so connections are dropped. reload asks the running process to re-read its configuration (via ExecReload, often SIGHUP) and keeps serving. Not every service supports reload.
How would you make an application start at boot, restart if it crashes, and run as a non-root user?
Write a unit with User=, ExecStart= and Restart=on-failure, put WantedBy=multi-user.target in [Install], then daemon-reload and enable --now.
What is a target?
A group of units that represents a system state, such as multi-user.target for a text-mode server or graphical.target with a desktop. It replaces the older "runlevels". systemctl get-default shows the default.
Summary
- systemd (PID 1) starts, orders and supervises services; its objects are units, controlled with
systemctl. - Own unit files go in
/etc/systemd/system/; rundaemon-reloadafter changing them; use drop-ins for packaged units. - A good service has a full-path
ExecStart, a non-rootUser,Restart=on-failureandWantedBy=multi-user.target. status,systemctl cat,systemd-analyze verifyand the journal are your first tools when a unit fails.
Going deeper
- systemd.service(5): every option in the [Service] section.
- systemctl(1): all commands.
- systemd.timer(5): timer units and calendar expressions.
See how much of this you can explain
A free mock interview asks you new questions on Linux for DevOps for DevOps Engineer and scores every answer, with what to fix.