Lesson 9 of 12 · 9 min · Troubleshooting

Reading logs: journalctl and /var/log

Covers: Ubuntu 24.04 LTS, systemd 255. Updated 13 Nov 2026. Part of the DevOps Engineer roadmap.

What you will learn

  • Where logs live: the systemd journal and the text files in /var/log
  • journalctl filters you will use in every incident: by unit, time, priority, boot and text
  • Following logs live and reading kernel messages
  • How log rotation works, and how to keep the journal from using all the disk

In simple words

A log is a diary that a program writes: what it did, when, and what went wrong. When something breaks, the answer is nearly always written there. This lesson shows where the diaries are kept and how to jump straight to the page you need instead of reading all of them.

Words to know

  • Log: a record of events that a program writes as it runs.
  • Journal: the central log that systemd keeps for all services.
  • journalctl: the command for reading the journal, with filters for time, service and seriousness.
  • Log rotation: automatically closing old log files and deleting the oldest, so logs never fill the disk.

Two places for logs

  1. The journal. systemd-journald collects the output (stdout and stderr) of every service systemd starts, messages sent through syslog, and kernel messages, and stores them in a structured, indexed form. You read it with journalctl.
  2. Text files in /var/log. Many programs write their own files: /var/log/nginx/access.log and error.log, /var/log/auth.log (logins and sudo on Debian/Ubuntu; /var/log/secure on RHEL family), /var/log/syslog (general messages, when rsyslog is installed), /var/log/dmesg, package manager logs.

Both exist side by side. For a systemd service, start with journalctl -u name. For web access logs or application-specific files, read the file.

journalctl: the filters

Run alone, journalctl prints everything, oldest first, in a pager. You almost never want that. Narrow it down.

journalctl -u nginx                     # one service (-u = unit)
journalctl -u nginx -n 50               # last 50 lines
journalctl -u nginx -f                  # follow new lines, like tail -f
journalctl -u nginx --since "10 min ago"
journalctl -u nginx --since "2026-11-12 09:00" --until "2026-11-12 09:30"
journalctl -b                           # this boot only (-b -1 = the previous boot)
journalctl -p err                       # priority err and worse
journalctl -k                           # kernel messages (like dmesg)
journalctl -g "timeout"                 # lines matching a pattern (-g = grep)
journalctl _PID=1234                    # by process id
journalctl -o short-iso                 # timestamps in ISO format; -o json for scripts
journalctl --no-pager | less            # when you want to control paging yourself

Filters combine: journalctl -u myapp -p err --since today.

Real examples. Messages sent with logger (the command a script can use to write to the journal):

logger -t demo "disk almost full"
logger -t demo -p user.err "database connection failed"
journalctl -t demo -n 3 --no-pager      # -t = syslog identifier
Nov 12 12:41:57 dd23a794e2e4 demo[653]: disk almost full
Nov 12 12:41:57 dd23a794e2e4 demo[654]: database connection failed

Only the second has priority err:

journalctl -t demo -p err --no-pager
Nov 12 12:41:57 dd23a794e2e4 demo[654]: database connection failed

And the typical output when a service cannot start (status 203 means the program could not be executed):

journalctl -u myapp -g "exited" -n 2 --no-pager
Nov 12 12:41:57 dd23a794e2e4 systemd[1]: myapp.service: Main process exited, code=exited, status=203/EXEC

Priorities

Log priorities follow the syslog levels from 0 to 7: emerg, alert, crit, err, warning, notice, info, debug. -p err shows err and everything more severe (crit, alert, emerg). -p warning..err shows a range.

Reading the previous boot

After an unexpected reboot or crash, the useful lines are from before it:

journalctl --list-boots
journalctl -b -1 -p err           # errors from the previous boot
journalctl -b -1 -e               # jump to the end (the last messages before it stopped)

This only works if the journal is persistent. If /var/log/journal does not exist, the journal lives in memory and is lost at reboot; create the directory and restart journald (sudo mkdir -p /var/log/journal && sudo systemctl restart systemd-journald), or set Storage=persistent in /etc/systemd/journald.conf. Many cloud images already have it.

Keeping the journal from filling the disk

journalctl --disk-usage
Archived and active journals take up 8.0M in the file system.

Limit it in /etc/systemd/journald.conf (SystemMaxUse=500M) or trim now with sudo journalctl --vacuum-size=500M or --vacuum-time=14d.

Text logs and rotation

Programs that write to files would fill the disk forever, so logrotate (run daily by a systemd timer or cron) renames old logs (app.log → app.log.1), compresses older ones, deletes the oldest and tells the program to reopen its file. Its configuration is in /etc/logrotate.conf and /etc/logrotate.d/<app>:

/var/log/myapp/*.log {
    daily
    rotate 14
    compress
    missingok
    notifempty
    postrotate
        systemctl reload myapp > /dev/null 2>&1 || true
    endscript
}

Read it: every day, keep 14 old files, compress them, do not complain if missing or empty, and reload the service afterwards so it writes to the new file. A common bug is a program that keeps writing into the old, renamed file because it was never told to reopen it.

Handy commands for files:

tail -f /var/log/nginx/error.log
grep -i "error" /var/log/myapp/app.log | tail -20
zgrep "timeout" /var/log/nginx/error.log.2.gz      # search a compressed log
sudo less +F /var/log/auth.log                      # less in follow mode (Ctrl+C to stop following)

Kernel messages

The kernel's own messages (hardware, drivers, out-of-memory kills, disk errors) are in journalctl -k or dmesg. For example, when the kernel kills a process because memory ran out, it logs a line like this:

Out of memory: Killed process 58226 (java) total-vm:... anon-rss:367256kB ... UID:1000 ...

dmesg -T prints human-readable timestamps. Searching dmesg -T | grep -i -E "oom|out of memory|error|fail" is a standard first step after a mysterious crash.

A method for reading logs

  1. Decide the time window. When did it start? Use --since/--until.
  2. Start at the failing component (-u), then widen to neighbours (database, proxy).
  3. Filter by severity first (-p err), then read the lines just before the first error: the cause is usually earlier than the first loud message.
  4. Find the first failure, not the last. Later errors are often consequences.
  5. Correlate across sources by time: app log, proxy log, kernel log.

Common mistakes

  • Reading a whole log from the top. Filter by unit and time.
  • Looking at /var/log/syslog for a systemd service that logs only to the journal.
  • Not noticing journal permissions. A normal user sees only their own messages unless in the systemd-journal or adm group; use sudo or join the group.
  • Running out of disk because of logs with no rotation or limits.

Try it yourself

  1. Use logger to write one info and one err message with the tag lab, then show only the error with journalctl.
  2. Find every sudo use in the last day from the journal or from /var/log/auth.log.
  3. Check journalctl --disk-usage and journalctl --list-boots on your machine. Is the journal persistent?
  4. Write a logrotate rule for /var/log/lab/*.log that keeps 7 days, then test it with logrotate -d /etc/logrotate.d/lab (debug mode: shows what would happen).

Check yourself

How do you see only the last 100 lines of a service's log and keep following it?

journalctl -u <service> -n 100 -f.

What does journalctl -b -1 -p err show, and why might it be empty?

Errors from the previous boot. It can be empty if the journal is not persistent (it was kept in memory and lost at reboot) or if nothing was logged at error level.

What is the difference between the journal and files in /var/log?

The journal is a structured, indexed store managed by journald and read with journalctl; /var/log holds plain text files written by rsyslog and by programs themselves. Both can be present.

Why does a log file keep growing even after logrotate renamed it?

The program still has the old file open and writes into it. Rotation must signal the program to reopen its log (a postrotate reload) or use copytruncate.

Interview questions

A service crashed at 3 a.m. Walk through how you find out why.

systemctl status for the exit status, then journalctl -u service --since "02:50" --until "03:10" for its own output, check journalctl -p err and dmesg -T around the same time (OOM kill, disk errors), and look at the proxy or dependency logs for the same window.

How do you stop logs from filling the disk?

Rotation with logrotate for files, journald size limits (SystemMaxUse, --vacuum-size), sensible log levels in the application, and shipping logs to a central system so local retention can be short.

What does exit status 203/EXEC mean in the journal?

systemd could not execute the ExecStart command: the path is wrong, the file is not executable, or the interpreter in its #! line is missing.

Summary

  • Service output goes to the journal (journalctl -u name); many programs also write files in /var/log.
  • Narrow down with -u, --since/--until, -p, -b, -g, follow with -f; kernel messages with -k or dmesg -T.
  • Find the first error and read what came just before it.
  • Control disk use with logrotate and journald limits.

Going deeper

See how much of this you can explain

A free mock interview asks you new questions on Linux for DevOps for DevOps Engineer and scores every answer, with what to fix.