Lesson 7 of 7 · 10 min · Reliable scripts
A complete script: a safe backup job
What you will learn
- How the pieces from the earlier lessons fit into one real script
- Validating input, locking against overlapping runs, writing safely and rotating old files
- How to test a script for success and for each failure path
- What to look for when you review someone else's script
In simple words
This lesson puts everything together in one real script: a backup job that checks its inputs, refuses to run twice at the same time, never leaves a half-written file behind, and deletes old copies. Read it like a worked example in a textbook: every line is there for a reason.
Words to know
- Validation: checking the inputs before doing anything, and stopping with a clear message if they are wrong.
- Lock: a marker that stops a second copy of the script from running at the same time.
- Retention: the rule for how many old backups to keep.
- Restore test: proving that a backup can really be brought back. A backup you never tested is only a hope.
The task
Back up a directory into a compressed archive, keep only the newest N archives, never leave a half-written archive behind, and never run two backups at the same time. It must be safe to run from cron and give clear exit codes.
The script
#!/usr/bin/env bash
# backup.sh - archive a directory, keep the newest N archives, and refuse to run twice at once.
set -euo pipefail
log() { printf '%s %s\n' "$(date +%T)" "$*" >&2; }
die() { log "ERROR: $*"; exit 1; }
usage() { echo "usage: $0 [-k KEEP] SOURCE_DIR DEST_DIR" >&2; exit 2; }
keep=7
while getopts "k:h" opt; do
case $opt in
k) keep=$OPTARG ;;
*) usage ;;
esac
done
shift $((OPTIND - 1))
[[ $# -eq 2 ]] || usage
src=$1 dest=$2
[[ $keep =~ ^[0-9]+$ ]] && (( keep >= 1 )) || die "-k must be a number of 1 or more"
[[ -d $src ]] || die "source is not a directory: $src"
mkdir -p "$dest"
exec 9>"$dest/.backup.lock"
flock -n 9 || die "another backup is already running"
name=$(basename "$src")
stamp=$(date +%Y%m%d-%H%M%S)
tmp=$(mktemp "$dest/.${name}-XXXXXX")
trap 'rm -f "$tmp"' EXIT
log "archiving $src"
tar -czf "$tmp" -C "$(dirname "$src")" "$name"
final="$dest/${name}-${stamp}.tar.gz"
mv "$tmp" "$final"
trap - EXIT
log "created $final ($(du -h "$final" | cut -f1))"
# keep only the newest $keep archives of this name
mapfile -t old < <(ls -1t "$dest/${name}"-*.tar.gz | tail -n +"$((keep + 1))")
for f in "${old[@]}"; do
log "removing old archive $f"
rm -- "$f"
done
log "done"
Walking through it
Header and strict mode. set -euo pipefail makes errors stop the script. log and die write to stderr, so stdout stays free for data.
Arguments. getopts "k:h" reads -k KEEP. After shift $((OPTIND - 1)) exactly two arguments must remain, otherwise usage prints help and exits 2 (the convention for wrong usage). Real failures exit 1.
Validation first. Before touching anything, the script checks that -k is a positive number (=~ regex plus (( ))) and that the source is a directory. A script that fails early and clearly is easier to run unattended.
Locking. exec 9>"$dest/.backup.lock" opens a lock file on file descriptor 9 and keeps it open for the life of the script; flock -n 9 tries to take an exclusive lock without waiting. If another run holds it, the script exits with a message. The lock is released automatically when the script ends, even if it crashes, which is why flock is better than "create a pid file".
Writing safely. mktemp creates a unique temporary file inside the destination, tar writes into it, and only when tar succeeded does mv rename it to the final name. A rename within one filesystem is atomic: other programs see either no archive or a complete one, never a half-written file. The trap 'rm -f "$tmp"' EXIT removes the temporary file if anything fails in between; trap - EXIT cancels it after the successful mv.
Retention. ls -1t lists newest first and tail -n +"$((keep + 1))" skips the first keep lines, leaving the older ones, which are deleted. Parsing ls is normally discouraged, but here every file name was generated by this script (no spaces or odd characters), and the pattern restricts to this source's archives. rm -- "$f" uses -- so a file name can never be read as an option.
Testing every path
A script is not finished until you have seen it fail properly. These runs are real:
$ ./backup.sh
usage: ./backup.sh [-k KEEP] SOURCE_DIR DEST_DIR
exit=2
$ ./backup.sh -k x /data/site /backups
ERROR: -k must be a number of 1 or more
exit=1
$ ./backup.sh /nope /backups
ERROR: source is not a directory: /nope
exit=1
$ ./backup.sh -k 2 /data/site /backups
archiving /data/site
created /backups/site-20261112-125215.tar.gz (4.0K)
done
exit=0
After four runs with -k 2, the oldest archives are removed (removing old archive ...) and the directory contains only the newest two. tar -tzf on the newest one lists site/, site/index.html, site/style.css, proving the contents. When a second copy starts while the first holds the lock:
ERROR: another backup is already running
exit=1
Test more: unwritable destination, full disk, source with spaces in its name, run twice in the same second (the name includes seconds; the lock normally prevents overlap).
Running it from cron
# m h dom mon dow command
30 2 * * * /usr/local/bin/backup.sh -k 7 /srv/app /var/backups/app >> /var/log/backup.log 2>&1
Use full paths, redirect both streams to a log, and have monitoring look at the exit code and the age of the newest archive, not only at whether the job ran. A backup that has not been restored is not a backup: test a restore regularly (tar -xzf into a scratch directory and compare).
Reviewing a script: a checklist
- Shebang and
set -euo pipefail? - Are all expansions quoted? Any
rm/mvwith a variable in the path, and is it guarded? - Does it validate input and tools before acting?
- Are temporary files made with
mktempand cleaned by atrap? - Can two copies run at once, and is that safe?
- Is it idempotent? What happens on the second run, or after a half-finished run?
- Does it exit non-zero on every real failure and log what happened?
- Does it run the same way under cron/CI (PATH, working directory, no terminal)?
- Does ShellCheck pass?
- Are secrets kept out of the command line and the logs?
Common mistakes
- No validation, so an empty variable becomes a dangerous path.
- Writing straight to the final file, leaving corrupt output after a failure.
- No lock on a job that can run long enough to overlap with the next one.
- Never testing a restore.
- Hard-coding secrets or passing them as command-line arguments (visible in
ps). Read them from a protected file or environment.
Try it yourself
- Type the script in, run ShellCheck on it, and write down each finding and whether it matters here.
- Add an option
-qthat silenceslogoutput (hint: a variable thatlogchecks). - Add a check that the destination has at least as much free space as the source (
df --output=availanddu -sk). - Change it to back up a PostgreSQL database with
pg_dump | gzip, keeping the same safety properties. What must you add to detect a failedpg_dump? (Hint:pipefail.)
Check yourself
Why write the archive to a temporary name and then mv it?
A rename on the same filesystem is atomic, so the final name never points to a partial file. If the job fails midway, only a temp file remains and the trap deletes it.
What advantage does flock have over a pid file for preventing overlapping runs?
The lock is held by an open file descriptor, so the kernel releases it when the process ends for any reason, including a crash. A pid file can be left behind and block future runs.
Why exit with 2 for a usage error and 1 for a runtime failure?
It is a common convention that lets callers tell "you called me wrongly" from "I tried and failed". Both are non-zero, so schedulers still see failure.
How would you know that backups are actually working in production?
Alert on the job's exit code, on the age of the newest archive, on its size (a zero-byte archive is wrong) and on a regular restore test.
Interview questions
How do you make a script safe to run from cron?
Full paths, explicit environment, locking against overlap, logging to a file with both streams, meaningful exit codes, cleanup traps, and monitoring of success and freshness.
How do you avoid leaving half-written files?
Write to a temporary file in the same directory and rename it into place after the command succeeds.
What is the risk of rm -rf "$dir/"* and how do you reduce it?
An empty or wrong dir deletes the wrong things. Validate it ([[ -d $dir ]], non-empty, expected prefix), use ${dir:?}, and avoid -f when not needed.
Summary
- A trustworthy script validates input, locks, writes atomically, cleans up, logs, and uses exit codes properly.
- Test failure paths as carefully as the success path.
- Schedule with full paths and both streams logged, and monitor outcome and freshness, including a restore test.
- Review scripts with a checklist; run ShellCheck.
Going deeper
- flock(1): locking from the shell.
- GNU tar manual: archive options and extraction.
- Google Shell Style Guide: a widely used set of conventions for readable scripts.
See how much of this you can explain
A free mock interview asks you new questions on Shell scripting with Bash for DevOps Engineer and scores every answer, with what to fix.