Lesson 3 of 7 · 8 min · Bash basics
Reading files, input and command-line options
What you will learn
- Reading a file or command output line by line without breaking on spaces
- Splitting delimited fields with
IFSandread - Here-documents and here-strings for multi-line text
- Parsing options with
getopts, and printing a proper usage message - Temporary files and directories
In simple words
Useful scripts take input: a file to read, an answer from a person, or settings typed after the script's name. This lesson shows how to read a file one line at a time without mistakes, and how to accept options such as -v, the way real commands do.
Words to know
- Argument: a value typed after the script's name, such as the file to work on.
- Option: a switch such as
-vthat changes how the script behaves. - Here-document: a way to put several lines of text inside a script and feed them to a command.
- Temporary file: a file a script creates for its own short use and deletes when it finishes.
Looping over lines
The correct pattern to read a file line by line:
while IFS= read -r line; do
echo "line: $line"
done < file.txt
readtakes one line from stdin into the variable.IFS=stopsreadfrom trimming leading and trailing spaces.-rstops it from treating backslashes as escape characters.< file.txtfeeds the file to the whole loop.
To loop over a command's output, use process substitution so the loop runs in the current shell (a plain pipe would run it in a subshell and lose variable changes):
while IFS= read -r line; do echo "line: $line"; done < <(printf 'one\ntwo words\n')
line: one
line: two words
Splitting fields
Set IFS (the input field separator) for the read command only:
printf 'name,qty\nweb,2\ndb,1\n' | while IFS=, read -r n q; do echo "$n has $q"; done
name has qty
web has 2
db has 1
This is fine for simple CSV. Real CSV with quotes needs a proper tool (Python's csv, jq for JSON). Never parse JSON, YAML or HTML with grep and sed: use jq, yq or a script language.
Looping over files
for f in *.log; do echo "file: [$f] lines: $(wc -l < "$f")"; done
file: [a b.log] lines: 1
file: [c.log] lines: 1
The brackets in the output show that the name with a space stayed in one piece, because every use of $f is quoted. For files in subdirectories use find ... -print0 | while IFS= read -r -d '' f; do ...; done, or find ... -exec.
Here-documents and here-strings
A here-document gives a command several lines of text. Quote the delimiter ('EOF') to stop expansion inside:
cat <<EOF > config.txt
host=$(hostname)
EOF
cat <<'EOF' > script.sh
echo "$HOME is not expanded here"
EOF
Without quotes the variables and commands inside expand, as for host= above. <<-EOF strips leading tabs so you can indent. A here-string passes one string as input: grep -c a <<< "banana".
Here-documents are the standard way to write config files or send multi-line input to commands (ssh host bash -s <<'EOF', mysql <<'EOF').
Asking for input
read -r -p "Delete everything? [y/N] " answer
[[ $answer == [yY] ]] || { echo "aborted"; exit 1; }
read -r -s -p "Password: " pw; echo # -s hides what is typed
In automation nobody is typing. A good script accepts values as arguments or environment variables and only asks if running in a terminal ([[ -t 0 ]] tests whether stdin is a terminal).
Temporary files
Never use fixed names like /tmp/data.txt: two runs collide, and another user could plant a symlink there. Use mktemp and clean up with a trap:
tmp=$(mktemp -d) # a new private directory
trap 'rm -rf "$tmp"' EXIT # removed when the script ends, however it ends
echo "working in $tmp"
mktemp creates the path with unpredictable name and mode 600/700. The trap command is explained in the next lesson.
Handling output and errors of commands
output=$(some_command 2>&1) || { echo "failed: $output" >&2; exit 1; } # capture both, handle failure
if ! command -v jq >/dev/null; then echo "jq is required" >&2; exit 1; fi # check a tool exists
mapfile -t lines < <(grep ERROR app.log) # read lines into an array
command -v name is the portable way to check that a program is installed. mapfile (also readarray) reads lines into an array.
Common mistakes
for line in $(cat file)splits on every space. Usewhile IFS= read -r.- Piping into
whileand expecting variables to persist: the loop ran in a subshell. Use< <(...)or a redirect. - Forgetting
-rand losing backslashes. - Fixed temp file names.
- Interactive prompts in scripts that run in CI or cron.
- Parsing structured data (JSON) with regular expressions.
Try it yourself
- Read
/etc/passwdand print the username and shell of every account whose shell ends inbash. - Write a script with options
-s SERVER(required) and-q(quiet) and a helpful usage message; test missing and unknown options. - Generate an nginx config snippet with a here-document using a variable for the port.
- Make a script that creates a temp directory, writes to it and always removes it, even when you press Ctrl+C halfway.
Check yourself
Why does for line in $(cat file) fail for lines with spaces?
$(cat file) is unquoted, so the shell splits the output at every space, tab and newline, giving words, not lines. while IFS= read -r line; do ...; done < file keeps whole lines.
What does IFS=, read -r a b do?
It splits one line from stdin at commas into a and b (the rest into b), setting IFS only for that command.
What is the difference between <<EOF and <<'EOF'?
With an unquoted delimiter, variables and command substitutions inside the here-document are expanded. With a quoted delimiter the text is passed exactly as written.
Why should a script use mktemp?
It creates a uniquely named, private file or directory, avoiding collisions between runs and predictable-name attacks.
Interview questions
How do you parse command-line options in Bash?
getopts for short options, with a case statement and shift $((OPTIND-1)) afterwards; a manual while/case over "$@" for long options.
How do you read a file line by line safely?
while IFS= read -r line; do ...; done < file.
How can a script check whether a required tool is installed?
command -v tool >/dev/null 2>&1 || { echo "tool is required" >&2; exit 1; }.
Summary
- Read lines with
while IFS= read -r; use< <(cmd)to avoid a subshell; split fields withIFS=,. - Here-documents write multi-line text; quote the delimiter to prevent expansion.
getoptsparses short options; print usage to stderr and exit 2 on misuse.mktempplustrap ... EXITgives safe, self-cleaning temporary files.
Going deeper
- Bash Reference Manual: Redirections (here documents): here-docs and here-strings.
- Bash Reference Manual: Bash Builtins (read, getopts, mapfile): the reference for these builtins.
- mktemp(1): options for temporary files.
See how much of this you can explain
A free mock interview asks you new questions on Shell scripting with Bash for DevOps Engineer and scores every answer, with what to fix.