Lesson 3 of 12 · 9 min · Working with files
Streams, pipes, redirection and searching
What you will learn
- The three standard streams: input, output and error
- Redirection with
>,>>,<,2>and2>&1 - Pipes, and why small commands chained together are powerful
findfor files andgrepfor text, with the options you need daily
In simple words
Linux commands are like small machines on a factory line. Each one does a single job, and you can join them so the result of one becomes the material for the next. You can also send a result into a file instead of the screen. This lesson shows how to join commands and how to search for files and for text.
Words to know
- Output: what a command prints. Normal results and error messages travel on two separate channels.
- Redirection: sending a command's output into a file (
>) instead of showing it on screen. - Pipe: the
|sign. It hands the output of one command to the next command. - grep: a command that finds the lines containing the text you ask for.
Three streams
Every Linux program starts with three open channels:
| Name | Number | Default | Purpose |
|---|---|---|---|
| stdin | 0 | keyboard | what the program reads |
| stdout | 1 | the terminal | normal output |
| stderr | 2 | the terminal | error and warning messages |
Errors have their own stream so you can save results to a file and still see problems on screen, or the reverse.
Redirection
echo "one" > a.txt # write stdout to a file (creates or REPLACES it)
echo "two" >> a.txt # append instead
cat a.txt
one
two
ls /nope 2> err.txt # send only stderr (stream 2) to a file
echo "exit code: $?" # $? holds the previous command's exit code; non-zero means failure
cat err.txt
exit code: 2
ls: cannot access '/nope': No such file or directory
Common forms:
cmd > out.txt 2> err.txtsplits the two streams into two files.cmd > all.txt 2>&1sends stderr to wherever stdout currently goes, so both end up in one file. (Order matters: redirect stdout first, then2>&1.) Bash also acceptscmd &> all.txt.cmd > /dev/nullthrows normal output away;cmd > /dev/null 2>&1throws everything away. Useful in cron jobs and scripts.cmd < input.txtfeeds a file to stdin.
Be careful: > truncates the file before the command runs. sort a.txt > a.txt leaves you with an empty file.
Pipes
A pipe | connects one program's stdout to the next program's stdin, so you build answers from small tools:
printf 'b\na\nc\na\na\nc\n' | sort | uniq -c | sort -rn
3 a
2 c
1 b
sort orders lines, uniq -c collapses adjacent duplicates and counts them (that is why you sort first), and sort -rn orders the counts from biggest down. Classic log questions follow this pattern, for example "which IPs hit us most":
awk '{print $1}' access.log | sort | uniq -c | sort -rn | head
Useful pipe companions:
| Command | What it does |
|---|---|
head -n N, tail -n N |
first or last N lines |
wc -l |
count lines |
sort, sort -n, sort -u |
sort text, numbers, unique |
cut -d, -f2 |
take the 2nd comma-separated field |
tr 'a-z' 'A-Z' |
translate characters |
tee file |
write to a file and pass on to the next command |
xargs |
turn lines on stdin into arguments of another command |
tee is how you save a long command's output while still watching it: make 2>&1 | tee build.log. With sudo, remember that sudo echo x > /etc/file fails, because the shell opens the file as you before sudo runs. Use echo x | sudo tee /etc/file instead.
Searching for files with find
find walks a directory tree and tests every entry:
find /tmp/demo -name '*.py' # by name (quote the pattern so the shell does not expand it)
find /var/log -type f -name '*.log' # -type f files, -type d directories
find . -type f -size +10M # bigger than 10 MB
find . -type f -mtime -1 # modified in the last 24 hours
find . -type f -name '*.tmp' -delete # act on the matches
find . -name '*.sh' -exec chmod +x {} + # run a command on the matches
Test it with -print or no action first, and only then add -delete.
Searching inside files with grep
grep prints the lines that match a pattern:
grep -n "two" a.txt # -n shows line numbers
2:two
The options you will use most:
| Option | Meaning |
|---|---|
-i |
ignore case |
-r |
search a directory recursively |
-v |
invert: lines that do not match |
-c |
count matching lines |
-l |
list only the names of matching files |
-E |
extended regular expressions: `grep -E 'error |
-A 2 -B 2 |
show 2 lines after and before each match |
grep -ri "connection refused" /var/log/myapp/ # search all logs for a phrase
grep -v '^#' /etc/ssh/sshd_config | grep -v '^$' # a config file without comments and blank lines
dmesg | grep -i error
A regular expression describes a text pattern: ^ start of line, $ end of line, . any one character, * repeat the previous item, [0-9] a digit. Learn these few and you can read most patterns. In scripting (course 2) you will meet sed and awk too.
Common mistakes
- Reading a file into a pipe unnecessarily.
cat file | grep xworks butgrep x fileis simpler and faster. - Forgetting that
uniqonly joins neighbours. Alwayssortfirst. >when you meant>>. You just erased the log.- Unquoted patterns.
find . -name *.pylets the shell expand*.pyin the current directory first and gives odd results. Quote it. - Ignoring stderr in a pipe.
cmd | grep xpipes only stdout; errors still show on screen. Usecmd 2>&1 | grep x.
Try it yourself
- Write the output of
ls /etcto a file, then count the lines in it withwc -l. - Find the five largest files under
/usrwithfind /usr -type f -size +1Mandsort/head, or withdu -a /usr 2>/dev/null | sort -rn | head -5. - In
/etc/passwd, count how many accounts use/usr/sbin/nologinas their shell, usinggrep -c. - Run a command that fails and capture only its error text in a file. Then run it again so both streams land in one file.
Check yourself
What is the difference between > and >>?
> creates the file or replaces its content; >> appends to the end. Mixing them up is a common way to lose logs.
What does cmd > out.txt 2>&1 do, and why does cmd 2>&1 > out.txt behave differently?
The first sends stdout to the file and then points stderr at the same place, so both go to the file. In the second, stderr is first pointed at the current stdout (the terminal) and only then is stdout moved to the file, so errors still show on screen.
Why do you run sort before uniq -c?
uniq only collapses adjacent duplicate lines, so equal lines must be next to each other first.
How do you list every file under /etc that mentions ssh, ignoring case, without printing the lines themselves?
grep -rli ssh /etc (-r recursive, -l file names only, -i ignore case). Add 2>/dev/null to hide permission errors.
Interview questions
How would you find the ten IP addresses that made the most requests in an access log?
awk '{print $1}' access.log | sort | uniq -c | sort -rn | head -10. This assumes the IP is the first field, which is true in the common web server log formats.
What is the difference between find and grep?
find selects files by properties such as name, type, size and time. grep selects lines by their content. They are often combined: find . -name '*.log' -exec grep -l ERROR {} +.
What do stdin, stdout and stderr mean, and what are their file descriptor numbers?
The three standard streams of a process: input (0), normal output (1) and error output (2). Redirection and pipes rewire them.
Summary
- Every program has stdin (0), stdout (1) and stderr (2);
>,>>,<,2>and2>&1redirect them. - A pipe feeds one command's output to the next;
sort | uniq -c | sort -rnis the counting pattern. findchooses files by properties and can act on them (test before-delete);grepchooses lines by pattern.- Quote patterns, and remember
>empties the target first.
Going deeper
- Bash Reference Manual: Redirections: every redirection form, exactly.
- GNU grep manual: all options and the regular expression syntax.
- find(1) manual page: the full list of
findtests and actions.
See how much of this you can explain
A free mock interview asks you new questions on Linux for DevOps for DevOps Engineer and scores every answer, with what to fix.