Lesson 3 of 12 · 9 min · Working with files

Streams, pipes, redirection and searching

Covers: Ubuntu 24.04 LTS, bash 5.2, GNU grep 3.11, GNU findutils 4.9. Updated 13 Nov 2026. Part of the DevOps Engineer roadmap.

What you will learn

  • The three standard streams: input, output and error
  • Redirection with >, >>, <, 2> and 2>&1
  • Pipes, and why small commands chained together are powerful
  • find for files and grep for text, with the options you need daily

In simple words

Linux commands are like small machines on a factory line. Each one does a single job, and you can join them so the result of one becomes the material for the next. You can also send a result into a file instead of the screen. This lesson shows how to join commands and how to search for files and for text.

Words to know

  • Output: what a command prints. Normal results and error messages travel on two separate channels.
  • Redirection: sending a command's output into a file (>) instead of showing it on screen.
  • Pipe: the | sign. It hands the output of one command to the next command.
  • grep: a command that finds the lines containing the text you ask for.

Three streams

Every Linux program starts with three open channels:

Name Number Default Purpose
stdin 0 keyboard what the program reads
stdout 1 the terminal normal output
stderr 2 the terminal error and warning messages

Errors have their own stream so you can save results to a file and still see problems on screen, or the reverse.

Redirection

echo "one"  > a.txt      # write stdout to a file (creates or REPLACES it)
echo "two" >> a.txt      # append instead
cat a.txt
one
two
ls /nope 2> err.txt      # send only stderr (stream 2) to a file
echo "exit code: $?"     # $? holds the previous command's exit code; non-zero means failure
cat err.txt
exit code: 2
ls: cannot access '/nope': No such file or directory

Common forms:

  • cmd > out.txt 2> err.txt splits the two streams into two files.
  • cmd > all.txt 2>&1 sends stderr to wherever stdout currently goes, so both end up in one file. (Order matters: redirect stdout first, then 2>&1.) Bash also accepts cmd &> all.txt.
  • cmd > /dev/null throws normal output away; cmd > /dev/null 2>&1 throws everything away. Useful in cron jobs and scripts.
  • cmd < input.txt feeds a file to stdin.

Be careful: > truncates the file before the command runs. sort a.txt > a.txt leaves you with an empty file.

Pipes

A pipe | connects one program's stdout to the next program's stdin, so you build answers from small tools:

printf 'b\na\nc\na\na\nc\n' | sort | uniq -c | sort -rn
      3 a
      2 c
      1 b

sort orders lines, uniq -c collapses adjacent duplicates and counts them (that is why you sort first), and sort -rn orders the counts from biggest down. Classic log questions follow this pattern, for example "which IPs hit us most":

awk '{print $1}' access.log | sort | uniq -c | sort -rn | head

Useful pipe companions:

Command What it does
head -n N, tail -n N first or last N lines
wc -l count lines
sort, sort -n, sort -u sort text, numbers, unique
cut -d, -f2 take the 2nd comma-separated field
tr 'a-z' 'A-Z' translate characters
tee file write to a file and pass on to the next command
xargs turn lines on stdin into arguments of another command

tee is how you save a long command's output while still watching it: make 2>&1 | tee build.log. With sudo, remember that sudo echo x > /etc/file fails, because the shell opens the file as you before sudo runs. Use echo x | sudo tee /etc/file instead.

Searching for files with find

find walks a directory tree and tests every entry:

find /tmp/demo -name '*.py'                # by name (quote the pattern so the shell does not expand it)
find /var/log -type f -name '*.log'        # -type f files, -type d directories
find . -type f -size +10M                  # bigger than 10 MB
find . -type f -mtime -1                   # modified in the last 24 hours
find . -type f -name '*.tmp' -delete       # act on the matches
find . -name '*.sh' -exec chmod +x {} +    # run a command on the matches

Test it with -print or no action first, and only then add -delete.

Searching inside files with grep

grep prints the lines that match a pattern:

grep -n "two" a.txt          # -n shows line numbers
2:two

The options you will use most:

Option Meaning
-i ignore case
-r search a directory recursively
-v invert: lines that do not match
-c count matching lines
-l list only the names of matching files
-E extended regular expressions: `grep -E 'error
-A 2 -B 2 show 2 lines after and before each match
grep -ri "connection refused" /var/log/myapp/      # search all logs for a phrase
grep -v '^#' /etc/ssh/sshd_config | grep -v '^$'   # a config file without comments and blank lines
dmesg | grep -i error

A regular expression describes a text pattern: ^ start of line, $ end of line, . any one character, * repeat the previous item, [0-9] a digit. Learn these few and you can read most patterns. In scripting (course 2) you will meet sed and awk too.

Common mistakes

  • Reading a file into a pipe unnecessarily. cat file | grep x works but grep x file is simpler and faster.
  • Forgetting that uniq only joins neighbours. Always sort first.
  • > when you meant >>. You just erased the log.
  • Unquoted patterns. find . -name *.py lets the shell expand *.py in the current directory first and gives odd results. Quote it.
  • Ignoring stderr in a pipe. cmd | grep x pipes only stdout; errors still show on screen. Use cmd 2>&1 | grep x.

Try it yourself

  1. Write the output of ls /etc to a file, then count the lines in it with wc -l.
  2. Find the five largest files under /usr with find /usr -type f -size +1M and sort/head, or with du -a /usr 2>/dev/null | sort -rn | head -5.
  3. In /etc/passwd, count how many accounts use /usr/sbin/nologin as their shell, using grep -c.
  4. Run a command that fails and capture only its error text in a file. Then run it again so both streams land in one file.

Check yourself

What is the difference between > and >>?

> creates the file or replaces its content; >> appends to the end. Mixing them up is a common way to lose logs.

What does cmd > out.txt 2>&1 do, and why does cmd 2>&1 > out.txt behave differently?

The first sends stdout to the file and then points stderr at the same place, so both go to the file. In the second, stderr is first pointed at the current stdout (the terminal) and only then is stdout moved to the file, so errors still show on screen.

Why do you run sort before uniq -c?

uniq only collapses adjacent duplicate lines, so equal lines must be next to each other first.

How do you list every file under /etc that mentions ssh, ignoring case, without printing the lines themselves?

grep -rli ssh /etc (-r recursive, -l file names only, -i ignore case). Add 2>/dev/null to hide permission errors.

Interview questions

How would you find the ten IP addresses that made the most requests in an access log?

awk '{print $1}' access.log | sort | uniq -c | sort -rn | head -10. This assumes the IP is the first field, which is true in the common web server log formats.

What is the difference between find and grep?

find selects files by properties such as name, type, size and time. grep selects lines by their content. They are often combined: find . -name '*.log' -exec grep -l ERROR {} +.

What do stdin, stdout and stderr mean, and what are their file descriptor numbers?

The three standard streams of a process: input (0), normal output (1) and error output (2). Redirection and pipes rewire them.

Summary

  • Every program has stdin (0), stdout (1) and stderr (2); >, >>, <, 2> and 2>&1 redirect them.
  • A pipe feeds one command's output to the next; sort | uniq -c | sort -rn is the counting pattern.
  • find chooses files by properties and can act on them (test before -delete); grep chooses lines by pattern.
  • Quote patterns, and remember > empties the target first.

Going deeper

See how much of this you can explain

A free mock interview asks you new questions on Linux for DevOps for DevOps Engineer and scores every answer, with what to fix.