Lesson 5 of 7 · 10 min · Text and scheduling

Text tools: grep, sed, awk and friends

Covers: Ubuntu 24.04 LTS, GNU grep 3.11, GNU sed 4.9, GNU awk 5.2. Updated 13 Nov 2026. Part of the DevOps Engineer roadmap.

What you will learn

  • Regular expressions you actually need
  • grep options for logs and code
  • sed for substitution, deleting and in-place edits
  • awk for columns, filters, sums and counts
  • Combining them, and knowing when to use something else

In simple words

Logs and config files are just text, often thousands of lines of it. Three small tools let you search, edit and add up that text without opening it: grep finds lines, sed changes text, and awk works on columns, like a tiny spreadsheet. This lesson teaches the part of each that you will use every week.

Words to know

  • Regular expression: a pattern that describes text, such as "lines that start with a number".
  • grep: prints the lines that match a pattern.
  • sed: edits text as it flows through, most often find-and-replace.
  • awk: splits each line into columns, so you can pick, filter or total them.

A sample access log

The examples use a small web server log in the standard "combined" style:

10.0.0.1 - - [12/Nov/2026:10:00:01 +0000] "GET /a HTTP/1.1" 200 512
10.0.0.2 - - [12/Nov/2026:10:00:02 +0000] "GET /b HTTP/1.1" 500 0
10.0.0.1 - - [12/Nov/2026:10:00:03 +0000] "POST /c HTTP/1.1" 200 2048
10.0.0.3 - - [12/Nov/2026:10:00:04 +0000] "GET /a HTTP/1.1" 404 128

Fields separated by spaces are: $1 client IP, $4 $5 time, $6 method (with a leading quote), $7 path, $8 protocol, $9 status, $10 bytes.

Regular expressions in five minutes

A regular expression (regex) describes a text pattern.

Pattern Matches
abc the text abc
. any single character
^ and $ start and end of the line
[0-9], [a-z], [^0-9] one character from a set, or not from it
* zero or more of the previous item
+, ? one or more, zero or one (extended regex: grep -E, sed -E, awk)
{3} exactly 3 of the previous item
(a|b) alternatives, with -E written `(a
\. a literal dot (a plain . means "any character")

Remember that the shell also treats * and ?, so put patterns in single quotes.

grep: find lines

grep -E ' 5[0-9]{2} ' access.log        # server errors
grep -c ' 200 ' access.log               # count
grep -v ' 200 ' access.log               # everything that is not 200
grep -oE '"[A-Z]+ [^ ]+' access.log      # -o print only the matched part
grep -rn "TODO" src/                      # recursive with file names and line numbers
grep -A2 -B2 "Traceback" app.log          # 2 lines of context around each match
grep -F 'a.b' file                        # fixed string, no regex (also faster)
grep -w error file                        # whole word only

The -o example prints:

"GET /a
"GET /b

sed: stream editor

sed reads text, applies an edit to each line, and prints the result. The workhorse is substitution s/old/new/flags:

echo "port=80" | sed 's/80/8080/'              # port=8080
echo "a-b-c"   | sed 's/-/_/g'                 # a_b_c     (g = every match in the line)
echo "version: 1.2.3" | sed -E 's/([0-9]+)\.([0-9]+)\.([0-9]+)/\3.\2.\1/'
port=8080
a_b_c
version: 3.2.1

The last one uses groups ( ) and refers to them as \1, \2, \3.

Other uses:

sed -n '2,3p' access.log                  # print only lines 2 to 3
printf '# c\nkey=1\n\nkey=2\n' | sed '/^#/d;/^$/d'     # delete comments and blank lines
key=1
key=2

Editing a file in place: -i changes the file itself. Keep a backup suffix until you trust the command:

printf 'mode=dev\n' > app.conf
sed -i.bak 's/mode=dev/mode=prod/' app.conf
cat app.conf; ls app.conf*
mode=prod
app.conf
app.conf.bak

When the pattern contains / (paths), use another delimiter: sed 's|/var/www|/srv/www|g'. Always test without -i first, since there is no undo. Note that macOS sed -i needs an argument (-i ''); in scripts that run on both, avoid -i or use perl -pi -e.

awk: columns and calculations

awk reads a line, splits it into fields ($1, $2, ... by whitespace, $NF is the last), and runs pattern { action } for each line.

awk '{print $1, $9}' access.log
10.0.0.1 200
10.0.0.2 500
10.0.0.1 200
10.0.0.3 404

A pattern before the braces filters lines. Only the errors:

awk '$9 >= 400 {print $1, $7, $9}' access.log
10.0.0.2 /b 500
10.0.0.3 /a 404

-F sets the separator, so awk also reads files like /etc/passwd:

echo 'asha:x:1001:1001::/home/asha:/bin/bash' | awk -F: '{print $1 " -> " $NF}'
asha -> /bin/bash

BEGIN { } runs before the first line and END { } after the last, which is how you calculate totals and counts with variables and arrays:

awk '{bytes+=$10} END {print "total bytes:", bytes}' access.log
awk '{c[$9]++} END {for (k in c) print k, c[k]}' access.log | sort
total bytes: 2688
200 2
404 1
500 1

The second command is a "group by": it counts lines per status code. awk can replace many sort | uniq -c pipelines, and it can also total and average values, which uniq cannot. For formatted output use printf: awk '{printf "%-10s %5d\n", $1, $10}' access.log prints the IP padded to 10 characters and the bytes right-aligned.

The small tools

cut -d, -f2 file.csv          # second comma-separated column
tr 'a-z' 'A-Z'                # translate characters; tr -d '\r' removes Windows line ends
sort -k2 -n file              # sort by 2nd column numerically
uniq -c                       # count adjacent duplicates (sort first)
xargs -I{} echo "item {}"     # run a command per input line
paste, join, column -t        # glue files together, format as a table

A classic pipeline, the five most common client IPs:

awk '{print $1}' access.log | sort | uniq -c | sort -rn | head -5
      2 10.0.0.1
      1 10.0.0.3
      1 10.0.0.2

xargs turns lines into arguments: find . -name '*.tmp' -print0 | xargs -0 rm (the -print0/-0 pair handles odd file names).

Choose the right tool

  • grep to select lines. cut for a fixed-delimiter column. sed to rewrite text. awk when you need columns, conditions or arithmetic.
  • JSON has structure, so use jq: curl -s URL | jq '.items[].name'. YAML: yq. CSV with quotes: Python's csv module. Using regex on structured formats breaks on the first unusual input.
  • When a script grows beyond 100 lines of text juggling, switch to Python.

Common mistakes

  • Unquoted patterns, which the shell expands.
  • sed -i without a test run or backup.
  • Using . when you meant a literal dot (1.2 also matches 142).
  • Wrong field numbers because a log format has an extra space (always print a sample line with awk '{print NF, $0}' to check).
  • grep for structured data such as JSON.
  • Forgetting that uniq needs sorted input.

Try it yourself

  1. From the sample log, print the paths that returned status 404 or 500, once each.
  2. Use awk to compute the average bytes of the 200 responses.
  3. Change every http:// in a config to https:// with sed, using a different delimiter, on a copy of the file.
  4. Remove comment lines and blank lines from /etc/ssh/sshd_config and show only what is active.

Check yourself

What does sed 's/a/b/' do versus sed 's/a/b/g'?

Without g only the first a on each line is replaced; with g every a is.

How do you count requests per HTTP status code in an access log?

awk '{c[$9]++} END {for (k in c) print k, c[k]}' access.log, or awk '{print $9}' access.log | sort | uniq -c.

Why quote a regular expression in single quotes on the command line?

Characters such as *, ?, $ and [ are special to the shell too. Quotes pass the pattern to the program unchanged.

When should you not use grep and sed?

For structured formats such as JSON, YAML, XML or complex CSV; use jq, yq, an XML tool or a script language that parses them properly.

Interview questions

How do you find the top 10 IP addresses in a web server log?

awk '{print $1}' access.log | sort | uniq -c | sort -rn | head -10.

What is the difference between grep, sed and awk?

grep selects lines that match; sed edits text with commands such as substitute and delete; awk processes records and fields with conditions and variables, so it can compute and report.

How do you replace a string in many files safely?

List files with grep -rl 'old' dir, test the sed command without -i, then run sed -i.bak 's/old/new/g' (or use version control so that git diff shows the change).

Summary

  • Learn the small regex set; quote patterns; use -E for extended syntax.
  • grep selects lines, sed 's/old/new/g' rewrites them (test before -i), awk handles columns, conditions, sums and counts.
  • Combine with sort | uniq -c | sort -rn for frequency questions.
  • Use jq for JSON; do not parse structured data with regex.

Going deeper

See how much of this you can explain

A free mock interview asks you new questions on Shell scripting with Bash for DevOps Engineer and scores every answer, with what to fix.