Lesson 5 of 7 · 10 min · Text and scheduling
Text tools: grep, sed, awk and friends
What you will learn
- Regular expressions you actually need
grepoptions for logs and codesedfor substitution, deleting and in-place editsawkfor columns, filters, sums and counts- Combining them, and knowing when to use something else
In simple words
Logs and config files are just text, often thousands of lines of it. Three small tools let you search, edit and add up that text without opening it: grep finds lines, sed changes text, and awk works on columns, like a tiny spreadsheet. This lesson teaches the part of each that you will use every week.
Words to know
- Regular expression: a pattern that describes text, such as "lines that start with a number".
- grep: prints the lines that match a pattern.
- sed: edits text as it flows through, most often find-and-replace.
- awk: splits each line into columns, so you can pick, filter or total them.
A sample access log
The examples use a small web server log in the standard "combined" style:
10.0.0.1 - - [12/Nov/2026:10:00:01 +0000] "GET /a HTTP/1.1" 200 512
10.0.0.2 - - [12/Nov/2026:10:00:02 +0000] "GET /b HTTP/1.1" 500 0
10.0.0.1 - - [12/Nov/2026:10:00:03 +0000] "POST /c HTTP/1.1" 200 2048
10.0.0.3 - - [12/Nov/2026:10:00:04 +0000] "GET /a HTTP/1.1" 404 128
Fields separated by spaces are: $1 client IP, $4 $5 time, $6 method (with a leading quote), $7 path, $8 protocol, $9 status, $10 bytes.
Regular expressions in five minutes
A regular expression (regex) describes a text pattern.
| Pattern | Matches |
|---|---|
abc |
the text abc |
. |
any single character |
^ and $ |
start and end of the line |
[0-9], [a-z], [^0-9] |
one character from a set, or not from it |
* |
zero or more of the previous item |
+, ? |
one or more, zero or one (extended regex: grep -E, sed -E, awk) |
{3} |
exactly 3 of the previous item |
(a|b) |
alternatives, with -E written `(a |
\. |
a literal dot (a plain . means "any character") |
Remember that the shell also treats * and ?, so put patterns in single quotes.
grep: find lines
grep -E ' 5[0-9]{2} ' access.log # server errors
grep -c ' 200 ' access.log # count
grep -v ' 200 ' access.log # everything that is not 200
grep -oE '"[A-Z]+ [^ ]+' access.log # -o print only the matched part
grep -rn "TODO" src/ # recursive with file names and line numbers
grep -A2 -B2 "Traceback" app.log # 2 lines of context around each match
grep -F 'a.b' file # fixed string, no regex (also faster)
grep -w error file # whole word only
The -o example prints:
"GET /a
"GET /b
sed: stream editor
sed reads text, applies an edit to each line, and prints the result. The workhorse is substitution s/old/new/flags:
echo "port=80" | sed 's/80/8080/' # port=8080
echo "a-b-c" | sed 's/-/_/g' # a_b_c (g = every match in the line)
echo "version: 1.2.3" | sed -E 's/([0-9]+)\.([0-9]+)\.([0-9]+)/\3.\2.\1/'
port=8080
a_b_c
version: 3.2.1
The last one uses groups ( ) and refers to them as \1, \2, \3.
Other uses:
sed -n '2,3p' access.log # print only lines 2 to 3
printf '# c\nkey=1\n\nkey=2\n' | sed '/^#/d;/^$/d' # delete comments and blank lines
key=1
key=2
Editing a file in place: -i changes the file itself. Keep a backup suffix until you trust the command:
printf 'mode=dev\n' > app.conf
sed -i.bak 's/mode=dev/mode=prod/' app.conf
cat app.conf; ls app.conf*
mode=prod
app.conf
app.conf.bak
When the pattern contains / (paths), use another delimiter: sed 's|/var/www|/srv/www|g'. Always test without -i first, since there is no undo. Note that macOS sed -i needs an argument (-i ''); in scripts that run on both, avoid -i or use perl -pi -e.
awk: columns and calculations
awk reads a line, splits it into fields ($1, $2, ... by whitespace, $NF is the last), and runs pattern { action } for each line.
awk '{print $1, $9}' access.log
10.0.0.1 200
10.0.0.2 500
10.0.0.1 200
10.0.0.3 404
A pattern before the braces filters lines. Only the errors:
awk '$9 >= 400 {print $1, $7, $9}' access.log
10.0.0.2 /b 500
10.0.0.3 /a 404
-F sets the separator, so awk also reads files like /etc/passwd:
echo 'asha:x:1001:1001::/home/asha:/bin/bash' | awk -F: '{print $1 " -> " $NF}'
asha -> /bin/bash
BEGIN { } runs before the first line and END { } after the last, which is how you calculate totals and counts with variables and arrays:
awk '{bytes+=$10} END {print "total bytes:", bytes}' access.log
awk '{c[$9]++} END {for (k in c) print k, c[k]}' access.log | sort
total bytes: 2688
200 2
404 1
500 1
The second command is a "group by": it counts lines per status code. awk can replace many sort | uniq -c pipelines, and it can also total and average values, which uniq cannot. For formatted output use printf: awk '{printf "%-10s %5d\n", $1, $10}' access.log prints the IP padded to 10 characters and the bytes right-aligned.
The small tools
cut -d, -f2 file.csv # second comma-separated column
tr 'a-z' 'A-Z' # translate characters; tr -d '\r' removes Windows line ends
sort -k2 -n file # sort by 2nd column numerically
uniq -c # count adjacent duplicates (sort first)
xargs -I{} echo "item {}" # run a command per input line
paste, join, column -t # glue files together, format as a table
A classic pipeline, the five most common client IPs:
awk '{print $1}' access.log | sort | uniq -c | sort -rn | head -5
2 10.0.0.1
1 10.0.0.3
1 10.0.0.2
xargs turns lines into arguments: find . -name '*.tmp' -print0 | xargs -0 rm (the -print0/-0 pair handles odd file names).
Choose the right tool
- grep to select lines. cut for a fixed-delimiter column. sed to rewrite text. awk when you need columns, conditions or arithmetic.
- JSON has structure, so use
jq:curl -s URL | jq '.items[].name'. YAML:yq. CSV with quotes: Python'scsvmodule. Using regex on structured formats breaks on the first unusual input. - When a script grows beyond 100 lines of text juggling, switch to Python.
Common mistakes
- Unquoted patterns, which the shell expands.
sed -iwithout a test run or backup.- Using
.when you meant a literal dot (1.2also matches142). - Wrong field numbers because a log format has an extra space (always print a sample line with
awk '{print NF, $0}'to check). grepfor structured data such as JSON.- Forgetting that
uniqneeds sorted input.
Try it yourself
- From the sample log, print the paths that returned status 404 or 500, once each.
- Use
awkto compute the average bytes of the 200 responses. - Change every
http://in a config tohttps://withsed, using a different delimiter, on a copy of the file. - Remove comment lines and blank lines from
/etc/ssh/sshd_configand show only what is active.
Check yourself
What does sed 's/a/b/' do versus sed 's/a/b/g'?
Without g only the first a on each line is replaced; with g every a is.
How do you count requests per HTTP status code in an access log?
awk '{c[$9]++} END {for (k in c) print k, c[k]}' access.log, or awk '{print $9}' access.log | sort | uniq -c.
Why quote a regular expression in single quotes on the command line?
Characters such as *, ?, $ and [ are special to the shell too. Quotes pass the pattern to the program unchanged.
When should you not use grep and sed?
For structured formats such as JSON, YAML, XML or complex CSV; use jq, yq, an XML tool or a script language that parses them properly.
Interview questions
How do you find the top 10 IP addresses in a web server log?
awk '{print $1}' access.log | sort | uniq -c | sort -rn | head -10.
What is the difference between grep, sed and awk?
grep selects lines that match; sed edits text with commands such as substitute and delete; awk processes records and fields with conditions and variables, so it can compute and report.
How do you replace a string in many files safely?
List files with grep -rl 'old' dir, test the sed command without -i, then run sed -i.bak 's/old/new/g' (or use version control so that git diff shows the change).
Summary
- Learn the small regex set; quote patterns; use
-Efor extended syntax. grepselects lines,sed 's/old/new/g'rewrites them (test before-i),awkhandles columns, conditions, sums and counts.- Combine with
sort | uniq -c | sort -rnfor frequency questions. - Use
jqfor JSON; do not parse structured data with regex.
Going deeper
- GNU grep manual: all options and regex syntax.
- GNU sed manual: every command and option.
- The GNU Awk User's Guide: the language in full.
- jq manual: processing JSON on the command line.
See how much of this you can explain
A free mock interview asks you new questions on Shell scripting with Bash for DevOps Engineer and scores every answer, with what to fix.