Text processing: grep, sed, awk
Searching files, transforming text and building reports.
Level 201 · 18 lessons · 18 practice labs
Lessons
grep and regular expressions
- Regular expressions: writing patterns with grep -E: You can describe formats such as emails, IPs or dates with a regular expression and extract only the matches from a log with grep -E.
- grep with context lines, file lists and binary files
- Comparing files: diff, comm and cmp: With diff, comm and cmp you can find what differs between two configurations or lists and show what a change affected.
- Reshaping columns and characters: tr, paste, join and nl: With tr, paste, join and nl you can merge records from two files on a shared field and produce clean numbered output.
- Building command chains: xargs and tee: With xargs and tee you can feed one command’s output to others as arguments and write it to both the screen and a file.
- Short task: find debug settings left switched on in config files
sed
- Substituting text with sed: You will use sed to replace a word or pattern in text and to preview the result before writing any change to a file.
- sed: deleting, inserting, in-place edits: You will use sed to delete certain lines, insert new ones and edit a file in place, doing it safely by keeping a backup.
- sed: addresses, ranges and multi-command scripts
- Short task: safely change the same line in a hundred config files
awk
- Extracting fields with awk: You will use awk to split each line into columns and pull out the field you want, such as listing only addresses or user names from a log.
- awk: conditions and patterns: You will use conditions and patterns in awk to select only lines meeting a rule, such as records that carry an error code.
- awk: sums, BEGIN/END and reports: You will use awk to compute totals and counters, use the BEGIN and END sections, and turn a raw log into a readable summary report.
- Short task: produce a disk-usage warning from df output
With real data
- Cleaning log data: You will learn to remove blank, broken and duplicate lines from messy log data and get it into a consistent state ready for analysis.
- Extracting HTTP status codes and error rates: You will count HTTP status codes in a web server log, calculate the error rate, and interpret what a sudden rise might mean.
- Masking sensitive data: You will learn to mask sensitive details such as IP addresses, emails and passwords before sharing logs, and why this matters for data protection.
- Capstone: build an incident summary from logs: From a fictional company's logs you will build a short incident summary: when it began, how many errors occurred, and which addresses stood out.
Practice labs
- Find debug settings left switched on: Use grep -rl to find config files with debug left on and write the list to the answer file.
- Pull the lines leading up to a crash: Extract every panic line in the application log together with the five lines before it into a separate file.
- Build an incident summary from logs: From three log files find the incident's start time, affected URLs and most frequent error and write a one-page summary.
- See the difference between two lists with diff: Sort yesterday's and today's package lists, compare them with diff and submit the number of lines added today.
- Find accounts not common to two lists: Compare the HR list with the server's accounts using comm and report accounts that are not on the list.
- Build an inventory table from three lists: Combine name, IP and role lists with paste and join, flag missing matches and produce a tab-separated table.
- Delete comment lines with sed: In a copy of the config file delete the lines starting with # and the blank lines with sed and write how many lines remain.
- Insert a missing header into every script: Find the scripts without a shebang and insert the right one as the first line of only those with sed.
- Append a settings block after a matching line: In only those of twenty virtual host files that lack the logging setting, append the two-line logging block right under the ServerName line with sed's a command; keep backups and show that a second run adds no duplicate lines.
- Replace an old host name with sed: Replace the old database host name in the config file with the new one using sed, leaving a backup.
- Safely change one line in a hundred configs: Change only the active timeout line in a hundred files with sed -i, leaving commented lines alone and keeping backups.
- Change settings inside one block only: In a file with several virtual host blocks, change the port only inside the named block using a sed range address.
- Produce a disk warning from df output: Use awk to find the partitions in df output above the fullness threshold and print them as warning lines.
- Filter slow requests with awk: Use awk to list the URL and duration of access-log requests whose response time exceeds the threshold.
- Report disk usage per user: Use awk to total size per user from a file listing and produce an aligned report with a header and grand total.
- Strip the broken lines from a log: Remove empty, duplicate and control-character lines from the log and save a clean copy.
- Work out the HTTP status code ratios: Compute the count of each status code in the access log and the 5xx error rate as a percentage.
- Mask sensitive data before sharing: Mask the email addresses, card numbers and tokens in a log headed to the support team using sed.
About this course
Searching files, transforming text and building reports.
What you will learn
- grep and regular expressions
- sed
- awk
- With real data
Prerequisites
Finishing the level 101 courses of the same path, or knowing those topics, is enough.
Lab environment
Lessons run in your own Kali Linux terminal that opens in the browser. Nothing to install; the lab starts clean in every lesson.
Frequently asked questions
- Is this course right for my level?
- The course is at level 201; it assumes you know the topics of the lower-level courses.
- Is there anything to install?
- No. Every lesson runs in your own lab environment that opens in the browser.
- How do lessons work?
- Each lesson has 8 to 10 steps and a short quiz. Early steps show the command to type, later steps give less help; if two attempts fail, the solution opens.
- Can I try it for free?
- Yes. The first three lessons of every course are free.