Programming with awk
You take awk beyond one-line filters and write real reporting programs with variables, arrays, functions and multi-file processing.
Level 301 · 18 lessons · 15 practice labs
Lessons
The language
- Structure of an awk program: pattern-action, BEGIN and END
- Built-in variables: NR, NF, FS, OFS, RS
- Field separators: CSV, tabs and regex
- Aligned output with printf
- Short task: count accounts by shell from /etc/passwd
Data structures
- Counting and summing with associative arrays
- Sorting results: asort and working with sort
- Multi-dimensional keys and SUBSEP
- Short task: the ten most requested URLs and their average response times
Flow and functions
- if, while, for and next
- String functions: sub, gsub, split, substr, match
- Writing your own functions
- getline, system and external commands: use with care
- Short task: turn an INI file into a section-key table
Several files
- Joining two files: the NR==FNR idiom
- Putting an awk program in a file and passing options
- gawk extras: time functions and CSV mode
- Capstone: an awk program that builds a monthly service report from three sources
Practice labs
- Count accounts by shell: Use awk to count how many accounts use each shell in /etc/passwd, sorted from most to fewest.
- Produce an aligned table from raw data: Turn a tab-separated server inventory into a fixed-width table with a header and right-aligned numbers using printf.
- Process multi-line records with RS: Turn a file of multi-line records separated by blank lines into one line per record using RS and FS.
- Find rows with the wrong field count: Use NF to find the line numbers of rows in an exported data file whose field count is not as expected.
- Select the last hour's records with time functions: Convert timestamps with gawk time functions and print only records from the hour before a given moment.
- Remove an unsafe system call: Show the injection risk in an awk script that passes an input field to the shell with system and rewrite it without external commands.
- Find the most requested URLs and average time: Compute the ten most requested URLs and each one's average response time from an access log with associative arrays.
- Total the bytes transferred per hour: Total the bytes transferred per hour in the access log and submit the busiest hour.
- Build a cross table by user and status: Use multi-dimensional keys to produce a table showing each user's successful and failed login counts side by side.
- Turn an INI file into a section-key table: Write an awk program that turns each setting in an INI file into one line of section, key and value.
- Normalise inconsistent fields with gsub: Bring a list whose phone and date fields are written in different formats into one format with sub, gsub and split.
- Humanise byte values with your own function: Write an awk function that converts byte counts to a suitable unit and apply it to the size column of a disk report.
- Join two files with NR==FNR: Replace the IDs in a session log with names using a user ID-to-name file.
- Turn an awk one-liner into a program with options: Move a long one-liner into a separate awk file and make it take the threshold from outside with -v.
- Build a monthly service report from three sources: Combine outage records, service owners and target values to report downtime and target status per service.
About this course
You take awk beyond one-line filters and write real reporting programs with variables, arrays, functions and multi-file processing.
What you will learn
- The language
- Data structures
- Flow and functions
- Several files
Prerequisites
You should be comfortable with the level 101 and 201 topics of the same path.
Lab environment
Lessons run in your own Kali Linux terminal that opens in the browser. Nothing to install; the lab starts clean in every lesson.
Frequently asked questions
- Is this course right for my level?
- The course is at level 301; it assumes you know the topics of the lower-level courses.
- Is there anything to install?
- No. Every lesson runs in your own lab environment that opens in the browser.
- How do lessons work?
- Each lesson has 8 to 10 steps and a short quiz. Early steps show the command to type, later steps give less help; if two attempts fail, the solution opens.
- Can I try it for free?
- Yes. The first three lessons of every course are free.