Parsing logs and text data
You split syslog, web server, JSON and Windows event logs into fields with regular expressions and Python, map them to a common schema and get them ready for analysis.
Level 201 · 19 lessons · 15 practice labs
Lessons
Log formats
- Why parse: from text to fields
- The syslog format: priority, time, host, message
- Web access logs: common and combined formats
- Structured JSON logs
- Windows event logs: event IDs and fields
- Quick task: identify log formats in mixed files
Regular expressions
- Regular expressions: characters, classes, repetition
- Groups and named captures
- Greedy matching, escaping and slow patterns
- Parsing with Python's re module
- Quick task: extract users and IPs from SSH logs
A common schema
- Common field names: source IP, user, action, outcome
- Unifying different time formats
- Multi-line and broken records
- Enrichment: joining asset and user tables
- Quick task: convert two sources to one schema
Scale and reliability
- Processing large files without loading them into memory
- Log injection: not trusting what is in a log
- Capstone: a unified event table from four sources
Practice labs
- Break down a syslog line: Split five syslog lines into priority, time, host, process and message fields and write them into a table.
- Extract fields from the combined format: Convert every field of a combined-format web log, including referrer and user agent, into CSV.
- Meaningful fields in Windows events: Extract logon type, account and source address from exported Windows events 4624, 4625 and 4720 and summarise them.
- Capture the user and IP: Pull the username and source IP out of an FTP server log with a grouped regular expression and write their counts.
- A parser with named groups: Write a Python function for a mail server log that uses named groups and returns sender, recipient and status as a dictionary.
- Speed up a slow pattern: Rewrite a greedy regular expression that hangs on long lines so it gives the same result and time both versions.
- Map field names to a common schema: Prepare a mapping table that maps fields such as src, client_ip and remote_addr from three sources to common schema names.
- Join multi-line records: Join application errors that span several lines, such as stack traces, into single records and count them by error type.
- Enrich with the asset table: Enrich parsed firewall events with the asset and user tables and list the sources with no known owner separately.
- Process line by line as a stream: Write a script that reads a very large log file with a generator, without loading it into memory, and counts the status codes.
- Compressed and rotated files: Write a script that reads rotated, partly gzipped log files in order and finds the time gap between them.
- Detect log injection: Find fake records created by injecting line breaks into the user-agent field and make the parser resistant to this attack.
- Fill in the missing year in syslog: Convert syslog timestamps that lack a year into full ISO 8601 using the file date.
- Four time formats into one: Write a function that turns Unix time, Apache format, ISO 8601 and local-time text into one UTC column and test it with unit tests.
- A log that runs backwards at New Year: Find where the order breaks in a yearless log spanning New Year, assign the year correctly and recompute the incident duration.
About this course
You split syslog, web server, JSON and Windows event logs into fields with regular expressions and Python, map them to a common schema and get them ready for analysis.
What you will learn
- Log formats
- Regular expressions
- A common schema
- Scale and reliability
Prerequisites
Finishing the level 101 courses of the same path, or knowing those topics, is enough.
Lab environment
Lessons run in your own Kali Linux terminal that opens in the browser. Nothing to install; the lab starts clean in every lesson.
Frequently asked questions
- Is this course right for my level?
- The course is at level 201; it assumes you know the topics of the lower-level courses.
- Is there anything to install?
- No. Every lesson runs in your own lab environment that opens in the browser.
- How do lessons work?
- Each lesson has 8 to 10 steps and a short quiz. Early steps show the command to type, later steps give less help; if two attempts fail, the solution opens.
- Can I try it for free?
- Yes. The first three lessons of every course are free.