Working with huge logs: speed, scale and accuracy
You process tens of gigabytes of logs without running out of memory and with correct results, making locale, sorting, parallelism and sampling decisions by measurement.
Level 501 · 15 lessons · 15 practice labs
Lessons
Speed
- Measuring a pipeline: time, pv and the bottleneck
- LC_ALL=C: how the locale affects speed and results
- Sorting big files: memory, temp dirs and parallelism
- Working directly on compressed logs: zgrep and zstdcat
- Short task: cut a twenty-minute report to two minutes
Scale
- Splitting and processing in parallel: split and parallel --pipe
- Streaming versus batch: continuous counting with tail -F
- Slicing time windows fast with binary search
- Merging logs from many hosts by time
- Short task: build one timeline from five servers' logs
Accuracy
- Time zones and clock skew
- Duplicate records and sampling: what you lose
- Computing percentiles and distributions on the command line
- Reproducible analysis: recording commands and versions
- Capstone: produce an outage report from a 30 GB access log, fast and correct
Practice labs
- Find a slow pipeline's bottleneck: Measure each stage of a five-stage report pipeline with pv and time and submit the slowest one.
- Tune a sort that fills the disk: Get a huge sort that fills the temp directory to finish using a memory limit, a separate temp directory and parallelism.
- Cut a twenty-minute report to two minutes: Rewrite a report script that re-reads compressed logs many times into a single pass and prove the result is unchanged.
- Speed up a sort by changing the locale: Run the same count with the default locale and with LC_ALL=C and report the difference in time and in results.
- Compute response time percentiles: Compute the median, 95th and 99th percentile response times from an access log on the command line.
- Find the top requesters within fixed memory: Find the twenty busiest addresses in an access log too large for memory while keeping memory use under a set limit, and verify the result.
- Count lines in compressed logs without unpacking: Count the 500 responses in a week of rotated gzip logs without unpacking to disk and list the count per day.
- Search compressed logs without unpacking: Find a request ID in a month of gzip and zstd logs without unpacking to disk and list the files it appears in.
- Run a per-minute error counter on a live log: Set up a counter that follows a rotating log with tail -F and prints errors per minute without breaking on rotation.
- Split a big file and count in parallel: Count status codes in one giant log in chunks with parallel --pipe and merge the results.
- Cut a time window without reading the whole file: Find the start and end offsets of the outage window in a huge time-sorted log by binary search and extract that slice.
- Build one timeline from five servers' logs: Merge five servers' individually sorted logs by time without re-sorting, tagging each line with its host.
- Align logs from different time zones: Convert the logs of two servers, one in local time and one in UTC, to UTC and work out the true order of events.
- Show how duplicates distort a count: Remove records written twice by retransmission using the request ID and compute the corrected error rate.
- Produce an outage report from a giant access log: Find the outage's start, duration, share of affected requests and worst-hit URLs and report them with the commands used.
About this course
You process tens of gigabytes of logs without running out of memory and with correct results, making locale, sorting, parallelism and sampling decisions by measurement.
What you will learn
Prerequisites
You should be experienced in the topics of the same path up to level 401.
Lab environment
Lessons run in your own Kali Linux terminal that opens in the browser. Nothing to install; the lab starts clean in every lesson.
Frequently asked questions
- Is this course right for my level?
- The course is at level 501; it assumes you know the topics of the lower-level courses.
- Is there anything to install?
- No. Every lesson runs in your own lab environment that opens in the browser.
- How do lessons work?
- Each lesson has 8 to 10 steps and a short quiz. Early steps show the command to type, later steps give less help; if two attempts fail, the solution opens.
- Can I try it for free?
- Yes. The first three lessons of every course are free.