Advanced Pandas: reshaping and performance
You analyse millions of log rows with multi-level indexes, reshaping, window calculations, chunked processing and memory and speed tuning.
Level 301 · 17 lessons · 15 practice labs
Lessons
Index and shape
- Multi-level indexes
- stack, unstack, melt and pivot
- Saving memory with categorical types
- Quick task: a host-by-hour matrix
Windows and time
- Rolling and expanding windows
- groupby transform and filter
- Time-based windows: the last 10 minutes
- Joining on nearest time with merge_asof
- Quick task: 20 failed logins in 5 minutes
Big data on one machine
- Reading in chunks: chunksize
- Measuring and shrinking memory use
- Vectorise instead of looping
- The Arrow backend and Parquet
- Readable code with method chaining
- Quick task: make a slow report ten times faster
Reliable analysis
- Testing Pandas code
- Capstone: data exfiltration analysis over a month of proxy logs
Practice labs
- Go long with melt: Turn an alert table with one column per day into long form with melt and compute the total per day.
- Querying a multi-level index: Build a MultiIndex of branch and date and select one branch's single week using xs and slicing.
- Missing combinations with stack and unstack: Build a host×hour matrix with unstack, fill combinations with no events with zeros and find the hosts that went silent.
- A cumulative mean with an expanding window: Compute the expanding mean of daily phishing reports and write on which day the mean settles.
- Per-person deviation with groupby transform: Turn each user's daily download volume into a z-score against their own mean and list the five most deviant days.
- Bursts with a time-based window: Use a time-based rolling window to find moments when one account received more than 15 password-reset requests within 10 minutes.
- Match on nearest time: Use merge_asof to match each alert to the nearest login record before it.
- Matching with a tolerance and a group key: Match DHCP leases to network events using by and tolerance and find which device held each IP at that moment.
- Align two sources with clock skew: Match events from two systems whose clocks differ by a few seconds with merge_asof, measure the skew and produce a corrected timeline.
- Measure memory use: Use memory_usage(deep=True) to find the most memory-hungry columns, convert them to category and report the saving.
- Vectorise an apply loop: Vectorise a risk-score calculation that runs row by row with apply and show identical results and the speed-up.
- Large data with Arrow and Parquet: Move a large table read from CSV to Arrow-backed Parquet and measure the difference in load time and memory.
- Readable code with method chaining: Turn cleaning code full of intermediate variables into one method chain using assign, query and pipe.
- Write a test for a pandas function: Write three tests using assert_frame_equal for a summary function, covering the empty-table and missing-value cases.
- A merge that is silently wrong: Catch a merge that multiplies rows because of duplicate keys with the validate argument and produce the correct total.
About this course
You analyse millions of log rows with multi-level indexes, reshaping, window calculations, chunked processing and memory and speed tuning.
What you will learn
- Index and shape
- Windows and time
- Big data on one machine
- Reliable analysis
Prerequisites
You should be comfortable with the level 101 and 201 topics of the same path.
Lab environment
Lessons run in your own Kali Linux terminal that opens in the browser. Nothing to install; the lab starts clean in every lesson.
Frequently asked questions
- Is this course right for my level?
- The course is at level 301; it assumes you know the topics of the lower-level courses.
- Is there anything to install?
- No. Every lesson runs in your own lab environment that opens in the browser.
- How do lessons work?
- Each lesson has 8 to 10 steps and a short quiz. Early steps show the command to type, later steps give less help; if two attempts fail, the solution opens.
- Can I try it for free?
- Yes. The first three lessons of every course are free.