Data analysis and log review with Pandas
You read, clean and summarise tabular data and turn security logs into meaningful reports.
Level 201 · 22 lessons · 18 practice labs
Lessons
First steps
- What Pandas is and setting it up: You see which jobs Pandas makes easier, install it in a virtual environment and run your first import.
- Creating a Series and a DataFrame: You build a one-column Series and a multi-column DataFrame from dictionaries and lists.
- Reading CSV and JSON files: You load files with read_csv and read_json and set the delimiter, header and data types.
- Getting to know data: head, info, describe: You size up a new table with a few commands and notice missing or suspicious columns.
- Quick task: load and profile a firewall log
Selecting and filtering
- Selecting columns and rows: loc and iloc: You select rows and columns by label and by position.
- Filtering by condition: You combine several conditions to pull, for example, failed logins out of a table.
- Sorting and finding the top values: You sort a table by one or more columns and list the sources producing the most events.
- Quick task: filter failed logins
Cleaning
- Finding and handling missing values: You count missing values and weigh how dropping or filling them changes the result.
- Duplicate records and type conversion: You remove repeated records of the same event and convert text-typed numbers and dates to proper types.
- Text and timestamp operations: You parse text in log lines and break timestamps into hours and days.
- Quick task: clean a messy user list
Summarising and combining
- Descriptive statistics: You describe a column’s distribution with the mean, median and percentiles.
- Grouping and summarising with groupby: You group events by user, IP or day and compute counts and averages.
- Combining tables: merge and concat: You match an event table with an asset inventory to see which server each event belongs to.
- Time series and resample: You bucket events by hour or day and spot trends.
- Pivot tables and crosstabs
- Quick task: the top talkers
Security and privacy
- Masking and pseudonymising personal data: You mask or pseudonymise personal data like emails and IPs with salted hashes before analysis.
- Safe export: CSV formula injection: You neutralise cells that could run as spreadsheet formulas when exporting a table.
- Capstone: an authentication log report: You clean a login log, group it and produce a report summarising suspicious account activity.
Practice labs
- Load a CSV exported from a spreadsheet: Read a customer file with Turkish characters and semicolon separators using the right encoding and delimiter and save the info() output.
- Turn nested JSON into a DataFrame: Flatten nested cloud audit records with json_normalize and produce a table with user, action and region columns.
- Read a large file in chunks: Read a proxy log too big for memory in chunks with chunksize and compute the total bytes per domain.
- Select rows with loc: Use loc to select only the blocked rows going to port 22 in a firewall log and write their count.
- Filter with isin and between: Filter VPN sessions that come from a given list of countries and fall outside business hours using isin and between and save them to CSV.
- Fix the chained-assignment warning: Fix filtering code that raises SettingWithCopyWarning and silently gives wrong results, and show the correct number of updated rows.
- Count and fill missing values: Count missing values per column in a staff list and fill blank departments with “Unknown”.
- Turn text numbers into real numbers: Convert amounts written like “1.250,50 TL” into numbers and move the rows that cannot be converted into a separate table.
- Remove duplicates wisely: In an alert table where the same event was written twice with different timestamps, remove only the true duplicates and report how many were removed.
- A user summary with groupby: Use groupby to produce the file count and total size per user from a file-sharing log.
- Merge two tables and find the orphans: Merge the account list with the HR list and find people who have left but still have an open account.
- Look for a link with a crosstab: Build a normalised crosstab of department against alert type and show which department stands out for which alert.
- Daily event counts: Set the timestamp as the index, use resample to compute daily event counts and write down the busiest day.
- Spikes with a rolling window: Compute a 24-hour rolling mean of hourly login counts and flag the hours that exceed three times that mean.
- Complete a time series with gaps: Fill the hours with no records with zeros in a series with gaps and show how the gaps had distorted the average.
- Mask e-mail addresses: Mask the e-mail addresses in a support table so only the first letter and the domain remain.
- Generate consistent pseudonyms: Turn usernames into pseudonyms with a salted hash and show that the same user gets the same pseudonym in two different tables.
- Export safely against formula injection: Write an export function that neutralises cells starting with =, +, - or @ and show the malicious rows stay harmless.
About this course
You read, clean and summarise tabular data and turn security logs into meaningful reports.
What you will learn
- First steps
- Selecting and filtering
- Cleaning
- Summarising and combining
- Security and privacy
Prerequisites
Finishing the level 101 courses of the same path, or knowing those topics, is enough.
Lab environment
Lessons run in your own Kali Linux terminal that opens in the browser. Nothing to install; the lab starts clean in every lesson.
Frequently asked questions
- Is this course right for my level?
- The course is at level 201; it assumes you know the topics of the lower-level courses.
- Is there anything to install?
- No. Every lesson runs in your own lab environment that opens in the browser.
- How do lessons work?
- Each lesson has 8 to 10 steps and a short quiz. Early steps show the command to type, later steps give less help; if two attempts fail, the solution opens.
- Can I try it for free?
- Yes. The first three lessons of every course are free.