Big data processing with Apache Spark
You read, clean and join large event data with PySpark and turn it into reusable batch jobs.
Level 301 · 21 lessons · 18 practice labs
Lessons
Fundamentals
- Spark architecture: driver, executors, lazy evaluation: You understand how Spark splits and distributes work and how transformations differ from actions.
- Your first PySpark DataFrame: You start a SparkSession, build a small DataFrame and display its contents.
- Reading CSV with an explicit schema: You define the schema yourself rather than letting Spark guess, avoiding wrong type inference.
- Reading JSON events: You flatten nested JSON event records into DataFrame columns.
- Quick task: explore authentication events
Transforming
- Selecting, filtering and transforming: You select columns, filter by condition and derive new columns.
- Cleaning missing, duplicate and invalid records: You remove broken records and prepare a cleaned dataset.
- Grouping and aggregating: You summarise events by source, day and type with groupBy and agg.
- Joining datasets: You match events with asset and user tables and choose between join types.
- Deriving sessions from raw events: You group timestamped events into per-user sessions and count sessions instead of raw events.
- Window functions: ranking and running totals
- Quick task: find brute-force attempts in sessions
SQL and storage
- Querying with Spark SQL: You register a DataFrame as a temporary view and query it with SQL.
- Writing and inspecting partitioned Parquet: You write the result as Parquet partitioned by date and check the resulting file layout.
- Execution plans and common errors: You read the plan with explain and find the causes of memory and schema errors.
- Quick task: a date-partitioned archive
Toward production
- Running a batch application with spark-submit: You submit a script as an application and rerun the same job safely.
- Introduction to Structured Streaming: You see how batch differs from stream processing and the logic of working with a Kafka source.
- Secrets and personal-data protection in Spark: You supply passwords without embedding them in config and strip personal columns from output.
- User-defined functions (UDFs) and their cost
- Capstone: a reusable event-processing pipeline: You build a batch job that reads, cleans and summarises raw events and writes partitioned output.
Practice labs
- Read a CSV with an explicit schema: Read a badge log in PySpark with an explicitly defined schema and print the column types and row count.
- Catch rows that break the schema: Read in PERMISSIVE mode, write the rows that break the schema to a separate file using the corrupt-record column and report their share.
- Flatten nested JSON events: Turn cloud audit events with nested structs and arrays into a flat DataFrame using explode and dot notation.
- Select, filter, derive a column: Filter only the blocked requests from a proxy log and add a new column that derives the domain from the URL.
- Clean duplicate and invalid records: Clean duplicate events, blank users and future timestamps and report the number of rows dropped at each step.
- Replace a slow UDF with built-ins: Rewrite an IP classification written as a Python UDF with built-in Spark functions and show the time difference and identical results.
- A summary per user: Use groupBy and agg to compute event count, distinct IP count and last-seen time per user.
- Join two data sets: Join authentication events with the asset inventory and rank the failed logins against critical servers.
- Unknown sources with an anti-join: Use a left anti-join to find source addresses missing from the allow list and produce each one's first-seen time and volume.
- Ranking with a window: Rank the three users who downloaded the most data within each department using a window function.
- Derive sessions from raw events: Split one user's events into sessions with a 30-minute gap rule and compute duration and event count per session.
- An exfiltration threshold with running totals: Compute each user's running upload volume within the day and report the moment the threshold was first crossed and the destinations up to then.
- Ask the same question in Spark SQL: Register a DataFrame as a temporary view and find the ten clients producing the most 404s with SQL.
- Write and inspect partitioned Parquet: Write events as date-partitioned Parquet and prove from the plan that a single-day query reads only the matching partition.
- Find the fault in an execution plan: Find the step in explain output that causes a full scan and a needless shuffle, fix the query and compare the plans.
- A batch job with spark-submit: Run a PySpark script that takes input and output paths as arguments with spark-submit and verify the output files.
- Streaming from a folder: Build a stream that reads JSON files landing in a folder with Structured Streaming and prints events per minute to the console.
- Remove the hard-coded password and exposed personal data: Move a hard-coded database password in a Spark job into an environment variable and pseudonymise the e-mail column in the output with a hash.
About this course
You read, clean and join large event data with PySpark and turn it into reusable batch jobs.
What you will learn
- Fundamentals
- Transforming
- SQL and storage
- Toward production
Prerequisites
You should be comfortable with the level 101 and 201 topics of the same path.
Lab environment
Lessons run in your own Kali Linux terminal that opens in the browser. Nothing to install; the lab starts clean in every lesson.
Frequently asked questions
- Is this course right for my level?
- The course is at level 301; it assumes you know the topics of the lower-level courses.
- Is there anything to install?
- No. Every lesson runs in your own lab environment that opens in the browser.
- How do lessons work?
- Each lesson has 8 to 10 steps and a short quiz. Early steps show the command to type, later steps give less help; if two attempts fail, the solution opens.
- Can I try it for free?
- Yes. The first three lessons of every course are free.