Advanced Spark: performance and stream processing
You read Spark's execution model in depth, tune shuffles, skew, memory and join strategies, and build stateful, windowed real-time detection with structured streaming.
Level 401 · 17 lessons · 15 practice labs
Lessons
Execution model
- Stages, tasks and shuffles
- Reading the Spark UI
- The Catalyst optimiser and adaptive execution
- Quick task: find a slow job's bottleneck
Performance
- Tuning partition counts and file sizes
- Data skew and salting
- Join strategies: broadcast and sort-merge
- Caching and memory management
- Quick task: fix a join piled onto one IP
Stream processing
- Structured streaming from Kafka
- Event time and watermarks
- Stateful aggregations and windows
- Stream–stream joins
- Checkpoints and exactly-once processing
- Quick task: detect brute force in a stream
Production
- Spark security: authentication, encryption, ACLs
- Capstone: a real-time detection and batch archive pipeline
Practice labs
- Read stages and tasks: Extract the stage count, the longest stage and the shuffle size from a finished job's Spark UI records.
- Find the bottleneck in the Spark UI: Look at the task-time distribution of a slow job and prove whether the bottleneck is the shuffle, one task or garbage collection.
- The effect of adaptive execution: Run the same query with adaptive execution on and off and report the difference in plan, partition count and time.
- Tune the partition count: Rework a write job that produces 200 tiny files with a suitable partition count and bring the file sizes into the target range.
- Salt a skewed key: Salt the key in an aggregation where one service account produces most of the events and show the drop in the longest task time.
- When a broadcast join is right: Join a small threat list to a large event table with sort-merge and broadcast joins and compare the plans and times.
- Measure the effect of caching: Cache an intermediate result used three times and measure how the total time changes.
- Remove needless caching: Remove the needless caches from a job that caches every DataFrame and fills memory and show the difference on the storage tab.
- An executor dying out of memory: Fix a job that runs out of memory because of collect and a wide aggregation so it produces the same result reliably.
- Read a stream from Kafka: Read login events from a Kafka topic with Structured Streaming, parse the JSON fields and write them to the console.
- Handle late events with a watermark: Count in five-minute event-time windows, add a watermark and show how late events are handled.
- Detecting password spraying in a stream: Catch cases in a stream where one source tries more than 20 different accounts in ten minutes with a stateful aggregation and write them to an alert topic.
- Resume from a checkpoint: Restart a stopped streaming job from its checkpoint and verify by count that no event was skipped or repeated.
- A stream-stream join: Join login and file-access streams with a time bound and match accesses made within two minutes of a login.
- Secure the Spark UI and traffic: Restrict a Spark UI left open without authentication using ACLs, encrypt traffic between nodes and show unauthorised access is refused.
About this course
You read Spark's execution model in depth, tune shuffles, skew, memory and join strategies, and build stateful, windowed real-time detection with structured streaming.
What you will learn
- Execution model
- Performance
- Stream processing
- Production
Prerequisites
You should have practised the topics of the same path up to level 301.
Lab environment
Lessons run in your own Kali Linux terminal in the browser, connected to a private lab network. Nothing to install.
Frequently asked questions
- Is this course right for my level?
- The course is at level 401; it assumes you know the topics of the lower-level courses.
- Is there anything to install?
- No. Every lesson runs in your own lab environment that opens in the browser.
- How do lessons work?
- Each lesson has 8 to 10 steps and a short quiz. Early steps show the command to type, later steps give less help; if two attempts fail, the solution opens.
- Can I try it for free?
- Yes. The first three lessons of every course are free.