Detection at scale: streaming analytics and correlation
You run detection rules on streams of millions of events and build a scalable detection engine with stateful correlation, enrichment, scoring and detection as code.
Level 501 · 15 lessons · 12 practice labs
Lessons
Detection engineering
- The detection lifecycle
- Detection as code: versioning, tests, review
- Translating Sigma rules into queries
- Quick task: a rule test harness
Correlation in the stream
- Stateful correlation: sequences and thresholds
- Enrichment at scale: caches and lookup tables
- Entity risk scoring
- Suppressing and grouping duplicate alerts
- Quick task: a sequence rule from phishing to execution
Scale
- Measuring throughput and latency
- Backpressure and late data
- Load-testing the detection pipeline
- Quick task: prepare the pipeline for ten times the load
Quality
- Detection metrics: coverage, precision, time
- Capstone: catch an attack simulation in real time
Practice labs
- Write test events for a rule: Write two events that should trigger a detection rule and two that should not, then run the tests.
- Translate a Sigma rule into a query: Translate a given Sigma rule into the platform's SQL and fix the field mapping against the common schema.
- Review a rule change: Run the tests on a rule change request, find the hidden bug that narrows its coverage and write the review note.
- Build a threshold rule on a stream: Write a streaming rule that alerts on ten failed logins for one account in five minutes and verify it with test data.
- A sequence rule: Write a stateful rule that catches failed logins followed by a success and then an immediate password change.
- Entity risk scoring: Add low-severity alerts into a weighted score per user and raise one combined alert for users who cross the threshold.
- Group duplicate alerts: Fold hundreds of alerts from the same source and rule into one within a time window and keep the count as an extra field.
- Enrichment with a lookup table: Enrich streaming events with a broadcast asset table and show the new value is used once the table is updated.
- Does suppression hide a real incident: Find the real attack hidden by an over-broad suppression rule and narrow the rule safely.
- Measure end-to-end latency: Compute median and 95th-percentile detection delay from the gap between event time and alert time.
- Diagnose backpressure: Watch consumer lag and processing time grow as the load generator speeds up and find the stage that is the bottleneck.
- Getting ready for ten times the load: Change partition count, parallelism and state-store settings and prove by load test that latency stays under target at ten times the load.
About this course
You run detection rules on streams of millions of events and build a scalable detection engine with stateful correlation, enrichment, scoring and detection as code.
What you will learn
- Detection engineering
- Correlation in the stream
- Scale
- Quality
Prerequisites
You should be experienced in the topics of the same path up to level 401.
Lab environment
Lessons run in your own Kali Linux terminal in the browser, connected to a private lab network. Nothing to install.
Frequently asked questions
- Is this course right for my level?
- The course is at level 501; it assumes you know the topics of the lower-level courses.
- Is there anything to install?
- No. Every lesson runs in your own lab environment that opens in the browser.
- How do lessons work?
- Each lesson has 8 to 10 steps and a short quiz. Early steps show the command to type, later steps give less help; if two attempts fail, the solution opens.
- Can I try it for free?
- Yes. The first three lessons of every course are free.