Job role capstone: security data engineer
As a security data engineer you onboard a new log source, build the pipeline, add quality and privacy checks, fix a midnight failure and document the handover.
Level 501 · 12 lessons · 9 practice labs
Lessons
Onboarding
- The role and the task list
- Onboarding a new log source
- Quick task: a data contract for the source
Building the pipeline
- Streaming in and mapping to the schema
- Writing to lake tables
- Adding quality and privacy checks
- Quick task: a security review of the pipeline
Operations
- On call: a pipeline stopped at midnight
- Cost and performance improvements
- Reviewing platform access
- The runbook and handover
- Capstone: an unbroken delivery from source to dashboard
Practice labs
- A data contract for a source: Write a YAML data contract with schema, volume, freshness and owner for a new endpoint source.
- A job that maps raw events to the schema: Write the streaming job that converts the new source's raw events to the common schema and routes unmappable ones to a dead-letter topic.
- Add quality and privacy checks: Add blank-ratio and freshness checks to the pipeline and turn the username into a keyed pseudonym before it reaches the lake.
- Why the pipeline stopped at night: Find from logs and metrics which task stopped the pipeline at midnight and why, and write it in the incident note.
- Backfill the lost hours: Reprocess the three hours missed during the outage without creating duplicates and verify the counts match the source.
- An improvement that cuts cost: Find the most expensive job, cut its run time and data scanned with partitioning and file-size fixes and report the saving.
- Review platform access: List the access belonging to leavers and unused service accounts across bucket, topic and table permissions.
- A security review of the pipeline: Review the new pipeline for secrets, permissions, encryption and logging and write the findings with severity to review.md.
- Runbook and handover: Write a runbook with symptoms, diagnostic commands and fixes for the three most common failures and resolve one failure using it.
About this course
As a security data engineer you onboard a new log source, build the pipeline, add quality and privacy checks, fix a midnight failure and document the handover.
What you will learn
- Onboarding
- Building the pipeline
- Operations
Prerequisites
You should be experienced in the topics of the same path up to level 401.
Lab environment
Lessons run in your own Kali Linux terminal in the browser, connected to a private lab network. Nothing to install.
Frequently asked questions
- Is this course right for my level?
- The course is at level 501; it assumes you know the topics of the lower-level courses.
- Is there anything to install?
- No. Every lesson runs in your own lab environment that opens in the browser.
- How do lessons work?
- Each lesson has 8 to 10 steps and a short quiz. Early steps show the command to type, later steps give less help; if two attempts fail, the solution opens.
- Can I try it for free?
- Yes. The first three lessons of every course are free.