Data pipeline security and operations
You threat-model data pipelines and protect, monitor and respond to incidents in them with secrets, identity, network, integrity and lineage controls.
Level 401 · 15 lessons · 12 practice labs
Lessons
Threat model
- Threat modelling a data pipeline
- The attack surface: connectors, interfaces, notebooks
- Lessons from real data-platform incidents
- Quick task: threat-model a log pipeline
Protection
- Service identities and short-lived credentials
- Secrets management in pipelines
- Integrity: checksums and signed data
- Network isolation and egress control
- Dependencies and the supply chain
- Quick task: remove hard-coded passwords from a pipeline
Monitoring and response
- Observability: metrics, logs, traces
- Lineage and the audit trail
- Detecting data tampering
- Incident response in a data pipeline
- Capstone: find, clean and harden a tampered pipeline
Practice labs
- Trust boundaries in a data flow: Mark the components and trust boundaries in a pipeline diagram and write each boundary crossing to boundaries.csv.
- An attack-surface inventory: Scan the running environment for exposed interfaces, connectors and notebook servers and list those that ask for no authentication.
- From threat model to mitigation plan: Score six threats to a log ingestion pipeline by likelihood and impact and assign each a workable mitigation and an owner.
- Scan for embedded secrets: Find embedded passwords and access keys in the pipeline repository's DAGs, configuration and scripts and list them with their locations.
- Split a shared account into service identities: Replace the single admin key every task uses with least-privilege service accounts per task and show the pipeline still runs.
- Rotate a leaked key: Revoke an access key pushed to the repository by mistake, have the new one read from the secrets store and keep the pipeline running.
- File integrity with checksums: Compare the checksums of files arriving in the raw layer against the manifest and find the one altered in transit.
- Restrict egress traffic: Define egress rules so pipeline workers can reach only the destinations they need and show a connection elsewhere is blocked.
- A malicious package hiding in dependencies: Find the unpinned, look-alike fake package in the pipeline's dependency list and pin the versions with hashes.
- Trace a change through lineage: Find from the audit trail which job and account last wrote a table and compare it with the expected job.
- Catch silently dropped events: Compare event counts at source and destination hour by hour and prove that a filter is dropping one IP's events.
- Responding to a tampered pipeline: Find the altered transform step, stop the pipeline, rebuild the affected partitions from clean data and write the response note.
About this course
You threat-model data pipelines and protect, monitor and respond to incidents in them with secrets, identity, network, integrity and lineage controls.
What you will learn
- Threat model
- Protection
- Monitoring and response
Prerequisites
You should have practised the topics of the same path up to level 301.
Lab environment
Lessons run in your own Kali Linux terminal in the browser, connected to a private lab network. Nothing to install.
Frequently asked questions
- Is this course right for my level?
- The course is at level 401; it assumes you know the topics of the lower-level courses.
- Is there anything to install?
- No. Every lesson runs in your own lab environment that opens in the browser.
- How do lessons work?
- Each lesson has 8 to 10 steps and a short quiz. Early steps show the command to type, later steps give less help; if two attempts fail, the solution opens.
- Can I try it for free?
- Yes. The first three lessons of every course are free.