Data pipelines with Airflow
Starting from ETL and ELT, you build scheduled, retryable, observable and secure data pipelines with Apache Airflow.
Level 301 · 17 lessons · 15 practice labs
Lessons
Pipeline concepts
- ETL and ELT: extract, transform, load
- Idempotency: safe to run again
- Scheduling, lateness and backfill
- Quick task: design the steps of a log pipeline
Airflow basics
- Airflow architecture: scheduler, executor, metadata database
- A first DAG: tasks and dependencies
- Operators and the TaskFlow API
- Templates and the logical date
- Quick task: a daily log ingestion DAG
Robust pipelines
- Retries, timeouts and alerts
- Sensors and external dependencies
- Passing data between tasks: XCom
- Dynamic tasks
- Quick task: recover a half-finished run
Operations and security
- Connections and secrets management
- Roles and audit logs in Airflow
- Capstone: an end-to-end pipeline that ingests, cleans and reports
Practice labs
- Order the ETL steps: Put eight shuffled pipeline steps into extract, transform, load order and write their dependencies to steps.csv.
- A task that duplicates rows on rerun: Make a load task that doubles the records on a second run idempotent and prove it by running it twice.
- Late data and backfill: Run a backfill for files that arrived three days late and verify that only the affected days were reprocessed.
- Enable a first DAG: Fix the dependency error in a supplied three-task DAG, run it successfully and report the task order.
- Write tasks with TaskFlow: Write a three-task DAG with the TaskFlow API that reads a file, filters it and writes a summary.
- Pick the right file with the logical date: Fix a faulty DAG that reads today's file on every run so it reads the logical date's file using template variables.
- Add retries and a timeout: Add retries, a delay and a timeout to a download task that fails now and then and show the behaviour in the logs.
- Wait until the file arrives: Make a pipeline that starts before its source file arrives wait with a sensor and define a timeout that raises an alert if the file never comes.
- Recover a half-finished run: In a run whose middle task failed, clear and rerun only the failed and downstream tasks to complete the output.
- Pass a small value with XCom: Pass the row count one task computes to the next via XCom and stop the pipeline if it is below a threshold.
- The mistake of putting big data in XCom: Fix a DAG that moves a whole table through XCom so it passes a file path instead and show the size difference in the metadata database.
- One task per source with dynamic tasks: Generate one task per log source from a list with dynamic task mapping and show the others carry on when one fails.
- Take the connection out of the code: Move a database address and password embedded in DAG code into an Airflow connection and show no secret remains in the code.
- Reduce roles to least privilege: In an Airflow setup where everyone is an admin, assign viewer, user and admin roles to the right people and test the access.
- Who changed what in the audit log: Go through the Airflow audit logs and report, with a timeline, the account that paused a DAG and changed a connection at midnight.
About this course
Starting from ETL and ELT, you build scheduled, retryable, observable and secure data pipelines with Apache Airflow.
What you will learn
- Pipeline concepts
- Airflow basics
- Robust pipelines
- Operations and security
Prerequisites
You should be comfortable with the level 101 and 201 topics of the same path.
Lab environment
Lessons run in your own Kali Linux terminal in the browser, connected to a private lab network. Nothing to install.
Frequently asked questions
- Is this course right for my level?
- The course is at level 301; it assumes you know the topics of the lower-level courses.
- Is there anything to install?
- No. Every lesson runs in your own lab environment that opens in the browser.
- How do lessons work?
- Each lesson has 8 to 10 steps and a short quiz. Early steps show the command to type, later steps give less help; if two attempts fail, the solution opens.
- Can I try it for free?
- Yes. The first three lessons of every course are free.