Data quality and validation
You measure the dimensions of data quality, write schema and business-rule tests, put quality gates into pipelines and trace quality problems to their root.
Level 301 · 16 lessons · 12 practice labs
Lessons
Quality dimensions
- Accuracy, completeness, consistency, timeliness
- Measuring quality: rates and thresholds
- Silent failures: when a log source goes quiet
- Quick task: a quality scorecard for a data set
Tests
- Schema contracts
- Rule-based checks: ranges, uniqueness, relationships
- DataFrame validation with pandera
- Detecting distribution drift
- Quick task: a test suite for an event table
Quality in the pipeline
- Quality gates: stopping bad data
- Quarantine tables and reprocessing
- Freshness and volume monitoring
- Lineage and root cause
- Quick task: an alert that catches a silent log source
Wrap-up
- Reporting and preventing a quality incident
- Capstone: add a quality layer to a security data pipeline
Practice labs
- Match the quality dimension: Match ten data problems to accuracy, completeness, consistency or timeliness and write the result to dimensions.csv.
- A quality scorecard: Compute rates across the four dimensions for an asset inventory and produce a scorecard that compares them against thresholds.
- Find the source that went quiet: From the hourly volume of thirty log sources, find the ones that silently stopped or halved and report when each first deviated.
- Write a schema contract: Write a YAML schema contract with column names, types and required flags for a login event table.
- Rule checks with pandera: Define range, uniqueness and allowed-value rules in a pandera schema and produce a report of the failing rows.
- Cross-table relationship checks: Write a relationship check that finds user and device IDs in the event table with no match in the reference tables.
- A freshness check: Write a check that computes each table's latest record time and lists the tables that exceed their expected delay.
- Catch distribution drift: Compare this week's event-type distribution with last week's and flag the types whose share changed markedly.
- Cut false alarms in a volume alert: Replace a fixed-threshold volume alert that fires every weekend with a day-of-week-aware threshold and show the drop in alarms.
- Add a quality gate: Add a quality gate to a load script that stops the load and states why when the blank ratio exceeds a threshold.
- Quarantine the bad rows: Move rule-breaking rows into a quarantine table, reprocess them after the fix and add them to the main table.
- Find the root cause with lineage: Trace a wrong number in a report step by step through its lineage to its source and fix the faulty transformation.
About this course
You measure the dimensions of data quality, write schema and business-rule tests, put quality gates into pipelines and trace quality problems to their root.
What you will learn
- Quality dimensions
- Tests
- Quality in the pipeline
- Wrap-up
Prerequisites
You should be comfortable with the level 101 and 201 topics of the same path.
Lab environment
Lessons run in your own Kali Linux terminal that opens in the browser. Nothing to install; the lab starts clean in every lesson.
Frequently asked questions
- Is this course right for my level?
- The course is at level 301; it assumes you know the topics of the lower-level courses.
- Is there anything to install?
- No. Every lesson runs in your own lab environment that opens in the browser.
- How do lessons work?
- Each lesson has 8 to 10 steps and a short quiz. Early steps show the command to type, later steps give less help; if two attempts fail, the solution opens.
- Can I try it for free?
- Yes. The first three lessons of every course are free.