Open table formats: Apache Iceberg
With Apache Iceberg tables you add transactional guarantees, schema and partition evolution, time travel and rollback to a data lake, and use table history for deletion requests and audits.
Level 401 · 15 lessons · 12 practice labs
Lessons
Table formats
- Why a table format: from files to tables
- Iceberg's metadata tree: snapshots and manifests
- Catalogues and table addressing
- Comparing open table formats: Iceberg, Delta Lake, Hudi
- Quick task: a first Iceberg table
Changes
- Transactional writes and concurrency
- MERGE, updates and deletes
- Schema and partition evolution
- Quick task: permanently delete one person's records
Time travel and maintenance
- Time travel and rollback
- Snapshot expiry and orphan files
- Compaction and sort maintenance
- Quick task: roll back a bad write
Finish
- Using table history for audits
- Capstone: migrate an event table to Iceberg
Practice labs
- Inspect a first Iceberg table: Create an Iceberg event table, add data and list its snapshots, manifests and data files from the metadata tables.
- From catalogue to table: Find a table's current metadata file from the REST catalogue and follow the chain from there down to the data files.
- Turn a plain Parquet folder into a table: Register an existing partitioned Parquet folder as an Iceberg table without rewriting the data and verify the row counts match.
- Add and rename columns: Add a new column to a table, rename another and show the old data still reads correctly.
- Late corrections with MERGE: Apply late-arriving correction records to a table with MERGE and report the rows updated and inserted.
- Change partitioning without rewriting data: Evolve a daily-partitioned table to hourly partitions and show from the plan that old and new data are pruned correctly in one query.
- Query a past snapshot: Query yesterday's state of a table with time travel and find the row difference from today.
- Roll back a bad write: Roll a table overwritten with wrong data back to the previous snapshot and verify the numbers are right again.
- Expire snapshots and remove orphans: Expire old snapshots by age, remove orphan files and report the storage saved and the history that remains.
- Who wrote when, from table history: List the past week's write operations from snapshot history with operation type and rows added.
- Delete one person's records: Delete every row of a user who asked for erasure and show they no longer exist in the current snapshot.
- Is deleted data really gone: Find the personal data still left in old snapshots and data files after a delete and remove it for good with maintenance operations.
About this course
With Apache Iceberg tables you add transactional guarantees, schema and partition evolution, time travel and rollback to a data lake, and use table history for deletion requests and audits.
What you will learn
- Table formats
- Changes
- Time travel and maintenance
- Finish
Prerequisites
You should have practised the topics of the same path up to level 301.
Lab environment
Lessons run in your own Kali Linux terminal in the browser, connected to a private lab network. Nothing to install.
Frequently asked questions
- Is this course right for my level?
- The course is at level 401; it assumes you know the topics of the lower-level courses.
- Is there anything to install?
- No. Every lesson runs in your own lab environment that opens in the browser.
- How do lessons work?
- Each lesson has 8 to 10 steps and a short quiz. Early steps show the command to type, later steps give less help; if two attempts fail, the solution opens.
- Can I try it for free?
- Yes. The first three lessons of every course are free.