Data lake design
You design a data lake with raw, clean and serving layers and store security data neatly and cheaply with object storage, Parquet, partitioning, a catalogue and access policies.
Level 301 · 16 lessons · 12 practice labs
Lessons
Lake concepts
- Data lake and data warehouse
- Layers: raw, clean, serving
- Object storage: buckets, keys, versions
- Quick task: set up the bucket and folder layout
File layout
- Parquet internals: row groups, columns, statistics
- Partitioning strategy and too many small files
- Compacting small files
- When the schema changes: new fields, type changes
- Quick task: speed up a slow query with partitioning
Management
- Data catalogues and metadata
- Retention and lifecycle rules
- Cost: storage classes and query cost
- Quick task: apply retention rules to log types
Security
- Bucket policies and least privilege
- Encryption and immutable storage
- Capstone: build a data lake for security logs
Practice labs
- Set up buckets and layers: Create the buckets and prefix layout for raw, clean and serving layers in MinIO and upload the sample files to the right layer.
- Bring back a deleted object with versioning: Turn on versioning for a bucket and restore the previous version of a log file that was overwritten and deleted.
- Move a job that writes straight to raw: Fix a transform job that modifies raw data in place so it writes to the clean layer and prove with checksums that the raw data is untouched.
- Look inside a Parquet file: Read the row-group count, column statistics and compression type from a Parquet file's metadata and report them.
- Compact small files: Merge the hundreds of small Parquet files in a day's partition into a few large ones and measure the difference in query time.
- Fix over-partitioning: Move a table partitioned by hour and user to date partitioning that fits the query pattern and compare file counts and bytes scanned.
- Apply a retention rule: Define a lifecycle rule that deletes raw web logs after 90 days and list the objects the rule will affect.
- Reads that survive a new field: Fix a query that fails after the source gained a new column using schema merging and read old and new files together.
- Prepare data catalogue entries: Write catalogue entries with owner, schema, freshness, sensitivity and retention for six data sets in the lake to catalog.csv.
- Close a public bucket: Find a log bucket left open to anonymous reads, fix the policy and show anonymous access is now refused.
- Least privilege by prefix: Write and test policies that let the analyst account only read the clean layer and the collector account only write to the raw layer.
- Protecting evidence with immutable storage: Enable object lock in compliance mode on the audit-log bucket and prove even the admin account cannot delete before retention ends.
About this course
You design a data lake with raw, clean and serving layers and store security data neatly and cheaply with object storage, Parquet, partitioning, a catalogue and access policies.
What you will learn
- Lake concepts
- File layout
- Management
- Security
Prerequisites
You should be comfortable with the level 101 and 201 topics of the same path.
Lab environment
Lessons run in your own Kali Linux terminal in the browser, connected to a private lab network. Nothing to install.
Frequently asked questions
- Is this course right for my level?
- The course is at level 301; it assumes you know the topics of the lower-level courses.
- Is there anything to install?
- No. Every lesson runs in your own lab environment that opens in the browser.
- How do lessons work?
- Each lesson has 8 to 10 steps and a short quiz. Early steps show the command to type, later steps give less help; if two attempts fail, the solution opens.
- Can I try it for free?
- Yes. The first three lessons of every course are free.