Introduction to big data
You learn why data outgrows one machine, the core ideas of distributed storage and processing, batch versus streaming, and the parts of a modern data architecture.
Level 101 · 17 lessons · 15 practice labs
Lessons
Concepts
- What big data is: volume, velocity, variety
- Scaling up and scaling out
- Distributed basics: nodes, partitions, replicas
- Failures and the consistency trade-off
- Quick task: estimate an organisation's daily log volume
Storage and formats
- Row and column storage
- File formats: CSV, JSON, Avro, Parquet
- Compression and splittable files
- Partitioning: foldering data by date
- Quick task: convert CSV to Parquet and compare sizes
Processing
- Batch and stream processing
- Split, process, combine: the MapReduce idea
- The ecosystem map: Hadoop, Hive, Spark, Kafka
- Data architecture: source, ingestion, storage, serving
- Quick task: do a word count by hand, MapReduce style
At security-data scale
- Why security data is big data
- Capstone: draw a data flow for a SOC
Practice labs
- Estimate daily log volume: Compute the daily and yearly log volume in GB of an organisation given its device count and average event size.
- Storage by retention period: Work out in a table the total raw storage Novaro Telecom needs for three log types with different retention periods and a replication factor.
- Planning for growth and peak load: Draw up a three-year storage and events-per-second capacity plan for an event stream growing 30 per cent a year and spiking fivefold on campaign days.
- Row or column: Read five sample queries and choose, with a reason, whether a row-based or column-based format suits each one better.
- Comparing CSV, JSON and Parquet sizes: Save the same alert data as CSV, JSON and compressed Parquet and record the sizes and the time to read one column in a table.
- Choosing a splittable compression: Choose between gzip, bzip2 and Parquet's internal compression for parallel reading and prove the choice with a small experiment.
- Folder by date: Write a script that splits a month of events delivered in one file into a year=/month=/day= folder layout.
- The wrong partition key: Move a data set partitioned by user ID, which produced thousands of tiny files, to date partitioning and compare the file counts.
- Design partitions from query patterns: Pick the partitioning scheme that skips the most data for a week's worth of queries and write down your reasoning.
- Map and reduce by hand: Count error codes separately in three small log chunks and combine the partial results into the totals.
- A mini MapReduce in Python: Write map, shuffle and reduce as separate functions and produce a script that counts requests per domain.
- The skewed key problem: Measure reducer load in a count where one IP makes up half the data and balance it by salting the key.
- Stream or batch: Read eight business needs and classify, with a reason, whether each needs stream or batch processing.
- Match the ecosystem component: Match six steps in a retail company's data flow to components such as Kafka, HDFS, Hive and Spark and write them to architecture.csv.
- A single point of failure: Find the single points of failure and unreplicated layers in a given data architecture diagram and propose a fix for each.
About this course
You learn why data outgrows one machine, the core ideas of distributed storage and processing, batch versus streaming, and the parts of a modern data architecture.
What you will learn
- Concepts
- Storage and formats
- Processing
- At security-data scale
Prerequisites
No prior knowledge is needed; the course starts from zero.
Lab environment
Lessons run in your own Kali Linux terminal that opens in the browser. Nothing to install; the lab starts clean in every lesson.
Frequently asked questions
- Is this course right for my level?
- The course is at beginner level; every term is explained where it first appears.
- Is there anything to install?
- No. Every lesson runs in your own lab environment that opens in the browser.
- How do lessons work?
- Each lesson has 8 to 10 steps and a short quiz. Early steps show the command to type, later steps give less help; if two attempts fail, the solution opens.
- Can I try it for free?
- Yes. The first three lessons of every course are free.