Hadoop and HDFS: distributed storage and batch processing
You store log data too big for one machine in HDFS, protect it with permissions and process it in batch jobs.
Level 201 · 19 lessons · 18 practice labs
Lessons
Core concepts
- What big data is and why distribution is needed: You understand the volume, velocity and variety problem and why data is split across many machines.
- Hadoop architecture: HDFS, YARN, MapReduce: You tell apart the roles of the storage, resource-management and processing layers.
- Getting to know a Hadoop environment: You check that services are running and locate the configuration files and web interfaces.
- Quick task: report the cluster status
Working with HDFS
- File operations with hdfs dfs: You create, upload, download, list and delete files and directories.
- The file lifecycle and the trash: You see how a deleted file moves to the trash and returns, and how that differs from permanent deletion.
- Blocks and replication: fsck: You inspect how a file is split into blocks and replicated with fsck and interpret missing blocks.
- Organising daily data-drop directories: You set up date-based directory layouts and keep raw and clean data in separate areas.
- Quick task: set up a daily log landing zone
Protection and recovery
- HDFS permissions and ACLs: You use chmod, chown and ACLs so only the right teams can access a shared dataset.
- Recovery points with snapshots: You take a snapshot before changes and restore a file that was damaged by mistake.
- HDFS security: Kerberos, encryption zones, audit log: You explain what authentication, encryption at rest and access audit logs each protect.
- Name and space quotas
- Quick task: protect and restore a log directory
Processing
- Map, shuffle and reduce: You understand the three-stage processing model through a word-count example over log lines.
- Submitting and monitoring jobs with YARN: You submit a batch job and follow its status and output with yarn commands.
- Diagnosing common Hadoop problems: You solve failures such as an existing output directory, low memory and permission errors from the logs.
- A Hadoop Streaming job in Python
- Capstone: a daily log-processing workflow: You land raw logs in HDFS, process them with a job and hand the result to the authorised team.
Practice labs
- HDFS folders for a new team: Create raw, processed and archive folders in HDFS for the analysis team and upload the sample files to the right place.
- Restore from the trash: Find a daily log folder deleted by mistake in the HDFS trash, move it back to its original place and verify the file count.
- Fix the small-files problem: Merge a folder of thousands of tiny files into one file per day and show the drop in the object count on the NameNode.
- Find a file's blocks: Use fsck to report how many blocks a given file is split into and which DataNodes hold them.
- Raise replication for critical data: Raise the replication factor of the audit-log folder to three and prove with fsck that no under-replicated blocks remain.
- Detect corrupt blocks: Find corrupt and missing blocks in fsck output, list the affected files and write which ones can be recovered from a replica.
- Tighten folder permissions: Fix the owner and permissions of a world-writable HDFS log folder so only the security team can write to it.
- Read access for one user with an ACL: Add an HDFS ACL that lets the auditor user read only one folder and verify they cannot reach any other.
- Protection with snapshots and quotas: Enable snapshots and set a space quota on a folder, then show an upload over the quota being refused and a rollback from the snapshot.
- List the running jobs: List running and failed applications with YARN commands and write down the ID and owner of the longest-running job.
- Read a failed job's logs: Pull the container logs of a failed MapReduce job and write the cause of the failure and the fix to failure.md.
- A job killed by memory settings: Fix the settings of a job that keeps getting killed for exceeding its memory limit, get it to finish and report the memory used.
- Counting status codes with Streaming: Run a Hadoop Streaming job with a supplied mapper and reducer and read the HTTP status-code counts from HDFS.
- Write the Python mapper: Write a Python mapper and reducer that count failed SSH logins per source IP and run them as a Streaming job.
- Repair a job that dies on bad lines: Fix a Streaming job that crashes on bad lines so it counts them with counters instead and verify the result.
- Report the cluster's state: Use NameNode and YARN commands to write the live node count, total and free capacity and running applications to status.txt.
- A cluster stuck in safe mode: Work out why a cluster refusing writes is stuck in safe mode, remove the cause and leave safe mode.
- Who is using the space on a filling disk: Find the largest folders and their owners on an HDFS nearly out of capacity and report them, accounting for the replication factor.
About this course
You store log data too big for one machine in HDFS, protect it with permissions and process it in batch jobs.
What you will learn
- Core concepts
- Working with HDFS
- Protection and recovery
- Processing
Prerequisites
Finishing the level 101 courses of the same path, or knowing those topics, is enough.
Lab environment
Lessons run in your own Kali Linux terminal in the browser, connected to a private lab network. Nothing to install.
Frequently asked questions
- Is this course right for my level?
- The course is at level 201; it assumes you know the topics of the lower-level courses.
- Is there anything to install?
- No. Every lesson runs in your own lab environment that opens in the browser.
- How do lessons work?
- Each lesson has 8 to 10 steps and a short quiz. Early steps show the command to type, later steps give less help; if two attempts fail, the solution opens.
- Can I try it for free?
- Yes. The first three lessons of every course are free.