Linux expert challenges
Long tasks that start with little information and span several subsystems: recovering lost data, solving performance mysteries, rebuilding identity and storage together.
Level 401 · 16 lessons · 15 practice labs
Lessons
Mysteries
- Disappearing disk space: df and du disagree
- A server that slows down every 17 minutes
- A process that comes back every time you kill it
- A job that crashes on the last day of the month
- A silent performance loss after a kernel update
- Short task: write up the root causes of the mysteries with evidence
Rescues
- Rescuing a server whose critical binaries were deleted
- Commands fail because of a broken shared library
- Root full and no login: the way back
- The LVM metadata of an encrypted volume is gone
- Short task: turn your rescue steps into a runbook
Rebuilds
- Central login broke and servers are locked
- Moving a running service to new storage with no downtime
- Configuration archaeology on an old service nobody understands
- Rebuilding a suspected-compromised server cleanly and moving its data safely
- Capstone: an eight-hour shift full of linked tasks
Practice labs
- Find where the disk space went: Prove the gap between df and du comes from a deleted file held open and reclaim the space without breaking the service.
- Find why a server slows every 17 minutes: Catch the slowdown moments with measurements, match them to timers and jobs and identify the trigger with evidence.
- Find what restarts a process each time it is ended: Find the mechanism restarting the process among units, timers, cron and the parent chain, document it and disable it properly.
- Rescue a server whose critical binaries were deleted: On a system missing basic commands, reinstall the packages with the tools that remain and verify the commands are back.
- Repair commands failing on a broken shared library: On a system where most commands fail, restore the library with a statically built tool or the rescue environment.
- Find the way back when root is full and nobody can log in: Reach a server refusing logins on a full root from the console, free space safely, fix the cause and set a limit against recurrence.
- Reproduce the bug in a job that crashes at month end: Run the script with different dates to find and fix the bug in its date arithmetic.
- Find a silent performance loss after a kernel update: Measure the same workload on both kernels, identify the changed default or driver and show the difference in numbers.
- Write up the root causes of the mysteries with evidence: For three mysteries you solved prepare a report with symptom, wrong hypotheses, the decisive evidence and the lasting fix.
- Restore access to servers when central login breaks: With the directory unreachable log in using the local break-glass account and determine why the identity service does not answer.
- Move a running service to new storage without downtime: Move a service's data to a new physical volume while it runs, remove the old disk from the group and show no request failed.
- Decode a legacy service nobody understands and rebuild it documented: Extract an undocumented service's files, dependencies and scheduled jobs from the running system, rebuild it on a clean server and compare outputs.
- Restore the lost LVM metadata of an encrypted volume: Bring the volume group back from the metadata archive, open the encrypted volume and verify the data is intact.
- Turn rescue steps into a runbook: Turn a rescue you performed into a runbook with prerequisites, steps, verification and rollback and test it in a clean environment.
- Cleanly rebuild a suspected-compromised server and move data safely: On your own lab server take an image preserving evidence, build a clean server, move only verified data and renew keys and passwords.
About this course
Long tasks that start with little information and span several subsystems: recovering lost data, solving performance mysteries, rebuilding identity and storage together.
What you will learn
Prerequisites
You should have practised the topics of the same path up to level 301.
Lab environment
Lessons run in your own Kali Linux terminal that opens in the browser. Nothing to install; the lab starts clean in every lesson.
Frequently asked questions
- Is this course right for my level?
- The course is at level 401; it assumes you know the topics of the lower-level courses.
- Is there anything to install?
- No. Every lesson runs in your own lab environment that opens in the browser.
- How do lessons work?
- Each lesson has 8 to 10 steps and a short quiz. Early steps show the command to type, later steps give less help; if two attempts fail, the solution opens.
- Can I try it for free?
- Yes. The first three lessons of every course are free.