Breakage scenarios: solve them yourself
On pre-broken servers you find the cause without hints, fix it and write a short report; each scenario tests a different subsystem.
Level 301 · 17 lessons · 15 practice labs
Lessons
Boot and disk
- The root file system came up read-only
- /var is full and services drop one by one
- The data disk doesn't mount at boot
- Short task: write up the three faults on one page
Services
- A service that keeps restarting
- After an update the service user can't read its own files
- Two services want the same port
- An app that runs by hand but not as a service
- Short task: report the findings and the lasting fixes
Accounts and access
- A broken sudoers file means nobody can become admin
- A user can't log in because of a wrong shell setting
- Broken home directory permissions log users straight back out
- Short task: find the common root of the access problems
Performance
- A swap storm: the server barely moves
- Cron jobs piling up on each other
- A log flood eating disk and CPU
- Capstone: fix five broken servers against the clock
Practice labs
- Bring back a data partition that will not mount: Read the mount error from the kernel log, fix the wrong file system type in fstab and mount the partition.
- Boot a server stuck in emergency mode: On a server dropping to emergency mode at boot find the failed mount unit, fix its cause and get a normal boot.
- Find why the root file system turned read-only: On a root that went read-only while running, trace the I/O error in the kernel log, check the file system safely and restore the services.
- Stop a service that keeps restarting and find why: Find the exit code and error line of a service in a restart loop and fix it by creating the missing directory.
- Fix an app that runs by hand but not as a service: Compare user, working directory and environment differences and complete what is missing in the unit file.
- Unblock a service denied by mandatory access control the right way: Find in the audit log why a service cannot read a custom directory and solve it with the right label and port definition without disabling protection.
- Find why a user cannot log in: Check the account's lock, password expiry and shell state to determine why login is refused, then fix it.
- Rescue a broken sudo: From the root console repair a sudo broken by a sudoers file with a syntax error and wrong permissions.
- Open a server nobody can reach over SSH from the console: On an SSH service refusing all logins after a config change, fix the allowed users, key file and PAM problems in turn.
- Find the cause when load is high but CPU idle: Show the high load average comes from I/O wait and identify the process loading the disk.
- Relieve a server deep in swap: Rank processes by swap use, limit the service consuming memory and measure that the server's response time recovers.
- Return a service slowed by a change to its old speed: Measure a service that slowed after yesterday's config change to find the faulty setting, fix it and compare before and after timings.
- Fix a server that cannot resolve names: On a server reachable by address but not by name find and fix the error in the resolver settings.
- Find what blocks a service that is listening: On a service answering locally but not remotely check and fix the listen address and firewall rule.
- Solve a three-layer fault in order: Fix a chain of a full log partition, the service it stopped and the port that closed with it, starting from the root cause, and close it documented.
About this course
On pre-broken servers you find the cause without hints, fix it and write a short report; each scenario tests a different subsystem.
What you will learn
- Boot and disk
- Services
- Accounts and access
- Performance
Prerequisites
You should be comfortable with the level 101 and 201 topics of the same path.
Lab environment
Lessons run in your own Kali Linux terminal that opens in the browser. Nothing to install; the lab starts clean in every lesson.
Frequently asked questions
- Is this course right for my level?
- The course is at level 301; it assumes you know the topics of the lower-level courses.
- Is there anything to install?
- No. Every lesson runs in your own lab environment that opens in the browser.
- How do lessons work?
- Each lesson has 8 to 10 steps and a short quiz. Early steps show the command to type, later steps give less help; if two attempts fail, the solution opens.
- Can I try it for free?
- Yes. The first three lessons of every course are free.