Hard failures in production
You solve hard failures such as memory leaks, OOM loops, hung I/O, descriptor exhaustion, clock skew and kernel warnings under pressure and on the evidence.
Level 501 · 17 lessons · 15 practice labs
Lessons
Memory
- Proving a memory leak: RSS and smaps over time
- The OOM loop: a service that restarts and dies again
- Kernel memory: slab and hidden consumers
- Short task: fix a service that leaks at midnight, both now and for good
Exhaustion
- "Too many open files": descriptor exhaustion
- Hitting the process and thread limits
- Ephemeral port and inode exhaustion
- Short task: find what has run out on a server that refuses new connections
Hangs
- Hung I/O and D state: storage or network
- hung_task and soft lockup warnings
- Clock skew and strange time-related bugs
- Short task: rescue a server where even SSH barely opens
Evidence and communication
- Collecting evidence under pressure: what to capture
- The incident commander role and status updates
- Writing up the root cause and lasting fixes
- Last resort: a safe reboot with SysRq
- Capstone: find the chained root causes in order and bring the service back
Practice labs
- Find the process whose memory grows over time: Sample processes' memory use at intervals and submit the name of the one that keeps growing.
- Analyse an OOM event from the kernel log: From the memory table in the kernel log determine which process was chosen and why, and who held the memory at that moment.
- Tell a leak from kernel and cache usage: Trace memory use not explained by the sum of processes through slab, shared memory and tmpfs figures and prove the real consumer.
- Confirm file descriptor exhaustion: Compare the descriptor count and limit of a service failing with too many open files.
- Tell process exhaustion from ephemeral port exhaustion: On a server that cannot open new connections check the task limit and ephemeral port range, find which is exhausted and its source.
- Diagnose invisible kernel tables filling up: Tie intermittent packet drops and watch errors to the connection tracking table and inotify limits, prove the limits and tune them persistently.
- Find the mount that freezes commands: Identify the unresponsive network mount that makes directory listings and disk reports hang.
- Tie uninterruptible processes to storage: On a server with high load but idle CPU, find from kernel stacks which device the processes in D state are waiting on.
- Solve a chain of failures caused by clock skew: Find the clock offset that is the shared cause of authentication, certificate and scheduled job failures and apply a plan to correct it without jumping time.
- Capture evidence before intervening: Before restarting a faulty service save process state, open files, connections and recent logs to a timestamped directory.
- Separate and apply mitigation and permanent fix: On a leaking service apply a mitigation that limits impact first, then write the permanent fix and its verification criterion.
- Write the root cause report with evidence: For the outage on Yelkovan Bankası's payment server prepare a report with timeline, evidence, five-whys analysis and actions.
- Set up triggered capture for an intermittent fault: Set up a small watcher script that records process and connection state automatically when load passes a threshold.
- Match a periodic slowdown to its trigger: Match a slowdown arriving at fixed intervals with timers, cron jobs and log rotation on a time axis.
- Untangle a fault with two independent causes: In an outage with two separate causes where fixing one reveals the other, separate the symptoms, prove each cause and fix them in turn.
About this course
You solve hard failures such as memory leaks, OOM loops, hung I/O, descriptor exhaustion, clock skew and kernel warnings under pressure and on the evidence.
What you will learn
- Memory
- Exhaustion
- Hangs
- Evidence and communication
Prerequisites
You should be experienced in the topics of the same path up to level 401.
Lab environment
Lessons run in your own Kali Linux terminal that opens in the browser. Nothing to install; the lab starts clean in every lesson.
Frequently asked questions
- Is this course right for my level?
- The course is at level 501; it assumes you know the topics of the lower-level courses.
- Is there anything to install?
- No. Every lesson runs in your own lab environment that opens in the browser.
- How do lessons work?
- Each lesson has 8 to 10 steps and a short quiz. Early steps show the command to type, later steps give less help; if two attempts fail, the solution opens.
- Can I try it for free?
- Yes. The first three lessons of every course are free.