Outage drills
You work like the on-call engineer: an alert arrives, the clock runs and problems chain across several servers; you handle diagnosis, communication and the lasting fix together.
Level 401 · 16 lessons · 15 practice labs
Lessons
Getting ready
- Drill rules: roles, timing and communication
- The on-call toolkit and the first five minutes
- Short task: a warm-up drill, one fault on one server
First drills
- A site whose certificate expired at midnight
- One full disk, chained errors in three services
- A bad release: roll back or fix forward
- The resolver changed and apps can't find each other
- The clock drifted and authentication broke
- Short task: draw the timeline of the first five drills
Hard drills
- An application tier slowing down under memory pressure
- Bringing deleted data back from backup
- An unknown cron job connects out at night
- A server that won't boot after a kernel update
- Short task: assess the status updates you wrote during the drills
Wrap-up
- From drill to improvement: update the runbooks
- Capstone: a two-hour, multi-server major outage drill
Practice labs
- Map the environment's services: Tabulate which ports the web, app, database and resolver hosts use to talk to each other.
- Record measurements of the healthy state: Write a check script recording normal response time, error rate and resource use for each tier.
- Set up roles and communication for a drill: Prepare a drill plan with incident commander, responder and scribe roles, decision rights and notification templates.
- Solve the web server down drill: In the injected fault find why the site does not answer, restore the service and record the outage duration.
- Restore service when the database disk fills: Safely free space on a full data partition causing write errors, start the database consistently and verify the application recovers.
- Manage a resolver failure's effect on every tier: Identify the tiers that slow and fail when resolution stops, apply a workaround, repair the resolver and improve the timeout settings.
- Identify affected users in a partial degradation: When only some requests fail find the faulty app instance from the logs and take it out of the pool.
- Trace cascading timeouts to their source: Trace timeouts at the web tier back through app and database metrics, find the slow query and ease the load.
- Manage two faults injected at once: With network delay and certificate expiry injected together, separate the symptoms, prioritise and fix both while keeping a log.
- Extract the drill's timeline: From the records list with minutes when the fault began, was detected, was first acted on and recovered.
- Measure the detection gap and add a check: Compute the time between the fault starting and being noticed and set up a check that catches the same fault within a minute.
- Produce a runbook from a drill and retest: Turn a solved drill into a step-by-step runbook, re-inject the fault, solve it by the runbook alone and report the time saved.
- Roll back a bad deployment: Match errors that began with a new release to the deploy time and verify recovery by returning to the previous version.
- Fix a bad config pushed to every server: Bring back services brought down by a bad setting pushed in bulk, one host at a time with verification at each step.
- Lead an end-to-end drill and write its review: In the Turnalı Perakende scenario lead a multi-tier outage with assigned roles, restore service and write a review with action items.
About this course
You work like the on-call engineer: an alert arrives, the clock runs and problems chain across several servers; you handle diagnosis, communication and the lasting fix together.
What you will learn
- Getting ready
- First drills
- Hard drills
- Wrap-up
Prerequisites
You should have practised the topics of the same path up to level 301.
Lab environment
Lessons run in your own Kali Linux terminal in the browser, connected to a private lab network. Nothing to install.
Frequently asked questions
- Is this course right for my level?
- The course is at level 401; it assumes you know the topics of the lower-level courses.
- Is there anything to install?
- No. Every lesson runs in your own lab environment that opens in the browser.
- How do lessons work?
- Each lesson has 8 to 10 steps and a short quiz. Early steps show the command to type, later steps give less help; if two attempts fail, the solution opens.
- Can I try it for free?
- Yes. The first three lessons of every course are free.