Server disaster recovery: back from bare metal
When a server is lost entirely, you bring it back from nothing with a documented, tested procedure: system images, configuration backups, Relax-and-Recover and recovery drills.
Level 401 · 14 lessons · 12 practice labs
Lessons
The plan
- DR scope and service tiers
- Writing a recovery runbook
- Backing up configuration: /etc, package lists, partition layout
- Short task: write the rebuild inventory for a critical server
Bare-metal restore
- Recovery images with Relax-and-Recover
- Restoring onto an empty machine: partitions, boot loader and data
- Restoring onto different hardware: driver and UUID problems
- Short task: bring back a deleted VM from backups alone
Drills
- Planning a recovery drill
- Measuring the real RTO and closing the gap
- Communication and decisions in a disaster
- Short task: a full recovery drill against a two-hour target
Wrap-up
- Updating the plan after a drill
- Capstone: bring three servers back in order after losing the data centre
Practice labs
- Draw up the service and dependency inventory: In a three-server environment tabulate the running services, their data directories and their dependencies on each other.
- Set the restore order from dependencies: For Kuzeyyıldızı Kargo's services write an ordered recovery plan showing what comes up first, with reasons.
- Find gaps in the runbook by following it: Follow a given recovery runbook step by step on an empty server and update it by fixing missing, outdated and vague steps.
- Record disk layout and package list for recovery: Export a server's partition table, LVM layout, fstab and installed package list in a form usable for recovery.
- Rebuild a wiped server from backup: On a server with a wiped disk recreate the partitions, restore the files from backup, install the boot loader and show it boots.
- Restore a backup onto a machine with a different disk layout: Restore a backup onto a machine with smaller, differently named disks, adapting fstab, initramfs and the boot loader to the new layout.
- Write the decision points of a tabletop drill: For a data-centre loss scenario list in order the decisions of the first hour, who owns them and the information needed.
- Measure recovery time against the target: Run a drill restoring the web and database servers from backup, record the time per stage and identify the slowest step.
- Measure real data loss and correct backup frequency: Find the time of the last record recovered in a drill to compute actual data loss, change the backup schedule if it exceeds the target and retest.
- Verify configuration is in the backup: Check the config files, certificates and scheduled jobs needed for recovery are in the backup and list what is missing.
- Test access to the keys needed in a disaster: Try to open the encrypted backup using only what is in the recovery envelope and identify the missing key or passphrase.
- Rebuild the whole environment from backups and report: Treat all three servers as wiped, bring them back in order from backups, pass the service checks and write a report with timings and issues.
About this course
When a server is lost entirely, you bring it back from nothing with a documented, tested procedure: system images, configuration backups, Relax-and-Recover and recovery drills.
What you will learn
- The plan
- Bare-metal restore
- Drills
- Wrap-up
Prerequisites
You should have practised the topics of the same path up to level 301.
Lab environment
Lessons run in your own Kali Linux terminal in the browser, connected to a private lab network. Nothing to install.
Frequently asked questions
- Is this course right for my level?
- The course is at level 401; it assumes you know the topics of the lower-level courses.
- Is there anything to install?
- No. Every lesson runs in your own lab environment that opens in the browser.
- How do lessons work?
- Each lesson has 8 to 10 steps and a short quiz. Early steps show the command to type, later steps give less help; if two attempts fail, the solution opens.
- Can I try it for free?
- Yes. The first three lessons of every course are free.