RHCE Enterprise Linux: troubleshooting and recovery
On a production Red Hat server you collect evidence first, find the root cause, and repair boot, storage, package, network and application faults.
Level 401 · 25 lessons · 21 practice labs
Lessons
Evidence, boot and hardware
- Baseline evidence: support bundles and the web console: You will gather a support bundle with sos and record system state in the Cockpit web console to preserve evidence.
- Narrowing the root cause with docs, logs and metrics: You will start from a symptom and combine logs and metrics to keep an investigation trail that tests a hypothesis.
- Central logging with rsyslog and a persistent journal: You will forward logs to a remote server and configure a journal that survives reboots.
- Repairing a service that fails at boot, and its dependency: You will find the fault in the systemd dependency chain and fix the unit that blocks the boot.
- Recovering administrator access and the boot target in emergency mode: You will enter emergency mode in an authorised maintenance window and restore administrator access and the right target.
- Repairing bootloader and kernel parameter drift: You will compare grubby and GRUB settings and restore a broken kernel argument and module setting.
- Reading hardware failure signals: You will spot disk, memory and device errors in dmesg, smartctl and kernel logs.
- Short task: bring back a server that won’t boot while collecting evidence
Storage and package recovery
- Repairing a corrupted file system: You will safely check and repair a corrupted file system with xfs_repair and e2fsck.
- Bringing back invisible LVM volumes and mount state: You will find a missing volume with pvs, vgs and lvs, activate the volume group and recover the mount.
- Recovering data from an encrypted volume: You will inspect the LUKS header and key slots and open the volume with the authorised passphrase to reach the data.
- Repairing iSCSI discovery and persistent login: You will fix target discovery and session settings so the disk attaches automatically at boot.
- Repository and dependency problems, and rolling back a failed update: You will read a dnf error, resolve the dependency and roll back a failed update with dnf history.
- Repairing the RPM database and transaction state: You will rebuild a damaged RPM database and clean up unfinished transactions.
- Finding and restoring modified package files: You will detect files that differ from their package with rpm -V and replace them with the original.
- Short task: recover a broken volume and a half-finished update without losing data
Network, identity and applications
- Repairing NetworkManager, DNS and routing drift: You will find and fix broken address, DNS and route settings with nmcli, resolvectl and ip route.
- Examining traffic and the firewall service path: You will show at which stage a connection drops using tcpdump and firewalld logs.
- Finding application library and runtime failures: You will diagnose a crash caused by a missing library or wrong version with ldd, ldconfig and journalctl.
- Repairing a containerised service under a SELinux denial: You will find why a containerised service is denied with ausearch and Podman and correct the context safely.
- Login problems from PAM and identity lookup (sssd, nsswitch): You will trace a user’s login failure through the PAM stack and the identity source configuration.
- Tracing process and memory problems with strace and SystemTap: You will follow system calls with strace and compile a read-only SystemTap probe to find the source of slowness.
- Collecting kernel crash evidence with kdump: You will enable kdump and prepare the dump from a crash for first inspection with the crash tool.
- Short task: find the shared cause of a user who can’t log in and a service that won’t run
Wrap-up
- Capstone: solve a multi-layer failure with evidence: You will collect evidence, repair in order a fault that combines boot, storage and service problems, and write an incident note.
Practice labs
- Read why a server dropped to emergency mode: On a server that boots into emergency mode you find the failed unit in the journal and report the cause in one line.
- Fix a server booting the wrong kernel: On a server that boots an old kernel with stray arguments after an update you repair the default entry and arguments with grubby.
- Make a system that cannot find its root disk boot again: You boot a system unable to mount root over a missing driver from the rescue environment, rebuild the initramfs and return it to normal while collecting evidence.
- List failed units and find why: With systemctl and journalctl you find two services that failed at boot and extract each one's error message.
- Order a service that starts before the network is ready: You give an application service that dies by starting before the network the right dependency and ordering directives in a drop-in and test with a reboot.
- Untangle a dependency cycle between three units: You visualise with systemd-analyze the cycle formed by three units waiting on each other, remove the wrong directive and compare boot times.
- Repair an XFS volume that won't mount with xfs_repair: You detach a data volume that errors on mount, repair it with xfs_repair and verify the data is in place.
- Bring back a volume group that has vanished: You reactivate a volume group hidden by a device filter and deactivated volumes and repair its mount state.
- Recover data from a LUKS volume with a damaged header: You restore the header backup of a LUKS volume with a damaged header, open it, fix the crypttab entry and verify the data.
- Undo a troublesome update from dnf history: You find the last update transaction in dnf history, undo it and verify the service runs on the old version.
- Rebuild a corrupted RPM database: On a server where package queries error out you back up and rebuild the RPM database and finish the interrupted transaction.
- Resolve a dependency conflict and pin the version: You resolve a conflict caused by a third-party repository package using repo priority and excludes, lock the working version and retry the update.
- Fix persistently a DNS setting NetworkManager overwrote: You fix persistently in the connection profile a DNS setting that was hand-edited and overwritten by NetworkManager.
- Fix traffic leaving by the wrong interface: You move traffic leaving via the backup network because of two default routes' metrics to the right interface through profile settings and prove it with a trace.
- Follow a packet along the path and find the rule dropping it: With tcpdump and firewalld logging you show a request arrives but no reply leaves and fix the zone assignment and rule error.
- Solve a user not found through the nsswitch order: You find the cause of a central user that getent cannot find in the nsswitch.conf order and fix it.
- Fix a locked account and its authselect setting: You clear the faillock record of a user unable to log in with the right password, find the wrong setting in the authselect profile and test login.
- Solve a podman service's volume denial without disabling SELinux: You trace in the audit log why a podman service cannot write to its mounted directory, fix it with the right label and boolean and prove the unit survives a reboot.
- Collect a support bundle and learn its layout: You generate an sos report from a faulty server and find the kernel version, mounted disks and latest error records in it.
- Trace a silently exiting application with strace: With strace you find the file a silently exiting application cannot open and its missing library and get it running.
- Gather crash and log evidence centrally: You forward logs over TCP to the central server with rsyslog, verify kdump's reserved memory and target and capture the dump of a test crash.
About this course
On a production Red Hat server you collect evidence first, find the root cause, and repair boot, storage, package, network and application faults.
What you will learn
- Evidence, boot and hardware
- Storage and package recovery
- Network, identity and applications
- Wrap-up
Prerequisites
You should have practised the topics of the same path up to level 301.
Lab environment
Lessons run in your own Kali Linux terminal in the browser, connected to a private lab network. Nothing to install.
Frequently asked questions
- Is this course right for my level?
- The course is at level 401; it assumes you know the topics of the lower-level courses.
- Is there anything to install?
- No. Every lesson runs in your own lab environment that opens in the browser.
- How do lessons work?
- Each lesson has 8 to 10 steps and a short quiz. Early steps show the command to type, later steps give less help; if two attempts fail, the solution opens.
- Can I try it for free?
- Yes. The first three lessons of every course are free.