Kernel debugging: kdump, crash and ftrace
You capture a memory dump when the kernel crashes and examine it with crash, read oops, panic and lock-up messages, and trace what happens inside the kernel with ftrace.
Level 501 · 15 lessons · 12 practice labs
Lessons
Reading the messages
- Anatomy of an oops and a panic
- Tainted kernel flags
- Turning a stack trace into source lines
- Short task: find the module to blame from three oops messages
Dumps
- Setting up kdump: reserved memory and a second kernel
- Taking a dump and verifying it
- Examining a dump with crash
- Catching last messages with netconsole and a serial console
- Short task: find the cause of the lock-up in a prepared dump
Live tracing
- ftrace: the function and function_graph tracers
- trace-cmd and KernelShark
- Dynamic debug messages
- Short task: prove a kernel latency with ftrace
Wrap-up
- A good bug report to the distribution and developers
- Capstone: capture a dump from a server that keeps crashing, find the cause and apply a workaround
Practice labs
- Read the responsible module from a kernel error message: In a given oops output identify the type of fault, the function name and the module involved.
- Decode the kernel's taint flags: Break the taint value into its flags and find in the kernel log which event tainted the kernel and when.
- Find the source of a soft lockup warning: Compare the call traces of recurring lockup warnings and identify the common function and the workload that triggers it.
- Set up kdump and test that it works: Reserve memory for the crash kernel, enable the kdump service and produce a dump with a test crash on a disposable VM.
- Examine the moment of the crash in a prepared dump: Open the dump with the crash tool and extract the panic message, the name of the crashing task and its backtrace.
- Derive the root cause from a dump with evidence: Combine kernel log, task list and memory state in a prepared dump to write up the chain of events that led to the panic.
- Trace a function's calls with ftrace: Filter the function tracer to one kernel function and count how often it is called in ten seconds.
- Record scheduling latency with trace-cmd: Record scheduler events and find in the report the longest delay between a task waking and running.
- Capture function arguments with a dynamic probe: Put a probe on a kernel function in the file-open path, record the file name and return value and pick out the failing calls.
- Get information from a hung system with SysRq: From the serial console use SysRq to dump memory and task state to the kernel log and perform the safe reboot sequence.
- Tie a hung task warning to storage: From the trace of a task blocked for 120 seconds find which device it waits on and correlate it with I/O errors.
- Bisect a regression between kernel versions: Narrow a hang that appeared between two prepared kernels to a single change by testing the intermediate builds, and document the finding.
About this course
You capture a memory dump when the kernel crashes and examine it with crash, read oops, panic and lock-up messages, and trace what happens inside the kernel with ftrace.
What you will learn
- Reading the messages
- Dumps
- Live tracing
- Wrap-up
Prerequisites
You should be experienced in the topics of the same path up to level 401.
Lab environment
Lessons run in your own Kali Linux terminal that opens in the browser. Nothing to install; the lab starts clean in every lesson.
Frequently asked questions
- Is this course right for my level?
- The course is at level 501; it assumes you know the topics of the lower-level courses.
- Is there anything to install?
- No. Every lesson runs in your own lab environment that opens in the browser.
- How do lessons work?
- Each lesson has 8 to 10 steps and a short quiz. Early steps show the command to type, later steps give less help; if two attempts fail, the solution opens.
- Can I try it for free?
- Yes. The first three lessons of every course are free.