Performance analysis with perf and eBPF
You sample with perf and draw flame graphs, watch the inside of the kernel and applications live with bpftrace and BCC, and find problems with evidence rather than guesses.
Level 401 · 17 lessons · 15 practice labs
Lessons
perf
- perf stat: hardware counters
- Sampling with perf record and report
- Flame graphs: where the time goes
- Symbols, stack unwinding and frame pointers
- Short task: find the CPU-burning function in a flame graph
eBPF basics
- What eBPF is: small programs that run safely in the kernel
- BCC tools: execsnoop, opensnoop, biolatency
- bpftrace one-liners
- Probe types: tracepoints, kprobes and uprobes
- Short task: catch the short-lived processes wearing the server down
Latency analysis
- Off-CPU time: measuring waiting
- Disk latency histograms
- Scheduler latency: runqlat
- Writing your own bpftrace script
- Short task: find the cause of a p99 latency spike
Wrap-up
- Tracing in production: overhead and safety
- Capstone: prove the root cause on a slowing API server
Practice labs
- Find the hottest function with perf top: Submit the name of the function with the most samples in a CPU-hungry process.
- Record and interpret a profile with call stacks: Sample a service for thirty seconds with call stacks and extract from the report the call chain spending most of the time.
- Compare two versions with a flame graph: Produce flame graphs from profiles of the old and new versions and use a differential graph to identify the function that brought the slowdown.
- Catch short-lived processes: Use execsnoop to find a short-lived command started dozens of times a second and the parent launching it.
- Trace failed file opens: Filter failing open calls with opensnoop and identify the path of a missing file a service keeps looking for.
- Write your own counter with bpftrace: Write one-liners that count system calls by process name and show the distribution of write sizes, and interpret the output.
- Measure the disk latency distribution: Produce the block I/O latency histogram with biolatency and submit the most populated bucket.
- Measure scheduler latency and off-CPU time: Measure run queue wait with runqlat and find what a service is waiting on with an off-CPU profile.
- Hunt the source of tail latency across layers: Trace slow requests in the 99th percentile with file system, block and scheduler tools and prove in which layer the delay arises.
- Trace outgoing TCP connections per process: Use tcpconnect to list which process connects to which address and port and mark the unexpected destination.
- Measure retransmissions and connection lifetimes: Use tcpretrans and tcplife to identify the destination with retransmissions and the service opening short-lived connections.
- Catch name resolution delay with your own tracer: With a bpftrace script probing entry and return of the resolver function produce a latency histogram and find the process issuing slow lookups.
- Measure a tracing tool's overhead: Run the same workload with tracing off and on and report the slowdown the tool adds as a percentage.
- Fix missing symbols and broken stacks: Find why a profile shows only addresses, then set up debug symbols and the unwinding method to get readable stacks.
- Solve an unknown slowdown end to end with the tools: Investigate a slow service in a prepared scenario with perf and eBPF tools, prove the root cause with three independent measurements and show the fix's effect.
About this course
You sample with perf and draw flame graphs, watch the inside of the kernel and applications live with bpftrace and BCC, and find problems with evidence rather than guesses.
What you will learn
- perf
- eBPF basics
- Latency analysis
- Wrap-up
Prerequisites
You should have practised the topics of the same path up to level 301.
Lab environment
Lessons run in your own Kali Linux terminal that opens in the browser. Nothing to install; the lab starts clean in every lesson.
Frequently asked questions
- Is this course right for my level?
- The course is at level 401; it assumes you know the topics of the lower-level courses.
- Is there anything to install?
- No. Every lesson runs in your own lab environment that opens in the browser.
- How do lessons work?
- Each lesson has 8 to 10 steps and a short quiz. Early steps show the command to type, later steps give less help; if two attempts fail, the solution opens.
- Can I try it for free?
- Yes. The first three lessons of every course are free.