Alerting and on-call: Alertmanager, SLOs and post-incident reviews
You write quiet alert rules, route and silence with Alertmanager, and decide what deserves an alert from data, using SLOs and error budgets.
Level 401 · 14 lessons · 12 practice labs
Lessons
Alert rules
- Good alerts and bad alerts: symptoms or causes
- Prometheus alert rules: for, labels and annotations
- Testing alert rules with promtool
- Short task: fix five pointless alerts that fire every night
Alertmanager
- The routing tree and receivers
- Grouping, inhibition and silences
- A runbook for every alert
- Short task: send database alerts to the right team without noise
SLOs
- SLI, SLO and SLA: reliability from the user's view
- Error budgets and burn-rate alerts
- Short task: define an SLO and a burn-rate alert for an internal web service
After the incident
- Healthy on-call: handovers, escalation and fatigue
- Blameless post-incident reviews
- Capstone: rebuild a service's alerting around its SLO
Practice labs
- Write a rule that fires when a target is down: Write an alert rule for a target unresponsive for two minutes, stop the target and watch the rule fire.
- Alert on the symptom instead of the cause: Replace a CPU threshold alert with rules on user-facing error rate and latency and test them with fault injection.
- Write unit tests for alert rules: Write tests with sample series for a rule file and verify the cases that should and should not fire.
- Route alerts by team: Write a routing tree that sends database alerts to one receiver and web alerts to another by label.
- Calm an alert storm with grouping and inhibition: Collapse dozens of alerts from one server failure into a single notification and inhibit lower alerts while the top-level one fires.
- Set up narrow, timed silences for maintenance: Script a timed silence covering only the affected server and alerts for planned maintenance and verify alerts return afterwards.
- Define an availability indicator for a service: Write the query giving the demo web service's successful request ratio and submit the last hour's value.
- Compute the error budget and what remains: Compute the thirty-day error budget for a 99.9 percent target and show what remains after an injected fault.
- Build a multi-window burn rate alert: Write two-window alerts for fast and slow budget burn and show a brief error burst does not page needlessly.
- Extract an incident timeline from metrics: From the graphs determine with times when an injected fault began, when the alert fired and when it recovered.
- Attach a runbook and context to an alert: Add to the alert annotation the affected service, the current value and a runbook link with the first three checks.
- Write a blameless incident review: For the outage on Kuzeyyıldızı Kargo's demo service write a review with timeline, detection delay, contributing factors and measurable actions.
About this course
You write quiet alert rules, route and silence with Alertmanager, and decide what deserves an alert from data, using SLOs and error budgets.
What you will learn
- Alert rules
- Alertmanager
- SLOs
- After the incident
Prerequisites
You should have practised the topics of the same path up to level 301.
Lab environment
Lessons run in your own Kali Linux terminal in the browser, connected to a private lab network. Nothing to install.
Frequently asked questions
- Is this course right for my level?
- The course is at level 401; it assumes you know the topics of the lower-level courses.
- Is there anything to install?
- No. Every lesson runs in your own lab environment that opens in the browser.
- How do lessons work?
- Each lesson has 8 to 10 steps and a short quiz. Early steps show the command to type, later steps give less help; if two attempts fail, the solution opens.
- Can I try it for free?
- Yes. The first three lessons of every course are free.