Server monitoring with Prometheus and Grafana
You collect metrics with node_exporter, query them with PromQL, build Grafana dashboards and watch your servers' health on one screen.
Level 301 · 18 lessons · 15 practice labs
Lessons
Collecting metrics
- Metrics versus logs: what a time series is
- Prometheus architecture: the pull model and targets
- Server metrics with node_exporter
- Scrape configuration and labels
- Adding your own metric: the textfile collector
- Short task: bring three servers into monitoring and check target health
PromQL
- PromQL selectors and range vectors
- rate and increase: reading counters correctly
- Aggregation: sum, avg and by
- Forecasting a full disk with predict_linear
- Short task: answer five operations questions with PromQL
Grafana
- Data source and the first dashboard
- Dashboard variables: a host picker
- Good dashboard design: the USE method
- Short task: an at-a-glance dashboard for the on-call engineer
Wrap-up
- Retention and disk planning
- Exporter security: who can see the metrics
- Capstone: set up monitoring for five servers and spot the troubled one from the dashboard
Practice labs
- Add a server as a Prometheus target: Add a server running node_exporter to the scrape config and verify the target shows healthy.
- Find why a target shows down: Starting from the error on the targets page find and fix a wrong port and a firewall problem.
- Publish your own metric through the textfile collector: Set up a script and timer that writes the last successful backup time as a metric and verify it appears with the right type and label.
- Compute free disk percentage with PromQL: Write the query that gives the free space percentage of each server's root file system.
- Produce CPU usage from a counter: Use rate on the CPU time counter to compute usage percentage per server and find the busiest one.
- Predict with a query when a disk will fill: Write a query that finds file systems due to fill within four hours from the last six hours' trend and filter out false positives.
- Create a first metric panel in Grafana: Add the Prometheus data source and save a time series panel showing servers' memory use.
- Build a templated dashboard with a host selector: Create a dashboard with a label-driven host variable containing CPU, memory, disk and network panels.
- Provision a dashboard from a file under version control: Export a dashboard as JSON, load it automatically through provisioning and show a manual UI change does not persist.
- Turn a heavy query into a recording rule: Write a recording rule for a frequently used CPU percentage query and verify the new series exists.
- Drop unneeded series with relabelling: Drop metrics for virtual and temporary file systems at scrape time and measure the fall in series count.
- Size retention and disk requirements: Compute daily disk use from series count and scrape interval, set retention accordingly and verify by measurement.
- Manage targets with file-based discovery: Move the server list to file-based discovery and show a newly added server is scraped without a restart.
- Add role and environment labels to targets: Tag servers with role and environment labels and write a query aggregating production web servers only.
- Protect metric endpoints with authentication and TLS: Run node_exporter with TLS and basic authentication, configure Prometheus to match and show unauthenticated access is refused.
About this course
You collect metrics with node_exporter, query them with PromQL, build Grafana dashboards and watch your servers' health on one screen.
What you will learn
- Collecting metrics
- PromQL
- Grafana
- Wrap-up
Prerequisites
You should be comfortable with the level 101 and 201 topics of the same path.
Lab environment
Lessons run in your own Kali Linux terminal in the browser, connected to a private lab network. Nothing to install.
Frequently asked questions
- Is this course right for my level?
- The course is at level 301; it assumes you know the topics of the lower-level courses.
- Is there anything to install?
- No. Every lesson runs in your own lab environment that opens in the browser.
- How do lessons work?
- Each lesson has 8 to 10 steps and a short quiz. Early steps show the command to type, later steps give less help; if two attempts fail, the solution opens.
- Can I try it for free?
- Yes. The first three lessons of every course are free.