Centralised log management: Loki, Promtail, Grafana
Collecting, keeping and reviewing logs with rsyslog.
Level 301 · 16 lessons · 15 practice labs
Lessons
Foundations
- Why central logging with Loki: You will understand why collecting server logs in one place with Loki matters and how scattered logs make investigating an incident harder.
- Sending logs over the network with rsyslog
- Setting up Loki: You will learn to install Loki, a tool that stores and searches logs, in your own lab and check that it runs.
- Shipping logs to Loki with Promtail: You will learn to send a server's logs to the central Loki with Promtail and confirm that they really arrive.
- Promtail pipeline stages: regex, json and labels
- Short task: ship auth.log to the centre and confirm it arrives
Querying
- Grafana Explore and the Loki data source: You will learn to add Loki as a data source in Grafana and browse the collected logs from the Explore screen.
- Loki LogQL: stream selectors and filters: You will learn to use LogQL in Loki to pick which server's log stream to look at and filter out the lines that matter.
- Loki LogQL: metrics from logs: You will learn to use LogQL in Loki to count log lines and turn them into measures, such as errors or login attempts per second, and chart them.
- Loki label design: You will understand how to organise logs in Loki with labels such as server, service and environment, and what goes wrong with too many labels.
- Short task: find the service producing the most errors with LogQL
Running it
- Catching an error spike with Loki: You will learn to notice a sudden jump in errors in your Loki logs and catch it automatically with an alert rule.
- Loki alert rules and notifications
- Retention, disk and cost
- Secrets and access control in the log pipeline
- Capstone: collect three servers' logs in one place with Loki: In a fictional company you collect the logs of three servers in Loki in one place and follow an event across them in a small task.
Practice labs
- Ship one server's logs to the centre: Configure Promtail to send the system log to Loki and verify with a query that the first entries arrived.
- Design labels without exploding cardinality: Add host, service and environment labels to two servers' logs and remove a high-cardinality field such as user ID from the labels.
- Parse logs in the pipeline and mask secrets: Split the access log into fields with pipeline stages, turn the status code into a label and mask tokens before shipping.
- Find a service's errors with LogQL: Use a label selector and line filter to list the payment service's error lines in the last hour.
- Produce an error rate metric from logs: Compute the per-minute request rate by status code and find the window in which the 5xx rate spiked.
- Follow one request across two servers' logs: Using the request ID join proxy and app logs and show with timings in which tier the delay arose.
- Create a log panel in Grafana: Add the Loki data source and save a panel showing the error lines of a selected server.
- Build a dashboard with host and service selectors: Create a dashboard with host and service variables showing an error rate graph next to the raw lines.
- Write an alert rule from a log query: Write a rule that fires when failed logins pass a threshold in five minutes, trigger it with test entries and tune a quiet threshold.
- Set the log retention period: Reduce Loki's retention to thirty days and verify the setting is active from the config and logs.
- Find why a server's logs are not arriving: Find and fix why one server sends no entries from the shipper's status, positions file and rejected requests.
- Measure log volume and cut noise at the source: Compute volume per service, drop the noisiest source's debug lines at shipping and report the disk saving.
- Encrypt log shipping with TLS: Switch the connection between shipper and centre to TLS and show the certificate is verified.
- Let teams see only their own logs: Define separate tenants for two teams, bind their dashboards and verify one team cannot read the other's entries.
- Build an audit-ready central logging setup: For two Yelkovan Bankası servers ship auth and sudo logs encrypted, keep them a year, restrict access and add a check that tests for lost entries.
About this course
Collecting, keeping and reviewing logs with rsyslog.
What you will learn
- Foundations
- Querying
- Running it
Prerequisites
You should be comfortable with the level 101 and 201 topics of the same path.
Lab environment
Lessons run in your own Kali Linux terminal in the browser, connected to a private lab network. Nothing to install.
Frequently asked questions
- Is this course right for my level?
- The course is at level 301; it assumes you know the topics of the lower-level courses.
- Is there anything to install?
- No. Every lesson runs in your own lab environment that opens in the browser.
- How do lessons work?
- Each lesson has 8 to 10 steps and a short quiz. Early steps show the command to type, later steps give less help; if two attempts fail, the solution opens.
- Can I try it for free?
- Yes. The first three lessons of every course are free.