Event streaming with Apache Kafka
You collect log and event streams with Kafka, replay them and consume them securely.
Level 301 · 18 lessons · 15 practice labs
Lessons
Fundamentals
- Why event streaming: You understand how request-response differs from event streaming and its place in security data.
- Topics, partitions and brokers: You grasp how Kafka divides and stores events and create and inspect topics with kafka-topics.
- Producing and reading events from the command line: You send and read your first event with kafka-console-producer and kafka-console-consumer.
- Quick task: create an alert topic and send events
Order and distribution
- Keys and partition routing: You see how events with the same key go to the same partition and keep their order.
- Offsets and replaying events: You track where a consumer stopped and re-read past events from the beginning.
- Sharing work with consumer groups: You let several consumers share partitions and process a stream together.
- Retention and compaction: You set how long events are kept and how old records are condensed.
- Quick task: replay the incident hour
With Python
- Producing JSON events with Python: You send structured events to Kafka from a Python program.
- Consuming and saving events with Python: You read events, validate them and write them to a local file.
- Bad events, redelivery and dead-letter topics: You set aside events that fail processing and account for redelivery possibly creating duplicates.
- Quick task: enrich events and write them to another topic
Security and operations
- Delivery guarantees and de-duplicating events: You learn what at-least-once delivery means and count logical events rather than raw records.
- Kafka security: TLS, SASL and ACLs: You learn to encrypt traffic, authenticate clients and grant per-topic permissions.
- Monitoring consumer lag
- Reviewing ACLs and access
- Capstone: a log collection pipeline: You stream log events into a topic, process them with consumer groups and write summary counts to a file.
Practice labs
- Create a topic and send events: Create a three-partition topic for badge-reader events, send ten events from the command line and read them from the beginning.
- Inspect partitions and leaders: Pull partition count, replication and leader information for existing topics and report the under-replicated ones.
- Fix a misconfigured topic: Reconfigure an audit topic created with one partition and one day of retention to meet the requirement and verify the change.
- Ordering by key: Send keyed events so one user's events keep their order and show they all land in the same partition.
- Replay from a point in time: Reset a consumer group's offset to the incident time, reprocess the events after it and verify the count.
- An unbalanced consumer group: Examine a group where one consumer gets no partitions, fix the partition-consumer ratio and show the lag clearing.
- Set the retention period: Reduce a raw log topic's retention to seven days and verify that old segments get deleted.
- Latest state with a compacted topic: Set up a compacted topic for device status updates and show only each device's latest status remains.
- Remove a record with a tombstone: Remove a user's record from a compacted topic with a tombstone and prove it can no longer be read after compaction.
- Produce JSON events with Python: Send a hundred schema-conformant login events with a Python producer and check with a consumer that the fields arrive correctly.
- Send bad events to a dead-letter topic: Write a consumer that, instead of crashing on broken JSON, writes the event and the error reason to a dead-letter topic.
- Prevent duplicates on redelivery: Write a consumer that removes, by event ID, the duplicates caused by a consumer crashing before it committed offsets and verify the result.
- Measure consumer lag: List every consumer group's lag and report the group and partition falling furthest behind.
- Write-only access to a topic with ACLs: Add ACLs that let the collector service account only write to the raw topic and show it cannot read any other.
- Audit overly broad ACLs: Find wildcard and unnecessary permissions in the current ACL list and write the commands that fix them for least privilege.
About this course
You collect log and event streams with Kafka, replay them and consume them securely.
What you will learn
- Fundamentals
- Order and distribution
- With Python
- Security and operations
Prerequisites
You should be comfortable with the level 101 and 201 topics of the same path.
Lab environment
Lessons run in your own Kali Linux terminal in the browser, connected to a private lab network. Nothing to install.
Frequently asked questions
- Is this course right for my level?
- The course is at level 301; it assumes you know the topics of the lower-level courses.
- Is there anything to install?
- No. Every lesson runs in your own lab environment that opens in the browser.
- How do lessons work?
- Each lesson has 8 to 10 steps and a short quiz. Early steps show the command to type, later steps give less help; if two attempts fail, the solution opens.
- Can I try it for free?
- Yes. The first three lessons of every course are free.