Advanced Kafka: Connect, Streams and operations
You connect sources and sinks with Kafka Connect, produce contract-bound events with a schema registry, enrich with Kafka Streams, and run the cluster securely, observably and resiliently.
Level 401 · 16 lessons · 12 practice labs
Lessons
Architecture in depth
- KRaft: controllers and metadata
- Replication, ISR and leader election
- Producer tuning: batching, compression, idempotence
- Quick task: avoid data loss when a broker fails
Integration and schema
- Kafka Connect: source and sink connectors
- The schema registry and Avro
- Schema compatibility rules
- Quick task: feed syslog into a topic with a connector
Streaming applications
- Kafka Streams: streams and tables
- Enriching a stream with a table
- Windowed counts and late events
- Quick task: match events against threat intelligence
Operations
- Quotas and multi-tenancy
- Monitoring with JMX metrics
- Cross-cluster mirroring and disaster recovery
- Capstone: build and harden an enterprise log backbone
Practice labs
- Read the ISR state: Inspect topics' leaders and in-sync replica lists and report the partitions whose in-sync replicas have shrunk.
- Producer settings that lose no data: Configure acks, min.insync.replicas and idempotence so no event is lost when a broker goes down and prove it by count.
- Watch a leader election: Stop a broker and report the leader election, the controller logs and the producer's interruption time.
- A file source connector: Set up a source connector that streams a log file into a topic with Kafka Connect and verify its status over REST.
- Register and validate an Avro schema: Add an Avro schema for a login event to the registry and show an event that breaks it being rejected at production.
- A schema change that breaks compatibility: Explain why a schema version that adds a required field is rejected, then redesign it to be backward compatible and register it.
- Filter and reroute a stream: Use a streams application to route only high-severity events to a separate topic and compare the counts.
- Enrich a stream with a table: Join an event stream with an asset table, add owner and criticality to each event and write to a new topic.
- Windowed counts and late events: Count per source in one-minute windows and show with test data how events arriving outside the grace period are handled.
- Put a quota on a client: Define a byte-rate quota for a noisy producer and show the throttling in the metrics.
- A health view from JMX metrics: Write a script that collects under-replicated partitions, request latency and consumer lag metrics and lists those over threshold.
- Cross-cluster mirroring and failover: Mirror a topic to a second cluster, fail the consumer over to it and verify with offset translation that no events were lost.
About this course
You connect sources and sinks with Kafka Connect, produce contract-bound events with a schema registry, enrich with Kafka Streams, and run the cluster securely, observably and resiliently.
What you will learn
- Architecture in depth
- Integration and schema
- Streaming applications
- Operations
Prerequisites
You should have practised the topics of the same path up to level 301.
Lab environment
Lessons run in your own Kali Linux terminal in the browser, connected to a private lab network. Nothing to install.
Frequently asked questions
- Is this course right for my level?
- The course is at level 401; it assumes you know the topics of the lower-level courses.
- Is there anything to install?
- No. Every lesson runs in your own lab environment that opens in the browser.
- How do lessons work?
- Each lesson has 8 to 10 steps and a short quiz. Early steps show the command to type, later steps give less help; if two attempts fail, the solution opens.
- Can I try it for free?
- Yes. The first three lessons of every course are free.