GuideIntermediate
Streaming With Kafka And Flink
This guide opens by proving, with real code, the most expensive and best-documented mistake in streaming: recomputing in Kafka+Flink the exact same session funnel Kiosko already calculated in batch — and landing on the identical `35.3%`, with more infrastructure and zero new value. From that proof, the guide draws the real line: streaming earns its cost on the handful of problems batch, by design, cannot solve, no matter how much you optimize it. Running Apache Kafka (4.x, KRaft mode, $0 locally) and Apache Flink/PyFlink for real in Docker, you solve four concrete Kiosko problems: detecting in minutes that a store stopped reporting (the star case, against a daily DAG that wouldn't notice until tomorrow), capturing a price change straight from Postgres's write-ahead log with Debezium (CDC) instead of hand-declaring it in Python, building event-time windows with watermarks over a continuous stream, and keeping incremental state without re-summing the entire history on every run. The guide is honest about Flink's market gap and answers it with fresh evidence (Kafka's salary premium, ~33% of UK senior streaming roles) without overstating Flink's role beyond what Kiosko genuinely needs.
- 64
- lessons
- 8
- modules
- English · Spanish
- available in
- Yes
- certificate
- Free
- access
Outcomes
What you'll be able to do
- Reproduce in Kafka+Flink the same `page_view → add_to_cart → purchase` funnel a batch pipeline already computed, and verify it lands on the same result — the empirical proof of when streaming adds NO value
- Install Kafka in KRaft mode with Docker (no ZooKeeper) and PyFlink locally, $0, no cloud account
- Explain topics, partitions, and offsets as the real unit of order and parallelism, and choose a Python Kafka client with criteria
- Build a deterministic, finite producer and consumer that turn fixed order files into a real stream, verifying the same known total
- Distinguish processing time from event time, and use watermarks to deterministically close tumbling windows over `order_ts`
- Contrast a tumbling window (recomputable) with keyed incremental state (a running total that never re-sums the full history)
- Detect live, with a session window that closes on silence, that a store stopped reporting — and quantify the contrast in minutes against a daily DAG
- Capture a real Postgres `UPDATE` with Debezium (CDC) and apply it with `upsert()` to an existing Iceberg table, closing the bridge between the transactional system and the warehouse
- Tell apart infrastructure metadata that isn't reproducible (Kafka/Debezium offsets, timestamps) from the business content that is
Before you start
What you need to bring
It's for you if...
- Data engineers who need to know, with evidence rather than trend-chasing, when a problem genuinely needs streaming and when it's batch in disguise
- Teams evaluating Kafka for use cases where latency IS the product (operational alerts, fraud, syncing with a transactional system)
- Anyone preparing for senior data engineering roles where Kafka/Flink already show up as a hard requirement, especially in fintech and logistics
- Data engineers who completed `lakehouse-and-iceberg-guide` and want to close the CDC thread that guide left named but unbuilt
Requirements and materials
- Intermediate Python; comfortable with the command line and Docker/Docker Compose
- Ideally completed `lakehouse-and-iceberg-guide` (the Iceberg `dim_product` table the CDC module feeds) and `dbt-analytics-engineering-guide` (the session funnel Module 1 recomputes)
- Docker installed and running locally (Kafka, plus Postgres + Debezium only in the CDC modules)
Content
The syllabus, module by module
Open any of them to see its lessons.
- Module introduction: when latency IS the product
- The weak flag: streaming that recomputes batch
- What the 2026 market really pays for
- When latency genuinely IS the product
- Installing Kafka in KRaft mode with Docker
- Installing PyFlink locally
- Proving it: the same 35.3% funnel in Kafka and Flink
- Project: Kiosko's first Kafka and Flink "hello world"
- Module introduction: Kafka's real anatomy
- A topic is an ordered, append-only log
- Partitions, and why order is only guaranteed within one
- Offsets: the bookmark of a consumer
- Choosing a Python client: `confluent-kafka` vs. `kafka-python`
- Creating Kiosko's `kiosko.orders` topic
- Producing and consuming your first message
- Project: Kiosko's first Kafka topic
- Module introduction: from fixed file to real stream
- From fixed files to a message stream
- Keying messages by `store_id`
- Building a deterministic, finite producer
- Building a consumer that verifies the count
- Replaying the canonical week through Kafka
- Verifying the same `106.15` total again
- Project: Kiosko's order stream in Kafka
- Module introduction: this guide's heart
- Processing time vs. event time
- Why the wall clock is banned in this guide
- Watermarks: how Flink knows it's safe to close a window
- Bounded out-of-orderness in PyFlink
- Tumbling windows on `order_ts`
- Reproducing Kiosko's daily totals with a windowed job
- Project: Kiosko's first deterministic window
- Module introduction: two ways to aggregate, one real advantage
- Revenue per window, per store
- Keyed state: the other way to aggregate
- Running units sold per product, without recomputing everything
- Why incremental beats recomputing the entire history
- Verifying both aggregates against the known totals
- What this gives you that batch doesn't
- Project: Kiosko's live sales and running inventory signal
- Module introduction: this guide's flagship case
- Session windows: a window that closes on silence
- Designing a "heartbeat" check for a store
- Kiosko's August 10: when S02 goes quiet
- Building the silence detector in PyFlink
- The alert: store gone silent, and when it fires
- Minutes vs. tomorrow: comparing against the daily DAG
- Project: Kiosko's live operational alert
- Module introduction: the thread another guide left open on purpose
- What CDC solves that a periodic `SELECT` can't
- Debezium and the write-ahead log
- Setting up Postgres, Kafka Connect, and Debezium with Docker
- Capturing `P002`'s price change as a real event
- Reading the change events' `before`/`after` payload
- Applying the change with PyIceberg's `table.upsert()`
- Project: Kiosko's first live CDC bridge
- Module introduction: the full assembly
- The brief: Kiosko needs to know in minutes, not tomorrow
- Assembling the full pipeline: from producer to alert
- Running the canonical week, and matching every batch guide
- Running the broken day, and watching the alert fire
- Running the CDC bridge end to end
- What Kiosko still needs
- Project: Kiosko's first streaming platform
Common questions
What people usually ask
No limit. It's a free guide: come in whenever you like, as often as you like.
No. Modules run from easier to harder, but you can jump to the one you need. Progress is saved per lesson.
Whatever is needed is listed under “What you need to bring”, above. If nothing is listed there, you can start from zero.
In the Club's WhatsApp group, and every two weeks there's a live with an instructor where questions get worked through.
Yes. It's issued automatically once you finish every lesson, with a verifiable code you can share on LinkedIn.
Start whenever you like
What students say
These reviews are from enrolled students who completed at least 50% of the course. We moderate reviews only on content grounds (spam, offensive language, personal data), never for being critical or negative.
No approved reviews yet.
Be the first to share your experience!