GuideIntermediate
Lakehouse And Iceberg
Parquet is a file format, not a table format — and this guide shows you, using the same Kiosko case, four times that limit already hurt: the non-atomic overwrite-partition pattern from the basics, the `valid_from`/`valid_to` columns maintained by hand to track a product's history, the same technique automated with `dbt snapshot`, and Spark's folder-based partitioning. You'll learn Apache Iceberg, the table format that solves all of that from the storage layer up: real ACID transactions, an immutable snapshot on every write, time travel (`AS OF` a snapshot, with zero history columns declared anywhere), schema evolution without rewriting data, hidden partitioning and partition evolution, and native `MERGE INTO`/upserts. The main vehicle is PyIceberg — 100% Python, $0 (a local SQLite-backed catalog); Spark shows up exactly once, in the `MERGE INTO` module, to run real SQL against an Iceberg table. Delta Lake is named by contrast, without building a second parallel implementation. The thread you resolve end to end is a price change on a Kiosko product, recovered purely with time travel, with zero history columns.
- 64
- lessons
- 8
- modules
- English · Spanish
- available in
- Yes
- certificate
- Free
- access
Outcomes
What you'll be able to do
- Name precisely the difference between a file format and a table format, and the four times plain Parquet already hit its ceiling in the Kiosko ecosystem
- Install PyIceberg, create a local catalog (SQLite + filesystem), and load your first real Iceberg table from an existing Parquet file
- Walk the anatomy of an Iceberg table: catalog → metadata file → manifest list → manifest files → data files
- Use time travel (`AS OF` a `snapshot-id`) to recover a table's prior state, with no `valid_from`/`valid_to` declared anywhere
- Evolve a table's schema (add, rename, drop columns) without rewriting a single existing data file
- Contrast visible folder-based (Hive) partitioning with Iceberg's hidden partitioning, and evolve a partition scheme forward without rewriting existing data
- Run native `MERGE INTO` with Spark SQL and PyIceberg's `table.upsert()` as a 100% Python alternative for the same problem
- Name production catalogs (REST, AWS Glue, Unity Catalog, Polaris) and apply basic maintenance: compacting small files and safely expiring old snapshots
- Decide with criteria when a lakehouse on Iceberg wins over a classic managed warehouse or over plain Parquet with nothing else
Before you start
What you need to bring
It's for you if...
- Data engineers who've already written Parquet or built a dimensional warehouse and want to solve versioning/history at the table-format level, not by hand with columns
- Devs evaluating whether their team needs Iceberg or Delta Lake, and want to understand the real difference (not just the trendy name)
- Anyone preparing for roles where "Apache Iceberg or Delta Lake" shows up as an explicit job requirement
- Data engineers who completed `data-modeling-for-analytics-guide`, `dbt-analytics-engineering-guide`, and/or `spark-and-distributed-processing-guide` and want to close the history thread those guides left open
Requirements and materials
- Intermediate Python; comfortable reading and writing Parquet
- Basic SQL (joins, `WHERE`, aggregations)
- Ideally completed `data-modeling-for-analytics-guide` (the hand-historized `dim_product` problem) and `spark-and-distributed-processing-guide` (the partitioned Parquet this guide converts to Iceberg); Spark + Java 17 is only needed for the `MERGE INTO` module
Content
The syllabus, module by module
Open any of them to see its lessons.
- Module overview: from a Parquet file to a real Iceberg table
- Four times Parquet alone wasn't enough
- File format vs. table format
- Installing PyIceberg and a local catalog
- Creating your first namespace and table
- Loading Kiosko's fact_orders into Iceberg
- Verifying the same 106.15 total
- Project: Kiosko's first Iceberg table
- Module overview: the anatomy of an Iceberg table
- The catalog: a pointer to the current metadata
- The metadata file: schema, partitioning, snapshots
- Manifest lists and manifest files
- The data files: the Parquet you already know
- Inspecting Kiosko's table on disk
- Reading snapshots and history with PyIceberg
- Project: Kiosko's table anatomy, mapped
- Module overview: snapshots and time travel
- Every write is a new snapshot
- Changing P002 with a plain overwrite
- Capturing the snapshot-id, never hardcoding it
- Time travel: AS OF a snapshot-id
- Recovering P002's history with zero extra columns
- What time travel does not replace
- Project: Kiosko's time-traveled dim_product
- Module overview: schema evolution without rewriting
- Why overwrite-partition was never atomic
- Adding a column without rewriting data
- Renaming and dropping columns safely
- Populating country in dim_store
- Reading old snapshots after a schema change
- What "ACID" actually guarantees in this exact context
- Project: Kiosko's evolved dim_store
- Module overview: hidden partitioning and partition evolution
- How Spark partitioned Kiosko by folder
- Hidden partitioning: the same query, without knowing the layout
- Partition transforms: `IdentityTransform`, `BucketTransform`, `DayTransform`
- Partitioning `fact_orders_at_scale`
- Evolving the `PartitionSpec` without rewriting
- Reading old and new partition schemes, together
- Project: Kiosko's `fact_orders_at_scale`, partitioned
- Module overview: `MERGE INTO` and native upserts
- Three ways Kiosko already solved this
- Setting up Spark with the Iceberg runtime
- `MERGE INTO` in Spark SQL
- Running P002's change as a MERGE
- PyIceberg's upsert: the Python-native alternative
- Choosing between MERGE in SQL and upsert in Python
- Project: Kiosko's native upsert
- Module overview: the capstone, Kiosko's complete lakehouse
- The brief: Kiosko needs a lakehouse, not seven demos
- Rebuilding the star schema as Iceberg tables
- Reproducing P002's history with time travel
- Evolving and partitioning at scale
- Merging updates the native way
- What Kiosko still needs
- Project: Kiosko's first lakehouse
Common questions
What people usually ask
No limit. It's a free guide: come in whenever you like, as often as you like.
No. Modules run from easier to harder, but you can jump to the one you need. Progress is saved per lesson.
Whatever is needed is listed under “What you need to bring”, above. If nothing is listed there, you can start from zero.
In the Club's WhatsApp group, and every two weeks there's a live with an instructor where questions get worked through.
Yes. It's issued automatically once you finish every lesson, with a verifiable code you can share on LinkedIn.
Start whenever you like
What students say
These reviews are from enrolled students who completed at least 50% of the course. We moderate reviews only on content grounds (spam, offensive language, personal data), never for being critical or negative.
No approved reviews yet.
Be the first to share your experience!