GuideBeginner
Data Engineering Foundations
Make the jump from "I can program in Python" to "I think and build like a data engineer." This guide opens the Data Engineering ecosystem: it teaches the data lifecycle (extraction, storage, transformation, serving), the batch vs streaming decision governed by the business SLA (not by the trendiest tool), ETL vs ELT and what changed with the cloud warehouse, columnar storage with Parquet, a first fact-and-dimension model, quality gates before loading, and the pillar that market evidence flags as the most critical one courses tend to skip: idempotency and backfill. The entire thread actually runs, in standard Python 3 (with DuckDB as the only optional dependency), against the Kiosko case, a fictional convenience-store chain with sales data (`orders`) and app clickstream events (`events`). It closes with a capstone that integrates everything into a bronze-silver-gold pipeline, partitioned, validated, and queried with DuckDB.
- 64
- lessons
- 8
- modules
- English · Spanish
- available in
- Yes
- certificate
- Free
- access
Outcomes
What you'll be able to do
- Place the data engineer role in 2026 within the data lifecycle (generation, ingestion, storage, transformation, serving), and read your first Kiosko file with `csv.DictReader`
- Apply the batch vs streaming decision framework based on business SLA, and extract a date range of daily files with `pathlib.Path.glob`
- Distinguish ETL from ELT and OLTP from OLAP, load raw data into SQLite without transforming it and transform it afterward with `SELECT`, and run your first DuckDB query against a Parquet file
- Design a first data model with a fact table (`fact_orders`) and two dimensions (`dim_store`, `dim_product`), joined in Python via dictionary lookup with a derived column (`revenue`)
- Build a quality gate (`validate_orders`) that separates valid rows from broken ones by schema, nulls, type, and range, without stopping the pipeline
- Guarantee a pipeline is safe to rerun with the overwrite-partition pattern, and write a backfill function that recomputes a date range
- Partition data by date on disk and chain extraction, validation, transformation, and loading into a single minimal orchestration function with logging
- Integrate everything into a bronze-silver-gold pipeline that processes a full week of Kiosko data and produces a sales report by store and product
Before you start
What you need to bring
It's for you if...
- Python developers who already know how to code and want to enter data engineering with solid fundamentals, not just a list of tool names
- Data analysts or BI professionals who want to move from querying pipelines to building them
- Devs who need to decide, with real criteria, between batch and streaming, ETL and ELT, before picking a tool
- Anyone who wants to understand idempotency and backfill before touching a real orchestrator
Requirements and materials
- Intermediate Python: functions, classes, file handling, list comprehensions
- A personal laptop with Python 3 installed — everything runs locally at $0, no cloud account required
- No prior experience with SQL, warehouses, or data tools required
Content
The syllabus, module by module
Open any of them to see its lessons.
- Module overview: the question that comes before any tool
- Batch processing: what it is and why it exists
- Streaming processing: what it is and why it exists
- The SLA question, before any tool
- Does Kiosko need batch or streaming?
- Extracting multiple files with pathlib
- Parsing a full week of orders
- Mini-project: Kiosko's daily extraction job
- Module 3 overview: what changed when storage stopped being expensive
- ETL vs ELT: what changed with the cloud warehouse
- OLTP vs OLAP: two different jobs
- The map: the warehouse, the lake, and the lakehouse
- Columnar storage and why Parquet is different
- Loading raw data with SQLite: your first ELT
- A first look at DuckDB
- Mini-project: Kiosko's landing zone
Common questions
What people usually ask
No limit. It's a free guide: come in whenever you like, as often as you like.
No. Modules run from easier to harder, but you can jump to the one you need. Progress is saved per lesson.
Whatever is needed is listed under “What you need to bring”, above. If nothing is listed there, you can start from zero.
In the Club's WhatsApp group, and every two weeks there's a live with an instructor where questions get worked through.
Yes. It's issued automatically once you finish every lesson, with a verifiable code you can share on LinkedIn.
Start whenever you like
What students say
These reviews are from enrolled students who completed at least 50% of the course. We moderate reviews only on content grounds (spam, offensive language, personal data), never for being critical or negative.
No approved reviews yet.
Be the first to share your experience!