GuideBeginner
Python For Data Engineering
Picks up directly from `data-engineering-foundations-guide`: same Kiosko case, same pipeline, same `orders` and `events` data — what changes is how the code that processes them is written. This guide takes the plain script from foundations and turns it, module by module, into a production Python package: packaged with `uv` (`pyproject.toml`, `src/` layout, `uv.lock`), `pytest` tests over every pipeline function, structured JSON logging instead of `print`, deterministic retries with `tenacity` for transient failures, and DuckDB and Polars in depth as the two daily-driver transformation engines that replace the plain Python loops and `sqlite3` from foundations. pandas is never used. It closes with a capstone that assembles the full package — installable, tested, logged, resilient — and compares three implementations of the same transformation (a Python loop, SQL with DuckDB, a Polars `LazyFrame`) to prove they all produce exactly the same result.
- 64
- lessons
- 8
- modules
- English · Spanish
- available in
- Yes
- certificate
- Free
- access
Outcomes
What you'll be able to do
- Turn a plain script into an installable package with `uv`: `pyproject.toml`, `src/` layout, `uv.lock`, and `uv run` as the way to execute it
- Write `pytest` tests (fixtures, parametrize, arrange-act-assert) over the extraction, transformation, and quality functions inherited from foundations
- Replace every `print()` with structured JSON logging carrying context (`run_id`, `partition_date`) using the `logging` module and `structlog`
- Distinguish errors that should retry from errors that should go to quarantine, and implement deterministic retries with `tenacity` (backoff, jitter, when to give up), tested without using `random`
- Use DuckDB as a daily-driver engine: direct SQL over Parquet/CSV, the relational API in Python, and file vs. in-memory persistence
- Use Polars in depth: eager DataFrames vs. `LazyFrame`, the optimized query plan via `.explain()`, expressions, `group_by`/`agg`, and joins
- Decide with real criteria when to use DuckDB and when to use Polars, and why both displaced pandas in production pipelines
- Assemble a final package with a command-line entry point, configuration with no hardcoded values, and an integration test that proves idempotency with an `assert`, not by eyeballing it
Before you start
What you need to bring
It's for you if...
- Anyone who already completed `data-engineering-foundations-guide` (or has an equivalent pipeline written as a plain script) and wants to bring it to production quality
- Junior data engineers writing pipelines that work but lack tests, structured logging, and retry handling
- Developers still using pandas out of habit who want to migrate to DuckDB/Polars with real criteria, not because it's trendy
- Anyone who needs to deliver a data pipeline that a real engineering team would accept in code review
Requirements and materials
- `data-engineering-foundations-guide` completed (or equivalent: Kiosko's extraction, quality, transformation, and load pipeline working as a plain script)
- Intermediate Python: functions, exception handling, list comprehensions
- A personal laptop with Python 3 and `uv` installable — everything runs locally at $0
Content
The syllabus, module by module
Open any of them to see its lessons.
- Guide overview: from a script you run to a package you install
- Why a script is not the same thing as a package
- uv and the modern Python toolchain
- pyproject.toml and the src/ layout
- Turning Kiosko's script into a package
- Imports, modules, and the __init__ file
- Running the package with uv run
- Project: Kiosko's installable pipeline
- Module overview: from "it works on my machine" to "a machine confirms it"
- Why data pipelines need tests
- Writing your first pytest test
- Fixtures and Kiosko's fixed test data
- Testing the extract and transform steps
- Parametrize and testing the quality gate
- Arrange-Act-Assert and organizing a growing suite
- Project: Kiosko's test suite
- Module overview: from print() to a record a machine can read
- Why print() doesn't scale
- Python's logging module: levels and handlers
- JSON logs, and why a machine can read them
- Adding context with structlog
- Logging a pipeline run, from start to finish
- What to log and what to never log
- Project: Kiosko's structured record of a complete run
- Module overview: what to retry, what to quarantine
- Errors that should retry, and errors that never should
- Custom exceptions for a pipeline
- Retrying with tenacity
- Backoff, jitter, and when to give up
- Quarantine instead of crash, revisited
- Testing retry logic deterministically
- Project: Kiosko's resilient load step
- Module overview: DuckDB as the daily engine
- DuckDB beyond the preview
- Querying Parquet and CSV directly with SQL
- The relational API and persistence
- Replacing Python joins with SQL
- DuckDB and the Polars and Arrow ecosystem
- Testing SQL transformations with pytest
- Mini-project: Kiosko's DuckDB transform layer
- Module overview: the package that ships
- A command-line entry point for the pipeline
- Configuration without hardcoded values
- Wiring logging, retries, and engines together
- An integration test for the whole pipeline
- Proving idempotency with a test, not a manual rerun
- What a code reviewer would still ask for
- Mini-project: Kiosko's shippable package
- Module overview: Kiosko's capstone
- The brief: Kiosko needs a pipeline a team can own
- Assembling the complete package, end to end
- Running the complete test suite
- Running the pipeline with structured logs
- DuckDB vs Polars vs the original script
- What Kiosko still needs
- Mini-project: Kiosko's first production package
Common questions
What people usually ask
No limit. It's a free guide: come in whenever you like, as often as you like.
No. Modules run from easier to harder, but you can jump to the one you need. Progress is saved per lesson.
Whatever is needed is listed under “What you need to bring”, above. If nothing is listed there, you can start from zero.
In the Club's WhatsApp group, and every two weeks there's a live with an instructor where questions get worked through.
Yes. It's issued automatically once you finish every lesson, with a verifiable code you can share on LinkedIn.
Start whenever you like
What students say
These reviews are from enrolled students who completed at least 50% of the course. We moderate reviews only on content grounds (spam, offensive language, personal data), never for being critical or negative.
No approved reviews yet.
Be the first to share your experience!