Data Engineering Ecosystem

Python For Data Engineering

Learn to turn a flat data script into a production Python package for data engineering: packaging with uv, tests with pytest, structured JSON logging, retries with tenacity, and DuckDB and Polars as transformation engines. It's the continuation of the Data Engineering Foundations guide (the previous course in this ecosystem), where you built this pipeline as a flat script: same Kiosko case, same pipeline, same `orders` and `events` data — what changes is how the code that processes them is written. This guide takes the plain script from foundations and turns it, module by module, into a production Python package: packaged with `uv` (`pyproject.toml`, `src/` layout, `uv.lock`), `pytest` tests over every pipeline function, structured JSON logging instead of `print`, deterministic retries with `tenacity` for transient failures, and DuckDB and Polars in depth as the two daily-driver transformation engines that replace the plain Python loops and `sqlite3` from foundations. pandas is never used. It closes with a capstone that assembles the full package — installable, tested, logged, resilient — and compares three implementations of the same transformation (a Python loop, SQL with DuckDB, a Polars `LazyFrame`) to prove they all produce exactly the same result.

64 lecciones

Data Modeling For Analytics

Learn dimensional modeling for analytics: Kimball's four-step process, star schema vs snowflake vs wide table, historized dimensions (SCD), and accumulating patterns in SQL on DuckDB. It's the continuation of the Data Engineering Foundations Guide (the previous course in this ecosystem): there you built the flat model — one fact and two dimensions — that the course deliberately left incomplete; here you take it to real dimensional modeling, on the same Kiosko case. You learn Kimball's four-step process (business process, grain, dimensions, facts), how to decide with real criteria between a star schema, a snowflake schema, and a wide One Big Table by comparing join cost with `EXPLAIN`, how to historize a changing dimension with SCD type 1 and type 2 using `MERGE INTO`, how to join facts with historical dimensions at the exact point in time (the most expensive mistake in a dimensional model), how to deduplicate repeated rows with `ROW_NUMBER()`/`QUALIFY`, and how to build cumulative patterns: Kimball's accumulating snapshot for a session funnel and Zach Wilson's cumulative table design with array-type columns. Everything runs as SQL executed directly against DuckDB, with Python as minimal glue code. It closes with a capstone that integrates a complete Kiosko analytics warehouse: a star with a historized dimension, an accumulating snapshot, cumulative activity, and a wide table for the BI team.

64 lecciones

Dbt Analytics Engineering

Learn analytics engineering with dbt: turn a dimensional warehouse into a real software project with dbt-core and dbt-duckdb — models, tests, snapshots, incremental materializations, and auto-generated docs. It's the continuation of the Data Modeling for Analytics Guide (the previous course in this ecosystem): there you designed and built Kiosko's dimensional warehouse by hand — star schema, a historized `dim_product` with SCD type 2, an accumulating snapshot, a wide table for BI — and here you turn it into a real software project with dbt-core + dbt-duckdb. You learn the anatomy of a dbt project (sources, staging, marts), the dependency graph via `ref()` and materializations (view, table, incremental), how to write declarative tests (the four built-in generic tests, a custom generic test, and singular tests for business rules), how to automate SCD type 2 with dbt snapshots (instead of a hand-written `MERGE INTO`), how to build idempotent incremental models, and how to generate documentation and lineage automatically with `dbt docs generate`. Everything runs locally with dbt-core and dbt-duckdb, with no dbt Cloud account and no cloud provider required. It closes with a capstone that ports the rest of Kiosko's marts into the project and runs `dbt build` end to end: sources, snapshot, every model in DAG order, and every test, with a literal `PASS/WARN/ERROR` report.

64 lecciones

Airflow And Declarative Orchestration

Learn to orchestrate data pipelines with Apache Airflow: DAGs, the TaskFlow API, scheduling, retries, sensors, and backfill. It's the continuation of the Python for Data Engineering Guide (the previous course in this ecosystem): same Kiosko case, same `kiosko_pipeline` package, already installable, tested, and equipped with code-level retries. What changes is who decides when and how the pipeline runs — until now a human decided by typing a command; from this guide on, Apache Airflow decides. You learn the anatomy of a DAG with the TaskFlow API (`@dag`/`@task`), wrapping the already-tested functions from `kiosko_pipeline` without rewriting their logic, `logical_date` as the orchestrator's own clock (never `datetime.now()`), scheduler-managed retries as a distinct, complementary layer to code-level retries, sensors for waiting on data that hasn't arrived yet, backfill with Airflow's real CLI, and DAG observability through its UI and logs. Everything runs locally with `airflow standalone`, no Docker, no cloud account. The guide honestly names market alternatives (Dagster, Prefect) without teaching them: what's transferable is idempotency, DAGs, scheduling, retries, backfill, sensors, and observability — the specific vendor is what expires. It closes with a capstone that assembles Kiosko's complete DAG: scheduled, with a sensor, per-task retries, and backfillable across a full week.

64 lecciones

Docker Essentials Guide

Master Docker for AI applications: build optimized images, containerize FastAPI + LLM apps, orchestrate multi-service stacks with Docker Compose (API + ChromaDB + Redis), and apply production best practices including multi-stage builds, secrets management, and health checks. Your gateway to production deployment.

64 lecciones

Spark And Distributed Processing

Find out exactly when your data problem outgrows a single node, and what actually changes once you distribute it. This guide picks up the Kiosko case (a convenience-store chain) right where the earlier guides in the ecosystem left it — a dimensional warehouse that runs perfectly fine on DuckDB — and adds a synthetic 10-million-row dataset so partitioning and shuffle become real, felt costs on your own laptop. You'll learn Apache Spark in local mode (`local[*]`, $0, no cluster, no cloud account) with the DataFrame API as the main vehicle: the driver/executor model, lazy evaluation, the real cost of a shuffle in `groupBy`/`join`, broadcast join vs. sort-merge join, window functions at scale, the Catalyst optimizer read through `.explain()`, when to cache (and when not to), partitioned Parquet, and why a plain Python UDF is slow compared to an Arrow-vectorized `pandas_udf`. No performance claim is ever measured with a stopwatch — every comparison is backed by the execution plan. The guide closes with the honest criterion most market Spark courses skip: when Kiosko (real, 40 rows) does NOT need Spark, and when a much bigger Kiosko does.

64 lecciones

Lakehouse And Iceberg

Learn the lakehouse model with Apache Iceberg: ACID transactions, immutable snapshots, time travel, schema and partition evolution, and native MERGE/upserts with PyIceberg. Parquet is a file format, not a table format — and this guide shows you, using the same Kiosko case, four times that limit already hurt: the non-atomic overwrite-partition pattern from the basics, the `valid_from`/`valid_to` columns maintained by hand to track a product's history, the same technique automated with `dbt snapshot`, and Spark's folder-based partitioning. You'll learn Apache Iceberg, the table format that solves all of that from the storage layer up: real ACID transactions, an immutable snapshot on every write, time travel (`AS OF` a snapshot, with zero history columns declared anywhere), schema evolution without rewriting data, hidden partitioning and partition evolution, and native `MERGE INTO`/upserts. The main vehicle is PyIceberg — 100% Python, $0 (a local SQLite-backed catalog); Spark shows up exactly once, in the `MERGE INTO` module, to run real SQL against an Iceberg table. Delta Lake is named by contrast, without building a second parallel implementation. The thread you resolve end to end is a price change on a Kiosko product, recovered purely with time travel, with zero history columns.

64 lecciones

Streaming With Kafka And Flink

Learn streaming data processing with Apache Kafka and Apache Flink: when streaming is justified over batch and how to solve what batch cannot. This guide opens by proving, with real code, the most expensive and best-documented mistake in streaming: recomputing in Kafka+Flink the exact same Kiosko session funnel you already calculated in batch in a previous course of this ecosystem — and landing on the identical `35.3%`, with more infrastructure and zero new value. From that proof, the guide draws the real line: streaming earns its cost on the handful of problems batch, by design, cannot solve, no matter how much you optimize it. Running Apache Kafka (4.x, KRaft mode, $0 locally) and Apache Flink/PyFlink for real in Docker, you solve four concrete Kiosko problems: detecting in minutes that a store stopped reporting (the star case, against a daily DAG that wouldn't notice until tomorrow), capturing a price change straight from Postgres's write-ahead log with Debezium (CDC) instead of hand-declaring it in Python, building event-time windows with watermarks over a continuous stream, and keeping incremental state without re-summing the entire history on every run. The guide is honest about Flink's market gap and answers it with fresh evidence (Kafka's salary premium, ~33% of UK senior streaming roles) without overstating Flink's role beyond what Kiosko genuinely needs.

64 lecciones

Data Reliability And Governance

Learn data reliability and governance: the six quality dimensions, declarative tests with Pandera, data contracts, lineage, and access control and PII masking. A green pipeline with bad data is worse than a red one — because nobody double-checks it. This guide teaches you to answer the question the previous courses in this ecosystem never asked: how do you know your data is actually correct, not just that the process that produced it finished without an error? The driving case is Kiosko opening its fourth store: the first sales file from `S04` arrives late and with 12 rows, six of which each break a different quality dimension — including one row that passes every schema/type/range check but is wrong by two orders of magnitude (the classic dollars-to-cents bug). You'll learn the six data quality dimensions as precise vocabulary, declarative tests with Pandera (chosen over Great Expectations and Soda with real license and vendor-risk evidence), data contracts versioned in YAML that generate those same tests, deterministic anomaly detection with no Machine Learning, table-level freshness and volume checks, lineage traced by hand (with OpenLineage named as the production version), and what to do when a check fails in production: quarantine, alert, runbook. It closes with governance — role-based access and deterministic PII masking — so you can say, with evidence instead of faith, whether your data is correct and who can see it.

64 lecciones

Monitoring & Observability Guide

Master the observability of AI systems in production with OpenTelemetry, the industry standard for 2026. Learn to instrument LLM applications with traces for prompts, embeddings, and tool calls, build dashboards for latency and cost, design alerting strategies, implement AI-specific monitoring (prompt quality, token usage, model drift), and debug production issues like hallucinations and cost spikes. Integrates with LangSmith and monitoring backends.

64 lecciones

Advanced Sql Querying

Learn to query a relational database with real SQL: JOINs in depth, aggregation, subqueries, CTEs (including recursive ones), window functions, and query optimization by reading the execution plan. You work on Reservo's database, a coworking room-booking system, and come out knowing how to answer complex business questions with a single query that's well written, readable, and efficient. It closes with a full analytical report: revenue by room and by month, top members by spend, occupancy, refund rate, and ranking.

64 lecciones

Terraform And Iac

Learn Terraform and Infrastructure as Code (IaC): declare your infrastructure in HCL with the init/plan/apply/destroy cycle and state management, instead of creating it by hand. It's the continuation of the AWS Core Services Guide (the previous course in this ecosystem): there you built the Andes Cargo stack by hand, command by command; here you stop creating infrastructure manually and start declaring it. It covers HCL in depth (`resource`, `data`, `variable`, `output`, `locals`, `module`), the `init`/`plan`/`apply`/`destroy` cycle and the idempotency behind it, and dedicates its heaviest module to Terraform's `state`: what it is, why it's the source of truth, how drift gets detected, how to import infrastructure that already exists (the exact real-world problem Andes Cargo has, since it was created by hand in the previous guide), and why a poorly secured state is one of the least-taught security risks in the discipline. It builds two reusable modules (`s3-bucket`, `iam-role`) and uses them to declare the same four canonical Andes Cargo resources — the bucket, both IAM roles, and the Lambda function with its DynamoDB table — proving that a single `terraform apply` recreates what used to require a manual checklist, and a single `terraform destroy` cleans it all up. The capstone runs the same code with OpenTofu, Terraform's open-source fork, and dedicates a lesson to a real March 2026 incident where an AI agent ran `terraform destroy` against production infrastructure without anyone carefully reviewing the plan — installing the rule that no `apply` or `destroy` ever runs without reading the full plan first. Everything runs $0 against the same LocalStack lab from the previous guide.

64 lecciones

Aws Core Services

Start AWS from zero and build a real end-to-end architecture with your own hands — without ever entering a credit card. The guide follows Andes Cargo, a fictional LATAM logistics company that needs to move its shipment manifests and inventory tracking off a laptop and into the cloud, and uses that thread to build the mental model of "the cloud" (managed services, the shared responsibility model, regions and availability zones) plus the four services that carry 80% of what gets built on AWS: IAM (identity and least privilege), S3 (object storage), EC2 + VPC (compute and the minimal network to run it), and Lambda + DynamoDB (event-driven processing and a managed NoSQL database). Everything runs against LocalStack, the emulator that replicates AWS's real API in Docker, so every command is the same `aws`/`awslocal` you'd run against a real account, with literal, verified JSON output. The thread is cumulative: the VPC from module 2 hosts the EC2 instance from module 5, the IAM roles from module 3 get attached to Lambda and EC2, the S3 bucket from module 4 triggers the Lambda function from module 6, which writes to the DynamoDB table from module 7 — and the module 8 capstone runs that full flow, from file upload to final query, with executed evidence at every step. This is the guide that opens the AWS Cloud ecosystem: it doesn't teach Terraform, containers, Kubernetes, CI/CD, SRE, or multi-account security — it builds the foundation those sibling guides build on.

64 lecciones

Deployment & System Design Guide

Take your Python API from localhost to production on real cloud platforms (Render, Railway, Fly.io), set up managed databases (Supabase, Neon), Redis in the cloud (Upstash), a reverse proxy with Nginx, and master the fundamentals of System Design: monolith vs microservices, caching layers, message queues, scaling strategies, and API versioning. Guide #15 and the LAST one in the Backend Python Developer with FastAPI Path — the capstone that closes out the whole path.

56 lecciones

Cost Optimization & Caching Guide

Master cost reduction for AI systems: understand cost drivers (tokens, API calls, embeddings), implement Redis caching and semantic caching (similar queries = cached responses using embeddings), apply prompt optimization and model selection strategies, and build a system that achieves 50-80% cost reduction with real numbers. The only guide in Spanish that covers semantic caching hands-on.

64 lecciones