GuideBeginner
Airflow And Declarative Orchestration
Picks up directly from `python-for-data-engineering-guide`: same Kiosko case, same `kiosko_pipeline` package, already installable, tested, and equipped with code-level retries. What changes is who decides when and how the pipeline runs — until now a human decided by typing a command; from this guide on, Apache Airflow decides. You learn the anatomy of a DAG with the TaskFlow API (`@dag`/`@task`), wrapping the already-tested functions from `kiosko_pipeline` without rewriting their logic, `logical_date` as the orchestrator's own clock (never `datetime.now()`), scheduler-managed retries as a distinct, complementary layer to code-level retries, sensors for waiting on data that hasn't arrived yet, backfill with Airflow's real CLI, and DAG observability through its UI and logs. Everything runs locally with `airflow standalone`, no Docker, no cloud account. The guide honestly names market alternatives (Dagster, Prefect) without teaching them: what's transferable is idempotency, DAGs, scheduling, retries, backfill, sensors, and observability — the specific vendor is what expires. It closes with a capstone that assembles Kiosko's complete DAG: scheduled, with a sensor, per-task retries, and backfillable across a full week.
- 64
- lessons
- 8
- modules
- English · Spanish
- available in
- Yes
- certificate
- Free
- access
Outcomes
What you'll be able to do
- Install Airflow locally with `uv`/pip, run `airflow standalone`, and write your first DAG with the TaskFlow API
- Wrap already-tested `kiosko_pipeline` functions as `@task`s, using XComs as a metadata bus (never a bus for full datasets)
- Schedule a DAG (`schedule`, `catchup`) and use `logical_date` as the source of truth for each run's business date, never the system clock
- Distinguish code-level retries (`tenacity`, already handled in the previous guide) from scheduler-managed retries (`retries`, `retry_delay`), and configure both layers
- Build a sensor (`poke`/`reschedule` mode, plus the deferrable variant with the triggerer) that waits for a file that hasn't arrived yet before triggering extraction
- Run a backfill with Airflow's real CLI (`airflow backfill create`) over a date range, relying on the overwrite-partition pattern already built into the package
- Read Airflow's UI (Grid/Graph) and a failed task's logs the way someone operating a production pipeline would, and trigger dbt as just another task in the DAG
- Assemble a complete DAG — scheduled, with a sensor, per-task retries, idempotent — and verify that orchestration doesn't change the pipeline's result, only how and when it's produced
Before you start
What you need to bring
It's for you if...
- Anyone who already completed `python-for-data-engineering-guide` (or has an equivalent tested, installable pipeline package) and needs to set it running on its own, on a schedule
- Data engineers still running pipelines by hand or with a plain cron job, with no managed retries or real observability
- Devs preparing for roles where Airflow is the de facto standard (it shows up in most job postings in the field)
- Anyone who needs to understand the difference between code-level and infrastructure-level retries before operating a pipeline in production
Requirements and materials
- `python-for-data-engineering-guide` completed (or equivalent: an installable, tested Python package with structured logging and code-level retries)
- Basic comfort with the terminal and installing Python packages (`uv`/pip)
- A personal laptop capable of installing and running Apache Airflow locally (`airflow standalone`) — everything runs locally at $0, no Docker, no cloud account
Content
The syllabus, module by module
Open any of them to see its lessons.
- Module overview: from a command you run by hand to an orchestrator that decides for you
- What a chained script cannot do for you
- What an orchestrator actually is
- The orchestration landscape in 2026
- Installing Airflow locally with uv
- airflow standalone and the web UI
- Your first DAG: hello_kiosko, with the TaskFlow API
- Project: hello_kiosko's first real run
- Module overview: from a trivial graph to a DAG that wraps real code
- Anatomy of a DAG: nodes, edges, and why no cycles at all
- The TaskFlow API in depth: @dag and @task
- Classic operators vs. TaskFlow: a quick map
- XComs and why they aren't a data bus
- Wrapping Kiosko's extract step as a task
- Wrapping Kiosko's validate step as a task
- Project: Kiosko's first real two-task DAG
- Module overview: the DAG stops needing someone to trigger it by hand
- The schedule parameter: cron, presets, and timetables
- start_date, catchup, and what a run represents
- logical_date versus real execution time
- Templating with Jinja: {{ ds }}
- Parameterizing Kiosko's partition by logical_date
- Running a DAG run for a specific date
- Project: Kiosko's scheduled daily DAG
- Module overview: when the supervisor, not the worker, decides to retry
- What can fail at the task level
- retries, retry_delay, and exponential backoff
- Code-level retries vs. orchestrator-level retries
- Retry policies and selective exceptions
- Simulating a transient failure deterministically
- Reading a TaskInstance's history and its logs
- Project: Kiosko's resilient load task
- Module introduction: when the pipeline learns to wait
- Why a pipeline needs to wait
- Sensors: poke mode and reschedule mode
- Deferrable operators and the triggerer
- Building a sensor for Kiosko's daily file
- Timeouts: what happens when a sensor gives up
- Gating extraction on file arrival
- Project: Kiosko's sensor-gated DAG
- Module introduction: reprocessing a week with no hand-written loop
- Manual backfill vs orchestrator-managed backfill
- The overwrite-partition pattern, seen at DAG level
- Running a backfill with the Airflow CLI
- `--reprocess-behavior` and what happens when a run already exists
- Proving idempotency across a week of runs
- When backfill goes wrong
- Project: Kiosko's week, reprocessed with the Airflow CLI
- Module introduction: eyes on the system you already built
- The DAG UI: the Grid and Graph views
- Reading task logs like an operator
- Failure callbacks and the alerting pattern
- SLAs, deadlines, and what they really mean on Airflow 3.3.1
- dbt as a task inside a DAG
- The DAG as a contract: idempotency, revisited
- Project: Kiosko's observable DAG, end to end
- Module introduction: the capstone, where seven pieces become one
- The brief: Kiosko needs a real orchestrator
- Assembling the complete DAG
- Scheduling it for production
- Running the complete week with a backfill
- Verifying idempotency end to end
- What Kiosko still needs: the ecosystem map
- Project: Kiosko's first production DAG
Common questions
What people usually ask
No limit. It's a free guide: come in whenever you like, as often as you like.
No. Modules run from easier to harder, but you can jump to the one you need. Progress is saved per lesson.
Whatever is needed is listed under “What you need to bring”, above. If nothing is listed there, you can start from zero.
In the Club's WhatsApp group, and every two weeks there's a live with an instructor where questions get worked through.
Yes. It's issued automatically once you finish every lesson, with a verifiable code you can share on LinkedIn.
Start whenever you like
What students say
These reviews are from enrolled students who completed at least 50% of the course. We moderate reviews only on content grounds (spam, offensive language, personal data), never for being critical or negative.
No approved reviews yet.
Be the first to share your experience!