GuideIntermediate
Spark And Distributed Processing
Find out exactly when your data problem outgrows a single node, and what actually changes once you distribute it. This guide picks up the Kiosko case (a convenience-store chain) right where the earlier guides in the ecosystem left it — a dimensional warehouse that runs perfectly fine on DuckDB — and adds a synthetic 10-million-row dataset so partitioning and shuffle become real, felt costs on your own laptop. You'll learn Apache Spark in local mode (`local[*]`, $0, no cluster, no cloud account) with the DataFrame API as the main vehicle: the driver/executor model, lazy evaluation, the real cost of a shuffle in `groupBy`/`join`, broadcast join vs. sort-merge join, window functions at scale, the Catalyst optimizer read through `.explain()`, when to cache (and when not to), partitioned Parquet, and why a plain Python UDF is slow compared to an Arrow-vectorized `pandas_udf`. No performance claim is ever measured with a stopwatch — every comparison is backed by the execution plan. The guide closes with the honest criterion most market Spark courses skip: when Kiosko (real, 40 rows) does NOT need Spark, and when a much bigger Kiosko does.
- 64
- lessons
- 8
- modules
- English · Spanish
- available in
- Yes
- certificate
- Free
- access
Outcomes
What you'll be able to do
- Install and run PySpark in local mode (Java 17+, `SparkSession.builder.master("local[*]")`) and verify it reads the same Kiosko data as previous guides
- Distinguish transformations from actions, understand lazy evaluation, and read the logical/physical DAG of a chain of operations before running it
- Rebuild `fact_orders` with the DataFrame API (`join`, `withColumn`, `groupBy().agg()`) and verify the result matches earlier engines in the ecosystem exactly
- Deterministically generate a synthetic dataset at scale (10 million rows) and feel the real cost of a shuffle in `groupBy`/`join`/`distinct`
- Choose between `repartition()` and `coalesce()`, and decide when a broadcast join beats a sort-merge join by reading both in `.explain()`
- Write and read window functions (`Window.partitionBy`/`orderBy`) for running revenue totals and product rankings at scale
- Read the stages of the Catalyst optimizer and the effect of Adaptive Query Execution (AQE) on a query plan
- Decide with criteria when `.cache()`/`.persist()` helps and when it just costs memory with no benefit
- Write partitioned Parquet at scale and explain why a plain Python UDF (`cloudpickle`) is slow compared to an Arrow-vectorized `pandas_udf`
- Apply a "do I need Spark?" decision tree to the real Kiosko (40 rows) and to a hypothetical much-larger Kiosko, with criteria instead of trend-chasing
Before you start
What you need to bring
It's for you if...
- Data engineers who already know Python/SQL/DuckDB on a single node and need to know when that ceiling stops being enough
- Devs preparing for roles or certifications that require Spark (AWS DEA-C01, Databricks, DP-700) without wanting to learn Scala or RDDs first
- Anyone who's been told "just use Spark" for a problem that probably doesn't need it, and wants a real cost criterion to decide
- Data engineers who already completed the earlier guides in the NIEVA ecosystem (foundations, Python, modeling, dbt) and want to take the same case to distributed scale
Requirements and materials
- Intermediate Python and comfort with DataFrames (DuckDB, Polars, or pandas)
- SQL: joins, aggregations, and basic window functions
- Java 17 or later installed locally (verified in Module 1)
- Ideally completed `python-for-data-engineering-guide` and `data-modeling-for-analytics-guide` (or equivalent): this guide directly builds on Kiosko's case and dimensional warehouse
Content
The syllabus, module by module
Open any of them to see its lessons.
- Module introduction: why distribute — the single-node ceiling
- The single-node ceiling: what DuckDB and Polars already solved
- A real cost criterion for when to distribute
- Installing PySpark and Java locally
- Your first SparkSession
- Reading Kiosko's orders with Spark
- Verifying the same forty rows arrive
- Project: Kiosko's first Spark session
- Module introduction: the Spark execution model
- The driver and the executors
- RDDs: the original abstraction, seen once and left behind
- The DataFrame API as the modern interface
- Lazy evaluation: transformations vs actions
- Building a transformation chain without running it
- Triggering execution with an action
- Project: Kiosko's first transformation chain
- Module introduction: rebuilding `fact_orders` with the DataFrame API
- Reading `orders`, `dim_store`, and `dim_product` with an explicit schema
- Joining facts and dimensions with the DataFrame API
- Computing revenue, the same way, four engines later
- Grouping by store and by product
- Verifying the same 106.15 total
- Writing `fact_orders` as Parquet
- Project: Kiosko's `fact_orders` in Spark
- Module introduction: partitions and the cost of shuffle
- What a partition actually is
- Why `groupBy`, `join`, and `distinct` trigger a shuffle
- Generating `kiosko_orders_at_scale`, deterministically
- Reading the shuffle in `.explain()` and the Spark UI
- `repartition()` vs `coalesce()`
- `spark.sql.shuffle.partitions` and Adaptive Query Execution
- Project: Kiosko at scale, partitioned
- Module introduction: joins and window functions at scale
- Broadcast join vs shuffle join
- Reading a broadcast join in `.explain()`
- Forcing and reading a `SortMergeJoin`
- Window functions: `partitionBy` and `orderBy`
- A running revenue total per store
- Ranking top products per store per day
- Project: Kiosko's scaled joins and rankings
- Module introduction: Parquet at scale and Python UDFs
- Parquet at scale: partitioned writes
- Predicate and column pushdown
- Why plain Python UDFs are slow: `cloudpickle` and the executor
- `pandas_udf` and Arrow-based vectorization
- Rewriting `margin_category` as a `pandas_udf`
- Naming Structured Streaming, without building it
- Project: Kiosko's partitioned Parquet and a vectorized UDF
- Module introduction: Kiosko's distributed capstone
- The brief: Kiosko at scale
- Assembling the distributed pipeline, end to end
- Verifying correctness against the original `106.15`
- Choosing partitioning, caching, and `JOIN` strategy, with criteria
- The decision tree: does Kiosko actually need Spark?
- What Kiosko still needs
- Project: Kiosko's first distributed pipeline
Common questions
What people usually ask
No limit. It's a free guide: come in whenever you like, as often as you like.
No. Modules run from easier to harder, but you can jump to the one you need. Progress is saved per lesson.
Whatever is needed is listed under “What you need to bring”, above. If nothing is listed there, you can start from zero.
In the Club's WhatsApp group, and every two weeks there's a live with an instructor where questions get worked through.
Yes. It's issued automatically once you finish every lesson, with a verifiable code you can share on LinkedIn.
Start whenever you like
What students say
These reviews are from enrolled students who completed at least 50% of the course. We moderate reviews only on content grounds (spam, offensive language, personal data), never for being critical or negative.
No approved reviews yet.
Be the first to share your experience!