GuideIntermediate
Data Reliability And Governance
A green pipeline with bad data is worse than a red one — because nobody double-checks it. This guide teaches you to answer the question the earlier guides in the ecosystem never asked: how do you know your data is actually correct, not just that the process that produced it finished without an error? The driving case is Kiosko opening its fourth store: the first sales file from `S04` arrives late and with 12 rows, six of which each break a different quality dimension — including one row that passes every schema/type/range check but is wrong by two orders of magnitude (the classic dollars-to-cents bug). You'll learn the six data quality dimensions as precise vocabulary, declarative tests with Pandera (chosen over Great Expectations and Soda with real license and vendor-risk evidence), data contracts versioned in YAML that generate those same tests, deterministic anomaly detection with no Machine Learning, table-level freshness and volume checks, lineage traced by hand (with OpenLineage named as the production version), and what to do when a check fails in production: quarantine, alert, runbook. It closes with governance — role-based access and deterministic PII masking — so you can say, with evidence instead of faith, whether your data is correct and who can see it.
- 64
- lessons
- 8
- modules
- English · Spanish
- available in
- Yes
- certificate
- Free
- access
Outcomes
What you'll be able to do
- Name the six data quality dimensions (completeness, uniqueness, validity, consistency, freshness, accuracy) and diagnose what slips through a basic validation gate
- Write declarative quality tests with Pandera (`DataFrameModel`, `Field`, `Check`) over data read from DuckDB via Polars
- Write a referential integrity check (anti-join) that catches an orphan foreign key a declarative schema alone can't detect
- Design a data contract versioned in YAML (schema + SLA + violation policy), parse it with pydantic, and generate it into an executable Pandera schema
- Build a price baseline from clean data and detect anomalies with rules and thresholds — no Machine Learning, always explainable in one sentence
- Write freshness and volume checks at the file/table level, using a fixed reference clock instead of the system clock
- Trace lineage by hand (where does each column come from) and know OpenLineage, the open standard, as its production version
- Quarantine broken rows instead of failing the whole process or silently letting them through, raise a structured alert, and write an incident runbook
- Define a role-based access policy and mask PII deterministically (same data, same hash every time), and publish a minimal data catalog
Before you start
What you need to bring
It's for you if...
- Data engineers whose pipelines "run green" but have no systematic way to prove the data inside is correct
- Teams evaluating adding Great Expectations or Soda to the stack who want to understand the landscape and the real trade-off between vehicles first
- Anyone preparing for roles or certifications where data quality and governance already carry real weight (AWS DEA-C01 Domain 4, Databricks)
- Data engineers who completed the earlier guides in the NIEVA ecosystem and want to close the trust question none of them answered
Requirements and materials
- Intermediate Python and comfort with DataFrames (Polars or pandas)
- Basic SQL (joins, aggregations)
- Ideally completed `data-engineering-foundations-guide` (the local validation gate this guide diagnoses and replaces) and `data-modeling-for-analytics-guide` (the `kiosko.duckdb` warehouse this guide audits)
Content
The syllabus, module by module
Open any of them to see its lessons.
- Module introduction: declarative data quality tests with Pandera
- What, precisely, a declarative quality test is
- Choosing a vehicle: Pandera vs. Great Expectations vs. Soda
- Installing Pandera and building the DuckDB → Polars bridge
- Writing your first DataFrameModel
- Completeness and uniqueness, declared
- Validity declared, and how to read a complete failure report
- Project: testing S04 with Pandera
- Module introduction: consistency and referential checks
- What consistency means across tables
- Why schema checks alone don't catch referential bugs
- Writing a referential integrity check
- Cross-column consistency rules
- Catching S04's orphan product
- Combining Pandera with custom checks
- Project: S04's complete consistency report
- Module introduction: data contracts as versioned artifacts
- What, precisely, a data contract is
- Writing orders_contract.yaml
- Parsing the contract with pydantic
- From contract to Pandera schema
- Versioning a contract that changes
- Should S04 be allowed to write without a contract?
- Project: Kiosko's first data contract
- Module introduction: accuracy and deterministic anomaly detection
- Accuracy: the hardest dimension to test
- Why a valid row can still be wrong
- Building a price baseline from Kiosko's clean week
- Threshold-based anomaly detection, without Machine Learning
- Catching the dollars-to-cents bug
- When thresholds are too strict or too loose
- Project: S04's accuracy audit
- Module introduction: freshness, volume, and lineage
- Freshness is a table-level question, not a row-level one
- A fixed reference clock, never `datetime.now()`
- Writing a freshness check for S04
- Volume checks: too few or too many rows
- What lineage answers that a contract does not
- Mapping Kiosko's lineage by hand
- Project: S04's freshness, volume, and lineage report
- Module introduction: the incident and data governance
- What happens when a check fails in production
- Quarantining, instead of crashing or silently passing
- Writing an alert and a runbook
- Who can see what: access control basics
- Masking PII deterministically
- A minimal data catalog for Kiosko
- Project: S04's incident response and access policy
- Module introduction: Kiosko's complete trust system
- The brief: Kiosko needs a trust system, not just a pipeline
- Assembling the contract-driven quality gate
- Running the full gate against S04's incident
- Running the full gate against a clean day
- Publishing the catalog and access policy
- What Kiosko still needs
- Project: Kiosko's trusted data platform
Common questions
What people usually ask
No limit. It's a free guide: come in whenever you like, as often as you like.
No. Modules run from easier to harder, but you can jump to the one you need. Progress is saved per lesson.
Whatever is needed is listed under “What you need to bring”, above. If nothing is listed there, you can start from zero.
In the Club's WhatsApp group, and every two weeks there's a live with an instructor where questions get worked through.
Yes. It's issued automatically once you finish every lesson, with a verifiable code you can share on LinkedIn.
Start whenever you like
What students say
These reviews are from enrolled students who completed at least 50% of the course. We moderate reviews only on content grounds (spam, offensive language, personal data), never for being critical or negative.
No approved reviews yet.
Be the first to share your experience!