GuideBeginner
Product Metrics And Experimentation
Teaches how to measure whether a bet worked, with rigor: the quantitative side of product. It covers what to measure (North Star, actionable vs. vanity metrics, Google's HEART and McClure's AARRR/pirate metrics frameworks), how to read a funnel (conversion and drop-off by step), how to measure cohort retention without an aggregate average lying to you, how to protect a metric with guardrails, and — the heart of the guide — how to run A/B tests correctly: the hypothesis, randomization, lift, statistical significance (p-value and confidence interval via a real proportions z-test), sample size (MDE, statistical power), and the pitfalls that invalidate a result that looks good (peeking, multiple comparisons, Simpson's paradox, novelty effect). It picks up where the Discovery guide left off: Mercado's `recommendations` bet, already validated with a qualitative signal (fake door), now launches as a controlled experiment (`control` vs. `variant`) and gets measured with real statistics, implemented and verified in Node.js. It closes with a capstone project that measures the full launch and decides whether to ship, revert, or iterate.
- 64
- lessons
- 8
- modules
- English · Spanish
- available in
- Yes
- certificate
- Free
- access
Outcomes
What you'll be able to do
- Choose what to measure: distinguish actionable metrics from vanity metrics using the HEART (Google) and AARRR (pirate metrics) frameworks
- Read a funnel: conversion rate and drop-off by step, run with `funnelAnalysis` over Mercado's checkout
- Measure cohort retention with `retentionCurve` and understand why aggregate averages lie
- Define a North Star metric and its guardrails (latency, complaints, churn) to avoid optimizing a metric at the product's expense
- Design and run a proper A/B test: control vs. variant, randomization, null hypothesis, and lift
- Calculate real statistical significance with a proportions z-test: p-value and 95% confidence interval, implemented and verified in Node (`abTest`)
- Calculate the sample size needed to detect a given effect (MDE, alpha, power) with `sampleSize`
- Spot the pitfalls that invalidate a result: peeking, multiple comparisons, Simpson's paradox, novelty effect
- Decide to ship, revert, or iterate with evidence, applied to Mercado's recommendations launch in the capstone project
Before you start
What you need to bring
It's for you if...
- Engineers who validated a bet in discovery and now need to measure the real result at scale, with statistics
- Devs who've watched a team declare "the A/B won" without checking whether the lift was statistically significant
- Backend or data-adjacent engineers who want to correctly instrument and read funnels, cohorts, and experiments
- Anyone about to own or review A/B test results who needs real statistical grounding, not intuition
Requirements and materials
- Product Thinking for Engineers Guide and Product Discovery and Prototyping Guide completed (or equivalent: a prioritized bet already validated qualitatively, ready to launch as an experiment)
- Comfort running simple Node.js scripts (no dependencies)
- Basic comfort with arithmetic/algebra — no prior statistics required, the guide teaches the practical z-test from scratch
Content
The syllabus, module by module
Open any of them to see its lessons.
- Module introduction: the funnel, the first picture of where you win and lose users
- What a funnel is
- The conversion rate per step
- Overall conversion: a product, not a sum
- Finding the biggest drop-off
- Micro-conversion vs. macro-conversion
- Reading a funnel in context: segments, not just the aggregate
- Mini-project: Mercado's checkout funnel, end to end
- Module introduction: from converting in the session to coming back next week
- What a cohort is
- The retention curve
- The retention plateau
- Why the aggregate average lies
- Reading a cohort table
- Retention as the best proxy for real value
- Project: read Mercado's cohort table and compare with recommendations vs. without
- Module introduction: the metric that governs, and what protects it
- What a North Star metric is
- Choosing the North Star with a criterion, not by eye
- Input metrics: the levers that move the North Star
- Goodhart's law: when the metric becomes the target
- Guardrails: what must not get worse while you optimize
- The metric that governs: North Star, inputs, and guardrails together
- Mini-project: Mercado's official North Star, and what its launch can't break
- Module introduction: from comparing two curves to designing the experiment that tests cause
- Correlation isn't causation: why you need an experiment
- Control and variant
- Randomization
- The null hypothesis
- Measuring lift
- What an A/B test CAN — and cannot — measure
- Project: design Mercado's recommendations launch A/B test
- Module introduction: is the lift real, or is it chance?
- Could the lift be pure coincidence?
- The null hypothesis and the p-value
- What the p-value does NOT mean
- The confidence interval
- The proportions z-test, implemented and verified
- Statistical significance vs. practical importance
- Project: is the recommendations launch real?
- Module introduction: from "the lift is real" to "how much sample did I need, and what pitfalls could have fooled me"
- The minimum detectable effect (MDE)
- Sample size and power
- The peeking problem
- Multiple comparisons
- Simpson's paradox
- Novelty effect and p-hacking
- Project: size the recommendations A/B test and audit its pitfalls
- Module introduction: the entire method, over a single launch
- The primary metric and the funnel: where the opportunity lives
- Baseline retention: do people who buy come back?
- The official North Star, and the guardrail that broke
- Size the experiment: was the sample enough?
- Run the experiment, and ask the z-test if it's real
- Check the pitfalls: does the result hold up to scrutiny?
- Final project: measure the launch of Mercado's recommendations
Common questions
What people usually ask
No limit. It's a free guide: come in whenever you like, as often as you like.
No. Modules run from easier to harder, but you can jump to the one you need. Progress is saved per lesson.
Whatever is needed is listed under “What you need to bring”, above. If nothing is listed there, you can start from zero.
In the Club's WhatsApp group, and every two weeks there's a live with an instructor where questions get worked through.
Yes. It's issued automatically once you finish every lesson, with a verifiable code you can share on LinkedIn.
Start whenever you like
What students say
These reviews are from enrolled students who completed at least 50% of the course. We moderate reviews only on content grounds (spam, offensive language, personal data), never for being critical or negative.
No approved reviews yet.
Be the first to share your experience!