GuideIntermediate

Evaluation Frameworks Guide

Your LLM assistant already answers users. How do you know whether it does it well, where it fails, and whether yesterday's change made it worse? This guide teaches evaluation in the order strong teams use: first you read the conversations and find out what fails; then you build an evaluator for each failure that matters; then you validate each evaluator against a human expert; and finally you decide, as a product call, how good is good enough to launch. You learn it with **Ridgeline Rentals**, a fictional shop that rents bikes, e-bikes, kayaks, and camping gear, and its chat assistant that answers policy questions, recommends gear, and makes bookings with tools. You read the 120 traces from its beta and write a failure taxonomy specific to the product; you monitor with code what code can measure; you write one binary LLM judge per failure mode and measure its TPR and TNR against the labels of Maya, the operations lead; you correct the rate the judge reports and report it with its interval; you generate synthetic data by dimensions to evaluate before you have users; you separate the retriever from the generator and the session from the turn to know which component to fix; you put an eval gate in CI that blocks a real regression; you watch production with sampling and alerts; and you close with the launch decision that Sam, the product owner, signs. All the code is standard-library Python and runs on your laptop without an API key, with a simulated model and deterministic data: you see exactly the same numbers as the guide. An optional `--live` mode uses the Claude API. No evaluation framework is required: frameworks are presented as options you adopt with judgment.

lessons
64
modules
8
available in
English · Spanish
certificate
Yes
access
Free
Outcomes

What you'll be able to do

  • Read traces from an LLM product, do open and axial coding, and write a failure taxonomy with operational definitions, severity, and priority
  • Know when to stop reviewing (saturation) and when to review again (criteria drift, incidents)
  • Decide for each failure whether to fix it now, monitor it with code, or give it an LLM judge, and tell a guardrail from an evaluator
  • Write code-based evaluators over the whole trace: assertions, tool calls, and reference facts, and explain why generic metrics mislead
  • Write one binary LLM judge per failure mode, with a definition, examples, a critique before the verdict, and structured output
  • Validate each judge against human labels with train, dev, and test splits, measure TPR and TNR, correct the observed rate, and report a confidence interval
  • Generate synthetic data by dimensions, filter it, red team the system, and state what a synthetic set cannot tell you
  • Evaluate by component: retriever recall@k and MRR, the generator on its own, multi-turn sessions, and agent trajectories
  • Put an eval gate in CI, compare two versions with a paired test, and watch production with sampling, human audits, and alerts
  • Treat the quality bar as a product decision: one bar per failure mode, the sample size you need, and a ship, stage, or hold rule written before you see the numbers
Before you start

What you need to bring

It's for you if...

  • Engineers who already have an assistant, a RAG system, or an agent in development or in production and want evidence that it works
  • Teams that evaluate "by eye" or with a generic score today and want evaluators they can trust
  • Technical and product leads who have to decide when an LLM feature is ready to launch

Requirements and materials

  • Intermediate Python: functions, dictionaries, list comprehensions, modules, `argparse`, and reading and writing JSONL. Python 3.12 or later
  • Having integrated an LLM from code (calls, tool calling, structured outputs); the Building with LLMs in Code guide covers it
  • Knowing what RAG is and what an agent with tools is, at a conceptual level. No framework required
  • No API key needed. For the optional `--live` mode: an Anthropic key and `pip install anthropic`
Content

The syllabus, module by module

Open any of them to see its lessons.

Bootcamps

Learn with live classes and reviews of your work.

In a group or 1:1, depending on your schedule.

Common questions

What people usually ask

Start whenever you like

Reviews

What students say

These reviews are from enrolled students who completed at least 50% of the course. We moderate reviews only on content grounds (spam, offensive language, personal data), never for being critical or negative.

No approved reviews yet.

Be the first to share your experience!