GuideIntermediate
Evaluation Frameworks Guide
Your LLM assistant already answers users. How do you know whether it does it well, where it fails, and whether yesterday's change made it worse? This guide teaches evaluation in the order strong teams use: first you read the conversations and find out what fails; then you build an evaluator for each failure that matters; then you validate each evaluator against a human expert; and finally you decide, as a product call, how good is good enough to launch. You learn it with **Ridgeline Rentals**, a fictional shop that rents bikes, e-bikes, kayaks, and camping gear, and its chat assistant that answers policy questions, recommends gear, and makes bookings with tools. You read the 120 traces from its beta and write a failure taxonomy specific to the product; you monitor with code what code can measure; you write one binary LLM judge per failure mode and measure its TPR and TNR against the labels of Maya, the operations lead; you correct the rate the judge reports and report it with its interval; you generate synthetic data by dimensions to evaluate before you have users; you separate the retriever from the generator and the session from the turn to know which component to fix; you put an eval gate in CI that blocks a real regression; you watch production with sampling and alerts; and you close with the launch decision that Sam, the product owner, signs. All the code is standard-library Python and runs on your laptop without an API key, with a simulated model and deterministic data: you see exactly the same numbers as the guide. An optional `--live` mode uses the Claude API. No evaluation framework is required: frameworks are presented as options you adopt with judgment.
- lessons
- 64
- modules
- 8
- available in
- English · Spanish
- certificate
- Yes
- access
- Free

What you'll be able to do
- Read traces from an LLM product, do open and axial coding, and write a failure taxonomy with operational definitions, severity, and priority
- Know when to stop reviewing (saturation) and when to review again (criteria drift, incidents)
- Decide for each failure whether to fix it now, monitor it with code, or give it an LLM judge, and tell a guardrail from an evaluator
- Write code-based evaluators over the whole trace: assertions, tool calls, and reference facts, and explain why generic metrics mislead
- Write one binary LLM judge per failure mode, with a definition, examples, a critique before the verdict, and structured output
- Validate each judge against human labels with train, dev, and test splits, measure TPR and TNR, correct the observed rate, and report a confidence interval
- Generate synthetic data by dimensions, filter it, red team the system, and state what a synthetic set cannot tell you
- Evaluate by component: retriever recall@k and MRR, the generator on its own, multi-turn sessions, and agent trajectories
- Put an eval gate in CI, compare two versions with a paired test, and watch production with sampling, human audits, and alerts
- Treat the quality bar as a product decision: one bar per failure mode, the sample size you need, and a ship, stage, or hold rule written before you see the numbers
What you need to bring
It's for you if...
- Engineers who already have an assistant, a RAG system, or an agent in development or in production and want evidence that it works
- Teams that evaluate "by eye" or with a generic score today and want evaluators they can trust
- Technical and product leads who have to decide when an LLM feature is ready to launch
Requirements and materials
- Intermediate Python: functions, dictionaries, list comprehensions, modules, `argparse`, and reading and writing JSONL. Python 3.12 or later
- Having integrated an LLM from code (calls, tool calling, structured outputs); the Building with LLMs in Code guide covers it
- Knowing what RAG is and what an agent with tools is, at a conceptual level. No framework required
- No API key needed. For the optional `--live` mode: an Anthropic key and `pip install anthropic`
The syllabus, module by module
Open any of them to see its lessons.
- 1. Introduction: read first, measure second
- 2. Traces: the raw material
- 3. Reading a trace: where the failure starts
- 4. Open coding: free notes and a verdict
- 5. Axial coding: from notes to a failure taxonomy
- 6. Counting and prioritizing: what to fix now and what deserves an evaluator
- 7. Saturation and criteria drift: when to stop and when to review again
- 8. Project: Ridgeline's failure report
- 1. Introduction: the cheapest evaluator is the one that needs no other model
- 2. Choosing an evaluator for each failure
- 3. Assertions over the whole trace
- 4. Checking tool calls
- 5. Reference facts and expected answers
- 6. Why generic metrics mislead
- 7. An eval harness in plain Python
- 8. Project: the code evaluator suite and the v2 prompt
- 1. Introduction: a global rate doesn't tell you what to fix
- 2. Evaluating the retriever on its own
- 3. Evaluating the generator when the retriever got it right
- 4. Multi-turn sessions
- 5. Agent trajectories and tool calls
- 6. The transition failure matrix
- 7. Frameworks are optional
- 8. Project: component evals and prompt v3
Learn with live classes and reviews of your work.
In a group or 1:1, depending on your schedule.
What people usually ask
No limit. It's a free guide: come in whenever you like, as often as you like.
No. Modules run from easier to harder, but you can jump to the one you need. Progress is saved per lesson.
Whatever is needed is listed under “What you need to bring”, above. If nothing is listed there, you can start from zero.
In the WhatsApp group, and every two weeks there's a live with an instructor where questions get worked through.
Yes. When you finish every lesson you get a statement of completion. Pass the guide's certification exam and it becomes a certificate, with a verifiable code you can share on LinkedIn.
No. This guide is self-paced with no dates. The bootcamp is live, by cohort, with work someone reviews.
Start whenever you like
What students say
These reviews are from enrolled students who completed at least 50% of the course. We moderate reviews only on content grounds (spam, offensive language, personal data), never for being critical or negative.
No approved reviews yet.
Be the first to share your experience!