Calibrated Trust

Capsule overview

You already know the problem (only 3% trust it) and the two dangerous extremes (accept everything vs reject everything). Now it's time for the middle ground. Calibrated trust is the professional stance that lets you use AI coding tools with maximum productivity and minimum risk. It's not a magic formula — it's a principle: your level of trust should adjust to the type of task, the impact of an error, and your ability to verify the result.

In this capsule you're going to understand what calibrated trust is, how it works in practice, and why it's the most important skill for working with AI-generated code. Think of it like learning to drive: you don't brake at every corner (reject everything) nor ignore the traffic lights (accept everything). You drive with attention proportional to the context.

---

What Calibrated Trust Is

Definition

Calibrated trust is the process of adjusting your level of trust in AI-generated code based on objective factors, not on intuition or habit.

The factors are:

Trust = f(
    task_type,          → What type of code is it?
    error_impact,       → What happens if there's a bug?
    verifiability,      → Can I verify it easily?
    familiarity,        → Do I know this domain well?
    complexity          → How complex is the code?
)

You don't need to calculate a formula. You need to ask yourself these questions before accepting or rejecting output.

The core principle

Invest review time proportional to the risk. Not proportional to the amount of code.

A 100-line endpoint that only does CRUD might require 3 minutes of review. A 10-line function that validates permissions might require 15 minutes. The risk determines the review, not the size.

---

The 5 Calibration Factors

Factor 1: Type of task

Not all tasks are equal. AI is consistently good at some and consistently unpredictable at others:

AI is consistently GOOD at:
├── Boilerplate and setup
├── Basic CRUD
├── Formatting and simple data transformations
├── Documentation and README
├── Standard configuration (Docker, CI/CD)
└── Code that follows well-established patterns

AI is UNPREDICTABLE at:
├── Specific business logic
├── Security and authentication
├── Performance optimization
├── Complex regex
├── Code that depends on context you didn't give it
└── Edge cases and exhaustive error handling

AI is consistently BAD at:
├── Code that requires understanding the system's global state
├── Integration with APIs that change frequently
├── Debugging issues that require reproducing at runtime
└── System-level architecture decisions

Factor 2: Impact of error

Ask yourself: "If this code has a bug, what happens?"

LOW impact:
- Error in documentation → Easy to fix
- Bug in a setup script → Someone reports it, it gets fixed
- Bad formatting → Cosmetic

MEDIUM impact:
- Bug in an endpoint → Incorrect response to the client
- Unhandled edge case → Runtime crash for certain inputs
- Incorrect test → False confidence in quality

HIGH impact:
- Security hole → Data breach
- Incorrect business logic → Financial loss
- Data corruption → Irreversible
- Compliance failure → Legal

Factor 3: Verifiability

How easy is it to verify that the code is correct?

EASY to verify:
- Code with visible output (print, return)
- Pure functions (same input → same output)
- CRUD with test data
- Code with existing tests

HARD to verify:
- Side effects (modifies the database, sends emails)
- Race conditions (only appear under load)
- Security (you need to think like an attacker)
- Business logic (you need to understand the domain)

Factor 4: Familiarity

How well do you know this domain?

If you KNOW the domain:
- You can detect subtle errors quickly
- Your review is more efficient
- You can trust your quick review more
→ Calibrate upward (more trust if your review is OK)

If you DON'T know the domain:
- Subtle errors can go unnoticed
- Your review is less reliable
- You need more time and tools
→ Calibrate downward (less trust, more verification)

Factor 5: Complexity

How complex is the generated code?

LOW complexity:
- Linear flow (step 1 → step 2 → step 3)
- Few dependencies
- No state management
→ High trust, quick review

HIGH complexity:
- Multiple branches and conditions
- Cross-dependencies between modules
- Complex state management
- Concurrency or async
→ Low trust, exhaustive review

---

The Calibration Table

Combining the 5 factors, this is what calibration looks like in practice:

Scenario	Trust	Review time	What to review
Generate README.md	90%	1-2 min	That the info is correct
Project boilerplate	85%	2-3 min	Dependency versions
CRUD endpoint	70%	5-8 min	Validations, error handling
Utility function	65%	5-10 min	Edge cases, naming
Complex SQL query	40%	10-15 min	Query logic, injection
Business logic	30%	15-20 min	That it does what the business needs
Auth / JWT	15%	20-30 min	Everything: tokens, expiration, secrets, roles
Payment validation	10%	30+ min	Every line + tests + compliance
Regex for validation	20%	15-20 min	Test with edge-case inputs
AI-generated tests	50%	10-15 min	That they test the right thing, not just that they pass

How to read the table

• 90% trust: Accept almost always. Visual review of 1-2 minutes.
• 50-70% trust: Review the important parts. 5-15 minutes.
• 10-30% trust: Don't trust it. Read every line, verify against documentation, write tests.

---

Calibrated Trust in Action

Practical example: Generating an API

Imagine you ask Claude Code to generate a task-management API. The output includes:

# Claude Code generated this
from fastapi import FastAPI, HTTPException, Depends
from pydantic import BaseModel, Field
from typing import Optional, List
from datetime import datetime
import uuid

app = FastAPI(title="Task Manager API")

tasks_db: dict = {}

class TaskCreate(BaseModel):
    title: str = Field(..., min_length=1, max_length=200)
    description: Optional[str] = None
    priority: str = Field(default="medium")

class Task(TaskCreate):
    id: str
    created_at: datetime
    completed: bool = False

@app.post("/tasks", response_model=Task)
async def create_task(task: TaskCreate):
    task_id = str(uuid.uuid4())
    new_task = Task(
        id=task_id,
        created_at=datetime.now(),
        **task.model_dump()
    )
    tasks_db[task_id] = new_task
    return new_task

@app.get("/tasks", response_model=List[Task])
async def list_tasks(completed: Optional[bool] = None):
    tasks = list(tasks_db.values())
    if completed is not None:
        tasks = [t for t in tasks if t.completed == completed]
    return tasks

@app.get("/tasks/{task_id}", response_model=Task)
async def get_task(task_id: str):
    if task_id not in tasks_db:
        raise HTTPException(status_code=404, detail="Task not found")
    return tasks_db[task_id]

@app.put("/tasks/{task_id}/complete")
async def complete_task(task_id: str):
    if task_id not in tasks_db:
        raise HTTPException(status_code=404, detail="Task not found")
    tasks_db[task_id].completed = True
    return tasks_db[task_id]

@app.delete("/tasks/{task_id}")
async def delete_task(task_id: str):
    if task_id not in tasks_db:
        raise HTTPException(status_code=404, detail="Task not found")
    del tasks_db[task_id]
    return {"message": "Task deleted"}

Applying calibrated trust

Step 1: What type of code is it?

• CRUD endpoint → Base trust: 70%
• No auth or security → No critical-risk factors

Step 2: What's the impact of an error?

• It's a task manager, not a financial system → Medium impact
• It doesn't handle sensitive data → No compliance risk

Step 3: What do I verify? With 70% trust, I review the risk areas:

✅ Verify quickly:
- Correct imports → Yes, they all exist
- Pydantic models → Field with validation, good
- UUID for IDs → Correct

⚠️ Verify carefully:
- Priority validation → Only "medium" as default,
  but it accepts any string. Should it be an Enum?
  → Depends on the requirements. If you need fixed values, 
  switch to Enum.

- Error handling → It has 404 for a task not found.
  It's missing handling for other errors (e.g., what happens if the body
  is malformed).
  → Pydantic handles body validation automatically.
  OK for a basic CRUD.

- Delete without soft-delete → Deletes permanently.
  → Is that what you want? For a basic task manager, OK.
  For production, maybe soft-delete.

❌ I don't need to verify:
- Syntax (the linter does it)
- Code style (it's consistent)
- Variable names (clear and descriptive)

Step 4: Decision

• I accept the code with one change: add an Enum for priority
• Total time: ~5 minutes of review + 2 minutes for the change = 7 minutes

That's calibrated trust. You didn't review line by line. You didn't accept without looking. You reviewed what mattered for the type of task.

---

Calibration Anti-Patterns

Anti-pattern 1: Static calibration

❌ "My rule is: I review 50% of all AI code"

→ 50% of a README is too much
→ 50% of auth code is too little
→ Calibration should vary by task

Anti-pattern 2: Calibration by volume

❌ "If it's few lines, I trust it. If it's many, I review."

→ 3 lines of SQL injection are worse than 300 lines of CRUD
→ Risk doesn't correlate with size

Anti-pattern 3: Calibration by a single prior experience

❌ "Claude Code generated a bug in auth once, 
    now I don't trust anything it generates"

→ One error in auth doesn't invalidate its capability for CRUD
→ Calibrate by type of task, not by isolated memories

Anti-pattern 4: Calibration by social pressure

❌ "My lead says to always review everything, 
    so I review every import statement"

→ Reviewing every import is a waste
→ Your lead probably wants you to review what's important
→ Show them your calibration table

---

Troubleshooting

Problem 1: "I don't know how to evaluate a task's risk"

Cause: Lack of experience classifying tasks by impact. Solution: Ask yourself a simple question: "If this code has a bug and reaches production, who gets called at 3 AM?" If the answer is "no one, we fix it tomorrow" → low risk. If it's "the security team" → high risk.

Problem 2: "My calibration table doesn't cover my specific case"

Cause: The table is a starting point, not an encyclopedia. Solution: Use the 5 factors directly. Evaluate: (1) type of task, (2) impact, (3) verifiability, (4) your familiarity, (5) complexity. The combination gives you a reasonable trust range.

Problem 3: "I calibrate well but my team doesn't, and their bugs affect me"

Cause: A team problem, not an individual one. Solution: Share your calibration table with the team. In Guide 7 (Git Workflows) you'll see how to integrate this into the team's code review flow.

---

Exercises

Exercise 1: Assign trust (Easy)

For each task, assign a % of trust and justify it with the 5 factors:

1. Generate a script that reads a CSV and converts it to JSON
2. Create an authentication function with OAuth2
3. Write docstrings for 10 existing functions
4. Implement a custom rate limiter

See solution

1. CSV to JSON → 75% trust.

   • Type: data transformation (AI is good)
   • Impact: low (not security, not business-critical)
   • Verifiability: high (you run it and see the output)
   • Review: that it handles empty files, encoding, and data types

2. OAuth2 auth → 15% trust.

   • Type: security (AI is unpredictable)
   • Impact: critical (a breach if it's wrong)
   • Verifiability: hard (you need to think like an attacker)
   • Review: EVERY line, token handling, secret management, scopes

3. Docstrings → 85% trust.

   • Type: documentation (AI is good)
   • Impact: low (an error in docs doesn't cause bugs)
   • Verifiability: high (you read and verify)
   • Review: that the descriptions are correct, not misleading

4. Rate limiter → 30% trust.

   • Type: infrastructure with edge cases (AI is unpredictable)
   • Impact: high (bad rate limiting = DoS or false positives)
   • Verifiability: hard (you need to test under load)
   • Review: algorithm, edge cases (what happens if the clock jumps), storage

Exercise 2: Calibrate a real output (Medium)

Claude Code generates this function. Apply the 5 factors and decide: do you accept, edit, or reject?

def calculate_discount(price: float, user_type: str, quantity: int) -> float:
    if user_type == "premium":
        discount = 0.20
    elif user_type == "regular":
        discount = 0.10
    else:
        discount = 0.0
    
    if quantity > 100:
        discount += 0.05
    elif quantity > 50:
        discount += 0.03
    
    final_price = price * (1 - discount)
    return round(final_price, 2)

See solution

Analysis with the 5 factors:

1. Type: Business logic (pricing) → Low base trust (30-40%)
2. Impact: High (an error = overcharging or undercharging) → Low trust
3. Verifiability: Medium (you can test with known inputs)
4. Familiarity: Depends on you — do you know your business's discount rules?
5. Complexity: Low (linear if/else)

Detected problems:

• Are the percentages correct? Only you know (business logic)
• What happens with a negative price? It doesn't validate
• What happens with a negative quantity? It doesn't validate
• Should user_type be an Enum? It accepts any string
• Can the discount exceed 25%? There's no maximum cap
• What happens if price is 0? It works, but does it make sense?

Decision: Edit. The structure is fine (I accept ~70% of the code). But I need to:

1. Verify the percentages against the real business rules
2. Add input validation (negative price/quantity)
3. Consider a maximum discount cap
4. Use an Enum for user_type

Exercise 3: Create your custom table (Medium)

Think about your current job (or a personal project). List 8 types of tasks you usually ask AI for and assign trust to each based on the 5 factors.

See solution

There's no single answer — it depends on your context. But your table should:

• ✅ Have at least 3 different trust levels (not everything at 50%)
• ✅ Security/auth tasks should be in the 10-25% range
• ✅ Boilerplate and documentation should be at 75-90%
• ✅ Business logic should be at 25-40%
• ✅ Each entry should have a "what to review" column

Example for a backend developer:

Task	Trust	What to review
Docker compose	85%	Versions, ports
API endpoints	65%	Validations, error handling
Database queries	45%	SQL correctness, N+1, injection
Auth middleware	15%	Everything
Business rules	30%	Against the requirements
Tests	55%	That they test the right thing
Config files	80%	Default values
Error messages	75%	That they don't expose sensitive info

Exercise 4: Case study (Hard)

A team has this policy: "All AI code must pass code review by another developer before merge." Is it a good policy? Argue the pros and cons using the concept of calibrated trust.

See solution

Pros:

• Double verification reduces the risk of bugs in production
• It distributes knowledge of the code (not only the author understands it)
• Consistent — it doesn't depend on each developer's individual judgment
• It surfaces problems the author didn't see (fresh eyes)

Cons (from a calibrated-trust view):

• It applies the same process to everything — a README PR gets the same review as an auth PR
• Bottleneck: if the reviewer is busy, the PR waits (even if it's boilerplate)
• False safety: if the reviewer doesn't calibrate either, they review everything superficially
• It doesn't incentivize the author to review well — "the reviewer will catch it"
• Costly: 2 people review what 1 person could calibrate correctly

Improved policy with calibration:

• Low-risk PRs (boilerplate, config, docs): auto-merge with a checklist
• Medium-risk PRs (CRUD, features): review by 1 developer
• High-risk PRs (auth, payments, data migration): review by 2 developers + required tests

Conclusion: The policy isn't bad, but it lacks calibration. A mandatory code review for ALL code treats everything as equally risky — exactly the static-calibration anti-pattern.

---

Summary

In this capsule you learned:

• Calibrated trust is adjusting your level of trust based on objective factors, not on habit
• The 5 factors of calibration: type of task, impact of error, verifiability, familiarity, complexity
• The core principle: invest review time proportional to the risk, not to the volume of code
• The calibration table gives you a concrete starting point for each type of task
• The anti-patterns to avoid: static calibration, by volume, by isolated experience, by social pressure
• Calibrated trust isn't a calculation — it's a thinking habit you develop with practice

Next capsule: Your First Verification Framework — the concrete framework you're going to use tomorrow.

---

Additional resources

1. Thinking, Fast and Slow — Daniel Kahneman - Theoretical basis for calibration and cognitive biases
2. Google Engineering Practices — Code Review - How Google calibrates code review by type of change
3. Anthropic — Claude Code Documentation - Official recommendations for working with Claude Code output
4. OWASP — Code Review Guide - A code review framework with a security focus
5. Linus Torvalds on Code Review - The code review philosophy of the creator of Linux
6. Risk-Based Testing Approach - Principles of risk-based testing applicable to code review

---

Debugging & Code Review with Claude Code — Module 1, Capsule 04 Claude Code Agentic Development Path — Guide #6 of 11