Hallucination Detection Tools

Capsule overview

Your human eye is the first line of defense against hallucinations — but it isn't infallible. In capsules 03 and 04 you trained your eye to detect fake imports, invented APIs, incorrect parameters, and fabricated logic. Now you're going to build your automated safety net: tools that detect what your eye doesn't see.

This capsule gives you a concrete toolkit. Not theory about which tools exist — but exact commands, configurations, and processes you can use tomorrow. By the end, you'll have a 4-layer verification process that catches hallucinations at every level.

The 4 Verification Layers

The defense-in-depth model

No single tool detects all types of hallucinations. That's why you use layers: if one fails, the next catches the error.

Layer 1: Static linters (ruff, flake8, pylint)
├── Detects: nonexistent imports, undefined variables, 
│            syntax errors, unused imports
├── Type it catches: Type 1 (Fake imports) — partial
├── Time: < 1 second
└── Effort: 0 (runs automatically)

Layer 2: Type checkers (mypy, pyright)
├── Detects: incorrect types, nonexistent attributes,
│            incompatible signatures
├── Type it catches: Type 2 (Invented APIs) — partial
│                    Type 3 (Incorrect parameters) — partial
├── Time: 2-5 seconds
└── Effort: initial configuration

Layer 3: Quick tests (pytest, python -c)
├── Detects: incorrect logic, edge cases, 
│            unexpected behavior
├── Type it catches: Type 4 (Fabricated logic) — partial
│                    Type 2 and 3 — at runtime
├── Time: 1-5 minutes (write + run)
└── Effort: medium (writing tests)

Layer 4: Official documentation
├── Detects: everything the other layers don't catch
├── Type it catches: All types
├── Time: 2-10 minutes
└── Effort: high (reading and comparing)

The idea: layers 1 and 2 are automatic and fast. Layer 3 requires effort but is the most effective against fabricated logic. Layer 4 is the last resort for checks no tool can do.

Layer 1: Static Linters

ruff — Python's fast linter

ruff is the fastest linter for Python. It detects hundreds of types of errors in milliseconds.

Installation:

pip install ruff

Basic usage:

# Analyze a file
ruff check app.py

# Analyze an entire directory
ruff check src/

# Show errors with context
ruff check app.py --show-source

# Fix errors automatically where possible
ruff check app.py --fix

What it detects that's relevant for hallucinations:

F821: Undefined name → Undefined variable or function
F401: Imported but unused → An import that's imported but not used
E902: Syntax error → A syntax error
F811: Redefined unused name → Redefinition of a variable

Example:

# file: app.py
from fastapi.security import OAuth2TokenValidator  # ← Hallucination?
from pydantic import BaseModel

class User(BaseModel):
    name: str

$ ruff check app.py
app.py:1:1: F401 `fastapi.security.OAuth2TokenValidator` imported but unused

Wait — ruff flags it as "imported but unused", not as "nonexistent import." This is because ruff analyzes text, it doesn't execute imports. To verify that the import exists, you need layer 2 or run Python directly.

Important limitation: ruff detects unused imports but doesn't verify that the import exists. A fake import that's used in the code won't be flagged by ruff. You need type checkers (layer 2) for that.

flake8 with plugins

If you prefer flake8 or your project already uses it:

pip install flake8 flake8-import-order flake8-bugbear

# Analyze a file
flake8 app.py

# With more detail
flake8 app.py --show-source --statistics

Recommended ruff configuration

Create a ruff.toml file in the root of your project:

# ruff.toml
line-length = 88
target-version = "py311"

[lint]
select = [
    "E",    # pycodestyle errors
    "F",    # pyflakes (imports, undefined names)
    "I",    # isort (import ordering)
    "N",    # pep8 naming
    "UP",   # pyupgrade (deprecated syntax)
    "B",    # bugbear (common bugs)
    "S",    # bandit (security issues)
    "T20",  # print statements
    "SIM",  # simplify
]

[lint.per-file-ignores]
"tests/*" = ["S101"]  # Allow assert in tests

What ruff catches and what it doesn't

Layer 2: Type Checkers

mypy — static type checking

mypy goes beyond ruff: it verifies that data types are consistent, that the attributes and methods you call exist on the declared types, and that function signatures are correct.

Installation:

pip install mypy

Basic usage:

# Check a file
mypy app.py

# Check more strictly
mypy app.py --strict

# Ignore imports without stubs
mypy app.py --ignore-missing-imports

What it detects that's relevant for hallucinations:

# file: app.py
from fastapi.security import OAuth2TokenValidator  # ← Hallucination

from pydantic import BaseModel, Field

class User(BaseModel):
    email: str = Field(..., unique=True)  # ← unique isn't a parameter of Field

$ mypy app.py
app.py:1: error: Module "fastapi.security" has no attribute "OAuth2TokenValidator"
app.py:5: error: Unexpected keyword argument "unique" for "Field"

mypy detects both hallucinations: the fake import AND the incorrect parameter. This makes it significantly more useful than ruff for detecting hallucinations.

pyright — a faster alternative

If you use VS Code, pyright (the engine behind Pylance) is another excellent option:

pip install pyright

# Check a file
pyright app.py

# Check with detailed output
pyright app.py --outputjson

mypy configuration for hallucination detection

Create mypy.ini in the project root:

[mypy]
python_version = 3.11
warn_return_any = True
warn_unused_configs = True
disallow_untyped_defs = True
check_untyped_defs = True
warn_unused_ignores = True
show_error_codes = True

# For libraries without type stubs
[mypy-uvicorn.*]
ignore_missing_imports = True

What mypy catches and what it doesn't

Complete example of mypy catching hallucinations

# file: auth_service.py
from fastapi import FastAPI, Depends, HTTPException
from fastapi.security import OAuth2PasswordBearer, OAuth2TokenValidator  # ← H1
from pydantic import BaseModel, Field
import jwt

app = FastAPI()

class UserCreate(BaseModel):
    email: str = Field(..., unique=True)  # ← H2
    name: str

def create_token(data: dict) -> str:
    return jwt.encode(data, "secret", algorithm="HS256")

def verify_token(token: str) -> dict:
    return jwt.decode(token, "secret", algorithms=["HS256"], verify=True)  # ← H3

$ mypy auth_service.py
auth_service.py:2: error: Module "fastapi.security" has no attribute 
    "OAuth2TokenValidator"  [attr-defined]
auth_service.py:6: error: Unexpected keyword argument "unique" for 
    "Field"  [call-arg]
auth_service.py:14: error: Unexpected keyword argument "verify" for 
    "decode"  [call-arg]
Found 3 errors in 1 file (checked 1 source file)

mypy detected all 3 hallucinations. This is why layer 2 is so valuable: a single run can find multiple hallucinations your eye might overlook.

Layer 3: Quick Tests

Why tests are essential against logic hallucinations

Layers 1 and 2 don't detect logic hallucinations — code that compiles, has correct types, but produces incorrect results. For that you need to run the code and verify the output.

The 3-minute process

For any suspicious function, write a quick test in 3 minutes or less:

# Method 1: python -c (for quick one-line checks)
python -c "
from app import validate_email
tests = [
    ('user@domain.com', True),
    ('@domain.com', False),
    ('user@.com', False),
    ('', False),
]
for email, expected in tests:
    result = validate_email(email)
    status = '✅' if result == expected else '❌'
    print(f'{status} validate_email(\"{email}\") = {result}, expected {expected}')
"

# Method 2: pytest with a temporary file
cat > test_quick.py << 'EOF'
from app import calculate_median
import statistics

def test_median_odd():
    assert calculate_median([1, 2, 3]) == statistics.median([1, 2, 3])

def test_median_even():
    assert calculate_median([1, 2, 3, 4]) == statistics.median([1, 2, 3, 4])

def test_median_single():
    assert calculate_median([5]) == 5.0

def test_median_empty():
    import pytest
    with pytest.raises(ValueError):
        calculate_median([])
EOF

pytest test_quick.py -v

Quick test templates by type of function

For validation functions:

def test_validation_function(validate_func):
    """Template for testing validation functions."""
    valid_inputs = [
        "normal_valid_input",
    ]
    invalid_inputs = [
        "",              # empty
        None,            # null
        " ",             # only spaces
        "a" * 10000,     # very long
    ]
    
    for inp in valid_inputs:
        assert validate_func(inp) == True, f"Should accept: {inp}"
    
    for inp in invalid_inputs:
        assert validate_func(inp) == False, f"Should reject: {inp}"

For calculation functions:

import math

def test_calculation_function(calc_func, reference_func):
    """Template for testing calculation functions against a reference."""
    test_cases = [
        [1, 2, 3, 4, 5],       # normal
        [1],                     # one element
        [0, 0, 0],              # all zero
        [-1, -2, -3],           # negatives
        [1.5, 2.7, 3.14],      # decimals
        list(range(1000)),      # large
    ]
    
    for data in test_cases:
        result = calc_func(data)
        expected = reference_func(data)
        assert math.isclose(result, expected, rel_tol=1e-9), \
            f"For {data[:5]}...: got {result}, expected {expected}"

For transformation functions:

def test_transformation_function(transform_func):
    """Template for testing data transformation functions."""
    assert transform_func("hello") is not None  # doesn't return None
    
    original = "test_input"
    result = transform_func(original)
    assert isinstance(result, str)  # correct type
    
    assert transform_func("") == ""  # empty produces empty

Verify imports with python -c

For the most basic layer of verification, run the imports directly:

# Verify a specific import
python -c "from fastapi.security import OAuth2TokenValidator" 2>&1
# ImportError: cannot import name 'OAuth2TokenValidator'...

# Verify multiple imports from a file
python -c "
imports_to_check = [
    ('fastapi.security', 'OAuth2PasswordBearer'),
    ('fastapi.security', 'OAuth2TokenValidator'),
    ('pydantic', 'BaseModel'),
    ('pydantic', 'EmailStr'),
]

for module, name in imports_to_check:
    try:
        exec(f'from {module} import {name}')
        print(f'  ✅ from {module} import {name}')
    except ImportError as e:
        print(f'  ❌ from {module} import {name} — {e}')
"

Verify signatures with inspect

# See a function's signature
python -c "
import inspect
import jwt

sig = inspect.signature(jwt.decode)
print(f'jwt.decode{sig}')
print()
for name, param in sig.parameters.items():
    print(f'  {name}: {param.kind.name} = {param.default}')
"

Output:

jwt.decode(jwt, key='', algorithms=None, options=None, ...)

  jwt: POSITIONAL_OR_KEYWORD = <class 'inspect._empty'>
  key: POSITIONAL_OR_KEYWORD = 
  algorithms: POSITIONAL_OR_KEYWORD = None
  options: POSITIONAL_OR_KEYWORD = None
  ...

If the code uses jwt.decode(..., verify=True) and verify doesn't appear in the signature, it's a confirmed hallucination.

Layer 4: Official Documentation

When to use the documentation

Use the documentation when:

• Layers 1-3 don't give a clear answer
• The parameter could be accepted via **kwargs (it doesn't appear in the signature but could work)
• You need to verify valid values for a parameter (not just that the parameter exists)
• The behavior depends on the library version

How to verify efficiently

To verify a parameter:

1. Open the library's official documentation
2. Look up the specific function/class
3. Verify:
   a. Does the parameter exist?
   b. Is the type correct?
   c. Is the value valid?
   d. Is it deprecated?

Example: verify pd.DataFrame.to_json(orient="dict")

1. Open: https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.to_json.html
2. Look up the orient parameter
3. Valid values: "split", "records", "index", "columns", "values", "table"
4. "dict" isn't in the list → hallucination confirmed

Example: verify requests.get(url, verify_ssl=True)

1. Open: https://requests.readthedocs.io/en/latest/api/#requests.get
2. Look up the accepted parameters
3. The SSL verification parameter is verify, not verify_ssl
4. Hallucination confirmed

Offline documentation with help()

If you don't have internet access or prefer to verify quickly:

# See the full documentation
python -c "import requests; help(requests.get)"

# See only the first part (signature and description)
python -c "
import requests
doc = requests.get.__doc__
print(doc[:500] if doc else 'No documentation')
"

# See a method's parameters
python -c "
import pandas as pd
help(pd.DataFrame.to_json)
" | head -30

Essential documentation links

For the FastAPI stack you'll use in the capstone project:

The Complete Process: From Code to Confidence

4-step verification workflow

When Claude Code generates code, apply this process:

STEP 1: ruff check (< 1 second)
├── Run: ruff check file.py
├── Look for: F401 (unused imports), F821 (undefined), E902 (syntax)
├── If it finds errors → Fix before continuing
└── If it passes → Continue to step 2

STEP 2: mypy (2-5 seconds)
├── Run: mypy file.py --ignore-missing-imports
├── Look for: attr-defined, call-arg, type errors
├── If it finds errors → Investigate each one
│   ├── attr-defined → Fake import or nonexistent method
│   ├── call-arg → Incorrect parameter
│   └── type error → Incorrect type
└── If it passes → Continue to step 3

STEP 3: Quick tests (1-5 minutes)
├── Identify high-risk functions:
│   ├── Validation
│   ├── Calculations
│   ├── Security
│   └── Data transformation
├── Write 3-5 quick tests for each one
├── Run: pytest test_quick.py -v
├── If any test fails → Logic hallucination found
└── If they all pass → Reasonable confidence

STEP 4: Documentation (2-10 minutes, only if needed)
├── For suspicious parameters that passed mypy (**kwargs)
├── For parameter values (mypy doesn't verify values)
├── For version-specific behavior
└── Verify against official documentation

Complete example: verifying AI-generated code

Claude Code generates this code:

from fastapi import FastAPI, HTTPException, Depends
from fastapi.security import OAuth2PasswordBearer
from pydantic import BaseModel, Field
import jwt
import requests

app = FastAPI()
oauth2_scheme = OAuth2PasswordBearer(tokenUrl="token")

class Item(BaseModel):
    name: str = Field(..., min_length=1, max_length=200)
    price: float = Field(..., gt=0)
    category: str = Field(..., unique=True)

def verify_token(token: str) -> dict:
    return jwt.decode(token, "secret", algorithms=["HS256"], verify=True)

def fetch_external_data(url: str) -> dict:
    response = requests.get(url, verify_ssl=True, timeout=30)
    return response.json()

STEP 1: ruff check

$ ruff check app.py
# (assuming all imports are used) → No errors

STEP 2: mypy

$ mypy app.py
app.py:13: error: Unexpected keyword argument "unique" for "Field"  [call-arg]
app.py:16: error: Unexpected keyword argument "verify" for "decode"  [call-arg]

Results:

• ❌ Field(..., unique=True) — unique isn't a Pydantic Field parameter
• ❌ jwt.decode(..., verify=True) — verify isn't a jwt.decode parameter

STEP 3: Quick test (to verify verify_ssl)

mypy didn't detect verify_ssl because requests uses **kwargs. Manual verification:

python -c "
import inspect
import requests
sig = inspect.signature(requests.get)
print(sig)
"
# (url, **kwargs) — we can't see the real parameters

python -c "
import requests
help(requests.get)
" | grep -i "verify"
# :param verify: ... Either a boolean, in which case...
# The parameter is "verify", not "verify_ssl"

Result:

• ❌ requests.get(url, verify_ssl=True) — the parameter is verify, not verify_ssl

STEP 4: Documentation (confirmation)

• Open https://requests.readthedocs.io/en/latest/api/
• Confirm: verify is the correct parameter

Summary: 3 hallucinations detected in a 20-line file. Total time: ~5 minutes.

Automation: Integrating It into Your Workflow

Pre-commit hooks

Configure automatic verification before every commit:

pip install pre-commit

Create .pre-commit-config.yaml:

repos:
  - repo: https://github.com/astral-sh/ruff-pre-commit
    rev: v0.4.0
    hooks:
      - id: ruff
        args: [--fix]
      - id: ruff-format

  - repo: https://github.com/pre-commit/mirrors-mypy
    rev: v1.9.0
    hooks:
      - id: mypy
        additional_dependencies:
          - fastapi
          - pydantic
          - types-requests

pre-commit install

Now, every time you git commit, ruff and mypy run automatically. If they detect hallucinations, the commit fails and you can fix them before pushing code.

Quick verification script

Create a script that runs the first 3 layers with a single command:

#!/bin/bash
# verify.sh — Quick verification of AI-generated code

FILE=${1:-"app.py"}

echo "=== Layer 1: ruff ==="
ruff check "$FILE" --show-source
echo ""

echo "=== Layer 2: mypy ==="  
mypy "$FILE" --ignore-missing-imports --show-error-codes
echo ""

echo "=== Layer 3: Import verification ==="
python -c "
import ast
import sys

with open('$FILE') as f:
    tree = ast.parse(f.read())

for node in ast.walk(tree):
    if isinstance(node, ast.ImportFrom):
        module = node.module or ''
        for alias in node.names:
            name = alias.name
            try:
                exec(f'from {module} import {name}')
                print(f'  ✅ from {module} import {name}')
            except ImportError as e:
                print(f'  ❌ from {module} import {name} — {e}')
    elif isinstance(node, ast.Import):
        for alias in node.names:
            try:
                __import__(alias.name)
                print(f'  ✅ import {alias.name}')
            except ImportError as e:
                print(f'  ❌ import {alias.name} — {e}')
"

echo ""
echo "=== Verification complete ==="

chmod +x verify.sh
./verify.sh app.py

VS Code / Cursor integration

If you use Cursor (as in this guide), configure these extensions:

1. Pylance (pyright built-in) — real-time type checking
2. Ruff — real-time linting
3. Python — IntelliSense and autocomplete

With Pylance enabled, you'll see type checking errors directly in the editor while Claude Code generates code. If you see lines underlined in red after Claude Code finishes, review them — they could be hallucinations.

Connection to the Project

Toolkit for the capstone project

In module 8, you'll receive a FastAPI codebase with ~15-20 problems. Your verification process should be:

1. ruff check src/ → Finds unused imports, syntax issues
2. mypy src/ → Finds fake imports, incorrect parameters
3. Quick tests for security functions → Finds fabricated logic
4. Documentation for suspicious parameters → Confirms hallucinations

Time estimate:

• Layer 1 (ruff): 30 seconds
• Layer 2 (mypy): 2 minutes
• Layer 3 (quick tests): 15-20 minutes
• Layer 4 (docs): 5-10 minutes as needed

Total: ~30 minutes for the project's hallucination-detection phase.

What you'd miss without tools

Without the automated layers, the only tool would be your eye. An average developer reviewing 500-800 lines of code would detect:

• ~80% of fake imports (the most obvious ones)
• ~50% of incorrect parameters (the ones they know)
• ~30% of fabricated logic (only in domains they master)

With the 4 layers:

• ~95% of fake imports (ruff + mypy + import verification)
• ~80% of incorrect parameters (mypy + docs)
• ~60% of fabricated logic (quick tests + domain knowledge)

The tools don't replace your eye — they complement it. Together they cover significantly more than either one separately.

Troubleshooting

Problem 1: "mypy gives too many errors in my code — I don't know which are hallucinations"

Cause: mypy in strict mode can give hundreds of errors in code that has no type annotations.

Solution: Don't use --strict for hallucination detection. Use the default mode and focus on these error codes:

[attr-defined] → Fake import or nonexistent method (HIGH priority)
[call-arg]     → Incorrect parameter (HIGH priority)
[import]       → Module not found (HIGH priority)
[name-defined] → Undefined variable (MEDIUM priority)

Ignore generic type errors [type-arg], [return-type] — they're typing issues, not hallucinations.

Problem 2: "The import passes python -c but mypy says it doesn't exist"

Cause: The package is installed but doesn't have type stubs.

Solution: Install type stubs for the main libraries:

pip install types-requests types-PyYAML types-redis

For libraries without official stubs, add to mypy.ini:

[mypy-library_name.*]
ignore_missing_imports = True

Problem 3: "The tools don't detect hallucinations in functions with **kwargs"

Cause: **kwargs accepts any argument, hiding hallucinations.

Solution: For functions with **kwargs (like requests.get()):

1. Don't trust mypy for these functions
2. Use inspect.signature() to see the documented parameters
3. Verify against the official documentation
4. Do a quick test to confirm that the parameter has an effect

Problem 4: "I don't have time to run 4 layers on every file"

Cause: Complete verification takes time.

Solution: Prioritize:

• Always: Layer 1 (ruff) — takes <1 second
• Always: Layer 2 (mypy) — takes 2-5 seconds
• For risky code: Layer 3 (quick tests) — takes 1-5 minutes
• Only if in doubt: Layer 4 (documentation) — takes 2-10 minutes

Layers 1 and 2 should be automatic (pre-commit hooks or editor extensions). You use layers 3 and 4 selectively for high-risk code.

Problem 5: "Do these tools work for JavaScript/TypeScript?"

Cause: This guide uses Python, but the principles apply to any language.

Solution: The equivalents in JS/TS:

• Layer 1: ESLint (linting)
• Layer 2: TypeScript compiler (type checking)
• Layer 3: Jest/Vitest (quick tests)
• Layer 4: MDN / npm package documentation

The principles of the 4 layers are universal.

Exercises

Exercise 1: Run the 4 layers (Easy)

Take this code and run the 4 verification layers. Document what each layer finds:

from fastapi import FastAPI, HTTPException
from fastapi.security import OAuth2PasswordBearer
from pydantic import BaseModel, Field
import jwt

app = FastAPI()

class UserLogin(BaseModel):
    email: str = Field(..., format="email")
    password: str = Field(..., min_length=8)

def create_token(user_id: str) -> str:
    return jwt.encode(
        {"sub": user_id}, 
        "secret", 
        algorithm="HS256"
    )

Exercise 2: Set up verification (Medium)

Set up ruff + mypy for a Python project. Create the configuration files and run against an example file. Document:

1. The configuration files you created
2. The commands you ran
3. The results

line-length = 88
target-version = "py311"

[lint]
select = ["E", "F", "I", "B", "S", "UP"]

[mypy]
python_version = 3.11
check_untyped_defs = True
show_error_codes = True

[mypy-uvicorn.*]
ignore_missing_imports = True

pip install ruff mypy
ruff check app.py --show-source
mypy app.py --show-error-codes

Exercise 3: Import verification script (Medium)

Write a script that takes a Python file and automatically verifies all the imports. The script should:

1. Extract all the imports from the file
2. Try to run each import
3. Report which ones work and which don't

import ast
import sys
import importlib

def verify_imports(filepath: str) -> None:
    """Verifies all imports in a Python file."""
    with open(filepath) as f:
        tree = ast.parse(f.read())
    
    print(f"Verifying imports in {filepath}:\n")
    
    passed = 0
    failed = 0
    
    for node in ast.walk(tree):
        if isinstance(node, ast.ImportFrom):
            module = node.module or ""
            for alias in node.names:
                name = alias.name
                try:
                    exec(f"from {module} import {name}")
                    print(f"  ✅ from {module} import {name}")
                    passed += 1
                except ImportError as e:
                    print(f"  ❌ from {module} import {name} — {e}")
                    failed += 1
        elif isinstance(node, ast.Import):
            for alias in node.names:
                try:
                    importlib.import_module(alias.name)
                    print(f"  ✅ import {alias.name}")
                    passed += 1
                except ImportError as e:
                    print(f"  ❌ import {alias.name} — {e}")
                    failed += 1
    
    print(f"\nResults: {passed} passed, {failed} failed")
    if failed > 0:
        print(f"⚠️  {failed} potential hallucination(s) found!")

if __name__ == "__main__":
    if len(sys.argv) != 2:
        print("Usage: python verify_imports.py <file.py>")
        sys.exit(1)
    verify_imports(sys.argv[1])

Exercise 4: Detect with mypy (Hard)

This code has 3 hallucinations. Use mypy to find at least 2 of the 3:

from fastapi import FastAPI, HTTPException, Depends
from fastapi.security import OAuth2PasswordBearer
from fastapi.middleware.cors import CORSMiddleware
from pydantic import BaseModel, Field
from sqlalchemy import create_engine, Column, Integer, String
from sqlalchemy.orm import Session, declarative_base, joinedload
from typing import Optional, List
import jwt

app = FastAPI()

app.add_middleware(
    CORSMiddleware,
    allowed_origins=["*"],
    allow_methods=["*"],
    allow_headers=["*"],
)

Base = declarative_base()

class UserDB(Base):
    __tablename__ = "users"
    id = Column(Integer, primary_key=True)
    email = Column(String, unique=True)
    name = Column(String)

class UserCreate(BaseModel):
    email: str = Field(..., unique=True)
    name: str = Field(..., min_length=2, max_length=100)

class UserResponse(BaseModel):
    id: int
    email: str
    name: str
    
    model_config = {"from_attributes": True}

oauth2_scheme = OAuth2PasswordBearer(tokenUrl="token")

def get_current_user(token: str = Depends(oauth2_scheme)):
    payload = jwt.decode(token, "secret", algorithms=["HS256"], verify=True)
    return payload.get("sub")

Exercise 5: Design your verification pipeline (Hard)

Design a custom verification pipeline for your work stack. It should include:

1. Tools for each layer
2. Minimum configuration
3. Commands to run
4. What type of hallucination each step detects
5. When it runs (manual vs automatic)

Summary

In this capsule you learned:

• The 4 verification layers: linters → type checkers → quick tests → documentation
• Layer 1 (ruff): detects unused imports, syntax errors — automatic, < 1 second
• Layer 2 (mypy): detects fake imports, incorrect parameters — automatic, 2-5 seconds
• Layer 3 (tests): detects fabricated logic — manual, 1-5 minutes
• Layer 4 (docs): verifies everything the others don't cover — manual, 2-10 minutes
• mypy is the most valuable tool against type 1-3 hallucinations
• Functions with **kwargs are a blind spot of all the automated tools
• Automation (pre-commit hooks, editor extensions) makes verification part of your natural flow

Next capsule: Exercise: Detecting Hallucinations — 5 snippets with hidden hallucinations. It's time to test everything you learned.

Additional resources

1. ruff — Python Linter - Official documentation of Python's fastest linter
2. mypy — Type Checker - Official documentation of the type checker
3. pyright - Microsoft's type checker, the engine behind Pylance
4. pre-commit - A framework for automated git hooks
5. pytest — Testing Framework - Official pytest documentation
6. Python inspect module - Inspect function signatures and metadata

Debugging & Code Review with Claude Code — Module 3, Capsule 05 Claude Code Agentic Development Path — Guide #6 of 11