Module 3: Detecting Hallucinations in Code

Module 3: Detecting Hallucinations in Code

Next →

Module 3: Detecting Hallucinations in Code

Capsule overview

A bug causes an error. A typo causes a crash. But a hallucination — code that looks perfectly correct, passes your visual review, even passes the linter, and only fails when you run it — is the most dangerous error AI can generate. In this module you're going to learn to detect hallucinations in code before they reach production.

This module closes Phase 1 of the guide. In module 1 you built awareness about trust in AI code. In module 2 you developed mental models to supervise AI output. Now you apply all of that to the most critical case: code that looks good but references something that doesn't exist.

If you can detect hallucinations, you've made the most important leap in this guide. Code review, debugging, and everything that comes in Phases 2 and 3 are refinements of a skill you already have.

Module context

Where are we?

This is Guide #6 of the Claude Code Agentic Development Path. Modules 1 and 2 built your mental frameworks — awareness about trust in AI code and mental models to supervise output. This module is where those frameworks are tested against the most subtle error.

Where are we headed?

This module closes Phase 1: Understanding the Problem. After this, Phase 2 gives you concrete tools: professional code review (module 4), common error patterns (module 5), and debugging with Claude Code (module 6). But those tools are more effective if you can first detect the invisible — hallucinations.

Why does this matter so much?

Think about the types of errors AI can generate:

Syntax errors
└── The linter catches them → Low risk

Logic errors
└── The tests catch them → Medium risk

Design errors
└── Code review catches them → Medium-high risk

Hallucinations
└── They look correct visually
└── They pass the linter (in dynamic languages)
└── They pass visual code review
└── They only fail at runtime
└── Sometimes, they only fail in SPECIFIC runtime conditions
└── → HIGH risk

Hallucinations are the most dangerous case because they're the hardest to detect. And they're exactly what LLMs produce most frequently when working with APIs, libraries, and specific functions.

The Critical Difference: Bugs vs Hallucinations

Before moving on, you need to clearly distinguish between a bug and a hallucination. The distinction matters because it changes how you look for the error and how you fix it.

A bug is an implementation error

# Bug: the developer meant to calculate the average but divided by the wrong number
def average(numbers):
    return sum(numbers) / (len(numbers) + 1)  # Error: should be len(numbers)

The developer knew that sum() and len() exist. They used them correctly. They simply got the formula wrong. The fix is to change + 1 to nothing. The concept of the function is correct, the implementation has a specific error.

A hallucination is an invention

# Hallucination: the LLM invented a function that doesn't exist
from sklearn.metrics import roc_auc_multiclass  # Doesn't exist in sklearn

score = roc_auc_multiclass(y_true, y_pred)  # Would never work

Here there's no implementation error — there's an invention. roc_auc_multiclass doesn't exist. The LLM generated a name that sounds real based on patterns (roc_auc_score exists, multiclass is a real concept), but it combined the two into something that was never real.

The differences table

AspectBugHallucination
OriginHuman or AI error in implementationInvention of something that doesn't exist
APIs usedRealInvented or incorrect
DetectionTests, debuggingVerification against documentation
FixCorrect the logicReplace with what actually exists
Exampleif x > 0 when it should be >=requests.get(url, verify_ssl=True)
DangerVariable (depends on the bug)High (looks correct, passes visual review)

This distinction is pragmatic, not academic. In practice, when you find an error in AI code, ask yourself: "did it use something that doesn't exist?" If the answer is yes, it's a hallucination. If it used everything correctly but the logic is wrong, it's a bug.

What a Hallucination in Code Is

Precise definition

A hallucination in code is code that:

  1. Is syntactically correct (doesn't cause a parsing error)
  2. Looks semantically valid (looks like it does something real)
  3. References something that doesn't exist or works differently than it appears

The key is in point 3. It's not a bug — it's an invention. The LLM generated something that sounds correct but doesn't correspond to reality.

Quick examples to calibrate

This is NOT a hallucinationThis IS a hallucination
import os (exists)from fastapi.security import OAuth2TokenValidator (doesn't exist)
requests.get(url, verify=True) (correct parameter)requests.get(url, verify_ssl=True) (invented parameter)
pd.DataFrame.to_json(orient="records") (valid value)pd.DataFrame.to_json(orient="dict") (invented value)
Bug in business logic (programmer's error)validate_email function that only checks for @ (invention)

Notice the pattern: hallucinations look like the correct thing. verify_ssl sounds like something that should exist — but the real parameter is verify. orient="dict" sounds reasonable — but the valid values are "records", "index", "columns", "values", "table", "split".

Why LLMs Hallucinate Code

The mechanics of the problem

LLMs don't consult documentation when they generate code. They don't run pip install to verify that an import exists. They don't open the API reference to confirm a parameter. They generate probable text based on patterns they saw during training.

This has specific consequences for code:

What a human developer does:
1. "I need to validate a JWT"
2. Opens the PyJWT docs
3. Reads the signature of jwt.decode()
4. Writes: jwt.decode(token, key, algorithms=["HS256"])

What an LLM does:
1. "I need to validate a JWT"
2. Has seen thousands of examples with jwt.decode()
3. Remembers patterns like "verify", "algorithms", "key"
4. Combines probable patterns
5. Generates: jwt.decode(token, algorithms=["HS256"], verify=True)
   ← "verify=True" is a parameter that sounds real but doesn't exist
   ← The real parameter is options={"verify_signature": True}

The LLM isn't "making a mistake." It's generating the most probable combination of tokens based on what it saw. And verify=True is a combination that sounds perfectly reasonable — it just doesn't exist in PyJWT.

The 4 main reasons for hallucinations in code

1. Mixing API versions

LLMs were trained on code from multiple versions of the same library. When they generate code, they can mix the v1 API with the v3 API.

# The LLM saw code from Pydantic v1 and v2
# It can generate this — which mixes both:
from pydantic import BaseModel, validator  # v1 style

class User(BaseModel):
    email: str

    @validator("email")  # v1 decorator
    def validate_email(cls, v):
        # But it uses model_dump() which is v2
        return v

user = User(email="test@test.com")
data = user.model_dump()  # v2 method — with a v1 validator

2. Inventing functions that "should exist"

If a pattern is common in a domain, the LLM can invent a function that fits that pattern even though it doesn't exist.

# sklearn has roc_auc_score for binary classification
from sklearn.metrics import roc_auc_score  # ✅ Exists

# The LLM reasons: "if roc_auc_score exists, 
# a multiclass version must exist"
from sklearn.metrics import roc_auc_multiclass  # ❌ Doesn't exist
# The real thing: roc_auc_score with the multi_class="ovr" parameter

3. Parameters that sound logical

LLMs invent parameters that fit a library's naming convention but don't exist.

# requests uses verify for SSL verification
requests.get(url, verify=True)  # ✅ Correct

# The LLM generates something that sounds more descriptive:
requests.get(url, verify_ssl=True)  # ❌ Doesn't exist
# "verify_ssl" sounds clearer than "verify", 
# but requests uses "verify"

4. Logic that appears to implement something but doesn't

The LLM generates a function with a descriptive name but the implementation is incorrect or incomplete.

def validate_email(email: str) -> bool:
    """Validates that the email address is properly formatted."""
    return "@" in email  # ← This is NOT email validation
    # Accepts: "@", "@@@@", "no-domain@", "@no-local"
    # Real validation uses regex or a library like email-validator

The Real Impact of Hallucinations

Scenarios that occur in production

Hallucinations aren't a theoretical problem. They're errors that reach production and cause real impact:

Scenario 1: Fake import in a deploy
├── Claude Code generates: from fastapi.security import OAuth2TokenValidator
├── Developer accepts without verifying
├── The local linter doesn't catch it (Python doesn't check static imports)
├── Push to GitHub → CI/CD runs tests → Tests don't cover that import
├── Deploy to production
├── First request that touches auth → ImportError → 500 Error
├── Impact: downtime until someone identifies the fake import
└── Cost: 30 minutes to 2 hours of downtime

Scenario 2: Silently ignored parameter
├── Claude Code generates: requests.get(url, verify_ssl=False)
├── Developer thinks they disabled SSL verification for testing
├── The code works (verify_ssl is ignored, verify=True by default)
├── Developer pushes to production with "verify_ssl=False" thinking 
│   they'll reactivate SSL verification later
├── In production, SSL was ALWAYS active (the parameter never worked)
├── Impact: none immediate (luckily), but future confusion
└── Cost: hours of debugging when someone tries to disable SSL

Scenario 3: Fabricated security logic  
├── Claude Code generates: a password hashing function with SHA-256
├── Developer accepts because SHA-256 is a known algorithm
├── The function is syntactically correct but insecure for passwords
├── Passes visual code review, passes tests (the tests only verify 
│   that the function produces a hash, not that it's secure)
├── Deploy to production with passwords hashed with SHA-256
├── 6 months later: breach, database compromised
├── Attacker does a rainbow table attack in hours (SHA-256 is very fast)
├── Impact: compromise of user data
└── Cost: incalculable (legal, reputational, financial)

The pattern that connects the 3 scenarios

In all 3 cases, the code:

  1. Was generated by AI
  2. Looked correct visually
  3. Was not verified against reality (documentation, execution, best practices)
  4. The error was detectable with the right tools

This module gives you those tools. Not to eliminate the risk (no tool eliminates it 100%), but to reduce it drastically.

Professional objective

By the end of this module you'll be able to:

  • ✅ Define "hallucination in code" with technical precision
  • ✅ Classify hallucinations into 4 types: fake imports, invented APIs, incorrect parameters, fabricated logic
  • ✅ Detect imports of packages or modules that don't exist
  • ✅ Detect APIs with incorrect or nonexistent parameters
  • ✅ Detect logic that compiles but doesn't do what it says
  • ✅ Use tools to verify: type checkers, linters, quick tests, official documentation
  • ✅ Find at least 4 of 5 hallucinations in the practical exercise

Module progression

Module map

CapsuleTopicWhat you'll learn
02Types of HallucinationsComplete taxonomy: imports, APIs, parameters, logic — with subtle examples
03Hallucinations in Imports and APIsHow to detect fake imports and APIs with invented signatures
04Hallucinations in LogicCode that "looks correct" but implements incorrect logic
05Detection ToolsType checkers, linters, quick tests, official documentation
06Exercise: Detecting Hallucinations5 snippets with hidden hallucinations — can you find them all?

Learning flow

First you'll understand the complete taxonomy of hallucinations with subtle examples of each type (capsule 02). Then you'll go deeper into import and API hallucinations — the most common ones — with specific techniques for detecting them (capsule 03). Then you'll tackle logic hallucinations — the most dangerous because they even pass static linters (capsule 04). With that foundation, you'll learn the tools that act as a safety net when your eye fails (capsule 05). Finally, you'll put it all to the test with 5 snippets that contain real hallucinations of increasing difficulty (capsule 06).

Troubleshooting

Problem 1: "I'm not sure when something is a hallucination vs when it's just a different way of doing the same thing"

Cause: Some variations are stylistic (valid), others are hallucinations. The line can be confusing.

Solution: The rule: if changing the code to the "correct form" changes the behavior, it's a hallucination. If both forms produce the same result, it's a stylistic variation. Example: from json import JSONDecodeError vs json.JSONDecodeError — both work, it's variation. from json.exceptions import JSONDecodeError — doesn't work, it's a hallucination.

Problem 2: "Do LLMs hallucinate more in certain libraries than others?"

Cause: Yes. Libraries with APIs that change between versions or with ambiguous naming conventions generate more hallucinations.

Solution: High-risk libraries for hallucinations: Pydantic (v1 vs v2), SQLAlchemy (1.x vs 2.x), sklearn (extensive APIs), FastAPI (confusion with Starlette). Low-risk libraries: Python standard modules, requests (stable API), pytest. Adjust your level of verification according to the library.

Problem 3: "Can I use Claude Code to verify whether its own code has hallucinations?"

Cause: Yes, you can, but with caution. Claude Code can identify many of its own errors if you ask it directly.

Solution: You can ask: "Does this import exist in FastAPI?" or "What are the real parameters of requests.get()?" But verify the answer with tools (python -c, docs). An LLM can hallucinate about its own hallucinations. The tools are the source of truth.

Connection to the Project

This module's exercise

You're going to receive 5 code snippets with a hidden hallucination in each one. Your job is to find all 5. The difficulty is progressive: 1 obvious, 2 medium, 2 subtle. The benchmark is to find at least 4 of 5.

Connection to the capstone project (Module 8)

The capstone project in module 8 includes hallucinations intentionally planted in a complete FastAPI codebase. The techniques you learn in this module are exactly the ones you need to find them. The difference: here you work with isolated snippets; in module 8, the hallucinations are hidden in a codebase with multiple interconnected files.

Limits: What this module does NOT cover

  • ❌ Complete code review — That's module 4. Here we focus exclusively on hallucinations.
  • ❌ Business logic bugs — If the function implements something incorrect but uses real APIs, it's a bug, not a hallucination.
  • ❌ Security holes — That's module 5. An endpoint without auth is a security hole, not a hallucination.
  • ❌ Debugging errors — That's module 6. Here you detect hallucinations before they cause errors.
  • ❌ Hallucinations in text/comments — We focus on executable code, not on incorrect docstrings or comments.

Signs of success

By the end of this module, you'll know you succeeded if:

  • ✅ You can explain why LLMs hallucinate code (they generate probable text, not verified code)
  • ✅ You correctly classify hallucinations into their 4 types
  • ✅ Faced with an unknown import, your first instinct is to verify that it exists
  • ✅ Faced with an API parameter, you verify against the official documentation
  • ✅ You can detect functions that claim to do something but whose implementation doesn't match
  • ✅ You have a toolkit of tools to verify suspicious code
  • ✅ You found at least 4 of 5 hallucinations in the exercise

The Tone of This Module: Detective

This module has a different tone from the previous ones. Modules 1 and 2 were about reflection and frameworks. This module is about detection. Think of yourself as a detective training your eye to see what others don't.

The dangerous hallucinations are the ones that look good. The obvious ones (import unicorn_magic) don't matter — nobody would accept them. The ones that matter are the subtle ones: from fastapi.security import OAuth2TokenValidator looks so real that an experienced developer might not question whether it exists. What looks correct at first glance is exactly where the danger is.

What changes after this module

Before this module, your process with AI code probably looks like this:

Claude Code generates code
   └── "Looks good" → Accept

After this module, your process will be:

Claude Code generates code
   ├── Imports I don't recognize? → Verify with python -c
   ├── API parameters I haven't used before? → Verify with docs
   ├── Security/validation functions? → Verify the implementation
   ├── Logic that should use a standard library? → Compare
   └── Everything verified → Accept with confidence

The difference isn't that you distrust everything — it's that you know where to look. A developer who can detect hallucinations works faster (not slower) because they know what to verify and what to trust. They don't review every line — they review the ones that matter.

Your advantage after this module: every time Claude Code generates an import, an API call, or a library function, your instinct will be to verify what you don't recognize. Not because you distrust AI — but because you know that verification is a professional skill that separates the ordinary developer from the exceptional one.

Summary

  • Hallucinations in code are the most dangerous error in AI-generated code because they look correct
  • LLMs don't verify what they generate — they produce probable text, not proven code
  • Hallucinations are classified into 4 types: fake imports, invented APIs, incorrect parameters, fabricated logic
  • The dangerous hallucinations are the subtle ones — the ones that look like the real thing
  • This module closes Phase 1: it combines awareness (module 1) and mental models (module 2) with the most critical case
  • By the end, you'll have a detection toolkit and you'll have tested your skill with 5 real snippets

Additional resources

  1. Anthropic — Claude Code Documentation - Official documentation of Claude Code and its capabilities
  2. arXiv — Code Hallucinations in Large Language Models - Research on hallucinations specific to code generation
  3. GitHub Blog — AI Code Generation Research - Studies on the quality of AI-generated code
  4. Python Package Index (PyPI) - Verify that a Python package exists before trusting an import
  5. FastAPI Official Documentation - Reference for verifying FastAPI APIs

Next capsule: Types of Hallucinations — the complete taxonomy with subtle examples of each category.

Debugging & Code Review with Claude Code — Module 3, Capsule 01 Claude Code Agentic Development Path — Guide #6 of 11

← PreviousIntegrative Exercise: Applying the 3 Mental ModelsNext →Types of Hallucinations in Code