Validating AI-Generated Tests

Capsule overview

Claude Code can generate 50 tests in 30 seconds. That sounds impressive. But if you don't know how to evaluate whether those tests are useful or trivial, you have a false sense of security. 40 of those 50 tests could verify the obvious and pass with a wrong implementation. The remaining 10 might not exist — and the critical bug in production stays hidden.

This capsule is the most critical skill of the module. It's not enough to know how to generate tests with specific prompts (capsule 02) or to master the pytest patterns (capsules 03-04). The question you must answer in 30 seconds is: Do these tests that Claude Code generated actually protect against bugs, or do they just produce green checkmarks? You'll learn the 5-point checklist for evaluating AI-generated tests, the red flags that indicate trivial tests, and the mutation-testing mental model that distinguishes line coverage from behavior coverage.

By the end, you'll be able to look at a suite generated by Claude Code and tell in seconds which tests add value and which are decorative. And you'll have concrete prompts to improve weak tests until they become a professional suite.

The Problem: False Confidence

"50 tests pass" ≠ "Your code is correct"

When you run pytest and see 50 passed in 0.3s, it's tempting to assume your code is fine. But that conclusion is only valid if the 50 tests verify relevant behavior. If 45 of them do assert result is not None or test the same happy path with cosmetic variations, you have 45 tests that pass even when your logic has serious bugs.

The goal of tests is to detect regressions. If you modify the implementation and introduce a bug, a well-written test fails. A trivial test doesn't fail — because it doesn't verify the logic you broke. The result: deployments with false confidence.

Line coverage ≠ Behavior coverage

Coverage tools (which you'll see in Module 5) report what percentage of code lines were executed. If your suite runs 100% of the lines, coverage says "100%." But that doesn't mean every line is verified.

Imagine this function:

def calculate_tax(price: float, rate: float) -> float:
    if price < 0:
        raise ValueError("Price cannot be negative")
    return price * rate

A test that calls calculate_tax(100, 0.1) and verifies result == 10.0 runs 100% of the lines. But what if someone changes the formula to return 0? The test fails. Good. And if they change the formula to return price * rate * 0.5 (a bug)? The test also fails. That's fine.

Now imagine these tests:

def test_calculate_tax_returns_number():
    result = calculate_tax(100, 0.1)
    assert isinstance(result, float)

def test_calculate_tax_not_none():
    result = calculate_tax(100, 0.1)
    assert result is not None

def test_calculate_tax_positive_input():
    result = calculate_tax(100, 0.1)
    assert result > 0

All three pass. Coverage: 100%. But if the implementation does return 0.001 (any small positive value), all three tests still pass. Line coverage says 100%. Behavior coverage says: the formula was never verified.

Trivial tests: green checkmarks with no real protection

A trivial test gives a green result but wouldn't catch a bug if you introduced one. Typical traits:

• It verifies that the function returns "something" instead of verifying the exact value.
• It always uses "friendly" inputs (2, 3, "hello", True).
• It doesn't cover edge cases (empty, None, zero, negatives, limits).
• It doesn't verify the error paths (exceptions).
• It could pass if the function returned a hardcoded value.

Complete example: 10 tests that pass and a critical bug hidden

The function to test:

# discounts.py
def apply_discount(price: float, discount_percent: float) -> float:
    """
    Apply discount and return final price.
    Raises ValueError for invalid inputs.
    """
    if price < 0:
        raise ValueError("Price cannot be negative")
    if not 0 <= discount_percent <= 100:
        raise ValueError("Discount must be between 0 and 100")
    return round(price * (1 - discount_percent / 100), 2)

AI-generated tests (weak version):

# test_discounts_weak.py
import pytest
from discounts import apply_discount


def test_apply_discount_returns_float():
    result = apply_discount(100, 10)
    assert isinstance(result, float)


def test_apply_discount_not_none():
    result = apply_discount(100, 10)
    assert result is not None


def test_apply_discount_positive():
    result = apply_discount(100, 10)
    assert result > 0


def test_apply_discount_less_than_price():
    result = apply_discount(100, 10)
    assert result < 100


def test_apply_discount_with_zero_discount():
    result = apply_discount(50, 0)
    assert result == 50  # This is the only strong assert


def test_apply_discount_with_hundred_percent():
    result = apply_discount(100, 100)
    assert result >= 0


def test_apply_discount_different_inputs():
    result = apply_discount(200, 25)
    assert isinstance(result, float)


def test_apply_discount_small_values():
    result = apply_discount(1, 1)
    assert result >= 0


def test_apply_discount_negative_price_raises():
    with pytest.raises(ValueError):
        apply_discount(-10, 5)


def test_apply_discount_invalid_discount_raises():
    with pytest.raises(ValueError):
        apply_discount(100, 150)

All 10 tests pass. But there's a critical bug: if the implementation has a rounding error or uses the wrong formula, most of the tests don't catch it.

Implementation with a subtle bug that the weak tests do NOT catch:

# discounts_buggy.py — the 10 tests pass with this incorrect implementation
def apply_discount(price: float, discount_percent: float) -> float:
    if price < 0:
        raise ValueError("Price cannot be negative")
    if not 0 <= discount_percent <= 100:
        raise ValueError("Discount must be between 0 and 100")
    # BUG: divides by 100 twice (typo)
    return round(price * (1 - discount_percent / 100 / 100), 2)

With apply_discount(100, 10):

• Correct: 100 * 0.9 = 90.0
• With the bug: 100 * (1 - 0.001) = 99.9

Only test_apply_discount_with_zero_discount verifies an exact value. The other 9 pass. 9 out of 10 tests are decorative.

The test that would catch the bug:

def test_apply_discount_ten_percent_returns_ninety():
    result = apply_discount(100, 10)
    assert result == 90.0  # Explicit assert of the expected value

The 5-Point Checklist for Evaluating AI-Generated Tests

Use this checklist every time Claude Code generates a suite. If a test fails on more than one point, strengthen it or replace it.

1. Does it test BEHAVIOR or IMPLEMENTATION?

Behavior: What the system does from the consumer's point of view. Given input X, it must return Y. It doesn't matter how it does it internally.

Implementation: How the system does it. Which internal functions it uses, which data structures, which order of operations.

Practical rule: If you refactor the implementation without changing the behavior, the tests shouldn't change. If they do, they probably test implementation.

2. Does it cover EDGE CASES?

Edge cases are the inputs where the behavior is less obvious or where bugs tend to hide:

• Empty: "", [], {}, None
• Zero: 0 (for numbers)
• Negatives: -1, -100
• Limit values: first element, last element, exactly at the boundary
• Incorrect types: None where a string is expected, int where a float is expected
• Extreme domain cases: maximum representable value, very long strings

If all the tests use "normal" inputs (2, 3, "hello", [1, 2, 3]), the suite is weak. At least 20-30% of the tests should cover edge cases.

Example for clamp(value, min_val, max_val):

3. Is the ASSERT meaningful?

The strength of the assert determines how much "room" an incorrect implementation has to pass.

Rule: Prefer asserts that verify the exact expected value. If you only verify "it's a number" or "it's not None," a broken implementation can pass.

4. Does it test the ERROR PATHS?

Not just the happy path. What about invalid inputs? Does the function raise the correct exception with the correct message?

# ❌ Doesn't verify the error path
def test_validate_email():
    assert validate_email("user@example.com") == True


# ✅ Verifies the error path
def test_validate_email_none_raises_type_error():
    with pytest.raises(TypeError, match="must be a string"):
        validate_email(None)

For functions that validate inputs, at least 1-2 tests per error type (wrong type, out-of-range value, invalid format).

5. Would an INCORRECT implementation pass these tests?

The definitive test: Imagine the function returns a hardcoded value.

def apply_discount(price: float, discount_percent: float) -> float:
    return 90.0  # Hardcoded

Run the suite. If all the tests pass, the tests are trivial. A useful test would fail because apply_discount(50, 0) should give 50, not 90.

Variant: Comment out a critical line of the implementation. Does any test fail? If not, that line has no behavior coverage — even if pytest-cov says the line was executed.

Red Flags in AI-Generated Tests

Recognize these patterns and act (strengthen or remove the test).

Red flag 1: All the tests use "friendly" inputs

• Always 2, 3, "hello", True, [1, 2, 3]
• Never 0, -1, "", None, [], float('nan')

Action: Ask Claude Code: "Add tests for edge cases: empty input, None, zero, negatives."

Red flag 2: Weak asserts

• assert result is not None
• assert isinstance(result, SomeType)
• assert len(result) > 0 when the exact value matters

Action: Replace with an assert that verifies the expected value. If you don't know the expected value, compute it independently (not using the same function).

Red flag 3: They call the function but don't assert the result

def test_process_data():
    result = process_data([1, 2, 3])
    # No assert — the test passes if there's no exception

Action: Add an assert that verifies the result. If the function returns something, verify that something.

Red flag 4: The expected value seems computed by the same logic

Sometimes Claude Code does:

def test_calculate():
    result = calculate(100, 10)
    expected = 100 * (1 - 10/100)  # Repeats the logic
    assert result == expected

If the implementation has a bug in that same formula, the test replicates the bug and passes. Compute the expected value by hand or from an independent source.

Red flag 5: They test Python built-ins instead of your logic

def test_sort_returns_list():
    result = my_sort([3, 1, 2])
    assert isinstance(result, list)  # Verifies it's a list

That verifies that Python returns lists. It doesn't verify that your function sorts correctly.

Action: The assert should verify the order: assert result == [1, 2, 3].

How to Improve AI-Generated Tests

Expansion prompt: "Which edge cases are NOT covered?"

When you have a suite that passes, ask Claude Code:

Here are the existing tests for [function]:
[paste the tests]

And the implementation:
[paste the function]

Which edge cases, boundary conditions, or error paths are NOT covered
by these tests? Generate only the missing tests.

Claude Code usually identifies gaps (None, empty, zero, limits) and generates additional targeted tests.

Adversarial prompt: "Try to break this function"

Here is the implementation of [function]:
[paste the function]

Act as an adversarial tester. Your goal is to find inputs that:
1. Cause unhandled exceptions
2. Produce silently incorrect results
3. Exploit edge cases the implementation might mishandle

Generate tests that try to break the function.

This prompt produces tests that actively hunt for bugs.

The mutation-testing concept: "If I change line X, does any test fail?"

For each significant line of your implementation, ask yourself: If I delete or modify this line, does any test fail?

If the answer is no, that line has no behavior coverage. pytest-cov can say 100% of lines executed, but if no assertion depends on that line, a bug there would go unnoticed.

Example:

def apply_discount(price: float, discount_percent: float) -> float:
    if price < 0:
        raise ValueError("Price cannot be negative")
    if not 0 <= discount_percent <= 100:
        raise ValueError("Discount must be between 0 and 100")
    return round(price * (1 - discount_percent / 100), 2)  # Critical line

If you comment out the return line and put return 0, do the tests fail? If your tests only verify result >= 0 or isinstance(result, float), they wouldn't fail. You need at least one test with assert result == 90.0 (or the exact value) so that line has real coverage.

Targeted prompt: "Add a test that fails if [X] is removed"

The apply_discount function has this line that applies the discount:
    return round(price * (1 - discount_percent / 100), 2)

Add a test that FAILS if someone changes or removes that formula.
The test must verify the exact value of the result for at least
two combinations of price and discount_percent.

This prompt forces tests with strong asserts over the critical logic.

The Mental Model: Mutation Testing

The key question for each line

For each line of your implementation:

If I delete this line (or change it to something incorrect), does any test fail?

• Yes → That line has behavior coverage. The tests protect it.
• No → That line is executed but not verified. A bug there wouldn't be caught.

Line coverage vs behavior coverage

Goal: Maximize behavior coverage. The tests should be such that any change that introduces a bug causes at least one failure.

Quick mental exercise

Take a 10-line function. Go through each line:

1. Validation line (if x < 0: raise): Is there a test with an invalid input that uses pytest.raises?
2. Calculation line: Is there a test that verifies the exact numeric result?
3. return line: Is there a test that asserts the returned value?

If for any line the answer is "no," you have a gap. Add the test that covers that line.

Practice: Evaluate and Improve a Complete Suite

Generated suite (initial version)

# text_utils.py
def truncate(text: str, max_length: int) -> str:
    """Truncate text to max_length, appending '...' if truncated."""
    if not isinstance(text, str):
        raise TypeError("text must be a string")
    if not isinstance(max_length, int):
        raise TypeError("max_length must be an integer")
    if max_length < 0:
        raise ValueError("max_length must be non-negative")
    if len(text) <= max_length:
        return text
    return text[:max_length - 3] + "..."

# test_text_utils_initial.py
import pytest
from text_utils import truncate


def test_truncate_returns_string():
    result = truncate("hello world", 5)
    assert isinstance(result, str)


def test_truncate_short_text():
    result = truncate("hi", 10)
    assert result is not None


def test_truncate_long_text():
    result = truncate("hello world", 5)
    assert len(result) <= 8  # 5 + 3 for "..."


def test_truncate_empty_string():
    result = truncate("", 5)
    assert result == ""


def test_truncate_exact_length():
    result = truncate("hello", 5)
    assert result == "hello"


def test_truncate_invalid_text_raises():
    with pytest.raises(TypeError):
        truncate(123, 5)

Evaluation with the checklist

Problems detected:

1. test_truncate_long_text doesn't verify the exact value. An incorrect implementation return "wrong" would pass if len("wrong") <= 8.
2. There's no test that verifies the truncation produces "he..." for "hello world" with max_length=5.
3. There's no test for max_length=0, negative max_length (error path), or non-integer max_length.

Improved suite

# test_text_utils_improved.py
import pytest
from text_utils import truncate


class TestTruncateHappyPath:
    def test_short_text_returned_unchanged(self):
        result = truncate("hi", 10)
        assert result == "hi"

    def test_exact_length_returned_unchanged(self):
        result = truncate("hello", 5)
        assert result == "hello"

    def test_long_text_truncated_with_ellipsis(self):
        result = truncate("hello world", 5)
        assert result == "he..."

    def test_truncation_boundary(self):
        result = truncate("abcdefgh", 5)
        assert result == "ab..."


class TestTruncateEdgeCases:
    def test_empty_string_returns_empty(self):
        result = truncate("", 5)
        assert result == ""

    def test_max_length_zero_returns_ellipsis_only(self):
        # "a" truncated to 0 would be "..."
        result = truncate("a", 3)
        assert result == "..."

    def test_unicode_text_truncates_correctly(self):
        result = truncate("café", 3)
        assert result == "caf..."


class TestTruncateErrorPaths:
    def test_none_text_raises_type_error(self):
        with pytest.raises(TypeError, match="text must be a string"):
            truncate(None, 5)

    def test_int_text_raises_type_error(self):
        with pytest.raises(TypeError, match="text must be a string"):
            truncate(123, 5)

    def test_negative_max_length_raises_value_error(self):
        with pytest.raises(ValueError, match="non-negative"):
            truncate("hello", -1)

    def test_float_max_length_raises_type_error(self):
        with pytest.raises(TypeError, match="integer"):
            truncate("hello", 5.0)

Mental mutation: If you change return text[:max_length - 3] + "..." to return text[:max_length], the test_long_text_truncated_with_ellipsis test fails. That line has behavior coverage.

Project Connection

In the Generated unit test suite project (capsule 06), you'll receive a data_utils.py module and use Claude Code to generate tests. The grade isn't based on the number of tests but on the quality.

Evaluation criteria you'll apply with this checklist:

• Do the tests verify behavior (exact value) or only type/format?
• Is there coverage of edge cases (empty, None, zero, limits)?
• Are the asserts strong (expected value) or weak (is not None)?
• Are the error paths tested with pytest.raises?
• Would a buggy implementation pass the tests? (mental mutation test)

Before delivering, go through the checklist for each test. Strengthen the ones that fail. The project requires applying this validation skill explicitly.

Troubleshooting

Problem 1: All the tests pass on the first try and I don't know if they're good

Cause: It could be simple code or trivial tests.

Solution: Use the manual mutation test. Modify a critical line of the implementation (e.g., change a * to a +). If all the tests still pass, they're trivial. Add tests with strong asserts over exact values for that logic.

Problem 2: Claude Code generates many tests with assert result is not None

Cause: A generic prompt, or the AI prioritizes "don't fail" over "verify the exact value."

Solution: In the prompt, ask explicitly: "Each assert must verify the exact expected value. Don't use assert result is not None or assert isinstance. Verify the result with assert result == expected_value."

Problem 3: I don't know which edge cases to ask for in my domain

Cause: A lack of domain experience.

Solution: Use the expansion prompt: "Which edge cases are NOT covered?" Claude Code usually proposes: empty, None, zero, negatives, incorrect types, very large values. For specific domains, describe the business rules and ask: "Given these contracts, which inputs could violate them or be at the limit?"

Problem 4: The tests verify implementation and break when refactoring

Cause: The tests are coupled to internal details (function names, call order).

Solution: Refactor the tests to verify only inputs and outputs. If the test uses inspect.getsource or mocks internal functions, reframe it as a black-box test: given these inputs, the output should be this. The behavior is the contract; the implementation can change.

Problem 5: I have 100% coverage but I know there's unverified code

Cause: Line coverage ≠ behavior coverage. The lines run but no assert depends on their result.

Solution: For the doubtful lines, apply the mental mutation: change the line. If no test fails, add a test whose assert depends directly on that line. Alternatively, use mutation testing tools (mutmut, cosmic-ray) that automate this process — covered in Module 5.

Exercises

Exercise 1: Identify weak asserts (Easy)

These tests have weak asserts. Rewrite them with strong asserts. The function is def count_words(text: str) -> int.

def test_count_words_returns_int():
    result = count_words("hello world")
    assert isinstance(result, int)

def test_count_words_not_empty():
    result = count_words("hello world")
    assert result > 0

def test_count_words_two_words_returns_two():
    result = count_words("hello world")
    assert result == 2

def test_count_words_single_word_returns_one():
    result = count_words("hello")
    assert result == 1

def test_count_words_empty_returns_zero():
    result = count_words("")
    assert result == 0

Exercise 2: The hardcoded-function test (Easy)

Imagine that safe_divide(a: int, b: int) -> float returns a / b or raises if b == 0. Claude Code generated these tests:

def test_safe_divide_returns_float():
    assert isinstance(safe_divide(10, 2), float)

def test_safe_divide_not_none():
    assert safe_divide(10, 2) is not None

def test_safe_divide_by_zero_raises():
    with pytest.raises(ZeroDivisionError):
        safe_divide(10, 0)

Implement an incorrect version of safe_divide that returns a hardcoded value (e.g., always 5.0) and run the tests. How many pass? Add the test that would fail.

# Incorrect implementation
def safe_divide(a: int, b: int) -> float:
    if b == 0:
        raise ZeroDivisionError("division by zero")
    return 5.0  # Hardcoded — incorrect

def test_safe_divide_returns_correct_quotient():
    result = safe_divide(10, 2)
    assert result == 5.0  # 10/2 = 5.0

def test_safe_divide_returns_correct_quotient_non_integer():
    result = safe_divide(7, 2)
    assert result == 3.5

Exercise 3: Apply the checklist (Medium)

Evaluate these tests for def is_palindrome(s: str) -> bool using the 5-point checklist. Point out what's missing and write the necessary additional tests.

def test_is_palindrome_returns_bool():
    result = is_palindrome("racecar")
    assert isinstance(result, bool)

def test_is_palindrome_known_true():
    assert is_palindrome("aba") == True

def test_is_palindrome_known_false():
    assert is_palindrome("abc") == False

def test_empty_string_is_palindrome():
    assert is_palindrome("") == True  # Common convention

def test_single_char_is_palindrome():
    assert is_palindrome("a") == True

def test_palindrome_with_spaces_depends_on_spec():
    # Depends: is "a ba" a palindrome? Usually spaces are ignored
    assert is_palindrome("race car") == True  # If the spec says spaces are ignored

def test_none_input_raises_or_returns_false():
    with pytest.raises(TypeError):
        is_palindrome(None)

Exercise 4: Mental mutation (Medium)

For this function, indicate for each line whether it has behavior coverage with the given tests. If not, write the test that would cover it.

def min_of_three(a: int, b: int, c: int) -> int:
    if a < b:
        smallest = a
    else:
        smallest = b
    if c < smallest:
        smallest = c
    return smallest

Current tests:

def test_min_first():
    assert min_of_three(1, 2, 3) == 1

def test_min_second():
    assert min_of_three(2, 1, 3) == 1

def test_min_third():
    assert min_of_three(3, 2, 1) == 1

def test_all_equal_returns_that_value():
    assert min_of_three(7, 7, 7) == 7

Exercise 5: Expansion prompt (Medium)

You have these tests for def parse_age(age_str: str) -> int and suspect edge cases are missing. Write the prompt you'd give Claude Code to generate the missing tests.

def test_parse_age_valid():
    assert parse_age("25") == 25

def test_parse_age_zero():
    assert parse_age("0") == 0

Here are the existing tests for parse_age:

def test_parse_age_valid():
    assert parse_age("25") == 25

def test_parse_age_zero():
    assert parse_age("0") == 0

And the implementation (or specification): parse_age receives a string that
represents an age and returns an int. It should raise ValueError for
invalid strings.

Which edge cases, boundary conditions, and error paths are NOT covered?
Generate additional tests for:
- Empty string
- String with spaces
- Non-numeric string ("abc", "25 years")
- Negative number
- Number with decimals ("25.5")
- None (if applicable)
- Very large number
- Format "  25  " with spaces around it

Only generate the missing tests. Use pytest.raises where appropriate.

Exercise 6: Complete suite with the checklist (Hard)

Given this module, generate a test suite with Claude Code. Then evaluate each test with the 5-point checklist. Identify the ones that fail and improve them. Finally, apply the mutation test: change a critical line. Does any test fail?

# validators.py
def validate_age(age: int) -> bool:
    """Return True if age is between 0 and 150 inclusive."""
    if not isinstance(age, int):
        raise TypeError("age must be an integer")
    if age < 0:
        return False
    if age > 150:
        return False
    return True

import pytest
from validators import validate_age


def test_valid_age_returns_true():
    assert validate_age(25) == True

def test_zero_age_valid():
    assert validate_age(0) == True

def test_max_age_150_valid():
    assert validate_age(150) == True

def test_negative_age_invalid():
    assert validate_age(-1) == False

def test_over_150_invalid():
    assert validate_age(151) == False

def test_float_raises_type_error():
    with pytest.raises(TypeError, match="integer"):
        validate_age(25.5)

def test_none_raises_type_error():
    with pytest.raises(TypeError, match="integer"):
        validate_age(None)

def test_boundary_149_valid():
    assert validate_age(149) == True

Summary

• ✅ "50 tests pass" doesn't imply correct code; what matters is which behaviors they verify
• ✅ Coverage of executed lines ≠ coverage of verified behavior
• ✅ 5-point checklist: behavior vs implementation, edge cases, strong assert, error paths, mutation test (would an incorrect implementation pass?)
• ✅ Red flags: always "friendly" inputs, weak asserts (is not None, isinstance), calls with no assert, expected value copied from the logic, tests of built-ins
• ✅ Prompts to improve: expansion (missing edge cases), adversarial (break the function), targeted (a test that fails if X is removed)
• ✅ Mutation mental model: if changing/deleting a line makes no test fail, that line has no behavior coverage
• ✅ The capsule 06 project is graded by test quality, not quantity; use this checklist

Next capsule: Project — Generated unit test suite for a utilities module.

Additional Resources

1. Mutation Testing (Wikipedia) - Concept and fundamentals
2. mutmut: Python mutation testing - A tool for mutation testing in Python
3. Test Quality - Martin Fowler - The test pyramid and quality
4. pytest: Good practices - Organization and best practices
5. Google: Testing on the Toilet - Short articles on test quality
6. Hypothesis: Property-based testing - Automatic edge case generation (Module 5)

Module 2, Capsule 05 — Testing with Claude Code Guide Quantity ≠ quality: validate before you trust