Migration Testing — Verifying Equivalence

Capsule description

The most important principle of any migration is: the new code must do exactly the same as the old code. Not "more or less the same" — exactly. The way to verify this is with migration tests: tests that call the old endpoint and the new one with the same inputs and compare the outputs.

In this capsule you're going to learn to write three types of migration tests: equivalence tests (old==new), contract tests (the API responds with the expected structure), and rollback tests (you can go back if something fails). Claude Code can generate all these tests by analyzing the old code and the new code.

Equivalence Tests

The principle

# For any valid input:
flask_response = flask_app.handle(request)
fastapi_response = fastapi_app.handle(request)
assert flask_response == fastapi_response  # They must be equal

Implementation with pytest

# tests/test_equivalence.py
import pytest
from flask_app import app as flask_app
from fastapi.testclient import TestClient
from fastapi_app import app as fastapi_app

flask_client = flask_app.test_client()
fastapi_client = TestClient(fastapi_app)

class TestEquivalence:
    """Verifies that Flask and FastAPI produce the same responses."""
    
    @pytest.mark.parametrize("path", [
        "/health",
        "/users",
        "/products",
    ])
    def test_get_endpoints_equivalent(self, path):
        flask_r = flask_client.get(path)
        fastapi_r = fastapi_client.get(path)
        
        assert flask_r.status_code == fastapi_r.status_code
        assert flask_r.get_json() == fastapi_r.json()
    
    def test_create_user_equivalent(self):
        data = {"name": "Test User", "email": "test@example.com"}
        
        flask_r = flask_client.post("/users", json=data)
        fastapi_r = fastapi_client.post("/users", json=data)
        
        assert flask_r.status_code == fastapi_r.status_code
        
        # Compare structure (IDs can differ)
        flask_body = flask_r.get_json()
        fastapi_body = fastapi_r.json()
        assert flask_body["name"] == fastapi_body["name"]
        assert flask_body["email"] == fastapi_body["email"]
    
    @pytest.mark.parametrize("bad_data,expected_status", [
        ({"name": ""}, 400),
        ({"email": "bad"}, 400),
        ({}, 400),
    ])
    def test_validation_errors_equivalent(self, bad_data, expected_status):
        flask_r = flask_client.post("/users", json=bad_data)
        fastapi_r = fastapi_client.post("/users", json=bad_data)
        
        # Both should reject with the same status code
        assert flask_r.status_code // 100 == fastapi_r.status_code // 100
        # Note: Flask returns 400, FastAPI can return 422
        # We compare the "class" of the error (4xx), not the exact code

What to compare and what to ignore

Contract Tests

Verify that the new API fulfills the contract

class TestFastAPIContract:
    """Verifies that FastAPI fulfills the API contract."""
    
    def test_create_user_returns_required_fields(self):
        data = {"name": "Ana", "email": "ana@test.com"}
        response = fastapi_client.post("/users", json=data)
        
        body = response.json()
        assert "id" in body
        assert "name" in body
        assert "email" in body
        assert isinstance(body["id"], int)
        assert isinstance(body["name"], str)
    
    def test_create_user_returns_201(self):
        data = {"name": "Ana", "email": "ana@test.com"}
        response = fastapi_client.post("/users", json=data)
        assert response.status_code == 201
    
    def test_invalid_input_returns_4xx(self):
        response = fastapi_client.post("/users", json={})
        assert 400 <= response.status_code < 500
    
    def test_not_found_returns_404(self):
        response = fastapi_client.get("/users/999999")
        assert response.status_code == 404

Rollback Tests

Verify that you can go back

class TestRollback:
    """Verifies that the Flask app is still functional (rollback viable)."""
    
    def test_flask_still_serves_all_endpoints(self):
        """Flask still responds correctly."""
        assert flask_client.get("/health").status_code == 200
        assert flask_client.get("/users").status_code == 200
        assert flask_client.post("/users", json=valid_data).status_code == 201
    
    def test_flask_database_still_consistent(self):
        """The shared DB is consistent from Flask."""
        flask_client.post("/users", json=valid_data)
        response = flask_client.get("/users")
        assert len(response.get_json()) > 0

Generating Migration Tests with Claude Code

# Prompt:
> "Analyze flask_app.py and fastapi_app.py. Generate complete
   migration tests:
   1. Equivalence tests for each endpoint
   2. Contract tests for FastAPI
   3. Rollback tests for Flask
   
   Use pytest with parametrize where it applies.
   Include happy paths and error cases."

Connection with the Project

In the Module Project (capsule 06), the equivalence tests are part of the deliverable. Each migrated endpoint must have a test that verifies old==new.

Troubleshooting

Problem 1: Flask returns 400, FastAPI returns 422

Cause: Pydantic validation errors are 422 by default.

Solution: Compare the error class (4xx), not the exact code:

assert flask_r.status_code // 100 == fastapi_r.status_code // 100

Problem 2: The JSON structure differs slightly

Cause: Flask wraps in {"error": "..."}, FastAPI in {"detail": "..."}.

Solution: Compare the meaning, not the key:

assert "error" in flask_r.get_json() or "detail" in flask_r.get_json()

Problem 3: Different JSON field order

Cause: JSON doesn't guarantee key order.

Solution: Compare as a dict (Python ignores order) or use json.dumps(sort_keys=True).

Exercises

Exercise 1: Write a parametrized equivalence test (Easy)

Write a parametrized test that verifies GET equivalence for 5 different routes.

@pytest.mark.parametrize("path", [
    "/health",
    "/users",
    "/products",
    "/orders",
    "/categories",
])
def test_get_equivalence(self, path):
    flask_r = flask_client.get(path)
    fastapi_r = fastapi_client.get(path)
    assert flask_r.status_code == fastapi_r.status_code
    if flask_r.status_code == 200:
        assert flask_r.get_json() == fastapi_r.json()

Exercise 2: Equivalence test for POST with edge cases (Medium)

Write equivalence tests for POST /orders that include: happy path, empty items, nonexistent user, and negative quantity.

class TestCreateOrderEquivalence:
    def test_happy_path(self):
        data = {"user_id": 1, "items": [{"product_id": 1, "qty": 2}]}
        flask_r = flask_client.post("/orders", json=data)
        fastapi_r = fastapi_client.post("/orders", json=data)
        assert flask_r.status_code == fastapi_r.status_code
    
    def test_empty_items(self):
        data = {"user_id": 1, "items": []}
        flask_r = flask_client.post("/orders", json=data)
        fastapi_r = fastapi_client.post("/orders", json=data)
        assert flask_r.status_code // 100 == fastapi_r.status_code // 100
    
    def test_invalid_user(self):
        data = {"user_id": 99999, "items": [{"product_id": 1, "qty": 1}]}
        flask_r = flask_client.post("/orders", json=data)
        fastapi_r = fastapi_client.post("/orders", json=data)
        assert flask_r.status_code == fastapi_r.status_code
    
    def test_negative_quantity(self):
        data = {"user_id": 1, "items": [{"product_id": 1, "qty": -1}]}
        flask_r = flask_client.post("/orders", json=data)
        fastapi_r = fastapi_client.post("/orders", json=data)
        assert flask_r.status_code // 100 == fastapi_r.status_code // 100

Common Errors in Migration Testing

Error 1: Comparing responses byte-by-byte

Symptom: 80% of your tests fail because Flask returns {"id": 1, "name": "x"} and FastAPI returns {"name": "x", "id": 1}. JSON doesn't guarantee order.

Why it happens: You compare literal JSON strings instead of comparing as dictionaries. Python compares dicts ignoring order, but strings don't.

How to fix: Always parse to a dict before comparing:

assert flask_r.get_json() == fastapi_r.json()  # ✅ compares as a dict
# NOT: assert flask_r.text == fastapi_r.text   # ❌ fails due to order

Error 2: Assuming the same status codes = the same errors

Symptom: Your test says assert flask.status == fastapi.status, passes with 200, but the bodies are completely different.

Why it happens: An equal status code doesn't imply equivalence. Equivalence includes status + body + side effects (DB writes, emails sent, etc.).

How to fix: Verify the three dimensions:

1. Status code (or the 4xx/5xx class)
2. Body structure (with comparable fields)
3. Side effects (was the record created? was the email sent?)

Error 3: Equivalence tests without cleanup between runs

Symptom: The tests pass the first time, fail the second. POST /users creates a user in Flask, another in FastAPI, the IDs collide or the DB fills up.

Why it happens: There's no DB rollback between tests. Each POST accumulates state.

How to fix: Use pytest transactional fixtures or reset the DB between tests. If you use the same DB for Flask and FastAPI, make sure the cleanup cleans both paths:

@pytest.fixture(autouse=True)
def reset_db():
    yield
    db.session.rollback()
    db.drop_all()
    db.create_all()

Error 4: Only testing happy paths

Symptom: You migrated, all the tests green, in production the 4xx errors have different formats and break the clients.

Why it happens: Happy paths are usually easy to migrate. The error cases are where Flask and FastAPI differ the most (Flask returns 400 with text, FastAPI 422 with structured JSON).

How to fix: For each endpoint, write at least:

• 1 happy path
• 1 validation error (invalid input)
• 1 not found (nonexistent resource)
• 1 unauthorized (if applicable)
• 1 domain edge case

This capsule's exercises show the pattern.

Error 5: Not running the tests during the cutover

Symptom: Cutover of Flask, everything feels fine, you discover in production that a subtle endpoint stopped working.

Why it happens: Once the migration is "finished," the equivalence tests feel redundant. But they're your last line of defense.

How to fix: Keep the equivalence tests until the cutover. Run them as part of CI until the day you remove Flask. On cutover day you can archive them (not delete — they're a historical reference).

Summary

• Equivalence tests verify that old==new for the same input
• Contract tests verify that the new API fulfills the expected structure
• Rollback tests verify that the old app is still functional
• Compare meaning, not exact format (400 vs 422, "error" vs "detail")
• Claude Code generates all these tests by analyzing both apps
• The 3 dimensions of equivalence: status + body + side effects
• Tests cover error cases, not just happy paths

Next capsule: Project — Complete Flask→FastAPI Migration.

Additional Resources

1. pytest - Parametrize - Parametrized tests for equivalence
2. FastAPI TestClient - Testing in FastAPI
3. Flask Testing - Testing in Flask
4. Contract Testing - Pact - A contract testing framework
5. API Compatibility Testing - OpenAPI to validate contracts
6. Hypothesis - Property-Based Testing - Generate inputs automatically

Module 5, Capsule 05 — Refactoring & Legacy Code with Claude Code Guide