Incremental vs Big Bang Refactoring

Capsule description

This capsule formalizes why incremental wins and how to execute it as a process. It's not just a preference — it's a strategy with measurable advantages in risk, debugging, and rollback.

Why Big Bang Fails

The scenario

Big Bang:
Monday: "I'm going to modernize the whole module"
       Change f-strings + type hints + patterns + dead code
       across 15 files simultaneously
Tuesday: 3 tests fail
        Which of the 4 types of change caused the failure?
        In which of the 15 files?
        → Combinatorial debugging: 4 types × 15 files = 60 possible causes
Wednesday: Still debugging
Thursday: "I'm going to revert everything and start over"

The real cost

• Exponential debugging: N types of change × M files = N×M possible causes
• Total rollback: if something fails, you lose all the changes
• Impossible code review: a PR with 15 files and 4 types of changes is unreviewable
• Merge conflicts: while you work, other developers modify the same files

Why Incremental Wins

The scenario

Incremental:
Monday AM:  f-strings in user_service.py → tests pass → commit
Monday PM:  f-strings in order_service.py → tests pass → commit
Tuesday AM: type hints in user_service.py → tests pass → commit
Tuesday AM: type hints in order_service.py → 1 test fails
           → The failure is from type hints in order_service.py
           → We know exactly what and where
           → Fix in 10 minutes → commit

The advantages

The Incremental Framework

Step 1: Prioritize

List of modernizations by priority:
1. Dead code removal (low risk, cleans up noise)
2. Import cleanup (low risk)
3. bare except → specific exceptions (medium impact)
4. %-formatting → f-strings (low risk)
5. Type hints on public functions (medium effort)
6. Manual classes → dataclasses (medium risk)
7. os.path → pathlib (medium risk)

Step 2: One type at a time

Sprint 1: Dead code + import cleanup (the whole module)
Sprint 2: Exception handling (the whole module)
Sprint 3: f-strings (the whole module)
Sprint 4: Type hints (the whole module)
Sprint 5: Dataclasses + pathlib (the whole module)

Step 3: Tests at each step

For each sprint:
1. Run tests before the change → green
2. Make the change (1 type)
3. Run tests after → green
4. Commit with a descriptive message
5. If it fails → immediate fix (the cause is obvious)

Implementation with Claude Code

The incremental prompt

> "We're going to modernize src/services/order_service.py
   INCREMENTALLY. We start with step 1.
   
   STEP 1: Remove dead code (unused imports,
   functions never called, unreferenced variables).
   
   Show the changes. Run tests. Wait for my
   confirmation before step 2."

After confirming:

> "Tests pass. Step 2: Convert bare except to
   specific exceptions. Show the changes. Run tests."

The anti-pattern with Claude Code

# BAD: ask for everything at once
> "Modernize order_service.py: f-strings, type hints,
   dataclasses, pathlib, and clean up dead code"

# Claude Code does everything at once → if it fails, you don't know what caused it

# GOOD: one step at a time
> "Only f-strings for now. Nothing else."

When Big Bang Is Acceptable

On rare occasions, big bang makes sense:

1. The module is tiny (< 100 lines) — the risk is minimal
2. You have 100% test coverage — you can verify everything
3. The changes are purely cosmetic — only formatting, no logic
4. You're rewriting from scratch — it's not modernization, it's a rewrite

If none of these apply, use incremental.

Connection with the Project

In the Module Project (capsule 05), you execute incremental modernization: one type of change at a time, tests at each step, a separate commit for each type.

Exercises

Exercise 1: Order modernizations (Easy)

Order these modernizations from lowest to highest risk:

1. Add type hints to functions
2. Remove unused imports
3. Replace bare except with specific exceptions
4. Convert manual classes to dataclasses
5. Remove functions never called

Exercise 2: Design an incremental plan (Medium)

Your module has 8 files with mixed tech debt. Design an incremental plan of 4 sprints.

Sprint 1 (low risk, cleanup):
- Dead code removal across the 8 files
- Import cleanup across the 8 files
- 1 commit per type

Sprint 2 (low risk, syntax):
- f-strings across the 8 files
- 1 commit: "modernize string formatting"

Sprint 3 (medium risk, patterns):
- bare except → specific exceptions
- open() → context managers
- 1 commit per type

Sprint 4 (medium effort, types):
- Type hints on public functions
- Dataclasses where it applies
- 1 commit per type

The Real Cost: Comparative Table

To quantify the difference beyond intuition, consider a typical 500-line module with 12 tech debt items:

Big bang seems faster on the surface (2-3 hours vs 4-5). But the total cost includes debugging, review, and incident risk. When you integrate all the costs, incremental is 30-50% more efficient — and radically less risky.

Common Errors with Incremental

Error 1: "Incremental" but everything in a single commit

Symptom: You applied incremental in steps, but at the end you did git add . && git commit. The history shows 1 commit with 50 changes.

Why it happens: The instinct to "clean up the history" overrides the incremental discipline. But the history is the value.

How to fix: One commit per step. If you need to combine them at the end, use git rebase --interactive to merge selectively, not to collapse everything.

Error 2: Tests green but they don't cover the change

Symptom: You modernized a pattern. Tests green. In production it fails. It turns out the tests don't touch the code you changed.

Why it happens: "Tests green" gets confused with "tests green that cover my change". If you modify function X and the tests don't call X, the tests are irrelevant.

How to fix: Before moving to the next step, verify the change's coverage. If you modified lines that don't have tests, write the test before modernizing.

Error 3: Steps that are too large ("incremental but fat")

Symptom: "Step 1: modernize everything syntax-related". But "all syntax" is f-strings + type comparisons + dict comprehensions + walrus + etc. If it fails, which one was it?

Why it happens: Intellectual laziness in defining the "type" of change.

How to fix: One specific type of change per commit. F-strings is one commit. type() == X → isinstance is another. Not "syntax modernization" as a general category.

Error 4: Skipping the prior test phase

Symptom: You start with the change, "the existing tests are enough". Then you don't know whether what you'll break existed before or you introduced it.

Why it happens: Writing tests feels like a "separate task" from the real refactoring.

How to fix: Tests are always step 1. Even if tests already exist, run the suite and confirm it's green before touching anything.

Error 5: Not committing when a test is "temporarily" red

Symptom: You made a change, a test fails "but I'll fix it now". 2 hours pass, you have 5 stacked changes, and reverting is impossible.

Why it happens: Optimism: "I'm almost fixing it". But the problems accumulate, they don't simplify.

How to fix: If the tests are red after 15 minutes, revert and rethink. Don't accumulate more changes on top of a failure.

Summary

• Incremental wins for debugging, rollback, review, and risk
• Big Bang fails because N changes × M files = impossible debugging
• The framework: prioritize → one type at a time → tests at each step → commit
• Claude Code should receive incremental instructions, not "modernize everything"
• Big bang acceptable only in tiny modules with 100% coverage
• The total cost of incremental is ~30-50% lower than big bang
• Tests green ≠ tests that cover your change — verify the change's coverage

Next capsule: Project — Modernizing a Legacy Module.

A Reflection: Why Your Instinct Deceives You

Human instinct says: "if I do everything together, I save time". It's the same logic as "I'm going to do 5 things at once this week" that produces the opposite of the expected result.

In programming, the instinct is especially deceptive with AI tools. Claude Code can generate code fast — and that reinforces the instinct: "I can change 10 things in a single prompt". Yes, you can. But the cost isn't in generating — it's in debugging when something fails.

Time for Claude Code to make 10 changes:         5 minutes
Time to verify that the tests pass:              2 minutes
Debugging time if ONE change fails:              30 min - 4 hours
                                                (depends on which of the 10)

Total expected time:                             38 min - 4h 7min

Versus incremental:
Time for Claude Code to make 1 change:           30 seconds
Time to verify tests:                            30 seconds
Debugging time if it fails:                      5-15 min (obvious cause)
Repeat 10 times:                                 10 × 1 min + 1 failure × 10 min = 20 min total

Incremental wins even if everything goes well, but it wins overwhelmingly when something fails. The instinct deceives you because it only considers the "everything goes well" case — and in real projects, something fails.

Additional Resources

1. Working Effectively with Legacy Code - Feathers on incremental changes
2. Ship Small PRs - Google engineering practices
3. Trunk Based Development - Frequent, small commits
4. Feature Flags for Gradual Rollout - For modernizations in production

Module 7, Capsule 04 — Refactoring & Legacy Code with Claude Code Guide