Module 4: Deployment Automation

Deployment Readiness Validation

Next →

Deployment Readiness Validation

Overview

"Tests pass" doesn't mean "ready for production". This capsule teaches you to build an automated checklist that Claude Code verifies before each deploy, covering what the tests don't catch: pending DB migrations, missing env vars, deprecated dependencies, undocumented breaking changes, and runtime mismatches between staging and production.

The question the module answers: is the system really ready to deploy? Not just "the code compiles", but "all the necessary pieces are in place".

By the end, you'll have a workflow that runs a pre-deploy readiness validation, reports any issues, and blocks the deployment if something critical is missing.

Why Green Tests Aren't Enough

GREEN TESTS VERIFY:
✅ The code compiles/interprets without errors
✅ Individual functions work as you expect
✅ Internal integrations work
✅ The covered happy paths pass

GREEN TESTS DON'T VERIFY:
❌ Whether the DB migration is applied in the target environment
❌ Whether the necessary env vars are configured in production
❌ Whether the new code's dependencies are installed in the container
❌ Whether breaking changes are documented and communicated
❌ Whether the new features are behind feature flags
❌ Whether the infrastructure resources (queues, buckets, etc.) exist

Readiness validation fills that gap. It's the difference between "the code works on my machine" and "the system is ready to serve real traffic".

The Standard Readiness Checklist

1. CODE
   □ Tests pass (all suites)
   □ Linting passes
   □ Type checking passes
   □ No TODOs marked as "blocker"

2. DATABASE
   □ If there are new migrations → they're in the correct order
   □ Migrations are idempotent (IF NOT EXISTS)
   □ Down migrations exist for rollback
   □ Destructive changes are coordinated (don't break production with a DROP)

3. CONFIGURATION
   □ New required env vars are documented
   □ Sensitive env vars are in the secret manager (not .env)
   □ Config changes have a safe default

4. DEPENDENCIES
   □ New dependencies in requirements.txt / package.json
   □ No dependencies with known critical CVEs
   □ Pinned versions (not `latest`)

5. BREAKING CHANGES
   □ If there's a breaking change → documented in the changelog
   □ If there's a deprecation → it has a migration path
   □ If it affects external clients → communicated to the team

6. INFRASTRUCTURE
   □ New resources (queues, buckets, etc.) exist in the target environment
   □ IAM/RBAC permissions updated if necessary
   □ Enough capacity/quotas for the expected change

7. OBSERVABILITY
   □ New metrics configured
   □ Appropriate alerts if it's a critical feature
   □ Adequate logging for post-deploy debugging

8. ROLLBACK
   □ Rollback plan documented
   □ Tag/release of the previous state accessible
   □ Target rollback time defined (RTO)

The Validation Workflow

# .github/workflows/deployment-readiness.yml
name: Deployment Readiness Check

on:
  push:
    branches: [main]
  workflow_dispatch:  # also manually triggerable

permissions:
  contents: read
  pull-requests: write

jobs:
  validate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
      
      - uses: actions/setup-python@v5
        with:
          python-version: '3.11'
          cache: 'pip'
      
      - run: pip install anthropic requests
      
      - name: Validate readiness
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
          GITHUB_REPOSITORY: ${{ github.repository }}
          TARGET_ENV: production
        run: python scripts/validate_readiness.py
      
      - name: Upload readiness report
        if: always()  # upload even if it fails
        uses: actions/upload-artifact@v4
        with:
          name: readiness-report
          path: readiness_report.json

The Validation Script

"""scripts/validate_readiness.py

Validates that the system is ready to deploy.
Combines programmatic checks with analysis by Claude Code.
"""
import json
import os
import subprocess
import sys
from dataclasses import dataclass, field
from pathlib import Path
from anthropic import Anthropic


@dataclass
class CheckResult:
    name: str
    category: str
    status: str  # "pass", "warning", "fail"
    message: str
    details: list[str] = field(default_factory=list)


def check_tests_passing() -> CheckResult:
    """Assumes CI already ran tests. Verifies that the last commit
    has a positive status check."""
    # For simplicity: verify that a test directory exists
    # In real production, you'd query the checks API
    has_tests = (
        Path("tests").exists() or
        Path("test").exists() or
        Path("__tests__").exists()
    )
    
    if has_tests:
        return CheckResult(
            name="Tests exist",
            category="code",
            status="pass",
            message="Tests directory detected",
        )
    return CheckResult(
        name="Tests exist",
        category="code",
        status="fail",
        message="No tests directory",
    )


def check_migrations() -> CheckResult:
    """Verify new migrations and their consistency."""
    migrations_dir = next(
        (d for d in [Path("migrations"), Path("alembic/versions"), Path("db/migrate")]
         if d.exists()),
        None,
    )
    
    if not migrations_dir:
        return CheckResult(
            name="Migrations",
            category="database",
            status="pass",
            message="No migrations directory (project without a DB)",
        )
    
    # See migrations added in the last commit
    result = subprocess.run(
        ["git", "diff", "--name-only", "HEAD~1", "HEAD", "--",
         str(migrations_dir)],
        capture_output=True, text=True,
    )
    new_migrations = [f for f in result.stdout.strip().split("\n") if f]
    
    if not new_migrations:
        return CheckResult(
            name="Migrations",
            category="database",
            status="pass",
            message="No new migrations",
        )
    
    return CheckResult(
        name="Migrations",
        category="database",
        status="warning",
        message=f"{len(new_migrations)} new migrations — verify before the deploy",
        details=new_migrations,
    )


def check_env_vars() -> CheckResult:
    """Detect new required env vars in the code."""
    # Simplified: look for os.environ references in new code
    result = subprocess.run(
        ["git", "diff", "HEAD~1", "HEAD", "--",
         "*.py", "*.ts", "*.js"],
        capture_output=True, text=True,
    )
    diff = result.stdout
    
    # Basic pattern: detect referenced env vars
    import re
    env_var_pattern = re.compile(r'os\.environ\["([A-Z_]+)"\]|process\.env\.([A-Z_]+)')
    new_vars = set()
    
    for line in diff.split("\n"):
        if line.startswith("+") and not line.startswith("+++"):
            for match in env_var_pattern.finditer(line):
                var = match.group(1) or match.group(2)
                if var:
                    new_vars.add(var)
    
    if not new_vars:
        return CheckResult(
            name="Env vars",
            category="configuration",
            status="pass",
            message="No new env vars detected",
        )
    
    return CheckResult(
        name="Env vars",
        category="configuration",
        status="warning",
        message=f"Detected {len(new_vars)} env vars referenced in new code",
        details=sorted(new_vars),
    )


def check_dependencies() -> CheckResult:
    """Verify changes in dependency files."""
    dep_files = ["requirements.txt", "package.json", "pyproject.toml", "Gemfile"]
    
    result = subprocess.run(
        ["git", "diff", "HEAD~1", "HEAD", "--"] + dep_files,
        capture_output=True, text=True,
    )
    
    if not result.stdout.strip():
        return CheckResult(
            name="Dependencies",
            category="dependencies",
            status="pass",
            message="No changes in dependencies",
        )
    
    return CheckResult(
        name="Dependencies",
        category="dependencies",
        status="warning",
        message="Dependency changes detected — verify audit",
        details=result.stdout.split("\n")[:20],
    )


def analyze_with_claude(diff: str, conventions: str) -> list[CheckResult]:
    """Additional analysis with Claude Code: breaking changes,
    risky changes, improvements to the checklist."""
    prompt = f"""Analyze this release diff to detect readiness
issues for deployment. PROJECT CONTEXT:

{conventions}

RELEASE DIFF:

{diff[:8000]}


Identify:
1. Breaking changes (removed functions/endpoints, changed schemas)
2. Risky changes (critical business logic modified)
3. Missing documentation (changes without doc updates)
4. Possible configuration problems (refs to env vars, paths)

Return JSON:
{{
  "issues": [
    {{
      "name": "short name",
      "category": "code|database|configuration|dependencies|breaking changes|infrastructure|observability|rollback",
      "status": "pass|warning|fail",
      "message": "actionable description",
      "details": ["point 1", "point 2"]
    }}
  ]
}}
"""
    
    client = Anthropic()
    response = client.messages.create(
        model="claude-sonnet-5",
        max_tokens=3000,
        messages=[{"role": "user", "content": prompt}],
    )
    
    text = response.content[0].text.strip()
    if text.startswith("```"):
        text = "\n".join(text.split("\n")[1:-1])
    
    try:
        data = json.loads(text)
        return [CheckResult(**i) for i in data.get("issues", [])]
    except (json.JSONDecodeError, TypeError) as e:
        print(f"WARNING: analysis with Claude failed: {e}", file=sys.stderr)
        return []


def aggregate_results(checks: list[CheckResult]) -> dict:
    """Combine results into a final report."""
    by_status = {"pass": 0, "warning": 0, "fail": 0}
    for c in checks:
        by_status[c.status] = by_status.get(c.status, 0) + 1
    
    overall = "ready"
    if by_status["fail"] > 0:
        overall = "blocked"
    elif by_status["warning"] > 0:
        overall = "ready_with_warnings"
    
    return {
        "overall_status": overall,
        "summary": by_status,
        "checks": [
            {
                "name": c.name,
                "category": c.category,
                "status": c.status,
                "message": c.message,
                "details": c.details,
            }
            for c in checks
        ],
    }


def main() -> int:
    print("=== Deployment Readiness Validation ===\n")
    
    # Quick programmatic checks
    checks = [
        check_tests_passing(),
        check_migrations(),
        check_env_vars(),
        check_dependencies(),
    ]
    
    # Complete diff for analysis with Claude
    diff_result = subprocess.run(
        ["git", "diff", "HEAD~1", "HEAD"],
        capture_output=True, text=True,
    )
    
    conventions = (
        Path("CLAUDE.md").read_text() if Path("CLAUDE.md").exists()
        else "No documented conventions."
    )
    
    # Analysis with Claude
    print("Analyzing with Claude Code...")
    claude_checks = analyze_with_claude(diff_result.stdout, conventions)
    checks.extend(claude_checks)
    
    # Aggregate
    report = aggregate_results(checks)
    
    # Print readable results
    print(f"\nOverall: {report['overall_status'].upper()}")
    print(f"Pass: {report['summary']['pass']}, Warning: {report['summary']['warning']}, Fail: {report['summary']['fail']}\n")
    
    for c in checks:
        emoji = {"pass": "✅", "warning": "⚠️ ", "fail": "❌"}[c.status]
        print(f"{emoji} [{c.category}] {c.name}: {c.message}")
        for d in c.details[:3]:
            print(f"    - {d}")
    
    # Save the report as an artifact
    Path("readiness_report.json").write_text(json.dumps(report, indent=2))
    
    # Exit code: block if there are fails
    if report["overall_status"] == "blocked":
        print("\n❌ DEPLOYMENT BLOCKED — critical issues detected")
        return 1
    
    if report["overall_status"] == "ready_with_warnings":
        print("\n⚠️  DEPLOYMENT READY with warnings — review first")
    else:
        print("\n✅ DEPLOYMENT READY")
    
    return 0


if __name__ == "__main__":
    sys.exit(main())

Cross-Environment Validation

An important check: verify that the target environment is in sync with what the deploy will expect. For example, if the new code needs STRIPE_API_KEY, verify that this env var is configured in production.

def check_env_vars_in_target(target_env: str, github_token: str, repo: str) -> CheckResult:
    """Verify env vars configured in the target environment."""
    import requests
    
    # List the environment's variables via the GitHub API
    url = f"https://api.github.com/repos/{repo}/environments/{target_env}/variables"
    headers = {"Authorization": f"Bearer {github_token}", "Accept": "application/vnd.github+json"}
    r = requests.get(url, headers=headers)
    
    if r.status_code != 200:
        return CheckResult(
            name="Env vars in target",
            category="configuration",
            status="warning",
            message=f"Couldn't read variables from {target_env}",
        )
    
    configured_vars = {v["name"] for v in r.json().get("variables", [])}
    
    # Detect required vars from the code (similar to the previous check)
    required = detect_required_env_vars()
    missing = required - configured_vars
    
    if missing:
        return CheckResult(
            name="Env vars in target",
            category="configuration",
            status="fail",
            message=f"Missing env vars in {target_env}: {sorted(missing)}",
            details=sorted(missing),
        )
    
    return CheckResult(
        name="Env vars in target",
        category="configuration",
        status="pass",
        message=f"All required env vars are in {target_env}",
    )

Project-Type-Specific Checks

Different projects have different readiness priorities:

REST API

  • Backwards compatibility of public endpoints
  • API versioning (header, path, query)
  • Rate limits configured

Web App with a DB

  • Pending migrations
  • Indexes needed for new queries
  • Recent backups confirmed

Microservice

  • Compatible service contracts (OpenAPI, gRPC)
  • Service discovery updated
  • Circuit breakers configured

CLI tool

  • Semver version bumped
  • Backwards compatibility of commands
  • Updated docs in man pages / README

Adapt the specific checks to your project type.

Common Pitfalls

Error 1: Treating warnings as passes

Symptom: The report shows warnings but the deploy continues without anyone reviewing them.

Why it happens: The workflow doesn't visually differentiate between pass and warning, or it doesn't have an approval gate afterward.

How to fix it: In strict mode, warnings should require human confirmation before the deploy. Only passes allow auto-deploy.

Error 2: Migrations check without verifying idempotency

Symptom: You detect that there are new migrations but you don't validate whether they're idempotent. A migration with CREATE TABLE (without IF NOT EXISTS) fails on re-runs.

Why it happens: The check is superficial — it only counts files, it doesn't analyze content.

How to fix it: Pass the content of the migrations to Claude for idempotency and rollback analysis.

Error 3: Trusting automation 100%

Symptom: Everything is ready according to the script, but something is missing that the human knew but wasn't coded.

Why it happens: The team's "tribal" knowledge isn't in the checklist.

How to fix it: The checklist is a baseline, it doesn't replace human approval. For deploys to production, keep a human gate (capsule 04 of the module).

Error 4: A report without action

Symptom: It generates a report but nobody reads it. Issues make it to the deploy.

Why it happens: The report is left as an artifact with no notification to the team.

How to fix it: Post the report's summary as a comment on the commit/PR. Block the deploy if there are fails.

Error 5: Env vars check only of new code

Symptom: You detect that the new code needs NEW_VAR, but you don't detect that the old code already needed OLD_VAR that was never configured.

Why it happens: You're scanning only the diff, not the whole codebase.

How to fix it: For readiness, scan all the code. For PR review, scan only the diff. Different contexts.

Diagnosis

Question 1: Does your team have a readiness checklist or does it rely on "tests pass"?

Only tests = risk. An explicit checklist = safer. If you don't have one written, the first step is to write it.

Question 2: Do you validate migrations and env vars before a deploy?

Migrations and env vars are the #1 causes of post-deploy incidents. Automating those checks is high leverage.

Question 3: What happens when the script reports warnings?

If "they carry on unreviewed", the warnings are noise. If "they require human approval", they work as they should.

Question 4: Is the readiness report read or is it left as an ignored artifact?

If nobody reads it, it doesn't help. Posting the summary to the PR/commit + blocking if there are fails is the correct pattern.

Question 5: Does your check consider the target environment (prod, staging) or only the code?

Only code = partial check. Cross-environment (env vars, resources in the target) = complete check.

Exercises

Exercise 1: Minimal programmatic checklist (Easy)

Implement the 4 programmatic checks from the script (tests, migrations, env vars, dependencies). Verify it produces readable output.

Exercise 2: Analysis with Claude (Medium)

Add the analysis with Claude Code to detect breaking changes and risky changes. Make sure the JSON parsing is robust to code fences.

Exercise 3: Cross-environment validation (Hard)

Implement the env vars check against GitHub Environments. Connect with the API, list the variables configured in production, and compare them with those required by the code.

See approach

You need:

  1. A PAT with the scope to read environments (or GITHUB_TOKEN with the environments:read permission)
  2. An API call to /repos/{repo}/environments/{env}/variables
  3. A diff between the required vars (parsed from the code) and the configured vars
  4. Report the missing ones as fail

Summary

  • Green tests ≠ ready to deploy — readiness is broader
  • 8 categories of checks: code, DB, config, dependencies, breaking changes, infrastructure, observability, rollback
  • Combine programmatic checks (fast, cheap) with Claude Code (analytical, deep)
  • Cross-environment validation detects gaps between code and the target environment
  • Differentiate pass / warning / fail and act based on severity
  • A report that gets read > a report left as an artifact
  • Human approval is still necessary for production

Next capsule: 04 — Staging → production flow with approval gates. You have readiness validation working. Now you learn to integrate this with the complete deployment flow: automatic deploy to staging, staging validation, human approval, deploy to production.

Additional Resources

  1. 12-Factor App: Config — Configuration principles
  2. The Twelve-Factor App: Build, release, run — Release model
  3. GitHub Environments API — For validating target configuration
  4. Database Migration Best Practices — Martin Fowler on DB evolution
  5. Site Reliability Engineering: Release Engineering — Google's chapter
  6. Anthropic on Code Review use cases — Patterns applicable to readiness
← PreviousChangelog Generation Based on DiffsNext →Staging → Production Flow with Approval Gates