Smart Post-Rollback Diagnosis

Overview

This is the capsule that closes the resilience cycle: when the automatic rollback runs (capsule 04), Claude Code generates a detailed diagnosis of what happened. Without a diagnosis, the rollback only prevents the immediate damage — but it doesn't generate learning. With a diagnosis, each incident becomes structured information the team can use to prevent the next one.

You'll learn to generate automatic diagnoses that: identify the commit/PR that introduced the problem, correlate metrics with code changes, suggest a likely fix, produce a postmortem draft, and notify the team with rich context — not just "rollback executed".

By the end, you'll have a system where a production incident automatically becomes a postmortem draft ready for review, with root cause analysis, a reconstructed timeline, and an action plan.

---

The Diagnosis That Matters

WITHOUT DIAGNOSIS (basic rollback):

🚨 Rollback executed
- Metric: error_rate
- Value: 5.2%
- Threshold: 2%

→ The team knows THAT something broke
→ Starts investigating from scratch
→ Loses 30-60 minutes identifying the root cause

WITH DIAGNOSIS (this module):

🚨 Rollback executed + Analysis

LIKELY ROOT CAUSE:
Commit a3b5c8d (PR #847 "Update payment validation")
modified the validate_amount() function in payment_service.py:42.
The change inverts the original condition — it used to reject
negative amounts, now it allows them.

EVIDENCE:
- Error rate rose 2 minutes after the deploy
- Errors concentrated on the /api/payments endpoint
- Stack traces point to payment_service.validate_amount()
- The commit diff shows: `if amount < 0` changed to `if amount > 0`

SUGGESTED FIX:
Revert the line 42 change to the original condition
`if amount < 0`. Re-apply the desired logic after the deploy
with tests covering the negative-amount case.

POSTMORTEM DRAFT:
[link to the generated draft]

→ The team has the complete context in 5 minutes
→ An informed decision about what to do next
→ A postmortem draft ready for review

The difference is ~45 minutes of team time per incident. Multiply it by 5-10 incidents/year = a lot of recovered time.

---

The 5 Pieces of the Diagnosis

1. CAUSE IDENTIFICATION
   Which commit/PR introduced the problem?
   - Time correlation: deploy time vs error onset
   - Diff analysis: what changed in that PR
   - Author + reviewer

2. EVIDENCE TRAIL
   Which signals show the problem?
   - Metrics that crossed the threshold (with values)
   - Affected endpoints/paths
   - Relevant stack traces
   - Error pattern logs

3. ROOT CAUSE ANALYSIS
   Why does the code fail?
   - Wrong logic in the change
   - An unanticipated side effect
   - Missing validation
   - An introduced race condition

4. FIX SUGGESTION
   How to fix it?
   - Revert the change (short-term, already done)
   - Re-apply with a correction (medium-term)
   - Structural improvement to prevent this class of bug (long-term)

5. POSTMORTEM DRAFT
   A document ready for human review
   - Reconstructed timeline
   - Impact assessment
   - Proposed action items

Each piece is valuable on its own. Together they turn "rollback" into "incident learning".

---

The Extended Workflow

# Continuation of the capsule 04 workflow
# In addition to the rollback, it adds diagnosis

diagnose-and-document:
  needs: monitor-and-rollback
  if: needs.monitor-and-rollback.outputs.rollback_needed == '1'
  runs-on: ubuntu-latest
  steps:
    - uses: actions/checkout@v4
      with: { fetch-depth: 0 }
    
    - uses: actions/setup-python@v5
      with: { python-version: '3.11' }
    
    - run: pip install anthropic requests
    
    - name: Generate diagnosis
      env:
        ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
        GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        GITHUB_REPOSITORY: ${{ github.repository }}
        METRICS_API_URL: ${{ secrets.METRICS_API_URL }}
        METRICS_API_TOKEN: ${{ secrets.METRICS_API_TOKEN }}
        DEPLOY_SHA: ${{ needs.deploy.outputs.new_release }}
        PREVIOUS_SHA: ${{ needs.deploy.outputs.previous_release }}
      run: python scripts/diagnose_incident.py
    
    - name: Create postmortem issue
      if: always()
      env:
        GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        GITHUB_REPOSITORY: ${{ github.repository }}
      run: python scripts/create_postmortem_issue.py
    
    - name: Upload diagnosis
      uses: actions/upload-artifact@v4
      with:
        name: incident-diagnosis
        path: |
          diagnosis.json
          postmortem_draft.md

---

The Diagnosis Script

"""scripts/diagnose_incident.py

Generates a complete diagnosis of the incident with context from:
- The deploy's commits
- Metrics during the degradation
- Error logs (if accessible)
"""
import json
import os
import subprocess
import sys
from datetime import datetime, timedelta
from pathlib import Path
from anthropic import Anthropic
import requests


def get_deploy_commits(deploy_sha: str, previous_sha: str) -> list[dict]:
    """Get the commits included in the deploy."""
    result = subprocess.run(
        ["git", "log", f"{previous_sha}..{deploy_sha}",
         "--format=%H||%an||%s", "--no-merges"],
        capture_output=True, text=True, check=True,
    )
    
    commits = []
    for line in result.stdout.strip().split("\n"):
        if not line:
            continue
        parts = line.split("||")
        if len(parts) >= 3:
            commits.append({
                "sha": parts[0],
                "author": parts[1],
                "message": parts[2],
            })
    return commits


def get_commit_diff(sha: str) -> str:
    """Get the complete diff of a commit (truncated)."""
    result = subprocess.run(
        ["git", "show", "--stat", "--format=", sha],
        capture_output=True, text=True, check=True,
    )
    stat = result.stdout
    
    # Complete diff (truncated)
    diff_result = subprocess.run(
        ["git", "show", "--format=", sha],
        capture_output=True, text=True, check=True,
    )
    diff = diff_result.stdout[:5000]  # truncate so it doesn't overflow the context
    
    return f"STATS:\n{stat}\n\nDIFF (truncated):\n{diff}"


def get_pr_for_commit(sha: str, repo: str, token: str) -> dict | None:
    """Find the PR associated with the commit."""
    headers = {"Authorization": f"Bearer {token}", "Accept": "application/vnd.github+json"}
    url = f"https://api.github.com/repos/{repo}/commits/{sha}/pulls"
    r = requests.get(url, headers=headers)
    if r.ok and r.json():
        pr = r.json()[0]
        return {
            "number": pr["number"],
            "title": pr["title"],
            "url": pr["html_url"],
            "body": pr.get("body", "")[:1000],
        }
    return None


def get_metrics_during_incident(
    deploy_time: str,
    rollback_time: str,
    api_url: str | None,
    api_token: str | None,
) -> dict:
    """Get metrics during the incident period."""
    if not api_url:
        # Mock for demo
        return {
            "error_rate_max": 5.2,
            "error_rate_baseline": 0.5,
            "latency_p95_max_ms": 2500,
            "latency_p95_baseline_ms": 800,
            "affected_endpoints": ["/api/payments", "/api/orders"],
            "error_samples": [
                "ValueError: Invalid amount",
                "ValidationError: amount must be positive",
            ],
        }
    
    # In production: query Prometheus/Datadog
    # ...
    return {}


def generate_diagnosis_with_claude(
    commits: list[dict],
    pr_info: dict | None,
    metrics: dict,
    rollback_reason: dict,
) -> dict:
    """Call Claude for a structured analysis."""
    
    # Build rich context
    commits_summary = "\n".join(
        f"- {c['sha'][:8]} ({c['author']}): {c['message']}"
        for c in commits
    )
    
    prompt = f"""Analyze this production incident and generate a structured diagnosis.

INCIDENT CONTEXT:
- Rollback triggered by: {rollback_reason.get('description', 'unknown')}
- Triggering metric: {rollback_reason.get('metric')}
- Value: {rollback_reason.get('current_value')}
- Threshold: {rollback_reason.get('threshold')}

DEPLOYED COMMITS:
{commits_summary}

{f'ASSOCIATED PR: #{pr_info["number"]} — {pr_info["title"]}' if pr_info else 'No clear associated PR'}
{f'Description: {pr_info["body"][:500]}' if pr_info else ''}

DIFF OF THE MOST RELEVANT COMMIT:
{get_commit_diff(commits[0]['sha']) if commits else 'No commits'}

METRICS DURING THE INCIDENT:
{json.dumps(metrics, indent=2)}

Generate a JSON diagnosis with this structure:

{{
  "root_cause": {{
    "likely_commit": "sha of the responsible commit",
    "explanation": "clear explanation of why this change caused the problem",
    "confidence": "high|medium|low"
  }},
  "evidence": [
    "evidence 1 (time correlation, stack trace, etc.)",
    "evidence 2",
    "..."
  ],
  "fix_suggestion": {{
    "short_term": "what to do now (typically: verify the rollback)",
    "medium_term": "how to re-apply the change without the bug",
    "long_term": "structural improvement to prevent this class of bug"
  }},
  "lessons_learned": [
    "lesson 1 (what failed in the process)",
    "lesson 2",
    "..."
  ],
  "action_items": [
    {{"item": "...", "priority": "high|medium|low", "owner": "team|individual"}}
  ]
}}

Be specific — use references to files, lines, real values.
Return ONLY the JSON.
"""
    
    client = Anthropic()
    response = client.messages.create(
        model="claude-sonnet-5",  # Sonnet for deep reasoning
        max_tokens=4000,
        messages=[{"role": "user", "content": prompt}],
    )
    
    text = response.content[0].text.strip()
    if text.startswith("```"):
        text = "\n".join(text.split("\n")[1:-1])
    
    return json.loads(text)


def generate_postmortem_draft(diagnosis: dict, commits: list, pr_info: dict | None,
                              rollback_reason: dict, metrics: dict) -> str:
    """Generate a postmortem draft in markdown."""
    
    incident_id = datetime.utcnow().strftime("%Y-%m-%d-%H%M")
    rc = diagnosis["root_cause"]
    fix = diagnosis["fix_suggestion"]
    
    md = f"""# Incident Postmortem: {incident_id}

**Status:** Draft (auto-generated)  
**Severity:** Auto-rolled back

## Summary

Auto-rollback executed in production due to degradation of **{rollback_reason.get('metric')}** ({rollback_reason.get('current_value')} vs threshold {rollback_reason.get('threshold')}).

Likely root cause: commit `{rc['likely_commit'][:8]}`{f' (PR #{pr_info["number"]})' if pr_info else ''}

## Timeline

- **T+0**: Deploy to production ({commits[0]['sha'][:8] if commits else 'unknown'})
- **T+~2min**: Metrics start to degrade
- **T+~5min**: Threshold crossed, automatic rollback executed
- **T+~7min**: Service stabilized on the previous version

## Impact

- Affected endpoints: {', '.join(metrics.get('affected_endpoints', []))}
- Error rate peak: {metrics.get('error_rate_max', 'unknown')}% (baseline: {metrics.get('error_rate_baseline', 'unknown')}%)
- Latency p95 peak: {metrics.get('latency_p95_max_ms', 'unknown')}ms (baseline: {metrics.get('latency_p95_baseline_ms', 'unknown')}ms)
- Estimated user impact: [TBD by on-call]

## Root Cause

**{rc['explanation']}**

Confidence: {rc['confidence']}

### Evidence

"""
    for ev in diagnosis["evidence"]:
        md += f"- {ev}\n"
    
    md += f"""
## Resolution

### Short-term (already done)
{fix['short_term']}

### Medium-term
{fix['medium_term']}

### Long-term
{fix['long_term']}

## Lessons Learned

"""
    for lesson in diagnosis["lessons_learned"]:
        md += f"- {lesson}\n"
    
    md += "\n## Action Items\n\n| Priority | Action | Owner |\n|----------|--------|-------|\n"
    for ai in diagnosis["action_items"]:
        md += f"| {ai['priority']} | {ai['item']} | {ai['owner']} |\n"
    
    md += f"""
## Commits Deployed

"""
    for c in commits:
        md += f"- `{c['sha'][:8]}` ({c['author']}): {c['message']}\n"
    
    md += f"""
---
*Auto-generated postmortem. Review and update with human insights before publishing.*
*Generated by Claude Code at {datetime.utcnow().isoformat()}*
"""
    return md


def main() -> int:
    deploy_sha = os.environ["DEPLOY_SHA"]
    previous_sha = os.environ["PREVIOUS_SHA"]
    repo = os.environ["GITHUB_REPOSITORY"]
    token = os.environ["GITHUB_TOKEN"]
    
    # 1. Load the rollback reason (from the monitoring script)
    rollback_reason = json.loads(Path("rollback_reason.json").read_text())
    
    # 2. Get the deploy's commits
    commits = get_deploy_commits(deploy_sha, previous_sha)
    print(f"Commits in deploy: {len(commits)}")
    
    # 3. PR info from the most recent commit
    pr_info = get_pr_for_commit(commits[0]["sha"], repo, token) if commits else None
    
    # 4. Metrics during the incident
    metrics = get_metrics_during_incident(
        deploy_time=rollback_reason.get("timestamp"),
        rollback_time=datetime.utcnow().isoformat(),
        api_url=os.environ.get("METRICS_API_URL"),
        api_token=os.environ.get("METRICS_API_TOKEN"),
    )
    
    # 5. Diagnosis with Claude
    print("Generating diagnosis with Claude...")
    diagnosis = generate_diagnosis_with_claude(commits, pr_info, metrics, rollback_reason)
    
    # 6. Postmortem draft
    postmortem = generate_postmortem_draft(diagnosis, commits, pr_info, rollback_reason, metrics)
    
    # Save
    Path("diagnosis.json").write_text(json.dumps(diagnosis, indent=2))
    Path("postmortem_draft.md").write_text(postmortem)
    
    print(f"\nRoot cause: {diagnosis['root_cause']['explanation']}")
    print(f"Confidence: {diagnosis['root_cause']['confidence']}")
    print(f"Suggested fix: {diagnosis['fix_suggestion']['medium_term']}")
    
    return 0


if __name__ == "__main__":
    sys.exit(main())

---

Create a Postmortem Issue

"""scripts/create_postmortem_issue.py

Creates a GitHub Issue with the postmortem draft.
"""
import json
import os
import sys
from pathlib import Path
import requests


def main() -> int:
    repo = os.environ["GITHUB_REPOSITORY"]
    token = os.environ["GITHUB_TOKEN"]
    
    postmortem_file = Path("postmortem_draft.md")
    diagnosis_file = Path("diagnosis.json")
    
    if not postmortem_file.exists():
        print("ERROR: postmortem_draft.md not found", file=sys.stderr)
        return 1
    
    body = postmortem_file.read_text()
    
    # Diagnosis to build the title
    diagnosis = json.loads(diagnosis_file.read_text()) if diagnosis_file.exists() else {}
    rc = diagnosis.get("root_cause", {})
    likely_commit = rc.get("likely_commit", "unknown")[:8]
    
    title = f"Incident postmortem: automatic rollback of {likely_commit}"
    
    headers = {
        "Authorization": f"Bearer {token}",
        "Accept": "application/vnd.github+json",
    }
    
    payload = {
        "title": title,
        "body": body,
        "labels": ["incident", "postmortem", "auto-generated"],
    }
    
    url = f"https://api.github.com/repos/{repo}/issues"
    r = requests.post(url, headers=headers, json=payload)
    
    if r.status_code == 201:
        issue = r.json()
        print(f"Issue created: {issue['html_url']}")
        return 0
    
    print(f"ERROR creating issue: {r.status_code} {r.text}", file=sys.stderr)
    return 1


if __name__ == "__main__":
    sys.exit(main())

---

Notification with Diagnosis

You add the diagnosis context to the capsule 04 notification script:

# Update to scripts/notify_rollback.py
def main() -> int:
    reason = json.loads(Path("rollback_reason.json").read_text())
    
    # Load the diagnosis if it's available
    diagnosis = None
    if Path("diagnosis.json").exists():
        diagnosis = json.loads(Path("diagnosis.json").read_text())
    
    # Generate a message with/without diagnosis
    if diagnosis:
        rc = diagnosis["root_cause"]
        message = f"""🚨 **Auto-rollback executed** (with diagnosis)

**Trigger:** {reason['description']}

**Root cause (likely):** {rc['explanation'][:200]}...
**Confidence:** {rc['confidence']}

**Suggested fix (medium-term):** {diagnosis['fix_suggestion']['medium_term']}

📋 Postmortem draft created as a GitHub Issue.
On-call: review the incident and validate the diagnosis.
"""
    else:
        message = generate_basic_notification(reason)
    
    # ... rest of the send to Slack

---

Calibrating the Diagnosis's Confidence

Claude doesn't always get the root cause right. The diagnosis structure must include a confidence level and the team should treat the output as a draft, not absolute truth.

HIGH CONFIDENCE (~70% of cases):
- A single commit in the deploy
- The PR description mentions the area that failed
- The commit diff shows a clearly related change

MEDIUM CONFIDENCE (~25% of cases):
- Multiple commits, several could be the cause
- Correlated metrics but not conclusive
- The change could be a secondary cause

LOW CONFIDENCE (~5% of cases):
- Indirect changes (e.g. a dependency update)
- A bug that was latent and gets activated by load
- A race condition or timing-specific bug

Important: the team should never trust the automatic diagnosis 100%. It's a starting point, not a verdict. Human validation is part of the postmortem process.

---

Common Pitfalls

Error 1: Treating the diagnosis as final truth

Symptom: The team applies the "suggested fix" without validating and breaks something else.

Why it happens: Blind trust in the model's output.

How to fix it: Always flag the diagnosis as "draft" or "automated suggestion". Human validation before acting.

Error 2: Diagnosis without enough context

Symptom: The diagnosis is generic ("check the last commit") because the script didn't pass useful info to the model.

Why it happens: Passing only "rollback executed" without diff, metrics, PR info.

How to fix it: Pass rich context: commits, diffs, metrics, PR descriptions, stack traces if accessible.

Error 3: Postmortem without concrete action items

Symptom: The postmortem says "review process" — it's not actionable.

Why it happens: The prompt doesn't emphasize specificity in the action items.

How to fix it: Ask explicitly: "concrete action items with owner and priority".

Error 4: Creating an automatic issue without review

Symptom: Issues with incorrect info fill the repo, the team stops reading them.

Why it happens: Blind trust in the auto-generation.

How to fix it: Issues clearly marked as "auto-generated, requires review". Assign someone specific to validate before circulating.

Error 5: Not including info from previous similar incidents

Symptom: Each incident is treated as new, we don't leverage patterns from previous incidents.

Why it happens: The system doesn't look for similar incidents in the history.

How to fix it: Before the analysis, query GitHub Issues with the incident label to find similar ones. Pass them as context to the model.

---

Diagnosis

Question 1: Does your system generate an automatic diagnosis of the rollback or only notify?

Only notification = the team spends 30+ min identifying the root cause. Diagnosis = the team starts with context.

Question 2: Does the diagnosis include a confidence level?

Without confidence, the team may apply incorrect fixes trusting blindly. With confidence, it knows when to validate more.

Question 3: Do you generate a postmortem draft or only a message to Slack?

Slack gets lost. A GitHub Issue = persistent, actionable, trackable.

Question 4: Does your diagnosis have concrete action items?

Vague action items = don't get done. Concrete ones with an owner = get executed.

Question 5: Does the team understand that the diagnosis is a draft, not final truth?

If they trust it 100%, the model's false positives cause more damage. Treat as a starting point.

---

Exercises

Exercise 1: Basic diagnosis generation (Medium)

Implement diagnose_incident.py that:

1. Loads rollback_reason.json
2. Lists the deploy's commits
3. Calls Claude with basic context
4. Generates a structured diagnosis.json

Verify with a simulated rollback.

Exercise 2: Postmortem draft (Medium)

Add postmortem markdown generation from the diagnosis JSON. The postmortem must have:

• Summary
• Timeline
• Root cause + evidence
• Resolution (short/medium/long term)
• Action items with priorities

Exercise 3: Automatic issue with a review workflow (Hard)

Implement:

1. Automatic creation of the Issue with the postmortem draft
2. Assignment to an on-call based on a schedule (cron or config)
3. A requires-human-review label
4. An auto-comment after 24h if nobody validated: "This issue needs review"

---

Summary

• Diagnosis transforms rollback into learning — the team starts with context, not from scratch
• 5 pieces of the diagnosis: cause, evidence, root cause analysis, fix, postmortem
• Confidence levels communicate the model's uncertainty to the team
• A postmortem draft is a starting point, not a verdict — always human review
• Concrete action items with an owner = get executed; vague ones = get ignored
• A persistent Issue > an ephemeral Slack message
• Patterns from previous incidents improve future diagnosis

Next capsule: Module 6 — Integrative Project: Complete CI/CD Pipeline. You combine everything learned in modules 1-5 into an end-to-end production-ready pipeline. It's the guide's close and the biggest portfolio piece.

---

Additional Resources

1. Site Reliability Engineering: Postmortem Culture — A chapter from Google's book on postmortem culture
2. Etsy: Code as Craft - Blameless PostMortems — A philosophy of blameless postmortems
3. Atlassian: Incident Management — An incident management framework
4. PagerDuty: Postmortem Template — A standard postmortem template
5. Honeycomb: Observability for Incidents — How observability speeds up diagnosis
6. Claude Code root cause analysis use cases — Applicable patterns