Module 6: Project — Complete CI/CD Pipeline with Claude Code
Failure Paths and Observability
Failure Paths and Observability
Overview
Capsules 03-04 covered the happy path: what happens when everything goes right. But a production-ready pipeline also handles the cases where something fails. This capsule covers the pipeline's failure paths and the observability necessary to detect, diagnose, and respond to them.
You're going to implement Phase 4 (post-deploy monitoring + auto-rollback + diagnosis) and design the pipeline's behavior for the 5 main failure modes: code review/security/tests block, staging deploy fails, metrics degrade in production, approval rejected, and deploy to production fails. Each with a clear and observable response.
By the end, you'll have a pipeline that fails gracefully in each failure mode, active post-deploy monitoring, and observability the team uses to make decisions with data.
The 5 Failure Modes
1. PR REVIEW BLOCKS (Phase 1)
- Code review found critical issues → blocks the merge
- Security scan found critical → blocks the merge
- Tests fail → block the merge
Behavior: the PR can't merge until resolved
Visibility: red status checks on the PR + comments
2. PRE-DEPLOY VALIDATION FAILS (Phase 2)
- Readiness validation detects missing env vars
- Problematic migrations
Behavior: deploy to staging doesn't proceed
Visibility: workflow run failed + readiness report as an artifact
3. STAGING DEPLOY FAILS (Phase 3)
- Deploy script error
- Staging smoke tests fail
- Staging metrics degrade
Behavior: the production gate doesn't activate
Visibility: workflow fails, team notified via Slack
4. APPROVAL REJECTED (Phase 3)
- Human reviewer clicks "Reject"
- Timeout (24 hrs without approval)
Behavior: deploy to production doesn't proceed
Visibility: PR comment + workflow status
5. PRODUCTION FAILS POST-DEPLOY (Phase 4)
- Production smoke tests fail
- Metrics degrade after the deploy
Behavior: automatic rollback + diagnosis + notification
Visibility: PagerDuty incident + Slack + postmortem Issue
For each one: expected behavior + visibility + recovery path.
Phase 4: Complete Implementation
# Continuation of the staged-deployment.yml workflow
monitor-post-deploy:
needs: deploy-production
runs-on: ubuntu-latest
outputs:
rollback_needed: ${{ steps.monitor.outputs.rollback_needed }}
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with: { python-version: '3.11', cache: 'pip' }
- run: pip install anthropic requests
- name: Monitor production for 10 minutes
id: monitor
env:
METRICS_API_URL: ${{ secrets.METRICS_API_URL }}
METRICS_API_TOKEN: ${{ secrets.METRICS_API_TOKEN }}
run: |
python scripts/monitor_post_deploy.py
if [ $? -eq 1 ]; then
echo "rollback_needed=true" >> $GITHUB_OUTPUT
else
echo "rollback_needed=false" >> $GITHUB_OUTPUT
fi
continue-on-error: true # ← rollback is separate
rollback:
needs: [deploy-production, monitor-post-deploy]
if: needs.monitor-post-deploy.outputs.rollback_needed == 'true'
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Execute rollback
env:
PREVIOUS_RELEASE: ${{ needs.deploy-production.outputs.previous_release }}
run: |
echo "🚨 Rolling back production to $PREVIOUS_RELEASE"
./scripts/rollback.sh production "$PREVIOUS_RELEASE"
- name: Verify rollback with smoke tests
run: ./scripts/smoke_tests.sh https://app.example.com
diagnose:
needs: [deploy-production, monitor-post-deploy, rollback]
if: always() && needs.monitor-post-deploy.outputs.rollback_needed == 'true'
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
with: { fetch-depth: 0 }
- uses: actions/setup-python@v5
with: { python-version: '3.11', cache: 'pip' }
- run: pip install anthropic requests
- name: Generate diagnosis
env:
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
GITHUB_REPOSITORY: ${{ github.repository }}
DEPLOY_SHA: ${{ needs.deploy-production.outputs.new_release }}
PREVIOUS_SHA: ${{ needs.deploy-production.outputs.previous_release }}
run: python scripts/diagnose_incident.py
- name: Create postmortem issue
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
GITHUB_REPOSITORY: ${{ github.repository }}
run: python scripts/create_postmortem_issue.py
- name: Notify team of rollback
env:
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
SLACK_WEBHOOK: ${{ secrets.SLACK_WEBHOOK }}
PAGERDUTY_TOKEN: ${{ secrets.PAGERDUTY_TOKEN }}
run: python scripts/notify_rollback.py
notify-success:
needs: [deploy-production, monitor-post-deploy]
if: needs.monitor-post-deploy.outputs.rollback_needed == 'false'
runs-on: ubuntu-latest
steps:
- uses: actions/setup-python@v5
with: { python-version: '3.11' }
- run: pip install requests
- name: Notify success
env:
SLACK_WEBHOOK: ${{ secrets.SLACK_WEBHOOK }}
run: python scripts/notify_success.pyWhat matters:
monitor-post-deploywithcontinue-on-error: true— failing here doesn't break the workflow, it only triggers rollbackrollbackonly runs ifrollback_needed == 'true'diagnoseruns even if rollback fails (if: always()) — the diagnosis is critical infonotify-successanddiagnoseare mutually exclusive
Behavior by Failure Mode
Failure Mode 1: PR Review blocks
WHAT HAPPENS:
1. Developer opens a PR
2. Phase 1 runs the 3 jobs in parallel
3. Security scan detects SQL injection (critical)
4. The security-scan job returns exit 1
5. The "security-scan" status check is marked red on the PR
6. Branch protection blocks the merge
7. A comment on the PR explains the finding
WHAT THE DEVELOPER SEES:
- Status checks: ❌ security-scan failed
- A bot comment with the specific finding
- "Merge" button disabled
- Instructions to resolve
WHAT THE TEAM SEES:
- Nothing until the developer resolves it
- If they need to escalate, they open a conversation on the PR
RECOVERY PATH:
1. Developer resolves the finding (or uses the override label if it's a false positive)
2. Push a fix
3. The workflow re-runs
4. Green status → merge enabled
Failure Mode 2: Pre-Deploy Validation Fails
WHAT HAPPENS:
1. PR merges to main
2. Phase 2 runs: readiness-validation
3. It detects that `STRIPE_API_KEY` isn't configured in the "production" environment
4. The job returns exit 1
5. Phase 3 doesn't start (deploy-staging has needs: readiness-validation)
WHAT THE TEAM SEES:
- Workflow run failed
- The `readiness-report.json` artifact with the detail
- A Slack notification if configured
RECOVERY PATH:
1. Configure the missing env var in GitHub Settings → Environments
2. Re-trigger the workflow manually:
- Actions → workflow run → "Re-run all jobs"
3. Or make a trivial push (`git commit --allow-empty -m "trigger ci"`)
Failure Mode 3: Staging Deploy Fails
WHAT HAPPENS:
1. The deploy to staging script fails (e.g. Docker image not found)
2. The deploy-staging job returns exit 1
3. validate-staging doesn't run (needs: deploy-staging)
4. The production gate never activates
WHAT THE TEAM SEES:
- Workflow failed on staging
- Deploy script logs
- Automatic notification (if configured)
- Production NOT affected (it didn't get deployed)
RECOVERY PATH:
1. Investigate the deploy script logs
2. If it's an infrastructure issue: fix and re-trigger
3. If it's a code issue: revert the commit on main
4. Re-merge when ready
Failure Mode 4: Approval Rejected
WHAT HAPPENS:
1. The pipeline reached "Waiting for approval to deploy: production"
2. Reviewer clicks "Reject" (or timeout 24hr without approval)
3. The deploy-production job is canceled
4. Phase 4 doesn't run
WHAT THE TEAM SEES:
- Workflow run in "cancelled" state
- The reviewer's comment (if they left a reason)
- Production NOT affected
RECOVERY PATH:
1. If it was an intentional rejection: review the reason, adjust the code if necessary
2. If it was a timeout without attention: communicate to the team, potentially delegate approval
3. Re-trigger the workflow after fixes
Failure Mode 5: Production Fails Post-Deploy
WHAT HAPPENS:
1. Deploy to production successful
2. Phase 4 runs monitor-post-deploy
3. Metrics degrade in the first 5 min:
- error_rate rises to 8% (threshold: 2%)
- latency_p95 rises 3x baseline
4. monitor-post-deploy returns exit 1
5. rollback triggers automatically
6. Rollback complete in ~3 min
7. diagnose runs and generates a postmortem
8. notify_rollback sends to Slack/PagerDuty
WHAT THE TEAM SEES:
- Slack: "🚨 Auto-rollback executed in production"
+ brief diagnosis
+ link to the postmortem issue
- PagerDuty: incident created (if configured)
- GitHub Issue created with the postmortem draft
- Production stabilized on the previous version
RECOVERY PATH:
1. On-call opens the postmortem issue
2. Validates the automatic diagnosis
3. Identifies the exact root cause
4. Decides: re-apply the fix vs investigate more
5. When ready: a new PR with the correct fix
6. Normal pipeline to production
Observability: What the Team Needs to See
NECESSARY DASHBOARDS:
1. Pipeline Health Dashboard
- Deploy success rate (%)
- Average pipeline duration
- Failures per phase
- MTTR (mean time to recovery) when there's a rollback
2. Cost Dashboard
- Cost per run (API + runners)
- Accumulated monthly cost
- Top jobs by cost
- Comparison vs budget
3. Quality Dashboard
- PRs with security findings (critical/high)
- Test coverage trend
- Categorized code review findings
- The bot's false positive rate
4. Production Health (post-deploy)
- Error rate trend (with deploy markers)
- Latency p95 trend
- Throughput
- Rollbacks executed (count + reasons)
Implement dashboards in your observability system:
# scripts/export_pipeline_metrics.py
"""Export pipeline metrics to your monitoring system."""
import os
import json
from datetime import datetime
import requests
def export_to_datadog(metrics: dict):
"""Push metrics to Datadog."""
api_key = os.environ.get("DATADOG_API_KEY")
if not api_key:
return
series = []
for name, value in metrics.items():
series.append({
"metric": f"ci.pipeline.{name}",
"points": [[int(datetime.utcnow().timestamp()), value]],
"tags": [f"workflow:{os.environ['GITHUB_WORKFLOW']}",
f"branch:{os.environ['GITHUB_REF_NAME']}"],
})
requests.post(
"https://api.datadoghq.com/api/v1/series",
headers={"DD-API-KEY": api_key, "Content-Type": "application/json"},
json={"series": series},
)
def main():
# Metrics to export at the end of the workflow
metrics = {
"duration_seconds": int(os.environ.get("WORKFLOW_DURATION", 0)),
"phase1_pass": 1 if os.environ.get("PHASE1_STATUS") == "success" else 0,
"phase2_pass": 1 if os.environ.get("PHASE2_STATUS") == "success" else 0,
"phase3_pass": 1 if os.environ.get("PHASE3_STATUS") == "success" else 0,
"rollback_executed": 1 if os.environ.get("ROLLBACK_EXECUTED") == "true" else 0,
}
export_to_datadog(metrics)
print(f"Pipeline metrics exported: {metrics}")
if __name__ == "__main__":
main()Add at the end of the workflow:
export-metrics:
needs: [deploy-production, monitor-post-deploy, rollback, diagnose]
if: always()
runs-on: ubuntu-latest
steps:
- uses: actions/setup-python@v5
with: { python-version: '3.11' }
- run: pip install requests
- name: Export pipeline metrics
env:
DATADOG_API_KEY: ${{ secrets.DATADOG_API_KEY }}
ROLLBACK_EXECUTED: ${{ needs.monitor-post-deploy.outputs.rollback_needed }}
run: python scripts/export_pipeline_metrics.pyAlerting: What Wakes People Up
ALERTING TIERS:
TIER 1 — WAKE ON-CALL (PagerDuty/SMS):
✅ Auto-rollback executed
✅ Production smoke tests failure
✅ Pipeline failure on the last step before prod
TIER 2 — HIGH PRIORITY NOTIFICATION (Slack channel #alerts):
⚠️ Critical security finding blocked the merge
⚠️ Multiple deploys failed (last day)
⚠️ Pipeline costs exceeded the monthly budget
TIER 3 — INFO (Slack channel #ci):
💬 Successful deploy to production
💬 Daily pipeline summary
💬 New high-severity findings (not critical)
NEVER ALERT:
❌ Individual successful deploy (it's not an alert, it's info)
❌ PR merged (it's the typical GitHub info)
❌ Bot comments (expected)
PagerDuty Integration
# scripts/notify_rollback.py — extend with PagerDuty
import requests
def trigger_pagerduty_incident(
summary: str,
severity: str,
details: dict,
routing_key: str,
):
"""Create an incident in PagerDuty."""
payload = {
"routing_key": routing_key,
"event_action": "trigger",
"payload": {
"summary": summary,
"source": "ci-pipeline",
"severity": severity,
"custom_details": details,
},
}
r = requests.post(
"https://events.pagerduty.com/v2/enqueue",
json=payload,
timeout=10,
)
return r.okCommon Pitfalls in Failure Paths
1. A rollback that also fails
Symptom: Metrics degrade, the rollback runs, but the rollback itself fails. Production is left broken.
Why it happens: The rollback script has no post-execution validation.
How to fix it: After the rollback, mandatory smoke tests. If they fail, escalate to a human (critical PagerDuty).
2. Ignored diagnosis
Symptom: Diagnose generates a postmortem issue, but nobody reads it. The next similar incident happens again.
Why it happens: The issue is left unassigned.
How to fix it: Auto-assign to a specific on-call. A Slack notification that links to the issue. SLA: review in 24hr.
3. Notification spam
Symptom: Slack full of notifs. The team stops reading. When there's a real alert, it gets lost.
Why it happens: Tier 1/2/3 mixed in the same channel.
How to fix it: Separate channels. #alerts only for tier 1. #ci-info for general info. PagerDuty only for tier 1.
4. Metrics exported but no dashboards
Symptom: You're exporting to Datadog/Prometheus, but nobody has ever looked at a dashboard.
Why it happens: They built exports before building consumers.
How to fix it: Build the dashboard first (even if empty). Then add exports. Without a consumer, there's no value in exporting.
5. Not testing failure paths before needing them
Symptom: The first real rollback reveals bugs in the rollback script. Under pressure, the rollback fails.
Why it happens: Testing only the happy path.
How to fix it: Regular game days — simulate failures intentionally to validate that the pipeline responds correctly. Chaos engineering applied to CI/CD.
Diagnosis
Question 1: Does your pipeline cover the 5 failure modes with explicit behavior?
If you only have the happy path, you'll improvise under pressure. Explicit coverage = predictable response.
Question 2: Does your automatic rollback have post-rollback smoke tests?
Without this, you can have a "successful rollback" according to the script but production broken.
Question 3: Do you have separate alerting tiers (PagerDuty / Slack alerts / Slack info)?
Without tiers, the team gets desensitized and the important alerts get lost.
Question 4: Did you build pipeline dashboards before exports?
If you have exports without dashboards, you spend time with no value. Dashboard first, then optimize exports.
Question 5: Did you test your failure paths intentionally?
If you never did, the first real one fails badly. Game days = practicing under control.
Exercises
Exercise 1: Implement Phase 4 (Hard)
Implement the 4 additional jobs: monitor-post-deploy, rollback, diagnose, notify-success/rollback. Verify that:
- monitor runs 10 min after the deploy
- rollback triggers if metrics degrade
- diagnose generates a postmortem issue
- notifications reach Slack
Exercise 2: Simulated game day (Medium)
Simulate a failure mode end-to-end:
- Configure a mock endpoint that returns 500 to X% of requests
- Configure low thresholds in monitoring
- Merge a PR that activates the endpoint
- Verify that the pipeline executes rollback + diagnose correctly
- Read the auto-generated postmortem
Exercise 3: Pipeline dashboard (Hard)
Create a dashboard in your observability system with:
- Pipeline duration trend
- Success rate per phase
- Cost per run (calculated from tokens + minutes)
- Rollbacks executed (count + reasons)
Summary
- 5 main failure modes: PR review, pre-deploy validation, staging deploy, approval rejected, production post-deploy
- Each mode with explicit behavior: what happens, what the team sees, recovery path
- Phase 4 integrates monitor + rollback + diagnose + notify
- 3 alerting tiers: PagerDuty (critical), Slack alerts (high), Slack info (routine)
- Dashboards first, exports after — without consumers there's no value
- Game days — test failure paths intentionally before needing them
if: always()on critical jobs like diagnose — they run even if previous jobs fail
Next capsule: 06 — Documentation, optimization, and retrospective. The module's last capsule. You turn the working pipeline into a transferable pipeline: operational documentation, cost optimizations, a retrospective with metrics. What separates "it works on my machine" from "production-ready".
Additional Resources
- Site Reliability Engineering: Postmortem Culture — Google's chapter
- PagerDuty Events API — For integration
- Datadog API — Custom metrics
- Prometheus + Grafana — Open-source stack
- Chaos Engineering Principles — For game days
- The Site Reliability Workbook: Incident Response — Applicable chapter