Module 5: Security Scanning and Smart Rollback
Automatic Rollback with Triggers
Automatic Rollback with Triggers
Overview
This capsule covers the last line of defense of the pipeline: automatic rollback when something goes wrong in production. Even if you have code review, security scanning, readiness validation, and human approval, there will always be a deploy that fails in production. The difference between a team that survives those incidents and one that suffers weeks of fallout is how long it takes to revert.
You'll learn to define which metrics to monitor post-deploy, configure automatic rollback triggers, execute the revert without human intervention when the signals are clear, and maintain the right balance between "aggressive rollback" (false positives = unnecessary downtime) and "conservative rollback" (slow response = more affected users).
By the end, you'll have a system where a failed deploy is reverted in less than 5 minutes automatically — and where the rollback explains what happened (which capsule 05 covers).
Why Manual Rollback Fails
MANUAL ROLLBACK IN A REAL INCIDENT:
T+0: Deploy to production
T+2min: Metrics start to degrade
T+5min: Alert reaches on-call (if configured)
T+8min: On-call opens the laptop
T+12min: Identifies it's post-deploy
T+15min: Decides to roll back
T+18min: Finds the exact command
T+22min: Runs the revert
T+25min: Verifies the metric came back
TOTAL TIME: ~25 minutes of degradation
AFFECTED USERS: variable, potentially thousands
Versus automatic rollback:
AUTOMATIC ROLLBACK:
T+0: Deploy to production
T+2min: Metrics degrade (error rate >5%)
T+3min: Trigger detects the threshold
T+3min: Automatic rollback executing
T+5min: Metrics stabilized
T+5min: On-call notified of the completed rollback
TOTAL TIME: ~5 minutes of degradation
AFFECTED USERS: an order of magnitude lower
5 min vs 25 min is the difference between an almost-invisible blip and an incident your team will remember.
The Metrics That Trigger Rollback
TIER 1 — Clear symptoms (automatic rollback appropriate):
✅ Error rate rises significantly
→ Typical threshold: >2% over baseline
→ Observation time: 2-5 minutes sustained
✅ Latency p95 rises significantly
→ Typical threshold: 2-3x the baseline
→ Observation time: 3-5 minutes sustained
✅ Health check fails
→ Threshold: 3+ consecutive failed health checks
→ Observation time: immediate
TIER 2 — Symptoms that require judgment (alert for a human):
⚠️ Throughput drops
→ Can be real (bug) or expected (behavior change)
→ Better: alert on-call, not automatic rollback
⚠️ Memory/CPU rises
→ Can be a memory leak or a change in workload
→ Better: alert on-call
TIER 3 — Indirect symptoms (don't trigger rollback):
❌ Business metrics (signups, conversions)
→ Too much noise from user behavior
❌ Errors from a third party (Stripe, AWS)
→ It's not our deploy
The rule: automatic rollback only on Tier 1. Other metrics → human alert that decides.
Monitoring Setup
Before configuring rollback, you need monitoring with metrics accessible via an API. Typical stack:
APPLICATION METRICS:
→ Prometheus + Grafana (open source, self-hosted)
→ Datadog (paid, full-featured)
→ New Relic (paid, focus on APM)
→ CloudWatch (AWS native)
LOGS / ERRORS:
→ Sentry (errors)
→ Loki / Splunk / Elasticsearch (logs)
INFRASTRUCTURE:
→ Same tools, cluster metrics
Any of them works. The important thing is to have an API accessible from the workflow that says: "in the last N minutes, was the error rate >X%?"
The Workflow with Rollback
# .github/workflows/deploy-with-rollback.yml
name: Deploy with Auto-Rollback
on:
push:
branches: [main]
permissions:
contents: read
deployments: write
jobs:
deploy:
runs-on: ubuntu-latest
environment: production
outputs:
previous_release: ${{ steps.deploy.outputs.previous_release }}
new_release: ${{ steps.deploy.outputs.new_release }}
steps:
- uses: actions/checkout@v4
- name: Capture previous release
id: previous
run: |
# Your mechanism: git tag, kubectl, etc.
PREV=$(./scripts/get_current_release.sh production)
echo "previous=$PREV" >> $GITHUB_OUTPUT
- name: Deploy
id: deploy
run: |
NEW=$(./scripts/deploy.sh production)
echo "previous_release=${{ steps.previous.outputs.previous }}" >> $GITHUB_OUTPUT
echo "new_release=$NEW" >> $GITHUB_OUTPUT
- name: Initial smoke tests
run: ./scripts/smoke_tests.sh https://app.example.com
monitor-and-rollback:
needs: deploy
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with: { python-version: '3.11' }
- run: pip install requests anthropic
- name: Monitor metrics for 10 minutes
id: monitor
env:
METRICS_API_URL: ${{ secrets.METRICS_API_URL }}
METRICS_API_TOKEN: ${{ secrets.METRICS_API_TOKEN }}
DEPLOY_TIME: ${{ needs.deploy.outputs.new_release }}
run: |
python scripts/monitor_post_deploy.py
echo "rollback_needed=$?" >> $GITHUB_OUTPUT
- name: Execute rollback if needed
if: steps.monitor.outputs.rollback_needed == '1'
env:
PREVIOUS_RELEASE: ${{ needs.deploy.outputs.previous_release }}
run: |
echo "🚨 Rolling back to $PREVIOUS_RELEASE"
./scripts/rollback.sh production "$PREVIOUS_RELEASE"
- name: Notify team of rollback
if: steps.monitor.outputs.rollback_needed == '1'
env:
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
SLACK_WEBHOOK: ${{ secrets.SLACK_WEBHOOK }}
run: python scripts/notify_rollback.py
- name: Notify success
if: steps.monitor.outputs.rollback_needed == '0'
run: echo "✅ Deploy stable, no rollback needed"The Monitoring Script
"""scripts/monitor_post_deploy.py
Monitors post-deploy metrics. Returns exit 1 if a rollback is needed.
"""
import json
import os
import sys
import time
from dataclasses import dataclass
from datetime import datetime, timedelta
import requests
@dataclass
class MetricThreshold:
name: str
threshold: float
description: str
sustained_minutes: int = 3 # how long it must be over the threshold
THRESHOLDS = [
MetricThreshold(
name="error_rate",
threshold=0.02, # 2%
description="Error rate > 2% sustained for 3 min",
sustained_minutes=3,
),
MetricThreshold(
name="latency_p95",
threshold=2.0, # 2x baseline
description="Latency p95 > 2x baseline sustained for 5 min",
sustained_minutes=5,
),
MetricThreshold(
name="health_check_failures",
threshold=3,
description="3+ consecutive failed health checks",
sustained_minutes=1,
),
]
def fetch_metric(name: str, window_minutes: int = 1) -> float | None:
"""Get a metric from the monitoring system."""
api_url = os.environ.get("METRICS_API_URL")
api_token = os.environ.get("METRICS_API_TOKEN")
if not api_url:
# Mock for testing
# In real production, query Prometheus/Datadog/etc.
return mock_metric(name)
headers = {"Authorization": f"Bearer {api_token}"}
params = {
"metric": name,
"window": f"{window_minutes}m",
}
try:
r = requests.get(f"{api_url}/query", headers=headers, params=params, timeout=10)
if r.ok:
data = r.json()
return data.get("value")
except Exception as e:
print(f"WARNING: failed to fetch {name}: {e}", file=sys.stderr)
return None
def mock_metric(name: str) -> float:
"""Mock for demo — in production replace with a real API."""
# Simulates normal values by default
return {
"error_rate": 0.005,
"latency_p95": 1.0,
"health_check_failures": 0,
}.get(name, 0.0)
def check_threshold(threshold: MetricThreshold) -> tuple[bool, float]:
"""Check whether the metric is sustainedly over the threshold.
Returns (is_over_threshold, current_value).
"""
samples = []
samples_needed = threshold.sustained_minutes
for i in range(samples_needed):
value = fetch_metric(threshold.name, window_minutes=1)
if value is not None:
samples.append(value)
else:
samples.append(0) # assume normal if there's no data
if i < samples_needed - 1:
time.sleep(60) # wait 1 min between samples
# All samples over the threshold
all_over = all(s > threshold.threshold for s in samples)
avg_value = sum(samples) / len(samples) if samples else 0
return all_over, avg_value
def main() -> int:
"""Monitor for 10 minutes and decide whether a rollback is needed."""
print(f"=== Post-deploy monitoring ({datetime.utcnow().isoformat()}) ===\n")
# Wait 1 min after the deploy so the metrics stabilize
print("Waiting 60s for metrics to stabilize...")
time.sleep(60)
# Check each threshold
for threshold in THRESHOLDS:
print(f"Monitoring {threshold.name}...")
is_over, value = check_threshold(threshold)
if is_over:
print(f" 🚨 ALERT: {threshold.description}")
print(f" Current value: {value}")
print(f" Action: ROLLBACK")
# Save rollback info for the following steps
with open("rollback_reason.json", "w") as f:
json.dump({
"metric": threshold.name,
"threshold": threshold.threshold,
"current_value": value,
"description": threshold.description,
"timestamp": datetime.utcnow().isoformat(),
}, f, indent=2)
return 1 # exit 1 → rollback needed
print(f" ✅ OK: {threshold.name} = {value}")
print("\n✅ Monitoring complete. Deploy stable.")
return 0
if __name__ == "__main__":
sys.exit(main())The Rollback Script
#!/bin/bash
# scripts/rollback.sh — execute a rollback to the previous release
set -euo pipefail
ENV="${1:?'Usage: rollback.sh <env> <previous_release>'}"
PREVIOUS_RELEASE="${2:?'Usage: rollback.sh <env> <previous_release>'}"
echo "[rollback] Rolling back $ENV to $PREVIOUS_RELEASE..."
# Adjust according to your platform:
# Kubernetes
# kubectl rollout undo deployment/app -n $ENV --to-revision=$PREVIOUS_RELEASE
# Heroku
# heroku releases:rollback v$PREVIOUS_RELEASE --app $ENV
# AWS ECS
# aws ecs update-service --cluster $ENV --service app \
# --task-definition app:$PREVIOUS_RELEASE
# Vercel / Netlify
# vercel rollback $PREVIOUS_RELEASE
# Custom deploy script
./scripts/deploy.sh "$ENV" --release "$PREVIOUS_RELEASE"
echo "[rollback] Rollback to $PREVIOUS_RELEASE complete"
# Verify smoke tests
./scripts/smoke_tests.sh https://app.example.com
echo "[rollback] Verification complete"Post-Rollback Notification
"""scripts/notify_rollback.py
Notify the team of the automatic rollback with context.
"""
import json
import os
import sys
from pathlib import Path
import requests
from anthropic import Anthropic
def generate_notification(reason: dict) -> str:
"""Generate a notification message with context."""
client = Anthropic()
prompt = f"""The system just did an automatic rollback in production.
CONTEXT:
{json.dumps(reason, indent=2)}
Generate a BRIEF message for Slack that:
1. Starts with "🚨 Auto-rollback executed"
2. Explains which metric triggered the rollback
3. Says what on-call does now (verify stabilization, investigate root cause)
4. Is professional and actionable, maximum 100 words
Output: only the message, with no additional text.
"""
response = client.messages.create(
model="claude-haiku-4-5",
max_tokens=300,
messages=[{"role": "user", "content": prompt}],
)
return response.content[0].text.strip()
def send_slack(message: str, webhook: str):
"""Send to Slack."""
payload = {"text": message, "username": "Auto-Rollback Bot", "icon_emoji": ":rotating_light:"}
r = requests.post(webhook, json=payload, timeout=10)
if not r.ok:
print(f"WARNING: Slack failed: {r.status_code}", file=sys.stderr)
def main() -> int:
reason_file = Path("rollback_reason.json")
if not reason_file.exists():
print("ERROR: rollback_reason.json not found", file=sys.stderr)
return 1
reason = json.loads(reason_file.read_text())
message = generate_notification(reason)
print("--- Notification ---")
print(message)
print("---")
webhook = os.environ.get("SLACK_WEBHOOK")
if webhook:
send_slack(message, webhook)
print("Notification sent to Slack")
# Additional: PagerDuty for urgent incidents
pd_token = os.environ.get("PAGERDUTY_TOKEN")
if pd_token:
# Create an incident in PagerDuty
# ...
pass
return 0
if __name__ == "__main__":
sys.exit(main())Calibrating the Thresholds
The threshold values are specific to your system. Calibrate them based on:
DATA TO COLLECT:
1. Normal baseline: what's the typical error rate? Latency p95?
2. Historical variability: how much does it naturally fluctuate?
3. Previous false positives: were there alerts that weren't real?
GENERAL RULE:
- Threshold = baseline_max + 2-3x standard deviation
- Sustained time: enough to rule out spikes (2-5 min typical)
- When in doubt, a more permissive threshold (rollback only if clear)
Calibration example
If your normal error rate is 0.5% with a standard deviation of 0.2%:
- Baseline + 3σ = 0.5% + 0.6% = ~1.1%
- Conservative threshold: 2% (clearly over baseline)
- Sustained: 3 min (rules out momentary spikes)
Common Pitfalls
Error 1: Aggressive rollback (false positives)
Symptom: Frequent rollbacks on deploys that had no real bug. The team loses confidence.
Why it happens: Thresholds too low or observation time too short.
How to fix it: Increase thresholds and/or sustained time. Better to be conservative with rollback than with a human alert.
Error 2: Conservative rollback (false negatives)
Symptom: A bug reaches production, metrics degrade, but the rollback doesn't trigger because "it didn't reach the threshold".
Why it happens: A threshold too high or a window too long.
How to fix it: Calibrate with historical data. Review past incidents — which thresholds would have detected them?
Error 3: Not capturing the previous release before the deploy
Symptom: You need to revert but you don't know to which version.
Why it happens: The deploy script doesn't save the previous release before applying the new one.
How to fix it: An explicit step that captures the current release before the deploy and exposes it as a job output.
Error 4: Rollback without verifying it stabilized
Symptom: Rollback executed, but the metrics are still bad because the rollback also failed.
Why it happens: The rollback script runs without verifying post-rollback smoke tests.
How to fix it: Smoke tests after the rollback. If the rollback itself fails, escalate to a human (PagerDuty).
Error 5: Static thresholds without updating
Symptom: Thresholds configured 1 year ago when the system was different. They're no longer appropriate.
Why it happens: Lack of periodic review.
How to fix it: Quarterly review of thresholds. Compare with recent incidents — would they have triggered correctly?
Diagnosis
Question 1: Does your pipeline have automatic rollback configured?
Without automatic rollback, the first incident in production takes 25+ minutes. With rollback, ~5 minutes.
Question 2: Which metrics trigger your rollback?
Tier 1: error rate, latency p95, health checks. If you only monitor "is the app up?", you're missing important signals.
Question 3: How long does your rollback take from detection to restored service?
Target: <5 minutes. If it's more, optimize the rollback script (image cache, etc.).
Question 4: Do you capture the "previous release" before the deploy?
If not, you can't revert programmatically. The capture step is essential.
Question 5: Are the thresholds calibrated to your system or are they generic?
Generic = false positives or false negatives. Calibrated to the real baseline = precise.
Exercises
Exercise 1: Capture previous release (Easy)
Implement a get_current_release.sh script that returns the release currently deployed in production. Adapt it to your platform (Kubernetes, Heroku, etc.).
Exercise 2: Basic monitoring with thresholds (Medium)
Implement monitor_post_deploy.py with:
- A mock of metrics (the
mock_metricfunction) - Thresholds for error_rate, latency, health checks
- A sustained time check (samples every minute)
- Exit code 1 if a rollback is needed
Test it by simulating metrics that exceed the threshold.
Exercise 3: Complete pipeline with rollback (Hard)
Combine:
- Deploy
- Capture previous release
- Monitor post-deploy (10 min)
- Automatic rollback if needed
- Notification with context
Test it in a staging environment — force a "fake fail" to verify the rollback.
Summary
- Manual rollback = 25 min, automatic rollback = 5 min — an order of magnitude difference
- Tier 1 metrics trigger rollback: error rate, latency p95, health checks
- Tier 2 metrics alert a human: throughput, memory, CPU
- Sustained time avoids false positives from momentary spikes
- Capture the previous release BEFORE the deploy — necessary to revert programmatically
- Verify smoke tests post-rollback — the rollback itself can fail
- Calibrate thresholds to your system, don't use generic values
Next capsule: 05 — Smart post-rollback diagnosis. The rollback works, but it doesn't teach. The module's final capsule: when the rollback runs, Claude Code analyzes what happened, identifies the root cause, and suggests a fix. The rollback becomes systematized learning.
Additional Resources
- Site Reliability Engineering: Effective Troubleshooting — A chapter from Google's book
- Argo Rollouts — Automatic rollback in Kubernetes
- Spinnaker Canary Analysis — For combined canary + rollback
- Datadog Synthetics — Continuous monitoring of endpoints
- Prometheus Alertmanager — Open source alerting
- Honeycomb Observability — Modern observability for distributed systems