Module 6: Project — Complete CI/CD Pipeline with Claude Code

Documentation, Optimization, and Retrospective

Documentation, Optimization, and Retrospective

Overview

This is the guide's final capsule. The pipeline works end-to-end. Now you turn it into something transferable: operational documentation a teammate can use, cost and speed optimizations, and a retrospective with real metrics that justifies the ROI to the team or management.

Without this capsule, you have a pipeline that works on your machine. With this capsule, you have a professional artifact that demonstrates senior capability in CI/CD with AI. It's the difference between "I did something" and "I delivered something the team can operate and maintain".

By the end, you'll have: complete operational documentation, a runbook for failures, real pipeline metrics, a prioritized optimization plan, and a reflection on lessons learned. It's what closes the integrative project as a portfolio piece.

The 4 Documents of the Final Deliverable

docs/
├── DEPLOYMENT.md      # How the pipeline operates (for users)
├── RUNBOOK.md         # What to do when X fails (for on-call)
├── METRICS.md         # Cost and time analysis (for stakeholders)
└── LESSONS.md         # Lessons learned (for the team)

Each one has a specific audience and a clear purpose.

Document 1: DEPLOYMENT.md

Audience: developers on the team who use the pipeline. Purpose: explain how it works and how to interact with it.

# Deployment Pipeline

An end-to-end CI/CD pipeline with Claude Code. Covers from opening a PR to production with automatic rollback.

## When it triggers

| Event | Workflow | Trigger |
|--------|----------|---------|
| PR opened/sync | PR Review (Phase 1) | `pull_request` |
| Push to main | Staged Deployment (Phase 2-4) | `push: main` |

## The 4 Phases

### Phase 1: PR Review (~5 min)
- Automatic code review with Claude Code
- Security scan (critical findings block the merge)
- Tests + linting + type checking

### Phase 2: Pre-Deployment (~3 min)
- Automatic changelog generation
- Readiness validation (env vars, migrations, deps)

### Phase 3: Deployment (~13 min + approval wait)
- Automatic deploy to staging
- Smoke tests + validation with Claude
- **Human approval gate before production**
- Deploy to production + smoke tests
- Notification to the team

### Phase 4: Post-Deploy (~10 min monitoring)
- Continuous metrics monitoring
- Auto-rollback if metrics degrade
- Smart post-rollback diagnosis

## How to Approve a Deploy

When the pipeline reaches `deploy-production`, GitHub shows:
> ⏸️ Waiting for approval to deploy: production

As a reviewer:

1. **Before approving**, review:
   - The `validate-staging` bot comment on the PR (summary + checklist)
   - Staging metrics in Grafana: [link]
   - The release changelog (workflow artifact)

2. **If all OK**, click "Approve and deploy"
3. **If in doubt**, click "Reject" + a comment explaining, or ask for more info on the PR

## How to See the Pipeline Status

- **Pipeline runs:** the repo's Actions tab
- **Aggregated metrics:** Grafana dashboard "Pipeline Health" [link]
- **Historical rollbacks:** GitHub Issues with the `incident` label

## How to Skip the Pipeline (Emergencies Only)

### Urgent hotfix (skip PR review)

```bash
git commit -m "hotfix: critical bug in payment

[skip ci]"
git push origin hotfix/critical-bug

⚠️ Emergencies only. Requires post-merge approval in the #emergency channel.

Bypass security scan (confirmed false positive)

Apply the security-scan-override label to the PR. Requires explicit approval from @security-team.

How to Modify the Pipeline

Changes to the pipeline are regular commits to the repo in .github/workflows/ or scripts/. These changes go through their own code review.

Important: changes to the pipeline itself must be tested on a testing branch before merging to main.

Costs

ItemExpected Cost
Per complete run~$0.30
Monthly (50 deploys)~$15
Monthly (200 deploys)~$60

Cost monitoring: the "CI Cost" dashboard in Grafana.

Support

  • Issues with the pipeline: open an issue with the pipeline label
  • Bot false positives: comment on the PR, assign to @ai-bot-maintainer
  • Incidents: active PagerDuty rotation

---

## Document 2: RUNBOOK.md

**Audience:** on-call during incidents.
**Purpose:** what to do when X fails. Concrete steps, not description.

```markdown
# Pipeline Runbook

## Failure Mode 1: Pipeline Fails in Phase 1 (PR Review)

### Symptoms
- Red status check on the PR
- Workflow run "failed"

### Steps
1. Identify which job failed:
   - GitHub PR → "Details" on the red status check
   
2. Per job:
   - **code-review failed:** Claude API error or JSON parse → re-run
   - **security-scan failed with critical:** review the findings, decide whether to fix or override
   - **tests-and-linting failed:** run tests locally, fix

3. If it's a pipeline bug (not the code):
   - Issue with the `pipeline-bug` label
   - Tag @pipeline-team

### Escalation
- If it affects multiple PRs simultaneously: post to #infrastructure
- If it blocks a critical release: PagerDuty rotation

---

## Failure Mode 2: Readiness Validation Fails

### Symptoms
- The `readiness-validation` job failed
- Workflow doesn't continue to deploy

### Steps
1. Download the `readiness-report.json` artifact:
   ```bash
   gh run download <RUN_ID> -n readiness-report

  1. Identify the check that failed:

    cat readiness_report.json | jq '.checks[] | select(.status == "fail")'

  2. Resolve by type:

    • Missing env var: Settings → Environments → production → add it
    • Migration issue: review migrations/ and apply
    • Undocumented breaking change: add it to the changelog before deploy

  3. Re-trigger the workflow:

    gh workflow run staged-deployment.yml --ref main

Escalation

  • If it's repetitive (>3 failures in a week): review the PR process

Failure Mode 3: Deploy to Staging Fails

Symptoms

  • The deploy-staging job failed
  • A comment on the commit with the error

Steps

  1. Review the job logs:

    gh run view <RUN_ID> --log

  2. Identify the type:

    • Docker image not found: check the registry, re-build if necessary
    • kubectl timeout: check cluster health
    • Dependency failed: check requirements.txt, dependency CVE

  3. If it's an infra issue:

    • Check the [Infrastructure status page]
    • Re-trigger after resolving

  4. If it's a code issue:

    • Revert the commit on main
    • Investigate offline
    • Re-merge when ready

Escalation

  • 30 min without resolution: PagerDuty
  • 1 hour without resolution: incident escalation team

Failure Mode 4: Automatic Rollback Executed

Symptoms

  • Slack alert: "🚨 Auto-rollback executed"
  • PagerDuty incident
  • GitHub Issue with a postmortem draft

Steps

  1. Verify production:

    • Metrics in Grafana (should be returning to baseline)
    • Manual smoke tests: ./scripts/smoke_tests.sh https://app.example.com

  2. Read the auto-generated postmortem:

    • GitHub → Issues → incident label
    • Validate the diagnosis

  3. If metrics are still bad:

    • Verify the rollback completed: kubectl get deployment app -n production
    • If the rollback failed: escalate to the infra team
    • Manual rollback as a last resort

  4. Communicate to the team:

    • Update in #incidents-channel
    • Confirm production stable

  5. Postmortem follow-up:

    • Assign the issue to someone for review in 24hr
    • Validate and complete the postmortem
    • Action items with owners

Escalation

  • Rollback fails: infra team + senior on-call
  • More than 1 rollback in 24hr: emergency process review

Failure Mode 5: Approval Gate Stuck

Symptoms

  • Workflow in "Waiting for approval" for more than 1 hour
  • Reviewers didn't respond

Steps

  1. Verify configured reviewers:

    • Settings → Environments → production → Required reviewers

  2. Notify reviewers:

    • Comment on the workflow
    • Slack DM to the senior on-call

  3. If urgent and nobody available:

    • Request an override in the #emergency channel
    • Another reviewer with permission can approve

  4. If timeout (24hr):

    • The workflow cancels automatically
    • Re-merge if the change is still valid

Common Commands

# See recent runs
gh run list --limit 10

# See a run's detail
gh run view <RUN_ID>

# Re-trigger the last run
gh run rerun <RUN_ID>

# Trigger the workflow manually
gh workflow run staged-deployment.yml --ref main

# Cancel a run in progress
gh run cancel <RUN_ID>

# Download artifacts
gh run download <RUN_ID>

On-Call Rotation

DayPrimarySecondary
Mon-Wed@on-call-1@on-call-2
Thu-Fri@on-call-2@on-call-3
Sat-Sun@on-call-3@on-call-1

PagerDuty: [link to the rotation]


---

## Document 3: METRICS.md

**Audience:** stakeholders, management, the team.
**Purpose:** justify the ROI with concrete data.

```markdown
# Pipeline Metrics & ROI Analysis

Pipeline data after [N runs in the last 90 days].

## Times

### Average per phase

| Phase | Average | P95 | Min | Max |
|-------|----------|-----|-----|-----|
| Phase 1 (PR Review) | 4.2 min | 6.5 min | 2.1 min | 9.8 min |
| Phase 2 (Pre-Deploy) | 2.8 min | 4.1 min | 1.5 min | 6.2 min |
| Phase 3 (Deployment) | 12.5 min | 18 min | 8 min | 25 min |
| Approval wait | 14 min | 1.2 hrs | 30s | 18 hrs |
| Phase 4 (Monitoring) | 10.1 min | 10.5 min | 9.8 min | 11.2 min |
| **Total (without approval)** | **29.6 min** | **38.6 min** | **22 min** | **55 min** |

### Distribution

Pipeline duration distribution (last 90 days):

10-20 min: ████ 22% 20-30 min: ████████████ 51% 30-40 min: ██████ 19% 40-60 min: ██ 6%

60 min: █ 2% (typically a long approval wait)


## Costs

### Per run

| Component | Average Cost |
|-----------|----------------|
| Anthropic API calls | $0.18 |
| GitHub Actions runners | $0.09 |
| Slack/PagerDuty | ~$0.01 |
| **Total** | **$0.28** |

### Monthly

| Month | Deploys | Total Cost | Average/Deploy |
|-----|---------|-------------|-----------------|
| Mar 2026 | 47 | $13.20 | $0.28 |
| Apr 2026 | 52 | $14.40 | $0.28 |
| May 2026 | 61 | $17.10 | $0.28 |

**Trend:** consistent. Predictable costs.

### Top costs per job
  1. claude-review: $0.06/run (Sonnet, ~3K tokens)
  2. security-scan: $0.05/run (Sonnet, ~2.5K tokens)
  3. validate-staging: $0.04/run (Sonnet, ~2K tokens)
  4. diagnose (when triggered): $0.06/run (Sonnet, ~3K tokens)
  5. Others: $0.07/run combined

## Quality Metrics

### Success rate

| Metric | Value |
|---------|-------|
| Successful deploys | 94% (172/183) |
| Rollbacks executed | 6 (3.3%) |
| Approvals rejected | 5 (2.7%) |

### Bot performance

| Metric | Value |
|---------|-------|
| Code review false positive rate | 12% |
| Security scan false positive rate | 8% |
| Diagnose accuracy (validated) | 78% |

### Time saved (estimated)

BEFORE (manual process):

  • Human code review per PR: ~30 min
  • Manual changelog per release: ~30 min
  • Manual readiness check: ~15 min
  • Deploy coordination: ~15 min
  • Incident investigation: ~45 min TOTAL: ~135 min of human time per deploy

AFTER (with pipeline):

  • Approval click: ~3 min
  • Validating bot comments: ~10 min (validated in human review)
  • Incident investigation: ~15 min (with auto diagnosis) TOTAL: ~28 min of human time per deploy

SAVINGS: ~107 min per deploy


### Applied to 200 deploys/month

Time saved/month: 200 × 107 min = 21,400 min = ~357 hours Value (at $100/hr of developer): $35,700/month

Pipeline cost: $14/month

ROI: 2,500x


## Trend: Pipeline Health

[Insert charts of:]
- Success rate trend (last 6 months)
- Deploy frequency trend
- MTTR when there's a rollback
- Accumulated monthly costs

## Action Items

Based on the data:

1. **Reduce the false positive rate** of code review (12% → target 5%)
   - Approach: refine the prompt + a more specific CLAUDE.md
   - Owner: @ai-bot-maintainer
   - Timeline: Q3 2026

2. **Increase diagnose accuracy** (78% → target 90%)
   - Approach: more context in the prompt (logs, historical metrics)
   - Owner: @ai-bot-maintainer

3. **Reduce approval wait time** (14 min average)
   - Approach: more visible Slack notification, automatic escalation at 1hr
   - Owner: @sre-team

4. **Reduce Phase 3 P95** (18 min → target 12 min)
   - Approach: optimize the deploy script, better caching
   - Owner: @sre-team

Document 4: LESSONS.md

Audience: the team and future maintainers. Purpose: capture learnings so mistakes aren't repeated.

# Lessons Learned: Building the Pipeline

## Decisions I Made Well

### 1. Starting with suggest-only mode (Module 2)
The bot operated without blocking merges during the first 4 weeks. It let us see how it behaved without generating friction. When we moved some checks to "blocking", the team already trusted the bot.

**Applicable to:** any introduction of automation that affects the team's workflow.

### 2. Capture previous_release before the deploy
Trivial but critical. Without this step, the rollback is manual. With this step, it's automatic and fast.

**Applicable to:** any system with rollback potential.

### 3. Design before implementing (Module 6 capsule 02)
The 2 hours I invested in the design document saved me days of rework. Especially for deciding parallelization and dependencies between stages.

**Applicable to:** any non-trivial technical project.

## Decisions I Made Poorly (And Corrected)

### 1. Initially I used Opus for everything
Costs grew 5x. Migrating to Haiku for basic code review and Sonnet for deep analysis reduced costs without losing quality.

**Lesson:** Default to the most economical model that works. Move up only when you have clear evidence it's worth it.

### 2. Lack of markers in comments
The first version generated a new comment for each push. After 5 pushes, the PR had 5 bot comments. The team complained.

**Lesson:** Unique markers (`<!-- bot-x -->`) are trivial to implement and prevent this problem from day 1.

### 3. Monitoring thresholds too low at the start
First week: 4 rollback false positives. The team lost confidence.

**Lesson:** Start with conservative (more permissive) thresholds. Tighten gradually with real data.

## Surprises

### 1. Approval wait is the biggest variable
The time from merge to production is dominated by the approval wait, not by the pipeline itself. The pipeline takes ~30 min; approval can take 0 to 24 hours.

**Implication:** optimizing the approval flow (notifications, delegation) has more impact on speed than optimizing the pipeline.

### 2. Diagnose is more valuable than expected
I thought automatic diagnosis would be a nice-to-have. It turned out to be what the team uses most in post-mortems. The auto-generated postmortem draft saves ~30 min in each incident.

**Implication:** investing in automatic diagnosis for critical systems is worth it.

### 3. Documentation is 30% of the project
I underestimated the time for docs. The working pipeline was ~70% of the work, professional docs were the remaining 30%. But without docs, the pipeline isn't transferable.

**Implication:** the project plan must include time dedicated to docs.

## What I Would Change

### 1. I would start with better observability
I built dashboards at the end. I would start with basic dashboards from day 1, even empty. They would have detected problems faster.

### 2. Test the rollback earlier
I didn't test the rollback until it happened by accident. Now there's a monthly game day scheduled where we simulate rollbacks.

### 3. CLAUDE.md from the start
I iterated the bot's conventions ad-hoc for 2 weeks before writing a formal CLAUDE.md. It would have been faster to make a CLAUDE.md on day 1 and iterate on it.

## Recommendations for the Next Team

If you're going to build a similar pipeline:

1. **Read this doc first** — it saves you the errors above
2. **Incremental iteration** — a working Phase 1 > Phase 1-4 half-done
3. **Metrics from day 1** — without data you can't optimize
4. **An active CLAUDE.md** — update it with every false positive
5. **Monthly game days** — practice controlled failures

## Things I Did NOT Try But Would Try

1. **Canary deployments** — currently we go 0→100% in production
2. **Multi-region deployments** — a single cluster for now
3. **Self-healing pipeline** — auto-fix of known issues before failing
4. **A/B testing of the pipeline** — testing prompt variations

These would be the next optimizations with a likely benefit.

Concrete Optimizations

Listed by cost/benefit:

Optimization 1: Pip cache (impact: high, effort: low)

- uses: actions/setup-python@v5
  with:
    python-version: '3.11'
    cache: 'pip'  # ← reduces ~25s per job

Benefit: ~75s saved per run across 3 parallel jobs = $0.01/run × 200 deploys = ~$2/month.

Optimization 2: Switch to Haiku where applicable (impact: high, effort: low)

# Code review: Haiku is enough
env:
  CLAUDE_MODEL: 'claude-haiku-4-5'

Benefit: 5x reduction in the API cost for that job.

Optimization 3: Skip draft PRs (impact: medium, effort: low)

jobs:
  review:
    if: github.event.pull_request.draft == false

Benefit: saves runs on PRs under construction.

Optimization 4: Custom Docker image with preinstalled deps (impact: medium, effort: medium)

FROM python:3.11-slim
RUN pip install --no-cache-dir anthropic>=0.39.0 requests

Benefit: eliminates pip install in each job. ~30s/job × 3 jobs × 200 deploys = ~$5/month.

Optimization 5: Concurrency cancel-in-progress (impact: high, effort: trivial)

concurrency:
  group: pr-${{ github.event.pull_request.number }}
  cancel-in-progress: true

Benefit: PRs with many pushes only complete the last one. Variable savings, typically 30-50% of runs canceled.

Common Pitfalls in Documentation

1. Docs that describe but don't guide

Symptom: "The pipeline has 4 phases" — but it doesn't explain what to do in each situation.

Why it happens: Documenting whats vs hows.

How to fix it: Each doc oriented to an action: "to do X, do Y, Z, W".

2. A generic runbook

Symptom: "If X fails, investigate".

Why it happens: A runbook written without thinking about who uses it under pressure.

How to fix it: Concrete steps, exact commands, links to dashboards. What the on-call needs while panicking.

3. Metrics without context

Symptom: "The pipeline takes 30 min" — but you don't know if that's good or bad.

Why it happens: Data without a baseline or comparison.

How to fix it: Show trends, compare with before, justify trade-offs. Calculated ROI, not just numbers.

4. Positive lessons learned only

Symptom: "Everything went well".

Why it happens: Fear of documenting your own mistakes.

How to fix it: Especially document the mistakes. It's where the team learns. A culture of blameless postmortems.

5. Docs that get out of date

Symptom: Accurate docs at the start, lies 3 months later.

Why it happens: There's no maintenance process.

How to fix it: Docs in the repo, versioned with the code. A PR that modifies the pipeline must update the docs as part of the change.

Diagnosis

Question 1: Does your DEPLOYMENT.md describe the pipeline or explain how to use it?

Describing = "it has 4 phases". Explaining = "to approve a deploy, do X". The second is more useful.

Question 2: Does your RUNBOOK have concrete steps or generic advice?

Under pressure, the on-call needs copy-paste commands, not advice.

Question 3: Did you calculate the pipeline's ROI with numbers?

Without numbers, "the pipeline saves time" is opinion. With numbers, it's a business case.

Question 4: Did you document your mistakes and corrections?

Only successes = future people repeat mistakes. Documented mistakes = collective learning.

Question 5: Did you apply at least 3 measurable optimizations?

Implementing without optimizing is half-done work. The optimizations above give visible savings.

Final Exercises

Exercise 1: Create the 4 documents (Hard)

For your pipeline, write the 4 documents:

  1. DEPLOYMENT.md oriented to users
  2. RUNBOOK.md oriented to on-call
  3. METRICS.md with real data (at least 1 month of pipeline)
  4. LESSONS.md honest about what worked and what didn't

Exercise 2: Apply 3 optimizations (Medium)

Choose 3 optimizations from the list, implement them, and measure the impact:

  1. Pipeline time before/after
  2. Pipeline cost before/after
  3. Document the result in METRICS.md

Exercise 3: Game day (Medium)

Schedule a monthly game day:

  1. Simulate each failure mode intentionally
  2. Verify the runbook responds correctly
  3. Document findings in LESSONS.md
  4. Improvements to the runbook if necessary

Final Summary of the Integrative Project

What You Built

END-TO-END PIPELINE:
- Phase 1: PR Review (parallel, ~5 min)
- Phase 2: Pre-Deployment (~3 min)
- Phase 3: Deployment with an approval gate (~13 min)
- Phase 4: Monitor + auto-rollback + diagnose (~10 min)

DOCUMENTATION:
- DEPLOYMENT.md (how it operates)
- RUNBOOK.md (what to do if X fails)
- METRICS.md (calculated ROI)
- LESSONS.md (learnings)

CHARACTERISTICS:
- Explicit failure paths for the 5 modes
- Pipeline health dashboards
- 3 alerting tiers
- Optimizations with measured impact

What It Demonstrates

This project in your portfolio demonstrates:

  1. Architectural design — you thought of the pipeline as a system, not as steps
  2. Trade-off awareness — decisions documented with a reason
  3. Operational excellence — runbook + dashboards + game days
  4. Business sense — calculated ROI, optimized costs
  5. Continuous improvement — lessons learned + action items

What Comes After

This is the last module of Guide #10. The natural next step in the Agentic Development path is Guide #11 (Security for AI-Generated Code):

  • This guide covered automation with Claude Code in CI/CD
  • Guide 11 covers the security of AI-generated code — both interactive and in pipelines

The transition is natural: you learned to automate, now you learn to do it securely.

Additional Resources

  1. The DevOps Handbook — The classic reference
  2. Site Reliability Engineering: Postmortem Culture — How to learn from incidents
  3. GitHub Engineering Blog — How GitHub builds its CI/CD
  4. DORA Research — Metrics that matter in DevOps
  5. Continuous Delivery: Reliable Software Releases — Jez Humble
  6. Anthropic API Best Practices — Applicable to CI/CD
  7. The Twelve-Factor App — Principles for deployable apps

Closing the Integrative Project

You built a production-ready end-to-end pipeline with Claude Code integrated. You have:

  • ✅ A working pipeline from PR to production
  • ✅ Failure paths covered for the 5 modes
  • ✅ Complete operational documentation
  • ✅ Real metrics and calculated ROI
  • ✅ Lessons learned for future maintainers
  • ✅ A prioritized optimization plan

This project is portfolio-worthy. Showing it demonstrates senior-level CI/CD with AI. And it's transferable — the pattern applies to any future professional project.

Congratulations on completing Guide #10.

← PreviousFailure Paths and Observability