Module 5: Security Scanning and Smart Rollback
Security Scanning with Claude Code in the Pipeline
Security Scanning with Claude Code in the Pipeline
Overview
This capsule teaches you to configure Claude Code as a pre-merge security gate: an automatic analysis that detects real vulnerability patterns in each PR's diff. It's not a generic security linter — it's analysis with context that understands what the code changes and predicts the impact on security.
The key difference from capsule 02 of Module 2 (general code review): here the focus is narrowly security. SQL injection, XSS, hardcoded secrets, missing validation on external inputs, dependencies with known CVEs. Recognizable patterns that the model can identify with high precision when given the right prompt.
By the end, you'll have a workflow that scans every PR looking for specific vulnerabilities, applies gradual policies (critical blocks, high warning, medium informs), and produces actionable reports that distinguish signal from noise.
Why Specific Security Scanning (vs General Code Review)
GENERAL CODE REVIEW (Module 2):
→ Detects a wide variety: bugs, naming, refactoring, etc.
→ Mixed severity: 1 critical issue among 20 suggestions
→ Prompt designed for broad capture
→ False positives tolerable (suggest-only)
SECURITY SCAN (this module):
→ Detects one specific category: vulnerabilities
→ Homogeneous severity: if it flags it, it matters
→ Prompt designed for high precision in one category
→ False positives costly (can block the merge)
Why separate it: a prompt that asks "do a code review" is good at detecting many things but mediocre at detecting specific vulnerabilities. A security-focused prompt detects more SQL injections — but it misses refactoring opportunities.
The rule: have two separate jobs in the pipeline. The code review one (Module 2) and the security scan one (this module). Each with its optimized prompt.
The Catalog of Patterns to Detect
1. INJECTION
- SQL injection (string concat in queries)
- Command injection (subprocess with unsanitized input)
- LDAP/NoSQL injection
- Template injection (SSTI)
2. XSS / OUTPUT ENCODING
- HTML render without escaping
- innerHTML with user data
- JSON injection in responses
3. HARDCODED SECRETS
- API keys in code
- Passwords in config files
- JWT secrets in string literals
4. MISSING VALIDATION
- External inputs without type/range validation
- Path traversal (file paths without normalize)
- SSRF (unvalidated external URLs)
5. AUTH/AUTHZ
- Sensitive endpoints without an auth check
- Authorization bypassed (missing role check)
- Insecure direct object reference (IDOR)
6. CRYPTO
- Deprecated algorithms (MD5, SHA1 for passwords)
- Non-cryptographically-secure random
- Comparing secrets without timing-safe equals
7. DEPENDENCIES
- Versions with known CVEs
- Abandoned dependencies
8. SENSITIVE ERROR HANDLING
- Stack traces exposed to clients
- Error messages that leak internal logic
Each category has recognizable patterns the model learns to identify.
The Specialized Prompt
"""scripts/security_scan.py — security scan with Claude Code."""
import json
import os
import re
import sys
from pathlib import Path
from anthropic import Anthropic
SECURITY_SCAN_PROMPT = """You are a security auditor analyzing a code diff.
Your only objective is to identify real security vulnerabilities.
CATEGORIES TO DETECT:
1. INJECTION
- SQL: string concatenation in SQL queries
- Command: subprocess/exec with unsanitized input
- Template: f-strings or templates with user input
- NoSQL: queries without parameterization
2. XSS
- innerHTML, dangerouslySetInnerHTML, document.write with input
- Server-side render without escaping (Jinja2 without autoescape, etc.)
3. HARDCODED SECRETS
- Patterns: sk-..., ghp-..., AKIA..., -----BEGIN PRIVATE KEY-----
- API keys, passwords, tokens in string literals
- JWT secrets as constants
4. MISSING VALIDATION
- Path traversal: paths without normalize() or validation against ../
- SSRF: external URLs without validation against an allowlist
- External inputs without type, range, format validation
5. AUTH/AUTHZ
- Sensitive endpoints without an auth decorator/middleware
- Missing role check before a privileged action
- IDOR: access to resources without verifying ownership
6. CRYPTO
- MD5, SHA1 for passwords (should be bcrypt, argon2, scrypt)
- random.random() for tokens (should be the secrets module)
- Comparing secrets with == (should be hmac.compare_digest or similar)
7. DEPENDENCIES
- Specific versions with known CVEs
- Known abandoned packages
8. SENSITIVE ERROR HANDLING
- returning stack traces to clients
- Logging of sensitive information
SEVERITY RULES:
- CRITICAL: confirmed bug or clear exposure (SQL injection, hardcoded secret, missing auth)
- HIGH: dangerous pattern but requires additional context to confirm
- MEDIUM: security debt (deprecated crypto, missing validation)
- LOW: a good practice that's missing but not an exploitable vulnerability
REPORTING RULES:
- Only flag if you SEE the pattern in the diff
- Do NOT infer vulnerabilities from file/function names
- Do NOT flag "there could be a problem" — flag "there's this specific problem"
- If in doubt, mark it as medium-severity, not critical
OUTPUT: JSON with this structure:
{
"findings": [
{
"category": "injection|xss|secrets|validation|auth|crypto|dependencies|error_handling",
"severity": "critical|high|medium|low",
"path": "path/to/file",
"line": 42,
"title": "SQL injection in the users query",
"description": "Technical description of the problem",
"evidence": "The code in question (snippet)",
"remediation": "How to fix it"
}
]
}
If there are no findings, return {"findings": []}.
DIFF TO ANALYZE:{diff}
"""
def run_security_scan(diff: str) -> list[dict]:
"""Run the scan and return findings."""
client = Anthropic()
response = client.messages.create(
model=os.environ.get("CLAUDE_MODEL", "claude-sonnet-5"), # Sonnet for precision
max_tokens=4000,
messages=[
{"role": "user", "content": SECURITY_SCAN_PROMPT.format(diff=diff[:30000])},
],
)
text = response.content[0].text.strip()
text = re.sub(r"^```(?:json)?\n?", "", text)
text = re.sub(r"\n?```$", "", text)
try:
data = json.loads(text)
return data.get("findings", [])
except json.JSONDecodeError as e:
print(f"WARNING: response is not valid JSON: {e}", file=sys.stderr)
return []
def main() -> int:
diff_file = Path("filtered_diff.txt")
if not diff_file.exists() or not diff_file.read_text().strip():
print("No diff to scan.")
Path("security_findings.json").write_text(json.dumps({"findings": []}))
return 0
diff = diff_file.read_text()
findings = run_security_scan(diff)
# Aggregate by severity
by_severity = {"critical": 0, "high": 0, "medium": 0, "low": 0}
for f in findings:
by_severity[f.get("severity", "low")] += 1
print(f"\nSecurity scan complete:")
print(f" Critical: {by_severity['critical']}")
print(f" High: {by_severity['high']}")
print(f" Medium: {by_severity['medium']}")
print(f" Low: {by_severity['low']}")
# Detail of critical/high
for f in findings:
if f.get("severity") in ["critical", "high"]:
print(f"\n {f['severity'].upper()}: {f['title']}")
print(f" {f['path']}:{f['line']}")
print(f" {f['description']}")
# Save the report
report = {
"findings": findings,
"summary": by_severity,
}
Path("security_findings.json").write_text(json.dumps(report, indent=2))
# Exit code based on severity
return apply_security_policy(by_severity)
def apply_security_policy(by_severity: dict) -> int:
"""Decide the exit code based on the severity policy."""
# Gradual policy:
# - Critical: blocks (exit 1)
# - High: visible warning but doesn't block (exit 0)
# - Medium/Low: only informs (exit 0)
if by_severity["critical"] > 0:
print("\n❌ BLOCK: critical findings detected")
return 1
if by_severity["high"] > 0:
print("\n⚠️ WARNING: high severity findings — review before merging")
return 0
if by_severity["medium"] > 0 or by_severity["low"] > 0:
print("\n💡 INFO: minor findings detected")
return 0
print("\n✅ No security findings")
return 0
if __name__ == "__main__":
sys.exit(main())
The Workflow
# .github/workflows/security-scan.yml
name: Security Scan
on:
pull_request:
types: [opened, synchronize]
permissions:
contents: read
pull-requests: write
security-events: write
jobs:
security-scan:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0
- uses: actions/setup-python@v5
with:
python-version: '3.11'
cache: 'pip'
- run: pip install anthropic requests
- name: Extract filtered diff
run: |
git diff origin/${{ github.base_ref }}...HEAD \
--diff-filter=ACMR \
-- '*.py' '*.js' '*.ts' '*.tsx' '*.go' '*.rb' '*.java' \
> filtered_diff.txt
- name: Run security scan
env:
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
run: python scripts/security_scan.py
- name: Post findings to PR
if: always()
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
GITHUB_REPOSITORY: ${{ github.repository }}
PR_NUMBER: ${{ github.event.pull_request.number }}
run: python scripts/post_security_findings.py
- uses: actions/upload-artifact@v4
if: always()
with:
name: security-findings
path: security_findings.jsonKey notes:
if: always()on the post step → publishes findings even if the scan reports critical (blocks exit 1)--diff-filter=ACMR→ only Added/Copied/Modified/Renamed (not Deleted)- Extension filter to focus on source code
Publishing Findings to the PR
"""scripts/post_security_findings.py — post findings to the PR."""
import json
import os
import sys
from pathlib import Path
import requests
SEVERITY_EMOJI = {
"critical": "🚨",
"high": "⚠️",
"medium": "💡",
"low": "ℹ️",
}
CATEGORY_EMOJI = {
"injection": "💉",
"xss": "🌐",
"secrets": "🔑",
"validation": "🛡️",
"auth": "🔐",
"crypto": "🔒",
"dependencies": "📦",
"error_handling": "🪲",
}
def build_summary(findings: list, summary: dict) -> str:
"""Build the PR comment."""
if not findings:
return """## 🔒 Security Scan
✅ **No vulnerabilities detected in this PR.**
---
*Automated analysis with Claude Code*
"""
body = f"""## 🔒 Security Scan
| Severity | Count |
|----------|-------|
| 🚨 Critical | {summary.get('critical', 0)} |
| ⚠️ High | {summary.get('high', 0)} |
| 💡 Medium | {summary.get('medium', 0)} |
| ℹ️ Low | {summary.get('low', 0)} |
"""
if summary.get('critical', 0) > 0:
body += "**🚨 This PR has critical findings. The merge is blocked until they're resolved.**\n\n"
# Detail by severity (critical and high first)
for severity in ["critical", "high", "medium", "low"]:
sev_findings = [f for f in findings if f.get("severity") == severity]
if not sev_findings:
continue
body += f"### {SEVERITY_EMOJI[severity]} {severity.title()}\n\n"
for f in sev_findings:
cat_emoji = CATEGORY_EMOJI.get(f.get("category", ""), "🔍")
body += f"#### {cat_emoji} {f['title']}\n"
body += f"**Location:** `{f['path']}:{f['line']}`\n\n"
body += f"**Description:** {f['description']}\n\n"
if f.get("evidence"):
body += f"**Evidence:**\n```\n{f['evidence']}\n```\n\n"
body += f"**Remediation:** {f['remediation']}\n\n---\n\n"
body += "*Automated analysis with Claude Code. Marker: `<!-- security-scan-bot -->`*\n"
return body
def find_existing_comment(comments: list, marker: str = "security-scan-bot") -> dict | None:
"""Find an existing bot comment."""
for c in comments:
if marker in c.get("body", ""):
return c
return None
def main() -> int:
findings_file = Path("security_findings.json")
if not findings_file.exists():
print("ERROR: security_findings.json not found.", file=sys.stderr)
return 1
data = json.loads(findings_file.read_text())
findings = data.get("findings", [])
summary = data.get("summary", {})
body = build_summary(findings, summary)
# Post or update the comment
repo = os.environ["GITHUB_REPOSITORY"]
pr_number = os.environ["PR_NUMBER"]
token = os.environ["GITHUB_TOKEN"]
headers = {
"Authorization": f"Bearer {token}",
"Accept": "application/vnd.github+json",
}
list_url = f"https://api.github.com/repos/{repo}/issues/{pr_number}/comments"
existing = requests.get(list_url, headers=headers).json()
bot_comment = find_existing_comment(existing)
if bot_comment:
# Update
update_url = f"https://api.github.com/repos/{repo}/issues/comments/{bot_comment['id']}"
r = requests.patch(update_url, headers=headers, json={"body": body})
else:
# Create
r = requests.post(list_url, headers=headers, json={"body": body})
if r.status_code in [200, 201]:
print(f"Security findings published to the PR")
return 0
print(f"ERROR posting comment: {r.status_code}", file=sys.stderr)
return 1
if __name__ == "__main__":
sys.exit(main())Gradual Policies
The most important rule: not everything blocks. Gradual policies based on severity:
SECURITY_POLICY = {
# Severity → action
"critical": "block", # blocks the merge
"high": "warn", # visible warning, doesn't block
"medium": "inform", # only informs
"low": "inform", # only informs
}
CATEGORY_OVERRIDES = {
# For certain categories, the effective severity is higher
"secrets": "block", # secrets always block
"auth": "block_if_high", # missing auth is critical
}Why it matters: if everything blocks, the team disables the bot or uses overrides constantly. Gradual policies keep the bot useful without being an obstruction.
Policy configuration
- name: Run security scan with policy
env:
SECURITY_POLICY_CRITICAL: "block"
SECURITY_POLICY_HIGH: "warn"
SECURITY_POLICY_MEDIUM: "inform"
run: python scripts/security_scan.pyOverride Mechanism
Even with gradual policies, you'll have occasional false positives. The override mechanism allows merging when the team confirms the bot was wrong:
- name: Check override label
id: override
run: |
if [[ "${{ contains(github.event.pull_request.labels.*.name, 'security-scan-override') }}" == "true" ]]; then
echo "override=true" >> $GITHUB_OUTPUT
fi
- name: Run security scan
if: steps.override.outputs.override != 'true'
run: python scripts/security_scan.pyWith the security-scan-override label applied to the PR, the scan is skipped. Important: this label should require approval from a security lead, not be applied freely.
Override auditing
- name: Log override usage
if: steps.override.outputs.override == 'true'
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: |
# Post a comment to the PR explaining the override
# Notify the security team via a webhook
python scripts/log_override.pyAny use of an override generates an auditable record: who applied it, when, justification (ideally in the comment).
Comparison with Other Tools
RULES-BASED SCANNERS (Snyk Code, Semgrep, Bandit):
✅ High speed
✅ Established and tested rules
✅ SARIF integration (standard)
❌ False positives on contextual patterns
❌ Don't understand the code's logic
❌ Limited to known patterns
CLAUDE CODE AS A SCANNER (this module):
✅ Understands context and semantics
✅ Detects patterns static rules don't
✅ Explains the "why" of the finding
❌ Slower than static scanners
❌ More expensive per scan
❌ Can have inconsistency between runs (mitigable)
It's not one or the other. A better pattern: use both, with static scanners as the first line (fast, cheap) and Claude Code as the second line (deep, contextual). Capsule 03 develops this integration.
Common Pitfalls
Error 1: Same prompt for general code review and security scan
Symptom: The bot detects naming issues mixed with vulnerabilities. The security signal is diluted.
Why it happens: Reusing the Module 2 bot expecting it to work well for security.
How to fix it: A specific prompt for security like the one in this capsule. Only vulnerability categories, homogeneous severity.
Error 2: Blocking everything the bot flags
Symptom: The team is frustrated because PRs with no real issues get blocked. They start using overrides constantly.
Why it happens: Without gradual policies, everything critical/high/medium blocks.
How to fix it: Only critical blocks. High = visible warning. Medium/low = report. Calibrate based on the team's false positive rate.
Error 3: Not having an override mechanism
Symptom: A false positive blocks an urgent merge. There's no way to skip it.
Why it happens: A "strict" design with no escape hatch.
How to fix it: A security-scan-override label with auditing. Allows merging when necessary, leaves a record.
Error 4: Reusing the complete diff (not filtered)
Symptom: The scan takes a long time and/or loses precision due to a full context.
Why it happens: Passing the whole diff including non-code files (markdown, yaml, etc.).
How to fix it: Filter by source code extensions before the scan. Capsule 02 of Module 2 covers the filtering.
Error 5: No notification of critical findings to the security team
Symptom: A critical finding blocks the merge, the developer investigates it alone, possibly deciding to override incorrectly.
Why it happens: The bot publishes the finding but only the PR author sees it.
How to fix it: Critical findings → notification to the #security channel on Slack. The override decision shouldn't be the PR author's alone.
Diagnosis
Question 1: Is your security scan prompt separate from the code review prompt?
If it's the same, the precision on security is suboptimal. A dedicated prompt gives better findings.
Question 2: Do you apply gradual policies (critical blocks, high warning) or does everything block?
Everything blocks = frustrated team. Gradual = useful without being an obstruction.
Question 3: Do you have an auditable override mechanism?
Without an override, false positives generate a crisis. With an override without auditing, the bot loses effectiveness.
Question 4: Do you combine Claude Code with static scanners (Snyk, Bandit) or use only one?
Both = better coverage. Only one = gaps.
Question 5: Do critical findings notify the security team or only the PR author?
Only the author = a potentially biased decision. With notification to the security team = better governance.
Exercises
Exercise 1: Specialized prompt (Easy)
Take the code review prompt from Module 2 and derive a security-specific prompt. Compare them: what changes?
Exercise 2: Implement the scan with policies (Medium)
Implement:
- A Python script with the security prompt
- Gradual policies (critical → exit 1, high → warning, medium/low → info)
- A JSON report
Test it with a PR that has an obvious SQL injection and verify it detects and blocks.
Exercise 3: Override with auditing (Hard)
Implement the flow:
- The
security-scan-overridelabel skips the scan - Any use of the label generates a comment on the PR explaining it
- A Slack notification to the security team
- A log of overrides for later auditing
Summary
- A security scan separate from general code review — a prompt focused on specific vulnerabilities
- 8 standard categories: injection, XSS, secrets, validation, auth, crypto, dependencies, error_handling
- Gradual policies: critical blocks, high warning, medium/low informs
- An audited override mechanism keeps the bot useful without being an obstruction
- Combining with static scanners (Snyk, Bandit) for better coverage — capsule 03
- Notifying the security team on critical findings, not just the PR author
Next capsule: 03 — Integration with existing security tools. Your scan with Claude Code is good but not unique. You'll learn to combine it with Snyk, Dependabot, audit tools — how to orchestrate all the tools into a coherent pipeline.
Additional Resources
- OWASP Top 10 — The 10 most common categories of web vulnerabilities
- CWE Top 25 — Common Weakness Enumeration
- SANS Top 25 — A complementary list
- Bandit (Python) — Static security linter
- Semgrep — Multi-language security rules
- Anthropic API best practices for security — Applicable patterns