Review Summary and Suggest-Only Mode

Overview

You have inline comments working (capsule 03). But a professional review includes two more things: a summary that gives general context before the inline comments, and a conscious decision about whether the bot blocks merges or only suggests. This capsule covers both.

The summary is the first thing the developer sees when opening the PR. It's where the bot says "I found 3 issues, the main ones are X and Y, overall the change is good/concerning/neutral". Without a summary, the inline comments show up in isolation — the developer doesn't know if they're critical issues or minor details.

The operating mode (suggest-only vs blocking) is an organizational decision more than a technical one. You'll learn why starting in suggest-only is almost always the right thing, how to gradually evolve toward selective blocking, and what metrics to use to make the decision based on data.

---

Anatomy of a Professional Review

┌────────────────────────────────────────────────┐
│ 🤖 Code Review (Claude Code)                  │
├────────────────────────────────────────────────┤
│                                                 │
│ ## Summary                                      │  ← summary
│ This PR refactors the payment flow to         │
│ extract the validation logic into a service.  │
│ The overall structure is solid, but there are  │
│ 2 issues worth resolving before the            │
│ merge:                                          │
│                                                 │
│ - 🚨 Potential SQL injection in auth.py:87     │
│ - ⚠️ Missing type validation in payment.py:42  │
│                                                 │
│ Other minor observations as inline             │
│ suggestions.                                    │
│                                                 │
│ ## Severity                                     │
│ - 1 critical                                    │
│ - 1 warning                                     │
│ - 3 suggestions                                 │
│                                                 │
│ ---                                             │
│ Tokens: 1,250 in / 380 out                     │
└────────────────────────────────────────────────┘

[+ inline comments on each affected file]

The summary answers three questions the developer asks when opening the PR:

1. Overall, is this change fine or does it have serious problems?
2. How many critical issues are there?
3. Is it worth reading each inline comment, or are they minor details?

---

Generating the Summary with the Same Prompt

The prompt that asks the model for the structured JSON (capsule 03) already included a summary field. Here we develop it so it has a professional shape.

Improved prompt

prompt = f"""Do a code review of this pull request diff.
Return a JSON with this structure:

{{
  "summary": {{
    "overview": "1-2 paragraphs describing what changes and the overall quality",
    "highlights": ["summary point 1", "point 2", "..."],
    "verdict": "ready_to_merge|needs_minor_changes|needs_major_changes"
  }},
  "comments": [
    {{
      "path": "...",
      "line": N,
      "severity": "critical|warning|suggestion",
      "body": "..."
    }}
  ]
}}

Rules:
- `overview` should contextualize (not just enumerate)
- `highlights` are the 2-4 most important points (not all of them)
- `verdict` is the bot's opinion about the merge
- Only `critical` for real bugs or security vulnerabilities
- Return ONLY the JSON

Diff:

{diff_text}

"""

Building the summary comment's body

def build_summary_markdown(review_data):
    """Generate the markdown for the review summary."""
    summary = review_data["summary"]
    comments = review_data["comments"]
    
    # Count by severity
    by_severity = {"critical": 0, "warning": 0, "suggestion": 0}
    for c in comments:
        by_severity[c.get("severity", "suggestion")] += 1
    
    # Verdict with emoji
    verdict_emoji = {
        "ready_to_merge": "✅",
        "needs_minor_changes": "⚠️",
        "needs_major_changes": "🚨",
    }.get(summary["verdict"], "💬")
    
    body = f"""# 🤖 Code Review (Claude Code)

## Summary

{summary["overview"]}

### Key points

"""
    for h in summary["highlights"]:
        body += f"- {h}\n"
    
    body += f"""
### Verdict

{verdict_emoji} **{summary["verdict"].replace("_", " ").title()}**

### Severity

- 🚨 Critical: {by_severity['critical']}
- ⚠️ Warning: {by_severity['warning']}
- 💡 Suggestion: {by_severity['suggestion']}

Details inline in the affected files.
"""
    return body

Publishing it as part of the review

The review API (capsule 03) has a body field for the general comment. That's where the summary goes:

review_payload = {
    "commit_id": commit_sha,
    "body": build_summary_markdown(review_data),  # ← general summary
    "event": "COMMENT",
    "comments": valid_comments,                    # ← inline comments
}

Result: a review with the summary at the start + inline comments. A single API call, complete output.

---

Suggest-Only Mode: The Default Configuration

Suggest-only mode means the bot doesn't block merges, it only comments. It's the right configuration for 99% of cases at the start.

Why start here

IF the bot blocks from day 1:
→ It generates 5 comments on the team's first PR
→ 2 are false positives (the bot didn't understand a legitimate pattern)
→ Frustrated developer: "the bot is blocking something that's fine"
→ Requests an override
→ The override generates friction
→ After 3 PRs like that, someone disables the bot
→ Result: no bot, no value

IF the bot operates in suggest-only:
→ It comments the same
→ The false positives are ignored (no blocking)
→ The real issues are seen and acted on
→ The team gains confidence gradually
→ After 4-6 weeks: the team already sees when the bot is right
→ Time to evolve to selective blocking (next section)

Configuring suggest-only

It's the default configuration of the review API. The event: "COMMENT" field requires no changes:

review_payload = {
    "commit_id": commit_sha,
    "body": summary_markdown,
    "event": "COMMENT",  # ← suggest-only: comments without blocking
    "comments": valid_comments,
}

COMMENT shows up as a review of type "Comment" on GitHub. It doesn't mark the PR as "approved" or "changes requested".

---

Evolving to Selective Blocking

After several weeks operating in suggest-only, you can have metrics on how well the bot works:

- True positive rate: X% of the issues marked as critical
  are confirmed as real bugs by human code review
- False positive rate: Y% of issues marked as critical
  are rejected (the bot was wrong)
- Precision by severity: how much it gets right on critical vs warning

When the precision on critical is >95%, you can evolve to blocking only on criticals:

# Decide the event based on the highest severity found
has_critical = any(c.get("severity") == "critical" for c in valid_comments)

review_payload = {
    "commit_id": commit_sha,
    "body": summary_markdown,
    "event": "REQUEST_CHANGES" if has_critical else "COMMENT",
    "comments": valid_comments,
}

REQUEST_CHANGES marks the PR as "changes requested". If the repo has branch protection with "require approvals" + "dismiss stale reviews", the merge is blocked until it's resolved.

Complementary branch protection

For REQUEST_CHANGES to actually block:

1. Settings → Branches → Branch protection rules → main
2. Enable "Require a pull request before merging"
3. "Require approvals: 1"
4. "Dismiss stale pull request approvals when new commits are pushed" (important)
5. "Require review from Code Owners" (optional but recommended)

Without this configuration, REQUEST_CHANGES only marks but doesn't block — anyone with write access can merge anyway.

---

Override Mechanism

Even with selective blocking, you need an override mechanism for the cases where the bot is wrong but the team wants to merge anyway.

Option 1: Human approval overrides REQUEST_CHANGES

GitHub lets a human approving review dismiss the bot's REQUEST_CHANGES. The developer:

1. Sees the bot block
2. Manually confirms it's a false positive
3. Requests a human review
4. Human reviewer approves (this overrides the bot's)
5. Merge enabled

It's the most natural mechanism — it keeps the human in the loop.

Option 2: Override label

Lets developers with authority add an override-bot-review label that skips the block:

- name: Run review
  if: |
    !contains(github.event.pull_request.labels.*.name, 'override-bot-review')
  # ...

Useful when:

• The team is small and everyone has authority
• There's urgency (a production hotfix)
• The bot has a temporary bug

Risk: the label becomes "always present". Monitor its use to detect when the bot needs adjustments.

Option 3: [skip ai] in the commit message

- name: Run review
  if: "!contains(github.event.head_commit.message, '[skip ai]')"

More casual, but it works for quick cases. Same overuse risk.

---

When NOT to Block

Some categories of findings should never block, even if the bot has high precision:

Category	Why not to block
Style / formatting	There are dedicated linters (Black, Prettier)
Naming preferences	Subjective, varies by team
Refactoring suggestions	An improvement, not a fix
Documentation gaps	Not a bug, it's debt
Performance optimizations	Premature, except in extreme cases

Only block on:

• Clear bugs (incorrect logic, null pointer, off-by-one)
• Security vulnerabilities (SQL injection, XSS, secrets)
• Undocumented breaking changes

---

Common Pitfalls

Error 1: Blocking from day 1

Symptom: Frustrated developers, the label override becomes the default, they eventually disable the bot.

Why it happens: "If the bot finds problems, we should block." But before it has confidence, the bot has false positives.

How to fix it: Always start in suggest-only. Measure precision for 4-6 weeks. Evolve to blocking only in categories where precision is >95%.

Error 2: Summary with no context

Symptom: The summary is just "I found 3 issues" without describing what changes or the general verdict.

Why it happens: The prompt doesn't ask for an explicit overview or verdict.

How to fix it: Structure the summary JSON with overview, highlights, verdict (the "Generating the Summary with the Same Prompt" section).

Error 3: REQUEST_CHANGES without branch protection

Symptom: The bot marks "changes requested" but the merge is still enabled.

Why it happens: Without "Require approvals" + "Dismiss stale" in branch protection, the REQUEST_CHANGES is decorative.

How to fix it: Configure branch protection before enabling blocking.

Error 4: Not having an override mechanism

Symptom: The bot blocks by mistake, the team can't merge, productivity drops.

Why it happens: They enabled blocking without thinking about how to unblock when the bot is wrong.

How to fix it: Document and communicate the override mechanism before enabling blocking. Human approval + emergency label.

Error 5: Metrics not measured

Symptom: The team argues about whether the bot works well without data.

Why it happens: There's no tracking of how often critical findings are confirmed or rejected.

How to fix it: Add to artifacts (capsule 04 of module 1) a JSON with each finding and its severity. Periodically review and calculate precision.

---

Diagnosis

Question 1: Does your bot operate in suggest-only or does it already block?

If it blocks without having measured precision >95% on critical, you're probably generating more friction than value.

Question 2: Does your summary have an overview, highlights, and verdict?

If it only enumerates issues, the developer has no general context. The overview+highlights+verdict structure gives the panorama at a glance.

Question 3: Does your repo have branch protection that respects REQUEST_CHANGES?

Without this, the bot blocking is theater. Configure it before enabling blocking.

Question 4: Did you document the override mechanism?

If not, the first serious false positive will generate chaos. Document it in .github/CONTRIBUTING.md or equivalent.

Question 5: Are you saving data to measure precision?

Without data, the "evolve to blocking" decision is opinion. With data, it's justifiable.

---

Exercises

Exercise 1: Implement the complete summary (Medium)

Modify the script so the review payload includes a summary with overview + highlights + verdict (not just a list of issues).

See solution

See the build_summary_markdown function in the "Building the summary comment's body" section.

Exercise 2: Configure suggest-only and verify (Easy)

Confirm that your bot operates with event: "COMMENT". Verify in a test PR that the review shows up as "Comment" type, not as "Approved" or "Changes requested".

Exercise 3: Evolution to selective blocking with override (Hard)

Implement:

1. The review uses REQUEST_CHANGES only if there are critical findings
2. There's an override mechanism with an override-bot-review label
3. Branch protection configured so REQUEST_CHANGES actually blocks

Document the override mechanism in CONTRIBUTING.md.

---

Summary

• The summary gives general context — overview, highlights, verdict
• Suggest-only mode is the right starting point in 99% of cases
• Evolution to blocking should be based on metrics: precision >95% on critical
• Branch protection is a prerequisite for REQUEST_CHANGES to block
• An override mechanism is a prerequisite for blocking (human approval, label, [skip ai])
• Only block on clear bugs and vulnerabilities — not on style or subjective matters

Next capsule: 05 — Customizing your team's conventions. You have a working bot that comments, and you operate consciously about when it blocks. What's missing is what separates a generic bot from an invaluable one: having it apply the conventions specific to your project.

---

Additional Resources

1. GitHub Branch Protection Rules — Setting up protection rules
2. GitHub Reviews API: events — Event types (COMMENT, APPROVE, REQUEST_CHANGES)
3. GitHub: Required reviews — Configure required approvals
4. GitHub Code Owners — Assign automatic reviewers
5. Anthropic: Structured outputs — Force JSON from the model
6. Google's "Code Review at Google" — Code review culture at Google (a reference for designing your team's)