Parsing Output and Generating Artifacts

Overview

Your workflow already runs Claude Code automatically and securely (capsules 02-03). But the analysis result still lives in the GitHub Actions logs — and developers don't read logs. If the finding doesn't show up on the PR page, it doesn't exist operationally.

This capsule solves the problem: how to turn Claude Code's raw output into something visible and actionable inside the team's workflow. You'll learn three output formats (PR comments, annotations, artifacts) and how to choose which to use in each case.

By the end, you'll be able to publish the analysis result as a comment on the PR, generate annotations that show up as check warnings, and save detailed reports as downloadable artifacts. The workflow stops being "a thing that runs in the background" and becomes "a thing the team sees and uses."

The Three Output Formats

1. PR COMMENT (general comment on the PR)
   → Visibility: high (everyone sees it when opening the PR)
   → Best for: summaries and general findings
   → Limitation: 65,536 characters maximum
   → API: POST /repos/{owner}/{repo}/issues/{pr}/comments

2. ANNOTATIONS (warnings/errors on specific lines)
   → Visibility: medium (they show up in the "Files changed" tab
     and as a check warning)
   → Best for: localized issues (this line has a problem)
   → Limitation: maximum 50 per workflow run
   → Output: ::warning file=X,line=N::message

3. ARTIFACTS (downloadable files)
   → Visibility: low (you have to go to the Actions tab and download)
   → Best for: detailed reports, data for later analysis
   → Limitation: 500 MB per artifact, 10 GB per workflow run
   → Action: actions/upload-artifact

General rule: combine all three. Executive summary as a PR comment, specific issues as annotations, complete report as an artifact.

Format 1: PR Comments

The PR comment is the most visible format. When someone opens the PR page, they see the comments automatically. It's the right place for the analysis summary.

Implementation with the GitHub API

Modify the Python script so that, after generating the analysis, it publishes it as a comment:

"""PR analysis + publishing as a comment."""
import os
import sys
from pathlib import Path
from anthropic import Anthropic
import requests

# 1. Generate analysis
client = Anthropic()
diff_text = Path("pr_diff.txt").read_text()

response = client.messages.create(
    model="claude-haiku-4-5",
    max_tokens=2000,
    messages=[
        {
            "role": "user",
            "content": (
                "Analyze this diff and generate a markdown summary with:\n"
                "## Summary\n[What changes]\n"
                "## Findings\n[Possible issues]\n"
                "## Suggestions\n[Concrete improvements]\n\n"
                f"Diff:\n```\n{diff_text}\n```"
            ),
        }
    ],
)

analysis = response.content[0].text

# 2. Publish as a PR comment
github_token = os.environ["GITHUB_TOKEN"]
repo = os.environ["GITHUB_REPOSITORY"]  # "owner/repo"
pr_number = os.environ["PR_NUMBER"]

url = f"https://api.github.com/repos/{repo}/issues/{pr_number}/comments"
headers = {
    "Authorization": f"Bearer {github_token}",
    "Accept": "application/vnd.github+json",
    "X-GitHub-Api-Version": "2022-11-28",
}

body = f"## 🤖 Claude Code Analysis\n\n{analysis}\n\n---\n*Generated automatically. Tokens: {response.usage.input_tokens} in / {response.usage.output_tokens} out*"

result = requests.post(url, headers=headers, json={"body": body})

if result.status_code != 201:
    print(f"ERROR publishing comment: {result.status_code} {result.text}", file=sys.stderr)
    sys.exit(1)

print(f"Comment published: {result.json()['html_url']}")

Updated YAML

- name: Run analysis and post comment
  env:
    ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
    GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
    GITHUB_REPOSITORY: ${{ github.repository }}
    PR_NUMBER: ${{ github.event.pull_request.number }}
  run: |
    pip install requests
    python .github/scripts/analyze_pr.py

Key note: secrets.GITHUB_TOKEN is an automatic secret that GitHub generates for each workflow run. You don't configure it — it's always available. It has permissions limited to the workflow's repo.

GITHUB_TOKEN permissions

By default, GITHUB_TOKEN has read permissions. To write comments, you have to expand them:

permissions:
  pull-requests: write
  contents: read

jobs:
  analyze:
    runs-on: ubuntu-latest
    # ... rest

pull-requests: write allows creating comments. contents: read is necessary for checkout. Always restrict to the minimum needed — don't use write-all.

Format 2: Annotations

Annotations show up as visible warnings or errors in the PR's "Files changed" tab, on the specific line of the file. They're ideal for flagging localized issues.

Syntax

GitHub Actions parses certain commands in stdout and turns them into annotations:

::warning file={path},line={line}::{message}
::error file={path},line={line}::{message}
::notice file={path},line={line}::{message}

Example: Generate annotations from the analysis

Modify the script so that, in addition to the comment, it emits annotations for each specific issue:

import re
import json

# ... (previous analysis code)

# We ask Claude to return structured issues
response = client.messages.create(
    model="claude-haiku-4-5",
    max_tokens=2000,
    messages=[
        {
            "role": "user",
            "content": (
                "Analyze this diff. Return a JSON with the issues found.\n"
                'Format: {"issues": [{"file": "...", "line": N, "severity": "warning|error", "message": "..."}]}\n'
                "Return only the JSON, with no additional text.\n\n"
                f"Diff:\n```\n{diff_text}\n```"
            ),
        }
    ],
)

# Parse the JSON from the response
text = response.content[0].text.strip()
# Clean up code fences if it added them
text = re.sub(r"^```json\n?", "", text)
text = re.sub(r"\n?```$", "", text)

try:
    issues_data = json.loads(text)
except json.JSONDecodeError:
    print(f"WARNING: response is not valid JSON:\n{text}", file=sys.stderr)
    issues_data = {"issues": []}

# Emit annotations
for issue in issues_data.get("issues", []):
    severity = issue.get("severity", "warning")
    file_path = issue.get("file", "")
    line = issue.get("line", 1)
    message = issue.get("message", "").replace("\n", " ")
    print(f"::{severity} file={file_path},line={line}::{message}")

How it looks in the PR:

In the "Files changed" tab, next to the affected file, a yellow banner (warning) or red one (error) shows up with the message. The workflow check is marked with a visible warning.

Annotation limitations

• Maximum 10 annotations per command and 50 per workflow run total. If you have more, the extras are silently ignored.
• They don't support markdown. Plain text only.
• They don't persist between runs. Each run generates its own annotations; they don't accumulate.

For issues above the 50 limit, combine: the 10 most critical as annotations, the rest in the PR comment.

Format 3: Artifacts

Artifacts are arbitrary files that get saved associated with the workflow run. Useful for:

• Detailed reports (HTML, PDF, large JSON)
• Structured logs for later analysis
• Intermediate outputs the team can download and review

Implementation

Generate the report as a file and upload it:

# In the Python script:
import json

report = {
    "pr_number": int(os.environ["PR_NUMBER"]),
    "analysis": analysis,
    "issues": issues_data.get("issues", []),
    "tokens_used": {
        "input": response.usage.input_tokens,
        "output": response.usage.output_tokens,
    },
}

Path("analysis_report.json").write_text(json.dumps(report, indent=2))

# In the workflow:
- name: Run analysis
  env:
    ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
    GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
    GITHUB_REPOSITORY: ${{ github.repository }}
    PR_NUMBER: ${{ github.event.pull_request.number }}
  run: python .github/scripts/analyze_pr.py

- name: Upload analysis report
  uses: actions/upload-artifact@v4
  with:
    name: analysis-report-pr-${{ github.event.pull_request.number }}
    path: analysis_report.json
    retention-days: 30

Result: in the workflow run's "Actions" tab, an "Artifacts" section shows up with a downloadable file. The team can download it, open it, run statistical analysis, etc.

When to use artifacts

• ✅ Reports that don't fit in a PR comment (>65K characters)
• ✅ Structured data for offline analysis (JSON, CSV)
• ✅ Visual outputs (HTML reports, charts)
• ❌ NOT for findings the team should see immediately (those go as a comment)

Combining the Three Formats

The professional pattern combines all three based on severity and volume:

ANALYSIS GENERATED BY CLAUDE CODE
        │
        ├─→ Executive summary (3-5 paragraphs)
        │   → PR Comment (everyone sees it)
        │
        ├─→ Localized issues (line X has problem Y)
        │   → Annotations (10 most critical, in files changed)
        │
        └─→ Complete report (all the details, raw data)
            → Downloadable JSON artifact

Example of an integrated script

# Summary → PR comment
post_pr_comment(summary_markdown)

# Top 10 issues → annotations
for issue in sorted(issues, key=lambda i: i["severity"])[:10]:
    print(f"::{issue['severity']} file={issue['file']},line={issue['line']}::{issue['message']}")

# Complete report → artifact
Path("analysis_report.json").write_text(json.dumps({
    "summary": summary_markdown,
    "issues": issues,
    "metadata": {...}
}, indent=2))

The team:

• Opens the PR → sees the comment with the summary
• Goes to "Files changed" → sees the inline warnings
• If they need the detail → downloads the artifact

Each developer finds what they need at the depth level they require.

Common Pitfalls

Error 1: Duplicate comments on every push to the PR

Symptom: Each synchronize (push to the PR) generates a new comment. After 5 pushes, the PR has 5 bot comments.

Why it happens: The script always creates a new comment, it doesn't reuse the previous one.

How to fix it: Before creating, check whether a bot comment already exists and update it:

# Find existing comment
list_url = f"https://api.github.com/repos/{repo}/issues/{pr_number}/comments"
existing = requests.get(list_url, headers=headers).json()

bot_comments = [c for c in existing if "🤖 Claude Code Analysis" in c["body"]]

if bot_comments:
    # Update the existing one
    update_url = f"https://api.github.com/repos/{repo}/issues/comments/{bot_comments[0]['id']}"
    requests.patch(update_url, headers=headers, json={"body": body})
else:
    # Create new
    requests.post(list_url, headers=headers, json={"body": body})

Error 2: Annotations with incorrect path or line

Symptom: The annotations show up but aren't associated with any visible line in "Files changed".

Why it happens: The path must be relative to the repo root (not absolute). The line must be a number of the file modified in the PR (not of an untouched file).

How to fix it: Validate paths before emitting. If the model suggests a file that isn't in the diff, ignore that annotation or fold it into the general comment.

Error 3: Artifacts without retention-days

Symptom: The artifacts take up space and accumulate. The org's quota runs out.

Why it happens: By default, artifacts are retained for 90 days. For PR analysis, that's excessive.

How to fix it: Configure retention-days: 30 (or less) explicitly. For ephemeral artifacts (temporary debug), use retention-days: 1.

Error 4: PR comment with broken HTML/markdown

Symptom: The comment shows the code of the markdown block instead of rendering it.

Why it happens: The model sometimes returns markdown with an incorrect triple backtick, or special characters that GitHub interprets literally.

How to fix it: Validate the markdown before publishing. If the comment is very long, consider splitting it into several or leaving only the summary and putting the details in an artifact.

Error 5: Missing GITHUB_TOKEN permissions

Symptom: The script fails with 403 Forbidden when trying to create the comment.

Why it happens: The YAML doesn't declare permissions: pull-requests: write. GitHub uses restrictive default permissions.

How to fix it: Add the explicit permissions: block to the workflow or the job. Minimum needed: pull-requests: write, contents: read.

Diagnosis

Exercises

Exercise 1: Implement a PR comment (Easy)

Take the script from capsule 02 and add publishing as a PR comment using the API. Verify it shows up in your test PR.

Exercise 2: Comment deduplication (Medium)

Modify the script so it updates the existing comment instead of creating a new one each time. Use a unique marker in the body (e.g. <!-- claude-code-bot -->) to identify the bot's comment.

MARKER = "<!-- claude-code-bot -->"
body_with_marker = f"{MARKER}\n\n## 🤖 Analysis\n\n{analysis}"

# Find
existing = requests.get(list_url, headers=headers).json()
bot_comments = [c for c in existing if MARKER in c["body"]]

if bot_comments:
    update_url = f"https://api.github.com/repos/{repo}/issues/comments/{bot_comments[0]['id']}"
    requests.patch(update_url, headers=headers, json={"body": body_with_marker})
else:
    requests.post(list_url, headers=headers, json={"body": body_with_marker})

Exercise 3: Combine the three formats (Hard)

Implement a script that combines the three formats: PR comment with a summary, annotations for the top 10 issues, artifact with a complete JSON report. Verify all three in a test PR.

Summary

• Three formats of output: PR comments (high visibility), annotations (localized issues), artifacts (detailed data)
• PR comments are the most visible place — use for the executive summary
• Annotations are ideal for issues on specific lines — maximum 50 per run
• Artifacts are for detailed reports that don't fit in comments
• Combining the three is the professional pattern
• Deduplicating comments avoids spam on PRs with many updates
• GITHUB_TOKEN permissions must be declared explicitly: pull-requests: write

Next capsule: 05 — Costs and rate limiting. Your workflow is already functional, secure, and produces useful output. The last piece is missing: making it economically sustainable. You'll learn to calculate costs per run, run selectively, and when to skip unnecessary runs.

Additional Resources

1. GitHub REST API: Comments — Documentation for the comments API
2. GitHub Actions: Workflow Commands — Annotation syntax
3. actions/upload-artifact — Official artifacts action
4. GitHub Actions: Permissions — Configure GITHUB_TOKEN permissions
5. Octokit — Official GitHub API client in JS/TS
6. GitHub API rate limits — Limits to consider