Running pytest in GitHub Actions

Capsule overview

In the previous capsule you configured a basic workflow: checkout, setup Python, install deps, run pytest. But in real projects you need more: seeing clearly which tests failed, saving reports as artifacts, running only unit tests or only integration tests depending on the context, and handling environment variables like API keys in a secure environment.

This capsule goes deeper into running pytest in CI: a breakdown of the steps, handling the output and failures, publishing results as artifacts, running specific test categories (unit, integration, e2e), interpreting logs in GitHub's UI, frequent problems and using secrets for sensitive values.

By the end you'll have a robust pipeline that gives you clear visibility when something fails and that correctly handles your project's dependencies and configurations.

A breakdown of the steps for pytest in CI

The typical sequence

Step 1: Checkout      → Get the code
Step 2: Setup Python → The correct version
Step 3: Install deps → pytest, pytest-cov, the project's dependencies
Step 4: Run pytest   → Run the tests
Step 5 (optional): Upload artifacts → Save the reports

Step 1: Checkout

- name: Checkout repository
  uses: actions/checkout@v4

Without a checkout, the runner is empty. This action clones the repo into the working directory. By default it uses the event's ref (the push's commit or the PR's head).

Step 2: Setup Python

- name: Set up Python
  uses: actions/setup-python@v5
  with:
    python-version: "3.12"

Indicate the version you use in development. If your project supports several versions, you'll use a matrix in capsule 04.

Step 3: Install dependencies

- name: Install dependencies
  run: |
    python -m pip install --upgrade pip
    pip install -r requirements.txt

If you have separate test dependencies:

# requirements.txt
fastapi>=0.100.0
uvicorn

# requirements-dev.txt (or in pyproject.toml [project.optional-dependencies] dev)
pytest
pytest-cov
httpx

Then:

- name: Install dependencies
  run: |
    python -m pip install --upgrade pip
    pip install -r requirements.txt
    pip install -r requirements-dev.txt

Step 4: Run pytest

- name: Run tests
  run: pytest tests/ -v

By default pytest looks for tests in test_*.py or *_test.py. The tests/ path indicates the directory. If your structure is different (e.g. test/ or tests at the root), adjust the path.

Useful flags for CI:

An example with the recommended flags for CI:

- name: Run tests
  run: pytest tests/ -v --tb=short --no-header -q

Handling the output and failures

What happens when a test fails

pytest returns exit code 1 when some test fails. The GitHub Actions runner interprets that as a step failure. The job fails and the workflow marks the run as failed.

Seeing the failure in the logs

In GitHub's UI, open the "Run tests" step that failed. You'll see something like:

============================= test session starts ==============================
tests/test_auth.py::test_login_success PASSED                            [ 25%]
tests/test_auth.py::test_login_invalid_credentials FAILED                  [ 50%]
tests/test_items.py::test_list_items PASSED                                [ 75%]
tests/test_items.py::test_create_item PASSED                               [100%]

=================================== FAILURES ===================================
_____________ test_login_invalid_credentials _____________
    def test_login_invalid_credentials():
        response = client.post("/login", json={"user": "x", "pass": "y"})
>       assert response.status_code == 401
E       AssertionError: assert 200 == 401
E        +  where 200 = <Response [200 OK]>.status_code

tests/test_auth.py:42: AssertionError
======================== 1 failed, 3 passed in 2.15s =========================

The key line is > assert response.status_code == 401. There pytest shows you the assert that failed and the actual vs expected value.

Using -x to fail fast

If you have 500 tests and the first one that fails breaks a chain of dependencies, sometimes it's worth stopping at the first failure:

- name: Run tests
  run: pytest tests/ -v -x

With -x, pytest stops at the first FAILED and doesn't run the rest. It saves time when the first failure already indicates a serious problem. For PR CI, many prefer to run all the tests to see the full picture; -x is useful in local development or in very long pipelines.

The exit code and the job's behavior

If pytest returns 1, the step fails. The following steps of the same job don't run. If you have an "Upload coverage" step after "Run tests", that upload won't run if the tests failed. That's why continue-on-error is sometimes used for uploads you want to run even with failures (to see partial coverage), but in general you want the job to fail when the tests fail.

Publishing results as artifacts

What is an artifact?

An artifact is a file or directory that the workflow saves after the job finishes. You can download it from GitHub's UI. Useful for HTML coverage reports, pytest results in JUnit XML format, or logs.

Uploading htmlcov as an artifact

- name: Run tests with coverage
  run: |
    pip install pytest-cov
    pytest tests/ -v --cov=src --cov-report=html

- name: Upload coverage report
  uses: actions/upload-artifact@v4
  if: always()  # Uploads even if the tests fail
  with:
    name: coverage-report
    path: htmlcov/

After the run, on the workflow run's page you'll see "Artifacts" with a link to download coverage-report.zip. Inside is the content of htmlcov/ — you unzip it and open index.html to see the report.

Uploading JUnit XML results

pytest can generate results in JUnit XML format, which GitHub Actions can parse to show a test summary in the UI.

- name: Run tests
  run: pytest tests/ -v --junitxml=test-results.xml

- name: Publish test results
  uses: EnricoMi/publish-unit-test-result-action@v2
  if: always()
  with:
    files: test-results.xml

This action shows a table with passed/failed tests in the UI. It requires installing the action; there are alternatives like dorny/test-reporter, or you can simply upload the XML as an artifact and download it for local analysis.

An artifact only if there are failures

- name: Upload logs on failure
  uses: actions/upload-artifact@v4
  if: failure()
  with:
    name: pytest-output
    path: test-results.xml

It only uploads the artifact when the job fails. Useful to avoid accumulating unnecessary artifacts in successful runs.

Running test categories

Marking tests with markers

In pytest.ini or pyproject.toml:

[tool.pytest.ini_options]
markers = [
    "unit: Unit tests (fast, no I/O)",
    "integration: Integration tests (DB, API, network)",
    "e2e: End-to-end tests (slow)",
]

In the tests:

import pytest

@pytest.mark.unit
def test_add():
    assert add(2, 3) == 5

@pytest.mark.integration
def test_api_create_item():
    response = client.post("/items", json={"name": "x"})
    assert response.status_code == 201

@pytest.mark.e2e
def test_full_user_flow():
    # A test that opens a browser, navigates, etc.
    pass

Running only unit tests in a fast CI

- name: Run unit tests
  run: pytest tests/ -v -m unit

It only runs tests marked with @pytest.mark.unit. Useful if you want a fast job that runs on every push, and a slower job with integration/e2e that only runs on PRs to main.

Running unit + integration in a standard CI

- name: Run tests
  run: pytest tests/ -v -m "unit or integration"

Or excluding e2e:

- name: Run tests (excluding e2e)
  run: pytest tests/ -v -m "not e2e"

Separate jobs for different categories

jobs:
  unit:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install -r requirements.txt
      - run: pytest tests/ -v -m unit

  integration:
    runs-on: ubuntu-latest
    needs: unit  # Only runs if unit passes
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install -r requirements.txt
      - run: pytest tests/ -v -m integration

The integration job only runs if unit passes. It saves time when unit already failed.

Interpreting failure logs in GitHub's UI

The typical structure of the log

1. pip's output: If the install step fails, you'll see pip's error (package not found, version conflict, etc.).
2. pytest's output: The test session, each test with PASSED/FAILED, and the summary at the end.
3. The traceback: When a test fails, pytest prints the traceback. The line with > indicates the assert that failed.

Common errors in the logs

The diagnostic flow when the job fails

1. Open the workflow run in the Actions tab.
2. Identify the job that failed (a red mark).
3. Click the job and look for the step that failed (the first one with a red X).
4. If it's "Install dependencies": review pip's error. It's usually a nonexistent package or a version conflict.
5. If it's "Run tests": read the traceback. The line with > indicates the assert. The value shown (E AssertionError: assert X == Y) tells you what you expected vs what you got.
6. Copy the relevant fragment (from ===== FAILURES ===== to the end of the traceback) and use it with Claude Code or in a search to diagnose it.

Searching the logs

On the step's page, use Ctrl+F (Cmd+F) to search for:

• FAILED — Quickly find which tests failed.
• Error or error: — Import or configuration errors.
• ModuleNotFoundError — To diagnose dependency problems.

Environment variables in CI

Basic environment variables

jobs:
  test:
    runs-on: ubuntu-latest
    env:
      DATABASE_URL: "sqlite:///./test.db"
      LOG_LEVEL: "WARNING"
      PYTHONUNBUFFERED: "1"
    steps:
      - uses: actions/checkout@v4
      - run: pip install -r requirements.txt
      - run: pytest tests/

The tests can read os.environ["DATABASE_URL"] or use os.getenv("DATABASE_URL"). Useful for integration tests that use a test DB.

Using GitHub Secrets for sensitive values

Never put API keys, tokens or passwords in the YAML. Use Secrets:

1. On GitHub: Settings → Secrets and variables → Actions.
2. New repository secret. Name: API_KEY, value: your key.
3. In the workflow:

jobs:
  test:
    runs-on: ubuntu-latest
    env:
      API_KEY: ${{ secrets.API_KEY }}
    steps:
      - uses: actions/checkout@v4
      - run: pytest tests/

If the secret doesn't exist, secrets.API_KEY will be empty. Make sure you create the secret before the workflow runs.

Secrets for tests that call real APIs

Some projects have tests that call external APIs (a sandbox) and need a token. Configure the secret in the repo and pass it as an env var. In the tests, if API_KEY is empty, you can skip the test:

import os
import pytest

@pytest.mark.skipif(not os.getenv("API_KEY"), reason="API_KEY not set")
def test_external_api():
    # A test that uses the real API with the token
    pass

Environment variables per step

steps:
  - name: Run unit tests
    run: pytest tests/ -m unit
    env:
      USE_MOCK: "1"

  - name: Run integration tests
    run: pytest tests/ -m integration
    env:
      DATABASE_URL: ${{ secrets.TEST_DATABASE_URL }}

Each step can have its own env block that overrides or extends the job's.

Guided practice: a pipeline with artifacts and markers

Follow these steps to build a complete pipeline with pytest, coverage and artifacts.

1. Create a test structure with markers

# tests/test_calc.py
import pytest
from src.calc import add, multiply

@pytest.mark.unit
def test_add():
    assert add(2, 3) == 5

@pytest.mark.unit
def test_multiply():
    assert multiply(3, 4) == 12

2. Register the markers in pyproject.toml

[tool.pytest.ini_options]
markers = ["unit: Unit tests"]
testpaths = ["tests"]

3. Add coverage and artifacts to the workflow

Modify the pytest step to generate coverage and upload the report:

- name: Run tests with coverage
  run: |
    pip install pytest-cov
    pytest tests/ -v -m unit --cov=src --cov-report=html --cov-report=term-missing

- name: Upload coverage report
  uses: actions/upload-artifact@v4
  if: always()
  with:
    name: coverage-report
    path: htmlcov/

4. Verify on GitHub

Push, wait for the workflow to finish, and on the run's page download the coverage-report artifact. Unzip it and open htmlcov/index.html to see the visual coverage report.

5. Test it with a failing test

Intentionally change an assert so it fails. Push. You'll see the workflow marked as failed, but the coverage artifact was uploaded anyway (thanks to if: always()). The "Run tests" step's logs will show the failing test's traceback.

Detailed interpretation: from the UI to the fix

When a workflow fails, the diagnostic flow is:

1. Actions → the failed run → Click the run.
2. The failed job → Click the job (e.g. "test") to see the steps.
3. The failed step → Click the step with the red X. It's usually "Run tests" or "Install dependencies".
4. The step's log → Scroll to the end. The error is usually there. Look for FAILED, Error, Traceback.
5. Copy the traceback → From def test_xxx to the assert's line. That block is what you need to fix it.

If the failure is in "Install dependencies", the problem is in the requirements or in the pip command's syntax. If it's in "Run tests", the problem is in the code or in the tests. The traceback tells you the exact file and line.

A summary of pytest flags for CI

The recommended combination for CI: pytest tests/ -v --tb=short — enough information without saturating the logs.

An example: diagnosing a ModuleNotFoundError

If the log shows:

ModuleNotFoundError: No module named 'src'

The steps:

1. Verify that the code is in src/ and that the tests import from src.X import Y.
2. The runner runs from the repo's root (after checkout). If src isn't in PYTHONPATH, Python doesn't find it.
3. Add env: PYTHONPATH: . to the job, or install the package with pip install -e . if you have a pyproject.toml.
4. Push and verify. If the error persists, the directory structure may be different (e.g. app/ instead of src/); adjust the path.

When to use markers vs separate jobs

For Module 7's project, a single job with all the tests is usually enough. If your suite takes more than 5 minutes, consider separating unit (fast, every push) from integration (slower, every PR).

A complete workflow with pytest in CI

# .github/workflows/tests.yml
name: Tests

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  test:
    runs-on: ubuntu-latest

    env:
      PYTHONUNBUFFERED: "1"

    steps:
      - name: Checkout repository
        uses: actions/checkout@v4

      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: "3.12"

      - name: Install dependencies
        run: |
          python -m pip install --upgrade pip
          pip install -r requirements.txt
          pip install pytest pytest-cov

      - name: Run tests
        run: pytest tests/ -v --tb=short --cov=src --cov-report=html --cov-report=term-missing

      - name: Upload coverage report
        uses: actions/upload-artifact@v4
        if: always()
        with:
          name: coverage-report
          path: htmlcov/

This workflow installs the deps, runs pytest with coverage, and uploads the HTML report as an artifact. The artifact is generated whether the tests pass or fail (if: always()), so you can inspect the coverage even when there are failures.

Exercises

Exercise 1: Add pytest flags (Basic)

Modify your workflow's pytest step to use -v --tb=short. Make a test fail intentionally, push, and verify that the log shows a readable traceback with the assert that failed.

- name: Run tests
  run: pytest tests/ -v --tb=short

>       assert False
E       AssertionError

Exercise 2: Publish htmlcov as an artifact (Intermediate)

Add pytest-cov to your project. Configure the workflow to generate htmlcov/ and upload it as an artifact. Run the workflow, download the artifact, unzip it and open htmlcov/index.html in your browser.

- name: Install dependencies
  run: |
    pip install -r requirements.txt
    pip install pytest-cov

- name: Run tests with coverage
  run: pytest tests/ --cov=src --cov-report=html

- name: Upload coverage report
  uses: actions/upload-artifact@v4
  with:
    name: coverage-report
    path: htmlcov/

Exercise 3: Run only unit tests (Intermediate)

Your project has tests marked with @pytest.mark.unit and @pytest.mark.integration. Configure the workflow to run only the unit tests. Add a unit marker in pyproject.toml if you don't have one.

[tool.pytest.ini_options]
markers = ["unit: Unit tests"]

- name: Run unit tests
  run: pytest tests/ -v -m unit

Exercise 4: An environment variable for a test (Intermediate)

You have a test that uses os.getenv("TEST_MODE") to decide whether to use mocks or a real DB. Add TEST_MODE=1 as an environment variable in the workflow's test job.

jobs:
  test:
    runs-on: ubuntu-latest
    env:
      TEST_MODE: "1"
    steps:
      - uses: actions/checkout@v4
      - run: pip install -r requirements.txt
      - run: pytest tests/

Exercise 5: Diagnose a ModuleNotFoundError (Advanced)

The workflow fails with ModuleNotFoundError: No module named 'src'. Your project has src/my_module/ and the tests do from src.my_module import something. What can you change in the workflow or in the structure to make it work?

env:
  PYTHONPATH: .

- name: Run tests
  run: pytest tests/
  env:
    PYTHONPATH: .

- run: pip install -e .

Exercise 6: A secret for an API key (Advanced)

You have an integration test that calls an external API and needs API_KEY. Configure a secret in the repo (even a fake value for practice) and use it in the workflow. Verify that the test receives it.

jobs:
  test:
    runs-on: ubuntu-latest
    env:
      API_KEY: ${{ secrets.API_KEY }}
    steps:
      - uses: actions/checkout@v4
      - run: pip install -r requirements.txt
      - run: pytest tests/

Troubleshooting

The tests pass locally but fail in CI

Cause: Environment differences: Python version, dependencies, environment variables, timezone, paths.

Solution: Check that the Python version in the workflow matches your local one. If you use pathlib or relative paths, verify that the working directory is the expected one (after checkout it's the repo's root). Run it locally in a clean venv: pip install -r requirements.txt && pytest tests/ to simulate CI.

"Fixture 'X' not found"

Cause: The fixture is defined in a conftest.py that pytest isn't discovering, or there's a typo in the name.

Solution: Make sure conftest.py is in tests/ or in the root. pytest discovers it automatically. If the fixture is in a subdirectory's conftest, verify that the tests using it are in that subdirectory or in its children. Check that the fixture's name matches exactly.

Dependencies missing in CI

Cause: Some package is installed globally or in your local venv but isn't in requirements.txt (or in pyproject.toml).

Solution: Run pip freeze in your local venv and compare it with what the workflow installs. Everything you import in the tests must be in the requirements. If you use pytest-cov, httpx, etc., add them to requirements-dev or to pyproject.toml's test section.

Import errors with a src/ structure

Cause: The src or app package isn't in PYTHONPATH when pytest runs.

Solution: Add PYTHONPATH: . to the job's env, or install the package with pip install -e .. In projects with a src/ layout, pip install -e . usually configures the path correctly.

The artifact isn't generated

Cause: The step that uploads the artifact failed earlier, or the path doesn't exist, or an if prevents it from running.

Solution: Verify that the previous step generates the directory/file (for example htmlcov/). If you use if: success() (implicit without an if), the upload only runs if the job passes; if the tests fail, it doesn't upload. Use if: always() to always upload. Check that the path is correct (e.g. htmlcov/ with the slash if it's a directory).

Project Connection

The pytest-in-CI pipeline you built here is the foundation of Module 7's project. In capsule 04 you'll add matrix testing (multiple Python versions) and caching to speed up builds. In 05 you'll integrate coverage into PRs and branch protection. The steps you defined — install, run pytest, upload artifacts — will remain the central blocks.

Summary

• The typical steps for pytest in CI: checkout, setup Python, install deps, run pytest
• When a test fails, pytest returns 1 and the job fails; the logs show the traceback
• You can publish reports (htmlcov, JUnit XML) as artifacts to download
• Use markers to run only unit, integration or e2e depending on the job
• Environment variables are configured in the job's or the step's env:
• Secrets are used as ${{ secrets.NAME }} for sensitive values
• Common problems: ModuleNotFoundError, fixture not found, path/PYTHONPATH

Next capsule: Matrix Testing and Caching — multiple Python versions and faster builds.

Additional Resources

1. pytest command line options - pytest flags
2. GitHub Actions: upload-artifact - Uploading artifacts
3. pytest markers - Markers documentation
4. GitHub Encrypted secrets - Using secrets
5. pytest JUnit XML - The JUnit format for results

Module 7, Capsule 03 — Testing with Claude Code Guide