Matrix Testing and Caching

Capsule overview

Your workflow runs on a single Python version. But does your project support 3.10, 3.11 and 3.12? Do you want to make sure the code works on all of them? And every time the workflow runs, it installs pip, creates the environment, downloads the dependencies from scratch. That can add up to minutes. If you change one line of code and only run the tests, why reinstall everything?

This capsule covers matrix testing (running the same job on multiple Python versions in parallel) and caching (saving pip dependencies between runs for faster builds). You'll learn the matrix strategy, fail-fast, how pip caching works, the key based on requirements.txt, a before/after comparison of caching, and a complete workflow that combines matrix and cache.

What Is Matrix Testing?

The definition

A matrix strategy in GitHub Actions lets you run a job multiple times, varying one or more parameters. Each combination runs on an independent runner, in parallel. For Python, the typical parameter is the version: 3.10, 3.11, 3.12.

Why it matters

• Compatibility: You ensure the code works on the versions your project declares it supports.
• Early detection: If you use an API that's deprecated in 3.10 and still works but was removed in 3.12, the matrix detects it.
• Confidence for upgrades: When you decide to raise the minimum from 3.10 to 3.11, you already know 3.11 passes.

When to use it

• Projects that support multiple Python versions (common in libraries).
• Projects that want to validate on the newest version before adopting it.
• When CI time isn't critical (each version adds ~1-2 min per run, but they run in parallel).

Matrix Syntax

A basic example

jobs:
  test:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        python-version: ["3.10", "3.11", "3.12"]
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: ${{ matrix.python-version }}
      - run: pip install -r requirements.txt
      - run: pytest tests/ -v

Each value in python-version generates a run of the job. Three versions = three parallel runs. matrix.python-version is substituted in each run with "3.10", "3.11" or "3.12".

Multiple dimensions

strategy:
  matrix:
    python-version: ["3.10", "3.11"]
    os: [ubuntu-latest, macos-latest]

That generates 2 × 2 = 4 combinations: (3.10, ubuntu), (3.10, macos), (3.11, ubuntu), (3.11, macos). Each one runs in parallel. Useful for multi-platform projects.

Excluding combinations

strategy:
  matrix:
    python-version: ["3.10", "3.11", "3.12"]
    os: [ubuntu-latest, macos-latest]
    exclude:
      - python-version: "3.10"
        os: macos-latest

It excludes (3.10, macos). Useful if a combination is problematic or unnecessary.

Including additional combinations

strategy:
  matrix:
    python-version: ["3.10", "3.11"]
    include:
      - python-version: "3.12"
        experimental: true

It adds an extra combination. experimental would be a parameter you can use in the steps if you need it.

fail-fast

The default behavior

By default, fail-fast: true in a matrix strategy: if one combination fails, the others get cancelled. It saves minutes when you already know there's a failure.

Disabling fail-fast

strategy:
  fail-fast: false
  matrix:
    python-version: ["3.10", "3.11", "3.12"]

With fail-fast: false, if (3.10) fails, (3.11) and (3.12) keep running. You'll see the complete result for all the versions. Useful for knowing which versions specifically fail.

When to use each one

• fail-fast: true (the default): Fast CI; when one version fails, the rest doesn't add much. Good for iterative development.
• fail-fast: false: When you need the complete report (e.g. a release that must support every version).

Caching pip dependencies

The problem

Every time the workflow runs, the runner is new. It has no pip cache, no installed packages. pip install -r requirements.txt downloads everything from PyPI. With 50 packages, that can be 1-2 minutes per run. If the workflow runs 10 times a day, that's 10-20 minutes of repeated downloads.

The solution: caching

GitHub Actions lets you cache directories between runs. The actions/cache@v4 action saves a folder (for example, pip's cache) under a key. If the key matches on the next run, it restores the cache. pip install then reuses the already-downloaded packages and only fetches what's new.

How the cache key works

The typical key is based on:

• The runner's operating system (ubuntu, macos, etc.)
• The Python version
• The content or hash of requirements.txt

If you change requirements.txt, the key changes, the cache doesn't match, and pip installs again. That's correct: new dependencies require a fresh install. If you don't change the requirements, the cache is reused.

A hash of requirements.txt

- name: Cache pip
  uses: actions/cache@v4
  with:
    path: ~/.cache/pip
    key: ${{ runner.os }}-pip-${{ hashFiles('requirements.txt') }}
    restore-keys: |
      ${{ runner.os }}-pip-

• path: Where pip stores its cache by default on Linux (~/.cache/pip).
• key: A unique key. runner.os distinguishes ubuntu from macos. hashFiles('requirements.txt') generates a hash of the file. If requirements.txt changes, the hash changes, a new key, a new cache.
• restore-keys: If there's no exact cache, it uses the ubuntu-pip- prefix to restore any pip cache on ubuntu. It may be a cache from a previous version of the requirements; pip will install only what's missing.

The cache's location on different OSes

The actions/setup-python action with cache: 'pip' handles this automatically. You'll see that option below.

Using actions/setup-python with cache

The simplest way to cache pip is to use actions/setup-python's built-in cache:

- name: Set up Python
  uses: actions/setup-python@v5
  with:
    python-version: "3.12"
    cache: "pip"
    cache-dependency-path: "requirements.txt"

With cache: "pip" and cache-dependency-path, setup-python caches automatically based on requirements.txt. You don't need a manual actions/cache for pip.

With pyproject.toml

- name: Set up Python
  uses: actions/setup-python@v5
  with:
    python-version: "3.12"
    cache: "pip"
    cache-dependency-path: "pyproject.toml"

It uses the hash of pyproject.toml for the key. If you have a separate requirements-dev.txt, you can use a string with several paths:

cache-dependency-path: |
  requirements.txt
  requirements-dev.txt

The fallback: a manual cache

If your project has a more complex dependency structure, use a manual actions/cache:

- name: Cache pip
  uses: actions/cache@v4
  with:
    path: ~/.cache/pip
    key: ${{ runner.os }}-py${{ matrix.python-version }}-pip-${{ hashFiles('requirements.txt') }}
    restore-keys: |
      ${{ runner.os }}-py${{ matrix.python-version }}-pip-

- name: Install dependencies
  run: pip install -r requirements.txt

Before and after caching

Without a cache (typical)

Set up Python:      8s
Install dependencies: 90s  ← downloads everything
Run tests:          25s
Total:              ~2 min

With a cache (the first time, a cache miss)

Set up Python:      8s
Cache pip:          2s (restore attempt, miss)
Install dependencies: 85s
Run tests:          25s
Total:              ~2 min (similar, but it saves the cache for later)

With a cache (the second time, a cache hit)

Set up Python:      8s
Cache pip:          15s (restore, hit)
Install dependencies: 12s  ← most of it is already in the cache
Run tests:          25s
Total:              ~1 min

You can save 50-70% of the "Install dependencies" time when the cache hits. In workflows that run many times a day, that translates into minutes or hours saved.

When not to use a matrix

A matrix isn't always the right option:

• Projects that only support one version: If your pyproject.toml declares requires-python = ">=3.12", you don't need a matrix with 3.10 or 3.11. A single job with 3.12 is enough.
• Very slow CI: Three jobs consume 3x the runner minutes (even if they run in parallel, each one uses its own runner). If your GitHub plan has minute limits, a matrix can exhaust them fast.
• Incompatible dependencies: Some libraries don't support old Python versions. If 3.10 fails because of a dependency that only supports 3.11+, exclude 3.10 or don't use a matrix.
• Internal projects: If the whole team uses 3.12 and the project isn't distributed, validating on 3.10 may be overkill. A single job is simpler.

Real metrics: an example with a typical project

A Python project with ~30 dependencies (FastAPI, SQLAlchemy, pytest, etc.):

The saving with a cache is ~1 minute per run. In 50 pushes a day, that's 50 minutes less waiting. The matrix triples the Actions minute consumption but gives confidence in multi-version compatibility.

A complete workflow: matrix + caching

# .github/workflows/tests.yml
name: Tests

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  test:
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        python-version: ["3.10", "3.11", "3.12"]

    steps:
      - name: Checkout repository
        uses: actions/checkout@v4

      - name: Set up Python ${{ matrix.python-version }}
        uses: actions/setup-python@v5
        with:
          python-version: ${{ matrix.python-version }}
          cache: "pip"
          cache-dependency-path: "requirements.txt"

      - name: Install dependencies
        run: |
          python -m pip install --upgrade pip
          pip install -r requirements.txt

      - name: Run tests
        run: pytest tests/ -v --tb=short

With pyproject.toml

      - name: Set up Python ${{ matrix.python-version }}
        uses: actions/setup-python@v5
        with:
          python-version: ${{ matrix.python-version }}
          cache: "pip"
          cache-dependency-path: "pyproject.toml"
      - name: Install dependencies
        run: pip install -e ".[dev]"

Guided practice: adding matrix and cache to an existing workflow

Follow these steps to transform a basic workflow into one with a matrix and caching.

1. Have a working workflow

Make sure you have checkout, setup Python, install, run pytest. And that it passes on at least one version.

2. Add the matrix strategy

Wrap the job with strategy and matrix.python-version:

jobs:
  test:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        python-version: ["3.10", "3.11", "3.12"]

3. Use the matrix in setup-python

Change the setup step to use ${{ matrix.python-version }}:

      - uses: actions/setup-python@v5
        with:
          python-version: ${{ matrix.python-version }}

4. Add caching to setup-python

Add cache and cache-dependency-path:

      - uses: actions/setup-python@v5
        with:
          python-version: ${{ matrix.python-version }}
          cache: "pip"
          cache-dependency-path: "requirements.txt"

5. Push and verify

Push. In Actions you'll see three jobs in parallel (3.10, 3.11, 3.12). The first time the cache will be a miss; the second, you should see "Cache hit" in the setup step and a shorter install time.

Time comparison: a real example

In a project with ~30 dependencies in requirements.txt:

The saving is ~45% on successive runs with no changes to the requirements. With a matrix (3 versions), there are 3 parallel jobs; each one with its own cache. The workflow's total time is that of the slowest job, not the sum.

When not to use a matrix

• A single-version project: If you only support Python 3.12 and don't plan to change, a matrix adds complexity with no benefit.
• Very slow CI: If it already takes 5+ minutes, adding 2 more versions may not be worth it. Validate at least on the minimum and maximum versions you support.
• Incompatible dependencies: If a critical library doesn't support 3.10, exclude it from the matrix or don't use that version.

The order of the steps: cache before install

The correct order is:

1. Checkout (to have requirements.txt)
2. Cache (restore) — if it exists, it restores ~/.cache/pip
3. Setup Python
4. Install — pip uses the restored cache and only downloads what's new

If you put the cache after the install, the restore has no effect on that run. If you use actions/setup-python with cache: "pip", setup-python does the caching for you; you don't need a separate step.

A decision flow: should I use a matrix?

Does your project support multiple Python versions?
├── No (only 3.12) → One job with python-version: "3.12"
└── Yes
    ├── Is it a library or a distributable package? → Yes → A matrix of 3.10, 3.11, 3.12
    └── Is it an internal app?
        ├── Does the whole team use the same version? → Yes → A single job
        └── Do you want to prepare for a future upgrade? → A matrix to validate

Caching with multiple dependency files

If you have requirements.txt, requirements-dev.txt and optionally requirements-prod.txt, the cache's key must reflect all the ones the job uses:

- uses: actions/setup-python@v5
  with:
    python-version: "3.12"
    cache: "pip"
    cache-dependency-path: |
      requirements.txt
      requirements-dev.txt

Or with a manual cache:

key: ${{ runner.os }}-pip-${{ hashFiles('requirements.txt', 'requirements-dev.txt') }}

If you only include one, changes in the other won't invalidate the cache and you could end up with outdated dependencies.

Frequent errors when configuring a matrix

"Invalid version" in setup-python

Cause: You wrote "3.12" but the expected syntax may vary. Use strings with the exact format: "3.10", "3.11", "3.12". Don't use 3.12 without quotes if it fails (although YAML sometimes accepts it).

The matrix generates 0 jobs

Cause: Every combination was excluded with exclude, or there's a typo in the matrix's definition.

Solution: Check that matrix.python-version has at least one value. If you use include without a base matrix, there can be unexpected behavior. Try with a minimal matrix first.

Different Python versions, the same cache key

Cause: The key doesn't include matrix.python-version. Python 3.10 and 3.12 can have different wheels; sharing a cache can cause incompatibilities.

Solution: Include the version in the key when you use a manual actions/cache: ${{ runner.os }}-py${{ matrix.python-version }}-pip-${{ hashFiles('requirements.txt') }}. With setup-python and cache: "pip", the action handles this automatically per Python version.

Tips for optimizing CI times

1. Cache whenever you have 5+ dependencies: With small projects, the saving is minimal (10-15s). With 20+ packages, the saving can be 1-2 min per run. In workflows that run dozens of times a day, that adds up.
2. Use fail-fast: true during iterative development: When you're fixing a failure and one version already failed, cancelling the rest saves minutes. Switch to fail-fast: false when you prepare a release and want the complete report.
3. paths-ignore for files that don't affect the tests: If you only change README.md, docs/ or .gitignore, avoid running CI. Configure paths-ignore in the triggers. Every run you avoid saves minutes of your Actions plan.
4. Parallelize jobs carefully: If unit and integration can run in parallel, do it. But if integration needs unit's build, use needs to avoid duplicating work.
5. Review your minute consumption: In GitHub Settings → Billing → Plans and usage you can see how many minutes your workflows consume. With the free plan (2000 min/month for private repos), a matrix of 3 versions can exhaust them fast in very active repos.

A summary of caching options for pip

A numeric example: a project with 15 dependencies

Let's say a project with a requirements.txt that includes FastAPI, SQLAlchemy, pytest, pytest-cov, httpx and about 10 more dependencies (about 15 in total).

Without a cache:

• Install dependencies: ~55 seconds
• Run tests: ~20 seconds
• Total per job: ~1 min 20 s

With a cache (the second run, a cache hit):

• Setup Python + restore cache: ~18 seconds
• Install dependencies: ~12 seconds (most of it from the cache)
• Run tests: ~20 seconds
• Total per job: ~50 seconds

The saving: ~30 seconds per job. With a matrix of 3 versions there are 3 jobs; each one saves ~30 s. In 20 pushes a day, that's 10 minutes less runner time. If your plan has a minute limit, the cache lets you do more runs within the limit.

When the cache doesn't help as much: Projects with 2-3 dependencies (pytest, requests) install in 10-15 s anyway. The saving may be only 5-8 s. It's still worth configuring; it costs nothing and in projects that grow the benefit appears.

Verifying that the cache works

After configuring the cache, do two pushes in a row without changing requirements.txt:

1. The first run: In the "Set up Python" (or "Cache pip") step you should see something like "Cache not found" or "Cache restored from key: ..." with a "miss" status.
2. The second run: You should see "Cache hit" with the same key. The install step will be noticeably faster.

If you always see "miss", check that the key is stable. If you use hashFiles('requirements.txt'), it shouldn't change between pushes if you didn't modify the file. If the key includes something random (a date, run_id), every run will have a different key and will never hit.

A quick reference: matrix and cache syntax

For quick consultation while implementing:

A basic matrix:

strategy:
  matrix:
    python-version: ["3.10", "3.11", "3.12"]
# Use it in the steps: ${{ matrix.python-version }}

A matrix with exclude:

strategy:
  matrix:
    python-version: ["3.10", "3.11", "3.12"]
    os: [ubuntu-latest, macos-latest]
    exclude:
      - python-version: "3.10"
        os: macos-latest

A cache with setup-python:

- uses: actions/setup-python@v5
  with:
    python-version: ${{ matrix.python-version }}
    cache: "pip"
    cache-dependency-path: "requirements.txt"

fail-fast:

strategy:
  fail-fast: false  # Every job runs even if one fails
  matrix:
    python-version: ["3.10", "3.11", "3.12"]

Exercises

Exercise 1: A basic matrix (Basic)

Add a matrix strategy to the test workflow so it runs on Python 3.10 and 3.11. Push and verify that both jobs appear in Actions.

jobs:
  test:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        python-version: ["3.10", "3.11"]
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: ${{ matrix.python-version }}
      - run: pip install -r requirements.txt
      - run: pytest tests/ -v

Exercise 2: fail-fast false (Basic)

Configure fail-fast: false in the matrix. Introduce a test that fails only on Python 3.10 (for example, using syntax that exists in 3.11+ but not in 3.10). Verify that the other jobs keep running when 3.10 fails.

strategy:
  fail-fast: false
  matrix:
    python-version: ["3.10", "3.11", "3.12"]

Exercise 3: Caching with setup-python (Intermediate)

Configure pip caching using actions/setup-python's built-in option. Run the workflow twice in a row without changing requirements.txt. Compare the "Install dependencies" step's time between the first and second run.

      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
          cache: "pip"
          cache-dependency-path: "requirements.txt"

Exercise 4: A manual cache with a hash (Intermediate)

Instead of cache: "pip" in setup-python, use actions/cache@v4 manually. Configure the key with hashFiles('requirements.txt') and the path ~/.cache/pip. Verify that the cache works.

      - name: Cache pip
        uses: actions/cache@v4
        with:
          path: ~/.cache/pip
          key: ${{ runner.os }}-pip-${{ hashFiles('requirements.txt') }}
          restore-keys: |
            ${{ runner.os }}-pip-

      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"

      - run: pip install -r requirements.txt

Exercise 5: A matrix with pyproject.toml (Intermediate)

Your project uses pyproject.toml with [project.optional-dependencies] dev = ["pytest", "pytest-cov"]. Configure a matrix for 3.10, 3.11, 3.12 and caching using cache-dependency-path: "pyproject.toml". The install step must use pip install -e ".[dev]".

jobs:
  test:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        python-version: ["3.10", "3.11", "3.12"]
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: ${{ matrix.python-version }}
          cache: "pip"
          cache-dependency-path: "pyproject.toml"
      - run: pip install -e ".[dev]"
      - run: pytest tests/ -v

Exercise 6: Excluding a combination (Advanced)

You have a matrix with python 3.10, 3.11, 3.12 and os ubuntu, macos. Exclude the (3.10, macos) combination because in your project that combination has known problems.

strategy:
  matrix:
    python-version: ["3.10", "3.11", "3.12"]
    os: [ubuntu-latest, macos-latest]
    exclude:
      - python-version: "3.10"
        os: macos-latest

jobs:
  test:
    runs-on: ${{ matrix.os }}
    # ...

Troubleshooting

The cache always misses

Cause: The key changes on every run (for example, it includes a timestamp) or the path is incorrect.

Solution: Use a stable key: hashFiles('requirements.txt') is deterministic. If the requirements don't change, the key is the same. Verify that path matches where pip stores the cache. On Ubuntu it's ~/.cache/pip.

A corrupt cache or an incorrect installation

Cause: Sometimes a restored cache can cause inconsistent installations (a package updated on PyPI, an old cache).

Solution: You can delete the cache from GitHub's UI: Settings → Actions → Caches → Delete. Or add a step that forces a reinstall when you need it. Most of the time the cache is reliable; if you see strange failures, try without a cache once.

A matrix job fails on one version

Cause: Code or a dependency incompatible with that Python version.

Solution: Review that job's error in the log. It could be unsupported syntax (e.g. match in 3.9), a dependency that doesn't support that version, or a test that assumes a newer version's behavior. Fix the code or exclude that combination from the matrix if you don't support it.

The CI time doesn't drop with the cache

Cause: The cache isn't being used (a miss), or requirements.txt is small and pip was already fast.

Solution: Verify that the cache step shows "Cache hit" on the second run. If it's always a miss, check the key. With few dependencies (2-3), the saving can be minimal (10-20s). With many (20+), the saving is usually significant (1-2 min).

Different Python versions share a cache

Cause: The cache key doesn't include the Python version.

Solution: Include matrix.python-version in the key when you use a matrix: ${{ runner.os }}-py${{ matrix.python-version }}-pip-${{ hashFiles('requirements.txt') }}. Each Python version gets its own cache.

The matrix generates jobs with long names

Cause: With a matrix, GitHub shows "test (3.10)", "test (3.11)", etc. In branch protection, the check can be called "test (3.10)" or simply "test" depending on the configuration.

Solution: For branch protection with a matrix, you normally need all the matrix's jobs to pass. Configure "Require status checks" and select all the job's checks (or the parent check if it exists). GitHub usually exposes one check per matrix combination.

Extended guided practice: measuring the cache's impact

To see the cache's real effect on your project:

1. Temporarily disable the cache

Comment out cache and cache-dependency-path in setup-python. Push. Note the "Install dependencies" step's time in the logs (e.g. 78s).

2. Re-enable the cache

Add cache: "pip" and cache-dependency-path back. Push again (without changing the requirements). The first time it'll be a miss. Do a second trivial push (e.g. a change in the README). The second time it should be a hit.

3. Compare

On the second run, "Install dependencies" should drop to 10-25s. The difference is the saving. In projects with many dependencies, it can be 1+ minute per run.

Project Connection

Matrix and caching are key components of Module 7's project. A professional CI pipeline validates on multiple Python versions and uses a cache to keep builds fast. In capsule 05 you'll add coverage in CI and branch protection. The workflow you built here — matrix + cache — will be the foundation you add coverage and thresholds to.

Summary

• A matrix strategy runs a job in multiple configurations (e.g. Python 3.10, 3.11, 3.12) in parallel
• fail-fast: true cancels the rest when one fails; fail-fast: false lets them all finish
• Caching pip reduces the install time by reusing packages between runs
• The cache key must include a hash of requirements.txt (or pyproject.toml) and, with a matrix, the Python version
• actions/setup-python with cache: "pip" simplifies the configuration
• Before the cache: the install can be 60-90s; afterwards (a cache hit): 10-25s is common

Next capsule: Coverage in CI and Branch Protection — coverage reports, comments on PRs, and branch protections.

Additional Resources

1. GitHub Actions: Workflow syntax - matrix - Official matrix documentation
2. actions/setup-python - caching - The built-in pip cache
3. actions/cache - The manual cache action
4. Caching dependencies to speed up workflows - The caching guide
5. hashFiles - The function to generate cache keys
6. Python version support - The Python version lifecycle

Module 7, Capsule 04 — Testing with Claude Code Guide