Chapter 20 · Production Practice: CI and Test Governance

Chapter goals: assemble the previous nineteen chapters into a reliable CI pipeline — a debugging arsenal for fast failure triage, flaky test discovery and governance, GitHub Actions matrix builds, coverage gates, and a test-code review checklist.

20.1 The Failure-Debugging Arsenal

Production repos run thousands of cases; your first reaction after a failure determines triage speed:

$ pytest --lf                # last-failed: rerun only the previously failing cases
$ pytest --ff                # failed-first: run failures first, then the rest
$ pytest -x                  # Stop immediately on the first failure
$ pytest --tb=short          # Shorter traceback; long/auto/line/no available
$ pytest -ra                 # Summarize reasons for all non-passed cases at the end (add to addopts)
$ pytest --pdb               # Drop into pdb on failure (pairs with set_trace breakpoints)
$ pytest test_pay.py::test_refund -vv   # Single case + maximum verbosity

--lf plus --pdb is the golden combo: converge on the failing set with --lf, then inspect variables live with --pdb. --lf state lives in .pytest_cache and works across processes; clear it with pytest --cache-clear if it muddies judgment.

Bake it into config

Put addopts = "-ra --tb=short" in pyproject.toml so everyone's default output includes the summary and short stacks — no more memorizing flags.

20.2 Discovering and Governing Flaky Tests

Flakiness is the number-one killer of CI trust. Governance has three steps: discover → attribute → eliminate.

Step one: stop the bleeding with a rerun plugin — but don't stop there.

pip install pytest-rerunfailures
$ pytest --reruns 3 --reruns-delay 1            # Up to 3 retries with a 1-second gap
$ pytest --reruns 5 --only-rerun AssertionError # Retry only specific exception types

Current versions (16.x) also offer --reruns-mode strict|append for counting semantics and --max-suite-reruns for suite-level caps. Grant surgical exemptions via @pytest.mark.flaky(reruns=5) on individual cases.

Step two: actively surface order dependencies.

Many flaky tests stem from shared mutable state between cases — green alone, red together. A shuffle plugin exposes exactly this:

pip install pytest-random-order      # Or the similar pytest-randomly
$ pytest -p random_order                        # Enable shuffling
$ pytest --random-order-bucket=global           # Global shuffle — maximum pressure
# Buckets: package/module/class/global — smaller granularity is more conservative

Pin it into a scheduled CI job; when a shuffle turns red, the seed in the logs reproduces that exact ordering.

Step three: eliminate by root cause. Four categories cover most cases: time dependencies (datetime.now() → injected clock), randomness (missing seeds), external dependencies (network/filesystem → Chapter 10 mocking), and resource leaks (uncleaned ports/temp dirs → tmp_path). --reruns is an anesthetic — without killing the root cause, retry counts will eventually lose to bad luck.

Standard fixes for the time and randomness root causes:

# conftest.py -- deterministic injection for time and randomness
import random

import pytest


@pytest.fixture(autouse=True)
def _deterministic_env(request):
    """Make time/randomness controllable for all cases by default; opt out explicitly when needed."""
    if "real_random" in request.keywords:
        yield                      # Cases marked real_random stay untouched
        return
    random.seed(42)                # Pin the global random source
    yield


# Code-under-test should not call datetime.now() directly; take time from an injection point:
class Clock:
    def now(self):
        from datetime import datetime
        return datetime.now()


def test_coupon_expired():
    clock = FakeClock(at="2025-06-01T00:00:00")   # A fake clock controlled by the test
    assert coupon_expired(coupon, clock=clock) is True

# test_flaky_demo.py -- self-check order dependencies with the shuffle mindset
# Run: pytest -p no:cacheprovider --random-order-bucket=module
_shared_cache = {}                 # Anti-pattern: module-level mutable state


def test_write_state():
    _shared_cache["token"] = "abc"


def test_read_state():
    # Fails standalone (no token yet) but may pass jointly by lucky ordering
    assert _shared_cache["token"] == "abc"

Commit such cases to the repo and wire them into a shuffle CI run — any change that breaks isolation shows up on the pipeline.

20.3 A Complete GitHub Actions Workflow

A matrix pipeline you can adopt as-is:

# .github/workflows/tests.yml
name: tests

on:
  push:
    branches: [main]
  pull_request:

jobs:
  test:
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false              # One failing version doesn't mask the others' reports
      matrix:
        python-version: ["3.11", "3.12", "3.13"]
        include:
          - python-version: "3.13"  # Only the newest version runs coverage, saving time
            coverage: true
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-python@v5
        with:
          python-version: ${{ matrix.python-version }}
          cache: pip                # Cache pip downloads to speed up installs

      - name: Install dependencies
        run: |
          python -m pip install --upgrade pip
          pip install -e ".[dev]"

      - name: Run tests
        run: >
          pytest -ra --tb=short
          ${{ matrix.coverage && '--cov=myapp --cov-report=xml --cov-fail-under=85' || '' }}

      - name: Upload coverage artifact
        if: matrix.coverage
        uses: actions/upload-artifact@v4
        with:
          name: coverage-xml
          path: coverage.xml

Key points:

• Matrix + fail-fast: false: three Python versions run without dragging each other down; compatibility issues surface immediately;
• Coverage runs in a single job only: avoiding redundant computation across versions;
• Inline gate: --cov-fail-under=85 directly decides job success — below threshold the PR cannot merge;
• If you use Chapter 14's pytest-xdist sharding, each shard emits its own .coverage.* data file; the merge step is:

      - run: coverage combine && coverage report --fail-under=85 && coverage xml

coverage combine merges shard data before the gate evaluates — the standard finish for distributed runs.

20.4 Baking Governance into conftest

Review-checklist items that can be automated belong in the root conftest. Here's a practical "slow-case patrol + Hypothesis profiles" snippet:

# conftest.py -- automatic slow-case registry

SLOW_THRESHOLD = 0.5          # Warn above 0.5s
_slow_report = []


@pytest.hookimpl(wrapper=True)
def pytest_runtest_makereport(item, call):
    rep = yield
    if rep.when == "call" and call.duration > SLOW_THRESHOLD:
        _slow_report.append((call.duration, item.nodeid))
    return rep


def pytest_terminal_summary(terminalreporter):
    if not _slow_report:
        return
    terminalreporter.write_sep("=", "slow tests top 10")
    for dur, nodeid in sorted(_slow_report, reverse=True)[:10]:
        terminalreporter.write_line(f"{dur:8.2f}s  {nodeid}")

# conftest.py -- Hypothesis local-full / CI-fast profiles (see Chapter 18)
from hypothesis import settings

settings.register_profile("local", max_examples=200)
settings.register_profile("ci", max_examples=25, deadline=None)
settings.load_profile("local")   # Switch in CI via HYPOTHESIS_PROFILE=ci

At runtime just flip an env var: HYPOTHESIS_PROFILE=ci pytest — deep exploration locally, fast regression in CI, same tests.

20.5 The Test-Code Review Checklist

When reviewing someone's tests (or your own three months later), walk through:

1. Assertion validity: any false greens? Does every test actually reach its key assertion (beware Chapter 17's coroutine false green)?
2. Isolation: module-level mutable state or leaky shared fixtures? Does it pass in any order (the 20.2 shuffle check)?
3. Determinism: are time, randomness, and filesystem paths injected or mocked? Using tmp_path instead of hardcoded /tmp?
4. Speed budget: a unit test belongs in milliseconds; sleeps/real IO need justification.
5. Naming and information: can you triage a failure without reading code? (Use assertion messages and meaningful ids.)
6. Duplication: should similar cases collapse into parametrize/Hypothesis properties?
7. Marker hygiene: are slow/integration markers registered and used correctly? Is CI tiering respected (fast set on PRs, full set nightly)?

Turn the checklist into automation

Items 1–4 can all be automated: false greens via -W error-style warning promotion, order dependencies via shuffle CI, determinism via time-machine/faker seeding, speed via --durations=10 patrols. Save human review for semantics.

Chapter Summary

• --lf/--ff/--pdb/--tb/-ra form the standard failure-debugging sequence;
• Flaky governance = retry to stop bleeding (rerunfailures) + shuffling to expose order dependencies + eliminating the four root causes;
• CI matrices use fail-fast: false for complete signals; the coverage gate sits in the single cheapest spot;
• Under xdist sharding, always coverage combine before gating;
• Automate whatever the checklist allows; humans review semantics only.

🧪 随堂测验

点击你认为正确的选项。答错时会展示正确答案与原因解析。

1. Which command reruns only the cases that failed in the previous session?

2. Which statement about pytest-rerunfailures is WRONG?

3. What is the correct coverage-gate flow after pytest-xdist sharding?

4. Why set fail-fast: false in a GitHub Actions test matrix?

🛠️ Hands-on Practice

1. Add a weekly --random-order-bucket=global shuffle job to your CI; deliberately build two cases sharing module-level state and verify it catches them.
2. Extend the 20.3 workflow with a nightly job: main branch only, daily full run with slow markers plus high-exploration Hypothesis settings.
3. Draft a ≤10-line "new test self-check list" for your team, tailoring this chapter's review checklist to your repo.

🎓 Congratulations on completing all 20 chapters of the pytest course! Revisit the course guide, or start the FastAPI course to put these testing skills to work on web projects.