Chapter 13 · Coverage and Quality Gates

Chapter goals: use pytest-cov to generate and read line/branch coverage reports, understand that "high coverage ≠ high quality", and enforce a --cov-fail-under quality gate in CI.

13.1 What pytest-cov Is and Why Not Just coverage run

pytest-cov plugs coverage.py's measurement capabilities into pytest. Compared with raw coverage run -m pytest, the official README lists these added values: automatic cleanup and merging of .coverage data files with a default report, per-test context recording (--cov-context=test can drill down to individual parametrized case names), and full xdist distributed support (still accurate across processes and even remote interpreters).

pip install pytest-cov
pytest --cov=mypkg tests/

Output looks like:

---------- coverage: platform linux, python 3.x ----------
Name                Stmts   Miss  Cover
---------------------------------------
mypkg/__init__.py       2      0   100%
mypkg/core.py         257     13    94%
---------------------------------------
TOTAL                 353     20    94%

The .coverage data file is wiped at the start of each run; add --cov-append when you need to accumulate across runs.

13.2 Report Formats and Locating Missing Lines

The three most used report forms:

pytest --cov=src --cov-report=term-missing     # Terminal table + missing line numbers column
pytest --cov=src --cov-report=html             # Generates htmlcov/ for line-by-line browser viewing
pytest --cov=src --cov-report=xml              # Machine-readable format consumed by CI tooling

The key of term-missing is the Missing column — it tells you exactly which statements never executed:

Name            Stmts   Miss  Cover   Missing
-------------------------------------------------
src/cart.py        42      6    86%   31-33, 47, 52-54

Reading example: lines 31–33 are entirely uncovered (likely an untested branch), while 47 is an isolated miss (possibly an error path). Pair it with --cov-report=html, open htmlcov/index.html, and inspect the highlighted lines — the most intuitive reconnaissance before writing new tests.

13.3 Branch Coverage: Stricter Than Line Coverage

Counting only "did this line run" misses half-finished tests. With --cov-branch, coverage additionally checks whether both directions of every branch were taken:

def discount(price: float, vip: bool) -> float:
    if vip:
        price *= 0.8
    return price

# content of test_cart.py
from cart import discount


def test_vip_discount():
    assert discount(100, vip=True) == 80.0    # Only exercised the if-true side

Without --cov-branch this file already reports 100% coverage; with it:

Name          Stmts   Miss Branch BrPart   Cover   Missing
----------------------------------------------------------
cart.py           4      0      2      1      88%   4->5

4->5 means the edge "line 4 is false → jump to line 5" never happened — i.e. nobody tested the vip=False path. In practice, make branch coverage your default metric.

13.4 Moving Options into Config

All coverage options can live in configuration to keep the command line clean:

# content of pyproject.toml -- enable from the pytest side
[tool.pytest.ini_options]
addopts = ["--cov=src", "--cov-branch", "--cov-report=term-missing"]

# coverage's own fine-grained config (coverage reads the same pyproject.toml)
[tool.coverage.run]
source = ["src"]

[tool.coverage.report]
show_missing = true
exclude_lines = [
    "pragma: no cover",
    "if TYPE_CHECKING:",
    'if __name__ == "__main__":',
]

exclude_lines removes idiomatic lines that "don't need testing", so you avoid writing meaningless tests just to cover a __main__ block:

# content of cli.py -- typical boilerplate paired with exclude_lines
def main():
    ...


if __name__ == "__main__":      # Excluded via exclude_lines; not counted as missing
    main()

Defensive branches that genuinely cannot be triggered reliably can be marked inline instead:

def parse_port(raw: str) -> int:
    port = int(raw)
    if not 0 < port < 65536:            # pragma: no cover -- validated upstream; defensive branch
        raise ValueError("port out of range")
    return port

Upgrade note

Since pytest-cov 7 the legacy .pth subprocess-injection mechanism is removed; to measure code inside subprocesses, use coverage's patch configuration (e.g. [run] patch = subprocess).

13.5 CI Quality Gate in Practice

The core of a gate is --cov-fail-under=N: below the threshold pytest exits non-zero and the pipeline fails outright. The recommended pattern is no gate locally, gate injected in CI, keeping the developer experience friction-free:

# .github/workflows/test.yml
name: test
on: [push, pull_request]

jobs:
  unit:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: {python-version: "3.12"}
      - run: pip install -e ".[test]"
      # Gate options injected via env vars; the repo's pyproject.toml contains no fail_under
      - run: pytest --cov=src --cov-fail-under=85
        env:
          PYTEST_ADDOPTS: "--cov-report=xml"

How do you pick a threshold? Two principles:

• Legacy projects start from reality: run once to get the true number (say 62%), set fail-under=60 to prevent regression, then ratchet upward;
• Do not chase 100% blindly: coverage measures which code your tests touched, not assertion quality. A suite with 100% line coverage but zero meaningful assertions still lets bugs through — coverage is necessary, not sufficient.

A chilling counterexample — perfect coverage, zero quality:

# ❌ These two tests push discount's line coverage to 100% but verify nothing
def test_discount_vip():
    discount(100, vip=True)           # Called it, asserted nothing


def test_discount_normal():
    discount(100, vip=False)

If someone someday changes the discount from 20% off to 10% off — or deletes the logic entirely — these two tests stay green. Team conventions should therefore demand both: a coverage gate + code review focused on assertion validity.

13.6 Chapter Summary

• pytest-cov advantages over raw coverage run: automatic cleanup/merge, test contexts, xdist support;
• The three report formats serve "quick location / human review / CI consumption" respectively;
• --cov-branch exposes one-sided branch blind spots and should be the default metric;
• A gate = --cov-fail-under + a non-zero CI exit code; tighten thresholds gradually from current reality, and never forget coverage isn't quality itself.

🧪 随堂测验

点击你认为正确的选项。答错时会展示正确答案与原因解析。

1. Which option shows you directly in the terminal report which lines were not tested?

2. A function has if/else branches but tests only trigger the truthy path. Line coverage is 100% — how do you expose this blind spot?

3. Which statement about --cov-fail-under is correct?

4. Which scenario indicates the coverage metric being misused?

🛠️ Hands-on Practice

1. Run --cov --cov-branch --cov-report=term-missing against the mocked get_json function from Chapter 10, find the uncovered exception-handling branch, and add a test that triggers it.
2. Configure [tool.coverage.report] exclude_lines in pyproject.toml and verify the if __name__ == "__main__": block no longer counts as missing.
3. Build a minimal GitHub Actions workflow (or simulate locally): inject --cov-fail-under=<current>-5 via environment variables and verify the gate fires (deliberately delete a test and watch the exit code).

The next chapter tackles "too many tests, too slow" — pytest-xdist multiprocessing and distribution strategies: Chapter 14.