Chapter 14 · Parallel and Distributed Execution

Chapter goals: spread your test suite across CPUs with pytest-xdist, understand when to use each of the four --dist distribution strategies, avoid parallel pitfalls around fixture isolation and shared resources, and keep xdist playing nice with coverage measurement.

14.1 Five Minutes to Start: -n auto

When your suite grows until "one run takes a coffee break", it's time for pytest-xdist. It distributes tests to multiple processes (workers) running in parallel:

pip install pytest-xdist
pytest -n auto        # One worker per physical CPU core
pytest -n 8           # Explicit worker count
pytest -n logical     # Logical core count (requires Python>=3.13 or psutil)
pytest -n 0           # Disable parallelism, back to a single process

The -n auto worker count can be customized via the PYTEST_XDIST_AUTO_NUM_WORKERS environment variable or by implementing the pytest_xdist_auto_num_workers(config) hook in conftest; there is also --maxprocesses=N to cap auto mode:

# content of conftest.py -- customize auto workers based on environment
import os


def pytest_xdist_auto_num_workers(config):
    if "CI" in os.environ:          # CI machines have few cores but heavy tests; pin at 4
        return 4
    return None                     # Returning None falls back to default behavior

# Cap local runs at 4 workers so the dev machine doesn't choke
pytest -n auto --maxprocesses=4

14.2 Distribution Strategies: The Four --dist Choices

The default strategy, load, hands each next test to whichever worker is free — no ordering guarantee. Some tests must run consecutively in the same process, however — say they depend on an expensive module-level fixture or have order coupling. That's when you switch strategies:

pytest -n 4 --dist load         # Default: free load balancing
pytest -n 4 --dist loadscope    # Functions grouped by module, methods by class; same group same worker
pytest -n 4 --dist loadfile     # All tests of one file pinned to a single worker
pytest -n 4 --dist loadgroup    # Group by the xdist_group marker
pytest -n 4 --dist worksteal    # Work stealing: idle workers pick up tasks from busy ones

With loadgroup, the xdist_group marker lets you hand-pick cases that "live and die together":

# content of test_cluster.py
import pytest


@pytest.mark.xdist_group(name="group1")
def test_create_resource():
    create_shared_resource()          # Guaranteed same worker as test_use_resource


class TestA:
    @pytest.mark.xdist_group("group1")   # Class-level markers work too
    def test_use_resource(self):
        assert shared_resource_ready()

Selection experience

• Default load suits the vast majority of pure unit tests;
• Expensive module/class-level fixtures → loadscope;
• Wildly uneven test durations (mixed fast/slow) → worksteal;
• Legacy code with cross-test shared state → transition via loadfile/loadgroup; the end goal is removing state coupling.

14.3 A Checklist of Parallel Pitfalls

Parallelism isn't free lunch. Beyond the limitations in the xdist docs, the most common production failures:

1. Shared external resources: two workers hitting the same database schema / writing the same file / grabbing the same port.

   # ❌ Every worker writes to a fixed path — guaranteed collisions
   LOG = open("/tmp/app.log", "w")
   
   # ✅ Isolate with tmp_path (a distinct directory per worker process)
   def test_write(tmp_path):
       target = tmp_path / "app.log"
       write_log(target)
       assert target.exists()

2. Session-scoped fixtures run once per worker: a scope="session" fixture executes once inside every worker process, not once globally. Truly global singletons (e.g. a database initialized exactly once) need file locks or pre-creation in the main process.

   # content of conftest.py -- ensure the database initializes only once via a file lock
   import filelock
   import pytest
   
   
   @pytest.fixture(scope="session")
   def database(tmp_path_factory):
       # A one-time lock file under rootdir; all workers contend on the same lock
       lock = tmp_path_factory.getbasetemp().parent / "db.lock"
       with filelock.FileLock(str(lock)):
           db = create_test_database()   # Initialized under lock; other workers wait then reuse
       yield db

3. Ordering assumptions: tests that depend on execution order or module-level mutable global state will fail randomly under load. Fix the tests rather than tuning strategies.

4. Interleaved output and reports: failure summaries are still aggregated, but print output's owning worker is hard to trace; add -v to see worker prefixes (gw0, gw1) when debugging.

# The standard three steps for debugging parallel-only failures
pytest -n 4                 # Run in parallel to reproduce
pytest -n 0                 # Run serially to confirm it's parallel-specific
pytest -n 4 -k "test_x"     # Narrow down to find the conflicting pair

14.4 xdist + coverage: Accurate Even in Parallel

Measuring per-process then naively summing is wrong (the same line may be partially executed by different workers). pytest-cov is designed to be xdist-compatible: as workers finish they automatically send data back for merging — you do nothing extra:

pytest -n auto --cov=src --cov-report=term-missing

The only hard requirement is that every worker must have pytest-cov installed (the plugin registers through entry points on each worker). To also measure subprocesses spawned by your code-under-test, configure coverage's subprocess patch as hinted in Chapter 13.

Mind CI CPU counts

CI machines often have only 2–4 cores, where -n auto yields little benefit or even slows down due to process overhead. Use explicit -n 4 in CI, and move integration tests with the biggest parallel wins into their own job.

14.5 Chapter Summary

• -n auto/N turns on multiprocessing with one flag; --maxprocesses caps auto mode;
• Four --dist strategies: load (default), loadscope/loadfile (ordered grouping), loadgroup (manual grouping via xdist_group), worksteal (work stealing);
• Three big pitfalls: isolate shared resources (tmp_path), session fixtures are per-worker, forbid ordering and global-state assumptions;
• pytest-cov merges multi-worker coverage automatically, provided the plugin is installed everywhere.

🧪 随堂测验

点击你认为正确的选项。答错时会展示正确答案与原因解析。

1. What determines the number of workers launched by -n auto?

2. A group of tests depends on an expensive module-level fixture and should always run in the same worker. Which strategy fits best?

3. What is true about scope="session" fixtures under xdist?

4. When using pytest-cov together with pytest-xdist, what is correct about coverage?

🛠️ Hands-on Practice

1. Build a suite of twenty time.sleep(0.1) parametrized cases; time -n 0 vs -n auto and record the speedup.
2. Create a real conflict: two tests writing and asserting the same fixed file path; observe intermittent failures under -n auto, then fix it with tmp_path.
3. Wrap a "create-then-delete" test pair in a named xdist_group; run repeatedly under both --dist loadgroup and default load and verify the delete-before-create race no longer occurs under loadgroup.

The next chapter dives into parameterization deep water: indirect fixtures, cartesian-product stacking, and custom ID systems: Chapter 15.