Chapter 15 · Parallel Execution and Sharding

Chapter goals: Parallelize the E2E suite with pytest-xdist, understand that "isolation is the prerequisite for parallelism", and master sharding and report merging in CI.

15.1 Why E2E Needs Parallelism Especially

Every Playwright test launches a browser context; serial execution of 200 cases easily eats half an hour. The official parallel solution is pytest-xdist, with the recommended process count at half the logical CPU cores:

pip install pytest-xdist
pytest --numprocesses 4        # Official recommended parallel flag
pytest -n auto                 # xdist shorthand: auto-detect CPU count

Each worker is an independent process with its own Playwright driver and browser instance — this is the source of speed, and also the source of every pitfall: any mutable state shared across tests becomes a flaky-failure generator under parallelism.

15.2 Isolation Is the Prerequisite for Parallelism

pytest-playwright creates a fresh context + page per test by default (browser instance reused按 scope), so the browser layer is naturally isolated; real conflicts come from the system under test:

Conflict Source	Symptom	Fix
Same account logged in concurrently	Sessions kick each other out, cart data cross-pollinates	Each worker uses an独立 test account (see fixture below)
Shared database records	Test A deletes data that test B is validating	Tests create and clean up their own data (API seeding, see Ch 17), or shard by worker/schema
Fixed ports / fixed file paths	Address already in use	Randomize ports; write to tmp_path
Global counters / leaderboard assertions	Asserting "rank #1" while another worker writes	Change to relative assertions (my created record is visible), avoid global state

Worker-scoped isolated accounts inject like this:

# conftest.py — each xdist worker gets a different account
import pytest

WORKER_ACCOUNTS = ["user0@t.io", "user1@t.io", "user2@t.io", "user3@t.io"]


@pytest.fixture(scope="session")
def worker_account(worker_id: str) -> str:
    # worker_id is "master" (serial) or gw0/gw1/gw2...
    idx = 0 if worker_id == "master" else int(worker_id[2:])
    return WORKER_ACCOUNTS[idx % len(WORKER_ACCOUNTS)]

15.3 Distribution Granularity and Order Pitfalls

xdist default dispatches collected tests per-case to workers, meaning two things must be kept in mind:

• Execution order is not guaranteed. If cases depend on each other (B's data is created by A), parallelism will randomly fail — the fix is not to tune order but to make each case self-contained;
• If you truly need "cases in a group fixed to the same worker" (e.g., sharing one expensive login), use --dist loadscope (group by module/class) or loadgroup (with @pytest.mark.xdist_group):

@pytest.mark.xdist_group(name="checkout")   # Cases in this group lock to one worker
class TestCheckoutFlow:
    def test_add_to_cart(self, page): ...
    def test_pay(self, page): ...

pytest -n 4 --dist loadgroup

Also note: -s output and print logs interleave across workers — adding -o log_cli=true or reading the failing case's own trace (Chapter 11) is more efficient for debugging.

15.4 Sharding and Report Merging in CI

单机 parallelism has a ceiling; large suites in CI commonly use multi-machine sharding: each machine runs a subset, results合并 at the end. Pure pytest has no built-in equivalent to Node's --shard; the common approach is xdist's --dist loadfile with environment variables:

# .github/workflows/e2e.yml key snippet
strategy:
  matrix:
    shard: [1, 2, 3]
steps:
  - name: Run shard
    run: |
      pip install pytest pytest-playwright pytest-xdist pytest-md-report
      # Static sharding by test file hash (simple and reliable)
      python scripts/pick_shard.py --total 3 --index ${{ matrix.shard }} > files.txt
      pytest $(cat files.txt) --numprocesses 4 \
        --tracing retain-on-failure
  - name: Upload traces
    if: ${{ !cancelled() }}
    uses: actions/upload-artifact@v5
    with:
      name: traces-shard-${{ matrix.shard }}
      path: test-results/

Key points:

• Shard key must be stable (by filename, not runtime order) — ensures re-running the same shard covers the same content;
• Failure artifacts (traces/screenshots) named per shard and uploaded for consolidated review;
• JUnit XML reports (--junitxml=report-{shard}.xml)聚合 at the CI layer for overall pass rate.

Rule of thumb

Start with -n auto to max out a single machine; introduce matrix sharding only when single-machine time exceeds 10 minutes — don't add infrastructure complexity prematurely.

15.5 Failure Localization Under Parallelism

When a parallel run fails, the trace is scoped to the failing worker — not a global mess.流程:

1. Check the CI artifact named traces-shard-N/ to identify which shard failed;
2. Download the trace zip for the failing case;
3. Open with playwright show-trace — DOM snapshots and network logs are worker-local, no cross-contamination.

15.6 Chapter Summary

• pytest-xdist parallelizes suites; recommended --numprocesses = half CPU cores;
• Browser layer is naturally isolated by pytest-playwright; conflicts come from shared backend state — isolate per-worker with独立 accounts and per-test data;
• --dist loadscope / --dist loadgroup control dispatch granularity;
• CI sharding: stable key, per-shard artifacts, JUnit XML merge;
• Never rely on execution order in parallel tests.

🧪 随堂测验

点击你认为正确的选项。答错时会展示正确答案与原因解析。

1. What is the recommended process count for pytest-xdist?

2. Why can shared database records cause flaky failures in parallel E2E?

3. What does @pytest.mark.xdist_group do?

4. What is the recommended threshold for introducing CI sharding?

🛠️ Hands-On Exercises

1. Run an existing test suite with pytest -n auto; measure the speedup vs serial and note any failures caused by shared state.
2. Write a worker_account fixture that assigns each xdist worker a unique test account; run 4 workers and verify no cross-account data leakage.
3. Add @pytest.mark.xdist_group(name="auth") to a login-dependent test class; run with --dist loadgroup and confirm cases in the group land on the same worker.

Proceed to Chapter 16: CI Integration and Docker.