Chapter 22 · Production Practice: E2E Strategy and Reporting

Chapter goals: Bring together the techniques from Chapters 1–21 into a sustainably operable E2E engineering system: layered strategy, environment and data management, reporting and alerting, and a maintenance checklist.

22.1 E2E Positioning: Few but Precise, Guarding Critical Paths

First recalibrate your mindset: E2E tests have the highest unit cost of any test layer (slow, fragile, hard to debug). Their value is verifying "the user's critical journey actually works", not chasing coverage. A healthy split looks like:

Layer	Scale	When to Run	Responsibility
Unit Tests	Thousands	Every push	Logic correctness (see pytest course)
API/Component Tests	Hundreds	Every PR	Service contracts, page fragments
E2E Critical Paths	20–50 cases	Every PR (smoke) + daily full run	Login, checkout, payment — core journeys

Based on this, tier E2E further and manage with pytest markers:

# pytest.ini
[pytest]
markers =
    smoke: Smoke set, must run on every PR (~5 min)
    critical: Critical paths, must run before merge
    full: Full regression, nightly scheduled

import pytest

@pytest.mark.smoke
def test_login(page):
    ...

@pytest.mark.critical
def test_place_order_end_to_end(page, login_as):
    ...

pytest -m smoke            # PR pipeline
pytest -m "critical or smoke"   # Merge pipeline
pytest -m full             # Nightly full run

22.2 Environment and Test Data Factories

The most common failure point in production E2E is data. Three principles:

1. Dedicated environment: never run write operations against production; test environments must be resettable quickly;
2. Self-seeded data: each test creates the data it needs and cleans up after (or use one-shot database snapshots);
3. API seeding preferred over UI seeding: UI data creation is slow and brittle.

Recommended "API prepare + UI verify" division of labor:

import requests
import pytest

API = "https://api.staging.example.com"

@pytest.fixture
def product(api_token):
    """Create a product via API; delete after test — UI only verifies display"""
    resp = requests.post(
        f"{API}/products",
        json={"name": "Test Product-Do Not Touch", "price": 9.9},
        headers={"Authorization": f"Bearer {api_token}"},
        timeout=10,
    )
    product = resp.json()
    yield product
    requests.delete(f"{API}/products/{product['id']}",
                    headers={"Authorization": f"Bearer {api_token}"}, timeout=10)

def test_product_visible_in_storefront(page, product):
    page.goto(f"https://staging.example.com/products/{product['id']}")
    expect(page.get_by_role("heading", name=product["name"])).to_be_visible()

Combined with storage_state auth reuse (Chapter 9) and parametric data expansion (Chapter 13), you have a stable data foundation.

22.3 Reporting and Alerting

Standard CI output chain:

JUnit XML → CI dashboard (pass/fail rate per layer)
    ↓
HTML report (pytest-html) → per-case screenshots
    ↓
Trace ZIP → per-failure root-cause replay

Alert rules of thumb:

• Smoke set failure → block merge;
• Critical set failure → P1 bug, investigate within 24h;
• Full set regression → ticket, schedule fix.

22.4 Maintenance Checklist

Check	Frequency	Action if Broken
Trace artifact upload	Every CI run	Fix if: !cancelled() condition
Browser engine version	After pip upgrade	Re-run playwright install
storage_state freshness	Weekly	Refresh token / extend expiry
Aria snapshot baseline	On UI refactor	Re-generate with --update-snapshots
Flaky test count	Per sprint	Root-cause and fix; never silently retry

22.5 Chapter Summary

• E2E is the tip of the pyramid: few, precise, guarding critical user journeys;
• Three-tier marker system (smoke/critical/full) matches CI stages;
• API seeding + UI verification is the performancegolden standard;
• Trace + screenshot + HTML report form the complete failure forensics chain;
• Maintenance is ongoing: versions, tokens, snapshots, flakiness — all need periodic attention.

🧪 随堂测验

点击你认为正确的选项。答错时会展示正确答案与原因解析。

1. What is the recommended scale for E2E tests in a healthy pyramid?

2. Why is API seeding preferred over UI seeding for test data?

3. When should the smoke test set block a merge?

4. What is the first action when a previously-green test starts flaking?

🛠️ Hands-On Exercises

1. Classify your existing test suite into smoke/critical/full markers; run each tier separately and measure time allocation.
2. Replace one UI data-creation step in an existing test with API seeding; measure the speedup.
3. Simulate a CI failure; walk through the full forensics chain: JUnit → HTML report → trace replay → root-cause identification.

Course complete. You now have a production-grade E2E testing system built on Playwright Python.