📚 AI 工程与智能体教程站

English

简体中文

English

简体中文

Chapter 22 · Capstone Project 2: Test Agent — Automated Execution and Reporting ​

Chapter goals: Move the test agent pipelines from Chapters 18–19 into GitHub Actions — auto-run smoke sets on push, full regression nightly; identify regression defects with baseline snapshots and auto-file Issues; push Slack summaries; keep cost and stability in check for CI scenarios.

22.1 Three Changes from Local Scripts to CI Tasks ​

The pipelines from the previous chapters run manually in the terminal; entering CI introduces three fundamental changes:

DimensionLocal RunCI Run
InteractionTerminal streaming output for humansNon-interactive; artifacts land on disk for machines to read
Result formatMarkdown reportJUnit XML (natively parsed and displayed by CI)
Failure semanticsStop after runningRegression判定: a failure is a real defect only if it passed last time

Core principle: in CI the Agent is an "unattended employee" — it must be reproducible, have a budget cap, and produce machine-readable reports.

22.2 Programmatic Entry: runPipeline and JUnit Output ​

Refactor the CLI logic in main.ts into importable functions and add JUnit report generation:

typescript
// pipeline/ci.ts —— CI programmatic entry
import { readFile, writeFile } from "node:fs/promises";
import { parseRequirements, reviewRequirements } from "./stages1-2.js";
import { makeTestPlan } from "./stage3.js";
import { generateCases } from "./stage4.js";
import { executeAll, type TestResult } from "../executor/index.js";

/** JUnit XML escaping: quotes and angle brackets in attribute values would break XML structure */
function esc(s: string): string {
  return s.replace(/&/g, "&amp;").replace(/</g, "&lt;")
          .replace(/>/g, "&gt;").replace(/"/g, "&quot;");
}

/** Serialize TestResult[] to JUnit XML recognizable by GitHub Actions */
export function toJUnitXml(suiteName: string, results: TestResult[]): string {
  const cases = results.map((r) => `
    <testcase name="${esc(r.caseId)}" classname="${esc(suiteName)}" time="${(r.durationMs / 1000).toFixed(2)}">
      ${r.status === "failed" ? `<failure message="${esc(r.error ?? "assertion failed")}">${esc(r.evidence ?? "")}</failure>` : ""}
      ${r.status === "skipped" ? `<skipped/>` : ""}
    </testcase>`).join("\n");
  const failed = results.filter((r) => r.status === "failed").length;
  return `<?xml version="1.0" encoding="UTF-8"?>
<testsuite name="${esc(suiteName)}" tests="${results.length}" failures="${failed}">
${cases}
</testsuite>`;
}

/**
 * Full pipeline: requirements → cases → execute → JUnit report.
 * mode=smoke runs only P0 cases (push-triggered); mode=full runs everything (scheduled).
 */
export async function runPipeline(opts: {
  requirementPath: string;
  mockups?: string[];
  mode: "smoke" | "full";
}) {
  const doc = await readFile(opts.requirementPath, "utf-8");
  const points = await parseRequirements(doc);
  const review = await reviewRequirements(points);
  if (!review.passed) throw new Error(`Requirement review not passed: ${review.blocker}`);
  const plan = await makeTestPlan(points, review);

  let cases = await generateCases(points, review, plan, opts.mockups);
  if (opts.mode === "smoke") {
    // Smoke set keeps only P0 to control pipeline duration and cost on push
    cases = cases.filter((c) => c.priority === "P0");
  }

  const results = await executeAll(cases);
  const failed = results.filter((r) => r.status === "failed").length;
  console.log(`::error title=TestFailures::${failed} cases failed`);   // Actions annotation
  return { results, junit: toJUnitXml(`test-agent-${opts.mode}`, results) };
}

::error is the GitHub Actions workflow command syntax; it annotates the summary directly on the PR's Checks panel.

22.3 GitHub Actions: Dual-Trigger Workflow ​

A single workflow file handles both triggers: push runs smoke; cron runs nightly full regression.

yaml
# .github/workflows/test-agent.yml
name: AI Test Agent

on:
  push:
    branches: [main]
    paths: ["src/**", "docs/requirements/**"]   # Trigger only on code or requirement changes
  schedule:
    - cron: "0 18 * * *"   # UTC 18:00 = 2 AM Beijing time, full regression

jobs:
  smoke:                          # ① Push-triggered: P0 smoke set
    if: github.event_name == 'push'
    runs-on: ubuntu-latest
    timeout-minutes: 30           # Hard timeout to防止 Agent infinite loops burning money
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with: { node-version: 22, cache: npm }
      - run: npm ci && npx playwright install --with-deps chromium
      - name: Run smoke pipeline
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
          MAX_TURNS: "12"                       # Limit turns in CI (see 22.6)
        run: npx tsx pipeline/ci-entry.ts smoke docs/shopping-cart.md
      - uses: actions/upload-artifact@v4        # Upload artifacts regardless of success or failure
        if: always()
        with: { name: junit-smoke, path: reports/junit-smoke.xml }

  nightly:                        # ② Scheduled-triggered: full regression + inspection
    if: github.event_name == 'schedule'
    runs-on: ubuntu-latest
    timeout-minutes: 60
    steps:
      - uses: actions/checkout@v4
        with: { fetch-depth: 0 }                # Need historical commits to read baseline snapshots
      - uses: actions/setup-node@v4
        with: { node-version: 22, cache: npm }
      - run: npm ci && npx playwright install --with-deps chromium
      - name: Full regression
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}   # For auto-filing Issues
          SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_URL }}
        run: npx tsx pipeline/ci-entry.ts full docs/shopping-cart.md
      - name: Publish test report
        uses: mikepenz/action-junit-report@v4   # Parse JUnit and display on the Checks page
        if: always()
        with: { report_paths: reports/junit-*.xml }

Key points: both jobs set timeout-minutes and if: always() artifact upload — Agent-class tasks must有硬超时兜底.

22.4 Regression Inspection: Baseline Snapshots and Auto-Filed Issues ​

The key to full regression is not "whether there are failures" but "whether they are new failures". Use baseline snapshots for diffing:

typescript
// pipeline/regression.ts —— Baseline comparison + auto-file Issues
import { readFile, writeFile } from "node:fs/promises";
import { Octokit } from "octokit";
import type { TestResult } from "../executor/types.js";

type Baseline = Record<string, "passed" | "failed">;   // caseId -> last status

export async function loadBaseline(): Promise<Baseline> {
  try { return JSON.parse(await readFile("reports/baseline.json", "utf-8")); }
  catch { return {}; }                                  // First run: no baseline
}

export async function saveBaseline(results: TestResult[]) {
  const b: Baseline = {};
  for (const r of results) b[r.caseId] = r.status === "passed" ? "passed" : "failed";
  await writeFile("reports/baseline.json", JSON.stringify(b, null, 2));
}

/** Diff classification: new failure = regression defect; persistent failure = existing issue; recovery = good news */
export function classify(results: TestResult[], baseline: Baseline) {
  return {
    regressions: results.filter(r => r.status === "failed" && baseline[r.caseId] === "passed"),
    chronic:     results.filter(r => r.status === "failed" && baseline[r.caseId] !== "passed"),
    recovered:   results.filter(r => r.status === "passed" && baseline[r.caseId] === "failed"),
  };
}

/** Auto-file Issues for regression defects with evidence links; avoids manual copy-paste */
export async function fileRegressionIssues(regressions: TestResult[]) {
  const octokit = new Octokit({ auth: process.env.GITHUB_TOKEN });
  for (const r of regressions) {
    await octokit.rest.issues.create({
      owner: process.env.GITHUB_REPOSITORY!.split("/")[0],
      repo:  process.env.GITHUB_REPOSITORY!.split("/")[1],
      title: `[Regression Defect] ${r.caseId} passed last time, failed this time`,
      labels: ["regression", "ai-test-agent"],
      body: [
        `### Failure Reason`, "```", r.error ?? "N/A", "```",
        `### Evidence`, r.evidence ?? "None",
        ``, `> Auto-created by the AI test agent nightly inspection`,
      ].join("\n"),
    });
  }
}

Note the distinction in handling strategies for two kinds of failures: regression defects auto-file Issues immediately (someone broke something); chronic persistent failures are only汇总 in weekly reports to avoid Issue spam.

22.5 Result Notification: Slack Summary ​

Push a summary that a human can读完 in 10 seconds after the inspection:

typescript
// pipeline/notify.ts —— Slack webhook notification
export async function notifySlack(summary: {
  total: number; passed: number; regressions: number; recovered: number; runUrl: string;
}) {
  const emoji = summary.regressions > 0 ? "🔴" : "🟢";
  const text = [
    `${emoji} *Nightly regression inspection complete*`,
    `• Total ${summary.total}, ${summary.passed} passed`,
    summary.regressions ? `• ⚠️ ${summary.regressions} new regression defects (Issues auto-filed)` : "• No new regressions ✅",
    summary.recovered ? `• 🎉 ${summary.recovered} historical failures recovered` : "",
    `• Details: <${summary.runUrl}|View Actions log>`,
  ].filter(Boolean).join("\n");

  // Slack Incoming Webhook: POST a Block Kit payload to send a message
  await fetch(process.env.SLACK_WEBHOOK_URL!, {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({ text }),
  });
}

Only regression defects > 0 or recoveries deserve to打扰 a human; a perfectly normal night can stay silent or send just a green one-liner.

22.6 Cost and Stability: Three保险 for Unattended Operations ​

In CI nobody is watching the terminal; an out-of-control Agent costs real money. Three lines of defense:

typescript
// executor/guard.ts —— Agent execution guardrails
import { Agent } from "@earendil-works/pi-agent-core";

export async function guardedRun(agent: Agent, prompt: string, opts: {
  maxTurns: number;       // ① Turn cap: max 8 tool calls per case
  retries: number;        // ② Failure retries: only infrastructure-class errors warrant retry
}) {
  for (let attempt = 1; attempt <= opts.retries; attempt++) {
    try {
      // maxTurns is injected via env var; CI and local can use不同的 tiers
      const cap = Number(process.env.MAX_TURNS ?? opts.maxTurns);
      return await agent.prompt(prompt, { maxTurns: cap });
    } catch (err: any) {
      const isInfra = /timeout|ECONNRESET|429/.test(err.message);   // Network-class errors
      if (!isInfra || attempt === opts.retries) throw err;
      await new Promise((r) => setTimeout(r, attempt * 5000));      // Exponential backoff
    }
  }
}
GuardMechanismEffect
① Turn capMAX_TURNS env var injected into prompt configPer-case cost封顶
② Selective retryOnly retry network/rate-limit errors; never retry assertion failuresAssertion failures retried只会浪费钱
③ Session cacheWhen the same requirement document hash has not changed, reuse cases.json and skip generationNightly inspection saves ~60% LLM calls

Item ③ is straightforward to实现: compare sha256(requirements.md) against the hash in the cached file; if identical, readFile("cases.json") directly — if the requirement hasn't changed, the cases won't either.

Chapter Summary ​

  • In CI, Agents must be非交互化: artifacts land on disk, output JUnit XML, write back to the Checks panel with ::error;
  • Dual-trigger design: push runs P0 smoke to控制 duration; cron nightly full regression to保 coverage;
  • Regression判定靠 baseline snapshot diffing: new failures only file Issues; chronic failures go to weekly reports; avoid告警疲劳;
  • Three unattended guards: MAX_TURNS caps cost, selective retry, requirement hash cache saves repeated generation;
  • Notifications should be "worth disturbing": push only on regression or recovery; keep summaries within 10 seconds to读.

🧪 随堂测验

点击你认为正确的选项。答错时会展示正确答案与原因解析。

1. Why output test reports as JUnit XML in CI?

2. In nightly full regression, a case "passed last time, failed this time" — how should it be handled?

3. Regarding failure retry strategy in CI, what is the correct approach?

4. How much LLM calls can the "requirement hash cache" optimization save during nightly inspections?

🛠️ Hands-on Practice ​

🛠️ Hands-on Practice ​

  1. Write .github/workflows/test-agent.yml for your project: push runs only the P0 smoke set; configure a 30-minute硬 timeout and MAX_TURNS=10; observe the JUnit report rendering on the Actions panel.
  2. Implement baseline snapshot diffing: intentionally change the expected status code in one API test to制造 a "regression"; verify that an Issue is auto-created with the failure reason and evidence. 3.接入 a Slack webhook;改造 notifySlack to silently skip sending when "all passed", and only push when regressions or recoveries exist; observe消息 quality over three consecutive nights.

The three-chapter capstone project is complete. You now have a full AI test pipeline from requirement document to CI gate — return to course overview to review the full journey, or deploy it to your own project and start monitoring.