Chapter 15 · Evals: Evaluation System

Chapter goals: Understand the Scorer mechanism, learn to use内置评估指标, write custom Scorers, and integrate evals into CI pipelines.

15.1 Why Traditional Tests Fall Short

Traditional tests have deterministic pass/fail; AI outputs are非确定性的 — the same question asked twice yields different wording but both correct. Mastra bridges this gap with Scorers: scoring outputs on a 0–1 numerical scale, quantifying "how good" an answer is.

Three Scorer implementation routes:

• Model-graded — use an LLM to score per criteria;
• Rule-based — keywords / regex / structure checks;
• Statistical — similarity and other数学 metrics.

15.2 Installation and Built-in Scorers

npm install @mastra/evals@latest

// src/mastra/agents/evaluated-agent.ts
import { Agent } from '@mastra/core/agent'
import {
  createAnswerRelevancyScorer,
  createToxicityScorer,
} from '@mastra/evals/scorers/prebuilt'

export const evaluatedAgent = new Agent({
  id: 'evaluated-agent',
  name: 'Evaluated Agent',
  instructions: 'You are a customer support assistant.',
  model: 'openai/gpt-5-mini',
  scorers: {
    // Key is a custom name for识别 in reports
    relevancy: {
      scorer: createAnswerRelevancyScorer({ model: 'openai/gpt-5-mini' }),
      weight: 0.7, // Weighted aggregate
    },
    toxicity: {
      scorer: createToxicityScorer({ model: 'openai/gpt-5-mini' }),
      weight: 0.3,
    },
  },
})

After mounting, each generate() call异步 scores; results are viewable in Studio and storage.

15.3 Custom Scorers

When built-in metrics don't match业务口径:

// src/mastra/scorers/policy.ts
import { createScorer } from '@mastra/core/evals'

// Rule型: the answer must contain the ticket format TK-XXXX
export const ticketFormatScorer = createScorer({
  name: 'ticket-format',
  description: 'Check whether the reply contains a规范 ticket number',
})
  .generateScore(({ run }) => {
    const text = run.output?.text ?? ''
    return /TK-\d{4,}/.test(text) ? 1 : 0
  })

// Model-graded型: have the LLM judge whether the tone符合品牌要求
export const toneScorer = createScorer({
  name: 'brand-tone',
  description: 'Is the tone friendly and professional?',
})
  .generateScore(({ run }) => {
    // Internally calls the configured model to score per the描述
    return judge(run.output?.text ?? '', 'friendly, professional, no imperative语气')
  })

15.4 Running Scorers Standalone in Scripts

No agent挂载 needed — score任意文本, suitable for批量回放历史数据:

// scripts/score-once.ts
import { ticketFormatScorer } from '../src/mastra/scorers/policy'

const result = await ticketFormatScorer.run({
  input: { text: 'Your ticket number is TK-20240801, please check.' },
  output: { text: 'Received, thanks for the feedback!' },
})
console.log(result.score) // 0 or 1

15.5 Running in CI

Treat evals as regression tests: after changing a prompt,自动跑 a batch of用例; fail the build if the score drops below threshold.

# .github/workflows/evals.yml
name: Agent Evals

on: [pull_request]

jobs:
  evals:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 20
      - run: npm ci
      - run: npx mastra evals run
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}

15.6 Chapter Summary

• Scorers量化 AI 输出质量, bridging the gap between deterministic tests and非确定性的 LLM output;
• Three routes: model-graded, rule-based, statistical;
• Custom Scorers via createScorer() + .generateScore();
• Standalone scoring for batch回放; CI integration as regression tests.

🧪 随堂测验

点击你认为正确的选项。答错时会展示正确答案与原因解析。

1. Why can't traditional pass/fail tests evaluate AI outputs directly?

2. What are the three Scorer implementation routes in Mastra?

3. What does a custom Scorer's `.generateScore()` callback receive?

4. How do you integrate evals into CI?

🛠️ Hands-On Exercises

1. Add two内置 scorers (relevancy + toxicity) to a support Agent; run three测试对话 and查看 the Studio scores.
2. Write a custom rule-based Scorer that checks for a specific keyword in responses; test on 10 sample inputs.
3. Add the eval job to your .github/workflows/ and verify it fails when a promptchange drops the score below threshold.

Next: Chapter 16 · Observability