Chapter 1 · Why Strong Models Still Fail
Chapter goals: Understand why the 50–60% pass rate on SWE-bench continues to drop on real tasks, and master the core definition of Harness Engineering and the five failure modes.
1.1 The Gap Between Benchmarks and Reality
At the end of 2025, the strongest coding agents on SWE-bench Verified achieved an average pass rate of 50–60%. At first glance this seems decent — but don't celebrate too early. These tasks are hand-picked: issue descriptions are clear, test cases are ready to run. Now swap in the real demands you face every day: vague specs, no tests, business rules scattered across the codebase. The pass rate on real tasks drops even further.
You confidently hand off a task, the agent runs for 20 minutes and tells you "done," but when you look at the code: the feature is added, tests are failing; a bug is fixed, new problems appear; worse, it implemented something completely different from what you wanted.
Most people's first reaction at this point is "the model isn't good enough — try a more expensive one." But before you open your wallet, consider: the problem may not be the model at all.
1.2 Anthropic's Controlled Experiment
Anthropic ran a controlled experiment that illustrates this point perfectly. Same prompt ("Build a 2D retro game editor"), same model (Opus 4.5), two runs.
First run: bare, no harness support. 20 minutes, $9, the game's core features simply don't work. Second run: full harness — planner + generator + evaluator three-agent architecture. 6 hours, $200, the game is fully playable.
The model didn't change. Opus 4.5 is still Opus 4.5. What changed is the harness.
OpenAI's Harness Engineering article puts it even more directly: in a well-configured repo, Codex can leap from "unreliable" straight to "reliable." Note the wording — not "a bit better," but a qualitative jump. The Harness here refers to all engineering infrastructure outside the model weights.
1.3 The Five Failure Modes
An agent's specific failure modes can be grouped into five categories:
1.3.1 Vague Requirements — the Agent Can Only Guess
"Add a search feature" — this says almost nothing. Search what? Full-text or structured query? Should results be paginated? Highlighted? You didn't make it clear, so the agent has to guess. Guessing right is luck; guessing wrong means rework, and the cost could be several times what it would have been to spell it out upfront.
1.3.2 Implicit Conventions Not Written Down — the Agent Has Nothing to Follow
Your whole team uses SQLAlchemy 2.0 syntax, but the agent defaults to writing 1.x code. All API endpoints must go through OAuth 2.0 authentication, but that rule only exists in your brain and a Slack message from three months ago. The agent doesn't refuse to comply — it has never seen this rule.
1.3.3 Incomplete Environment — the Agent Fixes the Environment Instead of Doing Work
An incomplete dev environment, missing dependencies, wrong tool versions — the agent's precious context window gets consumed by pip install errors and Node version conflicts instead of doing the actual work you assigned.
1.3.4 No Verification Method — the Agent Declares Done When It "Feels Finished"
No tests, no lint, or the verification commands were never told to the agent. The agent writes code, glances at it, thinks it looks fine, and announces completion. Anthropic also observed an interesting phenomenon: when the agent senses its context is about to run out, it exhibits rushed wrap-up behavior — hurrying to end current work, skipping verification steps, choosing the simple solution over the optimal one. Anthropic calls this context anxiety.
1.3.5 Cross-Session State Loss — Every New Session Starts from Zero
All findings from the last session are lost. Every new session has to re-explore the project structure and re-understand the code organization. Agents without persisted state see their failure rate spike sharply on tasks longer than 30 minutes.
1.4 Core Concepts
| Term | Meaning |
|---|---|
| Capability Gap | The large gap between a model's benchmark performance and its performance on real tasks. A 50–60% pass rate on SWE-bench means nearly half of real problems go unsolved |
| Harness | Everything outside model weights: instructions, tools, environment, state management, verification feedback. Anthropic refers to the Claude Agent SDK directly as a "general-purpose agent harness" |
| Harness-Induced Failure | The model has sufficient capability, but the execution environment has structural defects. Anthropic's controlled experiment has proven this |
| Verification Gap | The gap between an agent's confidence in its own output and the actual correctness. This is the most common failure mode |
| Diagnostic Loop | Execute → Observe failure → Attribute to a specific harness layer → Fix that layer → Re-execute |
| Definition of Done | A set of conditions verifiable by commands — tests pass, lint clean, type checks pass |
1.5 When It Fails, Fix the Harness First
There is one core principle: when something fails, don't swap the model first — check the harness. If the same model can succeed on a structurally sound similar task, assume it's a harness problem.
What does this mean in practice? Attribute each failure to a specific layer. Don't just say "the model isn't good enough." Ask yourself: Is the task unclear? Insufficient context? No verification method? Map to the five defense layers — task specification, context provision, execution environment, verification feedback, state management.
Then, write an explicit Definition of Done for each task:
Definition of Done:
- New endpoint GET /api/search?q=xxx
- Pagination supported, default 20 per page
- Results include highlighted snippets
- All new code passes pytest
- Type checks pass (mypy --strict)Place an AGENTS.md file at the repo root to tell the agent the project's tech stack, architecture conventions, and verification commands. This is the first step in harness engineering, and also the step with the highest ROI. An AGENTS.md may be more effective than upgrading to a more expensive model — this is not a joke.
1.6 Chapter Summary
- Strong models ≠ reliable execution; the 50–60% on SWE-bench comes from carefully chosen tasks, and the real-task pass rate is lower still;
- Anthropic controlled experiment: same model, same prompt, with harness vs. without harness — results are worlds apart;
- Five failure modes: vague requirements, unwritten implicit conventions, incomplete environment, no verification method, cross-session state loss;
- Core principle: when something fails, fix the harness first, not the model;
- Write an explicit Definition of Done for each task, and place an AGENTS.md.
🧪 随堂测验
点击你认为正确的选项。答错时会展示正确答案与原因解析。
1. According to Anthropic's controlled experiment, what is the main advantage of a full harness over a bare run with the same model and prompt?
2. What does "context anxiety" refer to?
3. Which of the following is NOT one of OpenAI's core works of Harness Engineering?
4. What is the core characteristic of a Definition of Done?
🛠️ Hands-on Practice
- Pick a project you've worked on and list the "implicit conventions the agent can't see" — the rules that only exist in verbal communication, Slack, or senior engineers' heads.
- Write a minimal
AGENTS.md(under 100 lines) for this project, including tech stack, directory structure, and verification commands. - Try using the
/goalcommand to have Claude Code or Codex complete a small task, and observe its behavior without an AGENTS.md.