Chapter 13 · Advanced Techniques: ToT and Prompt Chaining

Chapter goals: Understand the root cause of "the model is strong but execution is unreliable" — capability gap and harness-induced failure — and learn to use the diagnostic loop to locate the problem layer, laying the cognitive foundation for subsequent harness engineering.

13.1 The Truth About SWE-bench: What Does 50%–60% Really Mean?

At the end of 2025, the strongest coding agents on the SWE-bench Verified benchmark achieved a pass rate of about 50%–60%. This number looks decent at first glance — but don't celebrate yet.

This benchmark set has several hidden premises:

• The questions are hand-picked GitHub issues with clear descriptions; -配套 test cases are already written, verification criteria are explicit;
• The repo context is relatively clean, without ten years of technical debt.

Now swap in a real工作 scenario — vague requirement docs, implicit conventions scattered across the codebase, no test coverage — and the pass rate drops even lower. Both Anthropic and OpenAI have reported: in actual engineering team projects, even the strongest models without a harness often have success rates below 20%.

This is the capability gap — the massive落差 between a model's benchmark performance and its performance on real tasks.

Core Argument

The model not being strong enough is not the problem; an incomplete harness is. The same model, placed in different execution environments, can yield wildly不同的 results.

13.2 The Same Model, Two Different Fates

Anthropic ran a controlled experiment, running the same sentence ("build a 2D retro game editor") and the same model (Opus 4.5) twice:

Run	Duration	Cost	Result
Bare run	20 min	$9	Core features unusable
Full harness (planner + generator + evaluator)	6 hr	$200	Game fully playable

The model didn't change, the prompt didn't change — what changed is the harness, i.e. the engineering infrastructure surrounding the model.

OpenAI's 2025 harness engineering article uses an even more direct word: Codex in a repo with a complete harness doesn't perform "a bit better" — it jumps from "unreliable" straight to "reliable", which is a qualitative change.

13.3 The Five Failure Modes

When an agent says "done" but the result is unusable, the problem usually falls into one of the following five layers. Remember this checklist — every time something fails, first locate the layer, then think about how to fix it:

① Vague requirements (Task Specification)

"Add a search feature" — search what? Full-text or structured query? Should results be paginated? Highlighted? The agent can only guess; guessing right is luck, guessing wrong means rework costs several times more than spelling it out clearly.

② Unrecorded implicit conventions (Context Provision)

The team uniformly uses SQLAlchemy 2.0 syntax, but the agent defaults to writing 1.x; all APIs must go through OAuth 2.0 auth, but that rule only exists in your team's Slack history. The agent doesn't refuse to comply — it simply never saw that rule.

③ Incomplete environment (Execution Environment)

Dependencies not fully installed, Node version wrong, test command path incorrect — the agent spends its precious context budget on pip install errors and version conflicts instead of the actual work.

④ No verification method (Verification Feedback)

No tests, no lint, or the verification commands were never told to the agent. The agent writes code, glances at it, thinks "looks good enough," and declares done. Anthropic observed an interesting phenomenon: when the agent senses context is running low, it exhibits "anxious wrap-up" (context anxiety) — skipping verification steps, choosing simple solutions, rushing to finish.

⑤ Cross-session state loss (State Management)

Every new session starts from scratch. All analysis, decision rationale, and modified files from the previous round are lost. Without persistent state records, long tasks exceeding 30 minutes see a sharp spike in failure rate.

13.4 Key Terminology: Definition of Done

The remedy for the five failure modes is a new concept: Definition of Done (DoD).

A DoD is a set of conditions verifiable by command line — not subjective judgment. For example:

## Definition of Done
- [ ] New endpoint `GET /api/search?q=xxx` returns JSON
- [ ] Pagination supported, default 20 per page
- [ ] Search results highlight keywords
- [ ] All new code passes `pytest tests/`
- [ ] Type checks pass `mypy src/ --strict`

Note that all these conditions are executable — each condition maps to a command. An agent without a DoD will invent its own "I think I'm done" criteria, and those criteria are almost always overly optimistic.

13.5 The Diagnostic Loop: Every Failure Is an Improvement Opportunity

The core methodology of harness engineering is a closed loop — the diagnostic loop:

Execute → Observe failure → Attribute to specific layer → Fix that layer → Re-execute

Don't just stop at "the model isn't strong enough." Every time you encounter a failure, ask yourself:

1. Is the task unclear? → supplement the requirement description in AGENTS.md
2. Is context insufficient? → add tech stack versions, architecture constraints, relevant doc links
3. Is the environment non-reproducible? → pin with pyproject.toml / .nvmrc / Docker
4. Is verification feedback missing? → write make check commands in AGENTS.md
5. Is state lost? → introduce PROGRESS.md or similar state persistence files

After each attribution, fix one thing, then run again. After a few rounds, you'll clearly see where the bottleneck is, and you can focus your energy there.

Keep the records simple — just note "success/failure" and "which layer failed" per task; the bottleneck naturally emerges as data accumulates.

Chapter Summary

• SWE-bench 50%–60% is a carefully selected benchmark; in real scenarios, bare-run success rates are often below 20%;
• Same model, different harness experiment: bare run 20min/$9 vs. full harness 6h/$200 — results are worlds apart;
• Five failure modes: vague requirements, unrecorded implicit conventions, incomplete environment, no verification method, cross-session state loss;
• Definition of Done must be verifiable by executable commands, not subjective feeling;
• The diagnostic loop is the core methodology: failure → attribute to a layer → fix → re-run, iterate.

🧪 随堂测验

点击你认为正确的选项。答错时会展示正确答案与原因解析。

1. In Anthropic's controlled experiment, what was the result of the same model Opus 4.5 running bare for 20 minutes at $9?

2. What is the core characteristic of a "Definition of Done"?

3. What does Anthropic call the phenomenon when the agent senses context is about to run out?

4. Which of the following is the correct execution order of the "diagnostic loop"?

🛠️ Hands-on Practice

1. Pick a project you're familiar with (or create a small one), let Claude Code / Codex complete a task without any AGENTS.md, record the failure symptoms, then add AGENTS.md and run again; compare the two success rates.
2. Write a Definition of Done for your project by hand, requiring at least 5 verifiable-by-command conditions (tests, type checks, lint, builds, etc.).
3. Review a recent agent failure experience, attribute each failure to the five failure mode checklist, and write out exactly which layer needs to be supplemented.

After completing, proceed to Next chapter: What Exactly Is a Harness.