Chapter 20 · Testing and Evaluation: crewai test/train

Chapter goals: Master the mechanics and scoring metrics of crewai test, the human-feedback training flow of crewai train, how to build a regression evaluation set, and the trade-offs of running evaluations in CI.

20.1 Why LLM Apps Need "Evaluation" Not Just "Testing"

Traditional unit tests assert deterministic output (assert add(1, 2) == 3), but Crew outputs are natural language — they differ on every run. Therefore CrewAI divides quality assurance into three layers:

Layer	Method	Question Answered
Unit tests	pytest for custom tools, input parsing	Is the function logic correct?
Smoke tests	crewai run / kickoff() manual run	Does the pipeline run end-to-end?
Evaluation	crewai test / train	Is the output quality up to standard and stable?

Distinction matters

crewai test is not a unit test framework — it uses an LLM as judge to score each task's output (1–10 points),属于 LLM-as-judge evaluation; tool function logic correctness still requires real unit tests written per Chapter 8.

20.2 crewai test: Multi-Round Runs and Scoring

Run inside a crewai create crew scaffolded project directory:

# Defaults to 2 rounds; judge model gpt-4o-mini
crewai test

# Specify rounds and judge model (shorthand)
crewai test -n 5 -m gpt-4o

Two parameters:

• --n-iterations (shorthand -n): number of repeated runs, default 2. The point of multiple rounds is observing stability — one 9-point run could be luck; five runs averaging 9 is reliable;
• --model (shorthand -m): OpenAI model serving as judge, default gpt-4o-mini (currently the test feature only supports OpenAI-family judge models).

Old syntax --n_iterations is still accepted but deprecated and hidden from --help; use --n-iterations in new code.

After running, you get a score table like this (higher is better):

Tasks Scores (1-10 Higher is better)
| Tasks/Crew/Agents  | Run 1 | Run 2 | Avg. Total |
| Task 1             |  9.0  |  9.5  |   9.2      |
| Task 2             |  9.0  | 10.0  |   9.5      |
| Crew               |  9.00 |  9.38 |   9.2      |
| Execution Time (s) |  126  |  145  |   135      |

Three interpretation points: task scores pinpoint which环节 drags performance; Crew total score shows the overall trend; execution time is a cost signal — when scores are flat but time skyrockets, check tool call counts first.

You can also trigger the same flow programmatically:

from my_project.crew import MyProjectCrew

inputs = {"topic": "AI Agent Framework Comparison"}

try:
    # Programmatic evaluation entry equivalent to CLI
    MyProjectCrew().crew().test(
        n_iterations=3,
        inputs=inputs,
        openai_model_name="gpt-4o-mini",
    )
except Exception as e:
    raise Exception(f"Evaluation execution failed: {e}")

20.3 crewai train: Calibrating Agents with Human Feedback

After evaluation surfaces quality issues, train provides a "human-teaches-machine" path: each round pauses for you to give feedback on each task's output, then the Agent improves, and the "first draft → feedback → improved version" loop is recorded and沉淀 for future runs.

# Train 2 rounds; result saved to trained_agents_data.pkl (default name)
crewai train -n 2

# Specify save file (absolute path supported)
crewai train -n 3 -f /data/models/marketing_crew.pkl

Data flow during training (official docs description):

1. Enter training mode: all tasks auto-set to human_input=True; delegation disabled;
2. Each iteration records a three-piece set: initial_output (first draft), your human_feedback, improved_output (improved version), written to working file training_data.pkl;
3. After all rounds complete, aggregate by Agent into three things stored in the -f specified file:
   • suggestions[]: executable improvement instructions distilled from feedback;
   • quality: improvement score 0–10;
   • final_summary:操作清单 for future tasks.

The key mechanism is the closed loop: in subsequent normal (non-training) runs, the Agent auto-loads the suggestions under its role name and appends them as mandatory instructions in the task prompt — you get consistent improvement without changing any Agent definition:

import os
from crewai import Agent, Task, Crew, Process, LLM

llm = LLM(
    model="openai/deepseek-chat",
    base_url="https://api.deepseek.com/v1",
    api_key=os.getenv("DEEPSEEK_API_KEY"),
)

# Assuming training was completed with crewai train -f marketing.pkl,
# and each Agent's role matches the pkl records
writer = Agent(
    role="Senior Writer",          # role must match training to hit suggestions
    goal="Write high-quality marketing copy",
    backstory="B2B copy expert with ten years experience",
    llm=llm,
)

task = Task(
    description="Write a promotional short essay on topic {topic}",
    expected_output="Chinese copy under 300 words",
    agent=writer,
)
crew = Crew(agents=[writer], tasks=[task], process=Process.sequential)

print(crew.kickoff(inputs={"topic": "Smart Customer Service"}).raw)
# writer's prompt now auto-includes the improvement instructions沉淀 in trained_agents_data.pkl

Two pitfalls

① training_data.pkl is a temporary process file; what actually takes effect is trained_agents_data.pkl (or the file you specified with -f) — do not confuse them; ② training requires terminal interactive feedback input — cannot be run directly in non-interactive CI/containers.

20.4 Building Your Own Regression Evaluation Set

The crewai test judge model's scoring has randomness; production teams typically add a layer of deterministic-assertion regression sets. Approach: prepare fixed inputs + programmable-verify expected features; re-run and compare after every change:

import os
import re
import pytest
from my_project.crew import MarketingCrew

# Regression evaluation set: input -> structured checker (does not compare full text; asserts measurable features only)
EVAL_CASES = [
    {"topic": "Enterprise WeChat Customer Acquisition", "must_mention": ["WeChat"], "max_len": 500},
    {"topic": "SaaS Pricing Page Optimization", "must_mention": ["pricing"], "max_len": 400},
]


@pytest.mark.parametrize("case", EVAL_CASES)
def test_marketing_crew_regression(case):
    """Marketing Crew regression eval: keyword coverage + length constraint"""
    result = MarketingCrew().crew().kickoff(inputs={"topic": case["topic"]})
    text = result.raw

    for kw in case["must_mention"]:
        assert kw in text, f"Output missing required keyword: {kw}"
    assert len(text) <= case["max_len"], f"Output over length: {len(text)} > {case['max_len']}"
    # Output must be coherent Chinese sentences, not empty/noise/error messages
    assert re.search(r"[。！？]", text), "Output appears to be non-normal text"


def test_crew_cost_budget():
    """Cost guardrail: one kickoff's token usage must not exceed budget"""
    result = MarketingCrew().crew().kickoff(inputs={"topic": "Email Marketing"})
    usage = result.token_usage  # total_tokens field aggregates full Crew usage
    assert usage.total_tokens < 50_000, f"Token over budget: {usage.total_tokens}"

Design principle: assert features not full text (keywords, structure, length, language, JSON parsability); turn usage_metrics/token_usage into guardrails to prevent prompt changes from silently inflating costs.

20.5 Trade-offs of Running Evaluations in CI

Approach	Pros	Cons	Recommendation
Unit tests only	Fast, cheap, stable	Does not cover output quality	Mandatory on every PR
Unit tests + regression set (fixed inputs)	Can catch structural regression	Consumes real tokens, slow	Run before merging to main
crewai test -n N (LLM judge)	Directly measures quality	More expensive, judge noise, needs OpenAI key	Nightly scheduled / pre-release gate

落地建议: PR pipelines only run lint + tool unit tests; nightly jobs run the regression set and alert when scores drop below threshold; crewai test serves as a release-gate "acceptance checkpoint" with manual score-table review before放行. Judge model noise can be suppressed by increasing -n rounds to take the mean — but budget the cost carefully.

Minimal nightly alert script:

# nightly_eval.py — cron runs nightly: regression set, non-zero exit if threshold breached
import subprocess
import sys

THRESHOLD = 8.0

proc = subprocess.run(
    ["pytest", "tests/test_regression.py", "-q", "--tb=short"],
    capture_output=True, text=True,
)
print(proc.stdout)

if proc.returncode != 0:
    print("Regression set failed, blocking release")
    sys.exit(1)

# Optional: parse crewai test score table and compare against THRESHOLD,
# alert via DingTalk/Slack webhook when breached

Chapter Summary

• crewai test -n rounds -m judge_model runs multiple rounds and scores with an LLM; outputs per-task and per-Crew score tables plus execution time;
• crewai train -n -f iterates through "first draft → human feedback → improved version" to沉淀 suggestions into a pkl file; normal runs auto-inject them into prompts;
• Production-grade approach: build your own regression evaluation set — fixed inputs + feature assertions + token guardrails;
• CI分层: PRs run only unit tests; nightly runs regression set; release-gate runs LLM-judge evaluation.

🧪 随堂测验

点击你认为正确的选项。答错时会展示正确答案与原因解析。

1. What are crewai test's default judge model and number of rounds?

2. Regarding the two pkl files produced by crewai train, which is correct?

3. Why can crewai train not be put directly into an unattended CI?

4. When building a regression evaluation set, which assertion style is least maintainable?

🛠️ Hands-On Exercises

1. Run crewai test -n 3 on the CLI project created in Chapter 17; record average score per task and total time; find the lowest-scoring task and analyze the cause.
2. Follow the pattern in Section 20.4 to write a regression eval file for your project with at least 3 cases (pytest parametrized + keyword/length assertions + token guardrail).
3. Run crewai train -n 2 -f my_feedback.pkl; during training, intentionally give specific feedback like "too verbose, compress to under 200 words"; after training, re-run the Crew and observe whether the output style changes; inspect the pkl's suggestions content with pickle.

Next chapter makes every cent and every millisecond of your Crew's execution visible: Chapter 21 · Observability and Monitoring Integration.