Chapter 15 · Faux Provider and Unit Testing

Chapter goals: Write scripted LLM responses with fauxProvider(), perform deterministic testing of Agent loops in vitest, and eliminate real API dependencies.

15.1 Why LLM Tests Need Faux

Writing unit tests against a real API has three fatal problems: slow (second-level responses), expensive (token-based billing), and non-deterministic (same input, different outputs, impossible to assert). pi-ai's built-in fauxProvider() solves all of these — a fake provider in memory that returns responses according to a script you编排:

import {
  createModels,
  fauxAssistantMessage,
  fauxProvider,
  fauxText,
  fauxThinking,
  fauxToolCall,
} from '@earendil-works/pi-ai';

// Create a faux provider; optionally specify simulated output speed
const faux = fauxProvider({
  tokensPerSecond: 50, // Optional: simulate streaming speed
});

const models = createModels();
models.setProvider(faux.provider);

// Get the model object attached to this provider
const model = faux.getModel();

15.2 Scripting Responses

faux.setResponses() lays out a sequence of preset responses in call order; each request pops the next one. You can mix text, thinking blocks, and tool calls arbitrarily:

const context = {
  messages: [{
    role: 'user',
    content: 'Summarize package.json and then call echo',
    timestamp: Date.now(),
  }],
};

// Round 1: model thinks first, then issues a tool call
faux.setResponses([
  fauxAssistantMessage([
    fauxThinking('Need to inspect package metadata first.'),
    fauxToolCall('echo', { text: 'package.json' }),
  ], { stopReason: 'toolUse' }), // Crucial: declare that we are waiting for tool results
]);

const first = await models.complete(model, context, {
  sessionId: 'session-1',
  cacheRetention: 'short',
});
context.messages.push(first);

// We act as the real tool execution and feed the result back into context
context.messages.push({
  role: 'toolResult',
  toolCallId: first.content.find((b) => b.type === 'toolCall')!.id,
  toolName: 'echo',
  content: [{ type: 'text', text: 'package.json contents here' }],
  isError: false,
  timestamp: Date.now(),
});

15.3 Multi-Round Scripts and Streaming Assertions

Continue scripting round 2 — the model "sees" the tool result and gives a summary. Streaming events can同样 be fully consumed and asserted:

// Round 2: model summarizes based on the tool result
faux.setResponses([
  fauxAssistantMessage([
    fauxThinking('Now I can summarize the tool output.'),
    fauxText('Here is the summary.'),
  ]),
]);

// Consume the stream for the entire loop
const s = models.stream(model, context);
for await (const event of s) {
  console.log(event.type); // message_start → delta... → message_end
}

The value of the scripting mechanism is exact reproduction of multi-round tool loops: tool call → result feedback → regeneration — the most core path of the Agent can be validated in milliseconds.

15.4 Testing Agent Loops in vitest

Combining the Faux Provider with the Agent class lets you write assertions against complete Agent behavior:

// agent.test.ts
import { describe, it, expect } from 'vitest';
import { Agent } from '@earendil-works/pi-agent-core';
import { createModels, fauxAssistantMessage, fauxProvider, fauxText } from '@earendil-works/pi-ai';

describe('coding assistant agent', () => {
  it('should answer with scripted response', async () => {
    const faux = fauxProvider({ tokensPerSecond: 1000 }); // Speed up tests
    const models = createModels();
    models.setProvider(faux.provider);

    // Preset model response
    faux.setResponses([
      fauxAssistantMessage([fauxText('Hello from mock!')]),
    ]);

    const agent = new Agent({
      initialState: {
        systemPrompt: 'test',
        model: faux.getModel(),
        tools: [],
        messages: [],
      },
      streamFn: models.streamSimple.bind(models),
    });

    // Collect events for assertions
    const events: string[] = [];
    agent.subscribe((event) => events.push(event.type));

    await agent.prompt('Say hi');

    expect(events).toContain('agent_start');
    expect(events).toContain('agent_end');
  });
});

Testing pyramid recommendation

Use Faux to覆盖 95% of logic tests (fast, free, deterministic); leave only a few end-to-end tests hitting the real API to verify integration correctness.

Chapter Summary

• Real API testing is slow, expensive, and non-deterministic; Faux Provider provides a deterministic in-memory alternative;
• faux.setResponses() sequences responses in order; can混合 thinking/text/toolCall blocks;
• Manually construct toolResult feedback into context to reproduce the complete tool loop;
• Faux + vitest can make precise assertions on the Agent class's event sequence;
• Strategy: Faux covers the vast majority of tests; a few E2E tests hit the real API.

🧪 随堂测验

点击你认为正确的选项。答错时会展示正确答案与原因解析。

1. What is the primary reason for using a Faux Provider in LLM unit tests?

2. How does faux.setResponses([...]) work when multiple responses are set?

3. In a Faux scenario, after simulating the model issuing a tool call, how should the tool result be handled?

4. Which of the following is the officially recommended testing strategy?

🛠️ Hands-on Practice

1. Write a two-round scripted test for the file-reading tool from Chapter 6: round 1 issues a read tool call; round 2 summarizes the content.
2. Assert the complete event order of the Agent in vitest: agent_start → turn_start → ... → agent_end.
3. Add test cases for parameter validation failures in your tool functions (Faux returns incorrectly formatted parameters).

Next we explore a unique capability of pi-ai: switching models mid-conversation.