Chapter 12 · Image Input and Image Generation

Chapter goals: Learn to send image inputs to vision models, use image generation capabilities, and implement multimodal interactions in Agents.

12.1 Image Input Basics

Vision-capable models can process images. As with thinking capability, check the model property first; mismatched models are silently ignored:

import { readFileSync } from 'fs';
import { createModels } from '@earendil-works/pi-ai';

const models = createModels();
const model = models.getModel('openai', 'gpt-4o-mini')!;

// Check whether the model supports image input via the input property
if (model.input.includes('image')) {
  console.log('Model supports vision');
}

// Read a local image and convert to base64
const imageBuffer = readFileSync('image.png');
const base64Image = imageBuffer.toString('base64');

12.2 Sending Messages with Images

A user message's content can be a string or a content block array — mix text blocks and image blocks together to achieve "look at the image and talk":

const response = await models.complete(model, {
  messages: [{
    role: 'user',
    timestamp: Date.now(),
    content: [
      // Text block: ask a question
      { type: 'text', text: 'What is in this image?' },
      // Image block: base64 data + MIME type
      { type: 'image', data: base64Image, mimeType: 'image/png' },
    ],
  }],
});

// Iterate over response content blocks
for (const block of response.content) {
  if (block.type === 'text') {
    console.log(block.text);
  }
}

Sending images to a non-vision model does not error

If the target model's input does not include image, the image is silently ignored — the model sees only the text. Production code should explicitly check capability or add a routing fallback.

12.3 Sending Images in Agents

Agent.prompt()'s second parameter accepts an image attachment array directly:

import { Agent } from '@earendil-works/pi-agent-core';

const agent = new Agent({
  initialState: {
    systemPrompt: 'You are an image-description assistant; describe the image content in Chinese.',
    model,
    tools: [],
    messages: [],
  },
  streamFn: models.streamSimple.bind(models),
});

// Send text + image together as a single prompt
await agent.prompt('What is in this image?', [
  { type: 'image', data: base64Image, mimeType: 'image/jpeg' },
]);

Multiple images are also supported — place multiple image blocks in the array, and the model will understand them all.

12.4 Image Generation

Besides "reading" images, pi-ai also wraps image generation capability. The direction is reversed: text goes in, image data comes out. Typical flow:

// Image generation: the response's content contains an image data block
const gen = await models.complete(genModel, {
  messages: [{
    role: 'user',
    content: 'Draw a cat writing TypeScript, flat illustration style',
    timestamp: Date.now(),
  }],
});

for (const block of gen.content) {
  if (block.type === 'image') {
    // Save the generated image to disk
    writeFileSync('cat.png', Buffer.from(block.data, 'base64'));
  }
}

When selecting a generation model, also confirm via catalog lookup that the model has image output capability, avoiding routing generation tasks to pure-text models.

12.5 Multimodal Practice Recommendations

A few tips for building real multimodal applications:

1. Compress before uploading — resize large images to a reasonable resolution (e.g. longest side within 1568px) to save tokens and run faster;
2. Ask specifically — "describe this image" is less stable than "how many red cars are in this picture?";
3. Error fallback — prepare a pure-text fallback (e.g. OCR result concatenation) for models that do not support vision;
4. Cost control — images are billed by token at a higher rate than text; do not send HD originals unless necessary.

A practical capability-routing function:

// Simple router that auto-selects a model based on capability
function pickVisionModel(models: Models, preferCheaper = true) {
  const candidates = ['gpt-4o-mini', 'claude-sonnet-4-5', 'gemini-2.5-flash'];
  const found = candidates
    .map((name) => /* look up per provider */ models.getModel('openai', name))
    .filter((m): m is NonNullable<typeof m> => !!m && m.input.includes('image'));
  return preferCheaper ? found.at(-1) : found[0];
}

Chapter Summary

• Use model.input.includes('image') to check vision capability; sending images to non-vision models is silently ignored;
• User message content uses a block array mixing text + image; images are represented as base64 + mimeType;
• agent.prompt(text, images[]) is the shortcut for sending images in Agents;
• Image generation responses contain image blocks in content, decodable to files;
• Multimodal applications need compression, specific questions, capability routing, and cost control.

🧪 随堂测验

点击你认为正确的选项。答错时会展示正确答案与原因解析。

1. How do you判断 whether a model supports image input?

2. What happens when you send an image to a model that does not support vision?

3. What is the correct way to attach an image in Agent.prompt()?

4. Which of the following is NOT a recommended practice for multimodal applications?

🛠️ Hands-on Practice

1. Write a CLI tool: accept an image path argument, call a vision model to output an image description, and handle the "model does not support vision" case.
2. Implement a batch image classification script: read all png files in a directory, have the model tag each image, and write results to JSON.
3. Give the algorithm assistant from Chapter 11 screenshot analysis capability: paste a problem screenshot to get a solution walkthrough.

Next we handle a topic that cannot be avoided in production — Chapter 13 · Stop Reasons, Abort, and Error Handling.