Chapter 14 · Image Generation and Multimodal Input

Chapter goals:

• Master the basic usage of generateImage for image generation, including size and aspect ratio settings
• Learn advanced configurations: batch generation, seed specification, and providerOptions
• Understand wrapImageModel image model middleware and error handling
• Master the uploadFile file upload mechanism and the ProviderReference reference pattern
• Understand file reference merging strategies in multi-provider scenarios

14.1 Generating Images with generateImage

AI SDK provides the generateImage function to generate images based on a given prompt using an image model.

import { generateImage, createGateway } from 'ai';
import 'dotenv/config';

const gateway = createGateway({ apiKey: process.env.AI_GATEWAY_API_KEY ?? '' });

// A string model ID is resolved via the global default provider (AI Gateway);
// custom OpenAI-compatible providers use myProvider.image('your-image-model')
const { image } = await generateImage({
  model: gateway.image('openai/dall-e-3'),
  prompt: 'Santa Claus driving a Cadillac',
});

You can access the image data via base64 or uint8Array:

const base64 = image.base64; // base64-encoded image data
const uint8Array = image.uint8Array; // Uint8Array binary data

14.2 Basic Settings

Size and Aspect Ratio

Depending on the model, you can specify size or aspectRatio. Size is a string in {width}x{height} format — models support only a few sizes, varying by model and provider:

import { generateImage } from 'ai';
import 'dotenv/config';

const { image } = await generateImage({
  model: 'openai/dall-e-3',
  prompt: 'Santa Claus driving a Cadillac',
  size: '1024x1024',
});

Aspect ratio is in {width}:{height} format, with only a few values available:

import { generateImage } from 'ai';
import 'dotenv/config';

const { image } = await generateImage({
  model: 'openai/dall-e-3',
  prompt: 'Santa Claus driving a Cadillac',
  aspectRatio: '16:9',
});

Batch Generating Multiple Images

generateImage supports generating multiple images in one call:

import { generateImage } from 'ai';
import 'dotenv/config';

const { images } = await generateImage({
  model: 'openai/dall-e-3',
  prompt: 'Santa Claus driving a Cadillac',
  n: 4, // number of images to generate
});

generateImage auto-calls the model in parallel as needed to generate the requested number of images. Each image model has an internal limit on how many images can be generated per API call; the SDK自动 batches reasonably (e.g., DALL-E 3 generates 1 image per call, DALL-E 2 up to 10 per call).

You can override the default batch size with maxImagesPerCall — particularly useful for new or custom models:

const { images } = await generateImage({
  model: 'openai/dall-e-2',
  prompt: 'Santa Claus driving a Cadillac',
  maxImagesPerCall: 5, // Override the default batch size
  n: 10, // Will make 2 calls of 5 images each
});

Specifying a Seed

Providing a seed controls the image generation output. If the model supports it, the same seed always produces the same image:

import { generateImage } from 'ai';
import 'dotenv/config';

const { image } = await generateImage({
  model: 'openai/dall-e-3',
  prompt: 'Santa Claus driving a Cadillac',
  seed: 1234567890,
});

Provider-Specific Settings

Image models通常 have provider- or even model-specific settings, passed via the providerOptions parameter (options become request body properties). The following example follows the official docs' OpenAI pattern to show the providerOptions structure:

import { generateImage, createGateway } from 'ai';
import 'dotenv/config';

const gateway = createGateway({ apiKey: process.env.AI_GATEWAY_API_KEY ?? '' });

const { image } = await generateImage({
  model: gateway.image('openai/dall-e-3'),
  prompt: 'Santa Claus driving a Cadillac',
  size: '1024x1024',
  providerOptions: {
    openai: {
      style: 'vivid',
      quality: 'hd',
    },
  },
});

Abort Signal and Timeout

generateImage accepts an optional AbortSignal type abortSignal parameter, which can be used to abort image generation or set a timeout:

import { generateImage } from 'ai';
import 'dotenv/config';

const { image } = await generateImage({
  model: 'openai/dall-e-3',
  prompt: 'Santa Claus driving a Cadillac',
  abortSignal: AbortSignal.timeout(1000), // Abort after 1 second
});

Custom Headers and Warnings

The optional headers parameter (type Record<string, string>) adds custom headers to image generation requests; if the model returns warnings (e.g., for unsupported parameters), they are available in the result's warnings property:

const { image, warnings } = await generateImage({
  model: 'openai/dall-e-3',
  prompt: 'Santa Claus driving a Cadillac',
});

Provider Attachment Metadata

Some providers expose additional metadata on the overall result or per-image. The returned providerMetadata outer key is the provider name, inner is the metadata; it always includes an images key, an array同长度 as the top-level images array:

const prompt = 'Santa Claus driving a Cadillac';

const { image, providerMetadata } = await generateImage({
  model: 'openai/dall-e-3',
  prompt,
});

const revisedPrompt = providerMetadata.openai.images[0]?.revisedPrompt;

console.log({
  prompt,
  revisedPrompt,
});

14.3 Error Handling

When generateImage cannot generate a valid image, it throws AI_NoImageGeneratedError. Possible causes: the model failed to generate a response, or generated an unparseable response.

14.4 wrapImageModel Middleware

Use wrapImageModel to enhance image models with middleware, analogous to wrapLanguageModel:

import { wrapImageModel, generateImage } from 'ai';

const enhancedImageModel = wrapImageModel({
  model: gateway.image('openai/dall-e-3'),
  middleware: yourImageModelMiddleware,
});

const { image } = await generateImage({
  model: enhancedImageModel,
  prompt: 'Santa Claus driving a Cadillac',
});

14.5 uploadFile and ProviderReference

uploadFile uploads a file to the provider and returns a ProviderReference that can be passed to subsequent calls. This is useful for multi-modal inputs (e.g., passing an image to a vision model) without embedding large base64 strings directly:

import { uploadFile, createGateway } from 'ai';

const file = await uploadFile({
  model: gateway('openai/gpt-4o'),
  file: buffer, // Buffer or Blob
  mime: 'image/png',
});

// file.providerData contains the provider-specific reference

In multi-provider scenarios, file references are merged according to the provider's capabilities — the SDK handles cross-provider reference resolution automatically.

Chapter Summary

• generateImage creates images from text prompts; access via image.base64 or image.uint8Array;
• Size (1024x1024) and aspect ratio (16:9) are model-dependent; batch with n and control batch size with maxImagesPerCall;
• Seeds provide deterministic output when supported; providerOptions passes model-specific settings;
• abortSignal supports timeout and cancellation; warnings captures model-returned warnings;
• wrapImageModel applies middleware to image models; uploadFile handles multi-modal file uploads with ProviderReference.

🧪 随堂测验

点击你认为正确的选项。答错时会展示正确答案与原因解析。

1. How do you access the generated image data from generateImage?

2. What does the seed parameter do in generateImage?

3. What is the purpose of maxImagesPerCall?

4. What does uploadFile return that is useful for multimodal inputs?

🛠️ Hands-On Exercises

1. Generate 4 images with n: 4 and size: '1024x1024'; verify all 4 are returned and compare their base64 lengths.
2. Generate the same prompt twice with seed: 42; verify both images are identical (if the model supports deterministic generation).
3. Upload a file with uploadFile and pass the reference to a vision model; verify the model can "see" the image.

Next: Chapter 15 · Speech Transcription and Synthesis