Chapter 4 · Prompts and Message Models

Chapter goals:

• Master three prompt forms: text prompt, system prompt (instructions), and message prompt
• Understand the role system and content parts structure of ModelMessage
• Learn to carry image, PDF, audio, and other multimodal content in messages
• Understand the three levels of providerOptions: function-level, message-level, and part-level

4.1 What Is a Prompt

A prompt is your instruction to the large language model, telling it what to do. Like asking for directions — the clearer the question, the better the guidance.

Many LLM provider prompt interfaces are complex, involving different roles and message types. Although powerful, they are hard to understand and use. To simplify prompt writing, AI SDK supports text, message, and system three prompt forms.

4.2 Text Prompt

A text prompt is a string, suitable for simple generation scenarios such as repeatedly generating variants of the same prompt text. Set via the prompt property of AI SDK functions like generateText or streamText:

import { generateText, createGateway } from 'ai';

const gateway = createGateway({
  apiKey: process.env.AI_GATEWAY_API_KEY ?? '',
});

const result = await generateText({
  model: gateway('openai/gpt-5'),
  prompt: 'Invent a new holiday and describe its traditions.',
});

You can inject dynamic data into the prompt with template literals:

const result = await generateText({
  model,
  prompt:
    `I am planning a trip to ${destination} for ${lengthOfStay} days. ` +
    `Please suggest the best tourist activities for me to do.`,
});

4.3 System Prompt: instructions

The system prompt is the initial instruction set for the model, used to guide and constrain the model's behavior and response style, set via the instructions property. It can be used together with the prompt or messages property.

⚠️ Note: by default, system messages in prompt/messages are rejected; either pass system instructions via the instructions property, or explicitly set allowSystemInMessages: true when sending历史记录 containing system messages.

const result = await generateText({
  model,
  instructions:
    `You help planning travel itineraries. ` +
    `Respond to the users' request with a list ` +
    `of the best stops to make in their destination.`,
  prompt:
    `I am planning a trip to ${destination} for ${lengthOfStay} days. ` +
    `Please suggest the best tourist activities for me to do.`,
});

⚠️ Enabling allowSystemInMessages introduces prompt injection risk — users could override the system prompt by injecting system messages. In most cases, system instructions should only be set by trusted server-side code via instructions.

4.4 Message Prompt

A message prompt is an array of user, assistant, and tool messages, suitable for chat interfaces and more complex multimodal prompts, set via the messages property.

Each message has role and content properties. Content can be plain text or an array of content parts relevant to that message type:

const result = await generateText({
  model,
  messages: [
    { role: 'user', content: 'Hi!' },
    { role: 'assistant', content: 'Hello, how can I help?' },
    { role: 'user', content: 'Where can I buy the best Currywurst in Berlin?' },
  ],
});

⚠️ Not all models support all message and content types. For example, some models do not support multimodal input or tool messages — check the所用 model's capability documentation.

4.5 Three Levels of Provider Options

providerOptions passes provider-specific metadata through to the underlying API, at three granularities:

Function-call level — use when fine-grained control is not needed:

const { text } = await generateText({
  model,
  providerOptions: {
    openai: {
      reasoningEffort: 'low',
    },
  },
  prompt: 'Invent a new holiday and describe its traditions.',
});

Message level — precisely control a specific message's behavior, e.g. marking a system message with a cache breakpoint:

const result = await generateText({
  model,
  instructions: {
    role: 'system',
    content: 'Cached system message',
    providerOptions: {
      // Sets a cache control breakpoint on the system message
      anthropic: { cacheControl: { type: 'ephemeral' } },
    },
  },
  prompt: 'Invent a new holiday and describe its traditions.',
});

Message-part level — some provider-specific options must be configured on具体的 parts:

import { ModelMessage } from 'ai';

const messages: ModelMessage[] = [
  {
    role: 'user',
    content: [
      {
        type: 'text',
        text: 'Describe the image in detail.',
        providerOptions: {
          openai: { imageDetail: 'low' },
        },
      },
      {
        type: 'file',
        mediaType: 'image',
        data: 'https://github.com/vercel/ai/blob/main/examples/ai-functions/data/comic-cat.png?raw=true',
        // Sets image detail configuration for image part:
        providerOptions: {
          openai: { imageDetail: 'low' },
        },
      },
    ],
  },
];

💡 The UIMessage arrays returned by AI SDK UI hooks (such as useChat) do not support provider options; convert to ModelMessage first via convertToModelMessages before attaching.

4.6 User Messages: Multimodal Content Parts

Text Parts

Text is the most common content type. If you only need to send text, pass a string directly to content; you can also split into multiple parts:

const result = await generateText({
  model,
  messages: [
    {
      role: 'user',
      content: [
        { type: 'text', text: 'Part 1' },
        { type: 'text', text: 'Part 2' },
      ],
    },
  ],
});

Image Parts

Images can be passed as base64 data, binary Buffer, or URL:

import fs from 'node:fs';
import { generateText } from 'ai';

const result = await generateText({
  model,
  messages: [
    {
      role: 'user',
      content: [
        { type: 'text', text: 'Describe the image in detail.' },
        {
          type: 'file',
          mediaType: 'image',
          data: fs.readFileSync('./data/comic-cat.png'),
        },
      ],
    },
  ],
});

Base64 and URL examples:

// base64
{
  type: 'file',
  mediaType: 'image',
  data: fs.readFileSync('./data/comic-cat.png').toString('base64'),
}

// URL
{
  type: 'file',
  mediaType: 'image',
  data: 'https://github.com/vercel/ai/blob/main/examples/ai-functions/data/comic-cat.png?raw=true',
}

File Parts

User messages can also carry files (同样支持 base64, binary, and URL sources), requiring a MIME type. Currently only少数 provider/model support file parts (Google Generative AI, Google Vertex AI, OpenAI audio/PDF, Anthropic, etc.).

PDF file example:

import fs from 'node:fs';
import { generateText } from 'ai';

const result = await generateText({
  model,
  messages: [
    {
      role: 'user',
      content: [
        { type: 'text', text: 'What is the file about?' },
        {
          type: 'file',
          mediaType: 'application/pdf',
          data: fs.readFileSync('./data/example.pdf'),
          filename: 'example.pdf', // optional, not used by all providers
        },
      ],
    },
  ],
});

mp3 audio file example:

import fs from 'node:fs';
import { generateText } from 'ai';

const result = await generateText({
  model,
  messages: [
    {
      role: 'user',
      content: [
        { type: 'text', text: 'What is the audio saying?' },
        {
          type: 'file',
          mediaType: 'audio/mpeg',
          data: fs.readFileSync('./data/galileo.mp3'),
        },
      ],
    },
  ],
});

Custom Download Function (Experimental)

The default implementation auto-downloads URL files in parallel when the model does not support them. You can pass experimental_download to implement rate-limiting, retry, auth, caching, etc.:

const result = await generateText({
  model,
  experimental_download: async (
    requestedDownloads: Array<{
      url: URL;
      isUrlSupportedByModel: boolean;
    }>,
  ): PromiseLike<
    Array<{
      data: Uint8Array;
      mediaType: string | undefined;
    } | null>
  > => {
    // ... download the files and return an array with similar order
  },
  messages: [
    {
      role: 'user',
      content: [
        {
          type: 'file',
          data: new URL('https://api.company.com/private/document.pdf'),
          mediaType: 'application/pdf',
        },
      ],
    },
  ],
});

4.7 Assistant Messages

Assistant messages have role assistant, typically the助手's historical replies, which can contain text, reasoning, and tool-call parts.

Pure text reply:

const result = await generateText({
  model,
  messages: [
    { role: 'user', content: 'How many calories are in this block of cheese?' },
    {
      role: 'assistant',
      content: [
        {
          type: 'tool-call',
          toolCallId: '12345',
          toolName: 'get-nutrition-data',
          input: { cheese: 'Roquefort' },
        },
      ],
    },
  ],
});

4.8 Tool Messages and Multimodal Tool Results

For models supporting tool calls, assistant messages can contain tool-call parts, and tool messages carry tool outputs. One assistant message can call multiple tools in parallel; one tool message can contain multiple results:

import fs from 'node:fs';

const result = await generateText({
  model,
  messages: [
    {
      role: 'user',
      content: [
        {
          type: 'text',
          text: 'How many calories are in this block of cheese?',
        },
        {
          type: 'file',
          mediaType: 'image',
          data: fs.readFileSync('./data/roquefort.jpg'),
        },
      ],
    },
    {
      role: 'assistant',
      content: [
        {
          type: 'tool-call',
          toolCallId: '12345',
          toolName: 'get-nutrition-data',
          input: { cheese: 'Roquefort' },
        },
        // there could be more tool calls here (parallel calling)
      ],
    },
    {
      role: 'tool',
      content: [
        {
          type: 'tool-result',
          toolCallId: '12345', // needs to match the tool call id
          toolName: 'get-nutrition-data',
          output: {
            type: 'json',
            value: {
              name: 'Cheese, roquefort',
              calories: 369,
              fat: 31,
              protein: 22,
            },
          },
        },
        // there could be more tool results here (parallel calling)
      ],
    },
  ],
});

Tool results themselves can also be multi-part multimodal (e.g. text + image), expressed with output: { type: 'content', value: [...] }:

{
  type: 'tool-result',
  toolCallId: '12345', // needs to match the tool call id
  toolName: 'get-nutrition-data',
  // for models that support multi-part tool results:
  output: {
    type: 'content',
    value: [
      {
        type: 'text',
        text: 'Here is the nutrition data for the cheese:',
      },
      {
        type: 'file-data',
        data: fs
          .readFileSync('./data/roquefort-nutrition-data.png')
          .toString('base64'),
        mediaType: 'image/png',
      },
    ],
  },
}

For models that do not support multi-part results, the plain JSON output part can still be used as a fallback.

4.9 System Messages

System messages are sent to the model before user messages to guide the assistant's behavior. You can also use the instructions property (see Section 4.3):

const result = await generateText({
  model,
  messages: [
    { role: 'system', content: 'You help planning travel itineraries.' },
    {
      role: 'user',
      content:
        'I am planning a trip to Berlin for 3 days. Please suggest the best tourist activities for me to do.',
    },
  ],
});

4.10 Chapter Summary

• Three prompt forms: prompt (text), instructions (system prompt, recommended over system messages), messages (message array);
• System角色的 messages are rejected by default; allowSystemInMessages: true has injection risk and should be used cautiously;
• ModelMessage content can be组合 from text/image/file/tool-call/tool-result parts; one message can have multiple parallel tool calls;
• providerOptions supports three granularities: function-level, message-level, and message-part-level;
• UIMessage does not support provider options; convert via convertToModelMessages first.

🧪 随堂测验

点击你认为正确的选项。答错时会展示正确答案与原因解析。

1. What is the recommended property for setting system instructions in AI SDK?

2. Why is allowSystemInMessages: true marked as a risk option?

3. To set the imageDetail parameter for a specific image, at which level should providerOptions be configured?

4. Regarding file parts, which statement is FALSE?

🛠️ Hands-On Exercises

1. Implement the same travel planning request both with prompt + instructions and with a system message; compare the output style difference.
2. Construct a user message containing "text + image URL" and send it to a vision model; then try replacing the image with a base64 Buffer form and confirm both写法 produce consistent results.
3. Manually assemble a complete user → assistant(tool-call) → tool(tool-result) three-message sequence to模拟 tool history; observe how the model continues the conversation基于历史.

Next: Chapter 5 · Text Generation and Streaming Output