Chapter 6 · LLM Configuration Deep Dive and Third-Party Model Integration

Chapter goals: Fully understand CrewAI's model configuration system — provider/model naming rules, complete parameters for OpenAI-compatible endpoints, structured output via response_format, drop_params compatibility techniques, and the cost-control strategy of mixing models per role.

6.1 Model Naming: provider/model Rules

CrewAI unifies model access through the LLM class; the model parameter follows the provider/model-id format:

from crewai import LLM

# Common provider prefixes (routed through LiteLLM)
llm_openai = LLM(model="openai/gpt-4o")
llm_claude = LLM(model="anthropic/claude-sonnet-4-6")
llm_gemini = LLM(model="gemini/gemini-2.0-flash")       # Google Gemini API
llm_local  = LLM(model="ollama/llama3:70b",
                 base_url="http://localhost:11434")      # Local Ollama

Each provider requires its corresponding environment variable key (OPENAI_API_KEY, ANTHROPIC_API_KEY, GEMINI_API_KEY, etc.). The official team has built native SDK integrations for mainstream vendors like OpenAI / Anthropic / Google / Azure / Bedrock; the remaining long-tail providers are adapted by LiteLLM.

Course convention: all examples uniformly use OpenAI-compatible third-party endpoints (DeepSeek), which is also the most common pattern in domestic production environments.

6.2 Complete Configuration for Third-Party OpenAI-Compatible Endpoints

import os
from crewai import LLM

deepseek = LLM(
    # openai/ prefix means "speak the OpenAI protocol"; followed by the provider's model id
    model="openai/deepseek-chat",
    base_url="https://api.deepseek.com/v1",     # Must include /v1
    api_key=os.getenv("DEEPSEEK_API_KEY"),

    # ---- Sampling control ----
    temperature=0.7,     # Randomness: 0 most deterministic, 1 most divergent
    top_p=0.9,           # Nucleus sampling threshold
    max_tokens=2000,     # Output length cap
    seed=42,             # Fixed random seed for reproducibility
    stop=["END"],        # Custom stop sequences

    # ---- Behavior control ----
    stream=False,        # Streaming output
    timeout=120.0,       # Request timeout (seconds)
    max_retries=2,       # Request-level automatic retry count
)

Common pitfall: missing /v1 in base_url

Most OpenAI-compatible gateways mount chat/completions under /v1. Omitting it gives 404; writing too much as /v1/chat/completions is also wrong — CrewAI拼s the path itself.

If a third-party gateway requires a non-openai model name prefix, CrewAI 1.x also provides an explicit way to declare "this is an OpenAI-protocol endpoint":

gateway_llm = LLM(
    model="anthropic/claude-sonnet-4-6",        # Model name exposed by the gateway
    custom_openai=True,                          # Force OpenAI-protocol invocation
    base_url="https://your-gateway.example.com/v1",
    api_key="your-gateway-api-key",
)

6.3 Structured Output: response_format

Have the LLM return a parseable object directly, instead of relying on regex to extract text:

import os
from pydantic import BaseModel
from crewai import LLM

class Sentiment(BaseModel):
    label: str      # positive / negative / neutral
    score: float
    reason: str

judge = LLM(
    model="openai/deepseek-chat",
    base_url="https://api.deepseek.com/v1",
    api_key=os.getenv("DEEPSEEK_API_KEY"),
)

result = judge.call(
    "Analyze the sentiment of this review: 'Shipping was fast, but the packaging was damaged.'",
    response_format=Sentiment,
)
print(type(result))   # <class 'Sentiment'>
print(result.label)   # mixed / negative etc. structured fields

response_format accepts a Pydantic model; CrewAI auto-injects the schema and validates the parsed result. At the Task level, there is also the more commonly used output_pydantic (compare both in Chapter 12).

6.4 drop_params: Compatibility Life Preserver

Different third-party implementations support different parameter sets: some don't support stop, others don't support frequency_penalty. Passing them directly raises errors. Two switches solve this:

compatible_llm = LLM(
    model="openai/deepseek-chat",
    base_url="https://api.deepseek.com/v1",
    api_key=os.getenv("DEEPSEEK_API_KEY"),
    drop_params=True,                    # Auto-discard parameters the target endpoint doesn't support
    additional_drop_params=["stop"],     # Explicitly append a parameter blacklist to discard
)

• drop_params=True: the LiteLLM layer auto-filters parameters the endpoint doesn't recognize;
• additional_drop_params=[...]: manually specify extra items to discard, for bypassing individual gateways' pseudo-compatibility behavior.

When onboarding a new provider, start with drop_params=True, get it running, then gradually converge parameters — this is the most efficient debugging path.

6.5 Mixing Models by Role: Cost Engineering

Having all roles in a production crew share the flagship model is the biggest cost waste. The recommended three-tier configuration:

import os
from crewai import Agent, LLM

cheap = LLM(          # For rough work: classification, extraction, format conversion
    model="openai/deepseek-chat",
    base_url="https://api.deepseek.com/v1",
    api_key=os.getenv("DEEPSEEK_API_KEY"),
    temperature=0.1,
)
smart = LLM(          # For精细 work: reasoning, writing, complex decisions
    model="openai/deepseek-reasoner",       # Deep thinking model
    base_url="https://api.deepseek.com/v1",
    api_key=os.getenv("DEEPSEEK_API_KEY"),
    temperature=0.7,
)

classifier = Agent(role="Ticket Classifier",
                   goal="Categorize tickets into a single class with confidence",
                   backstory="Classify strictly according to the category table.",
                   llm=cheap, max_iter=5)          # Simple tasks: cheap model + low temperature

strategist = Agent(role="Solution Designer",
                   goal="Design handling plans for疑难 tickets",
                   backstory="Ten years of customer service expertise.",
                   llm=smart)                      # Complex tasks: powerful model

Supporting principles:

• Match temperature to task: extraction/classification use 0–0.2; creation uses 0.7+;
• Match max_iter to complexity: lower for simple tasks to prevent idle loops;
• Use crew.usage_metrics from Chapter 5 to跑 both configurations side by side and let data decide the tiering strategy.

6.6 Chapter Summary

• model always written as provider/model-id; OpenAI-compatible endpoint = openai/ prefix + base_url (with /v1) + api_key;
• Key sampling parameters: temperature/top_p/max_tokens/seed/stop; network layer has timeout and max_retries;
• response_format=PydanticModel lets the LLM吐 directly verifiable objects;
• drop_params=True + additional_drop_params is the compatibility weapon for integrating third-party gateways;
• Three cost-optimization moves: model tiering (chat/reasoner), temperature matching tasks, usage_metrics closed-loop measurement.

🧪 随堂测验

点击你认为正确的选项。答错时会展示正确答案与原因解析。

1. In model="openai/deepseek-chat", what does the openai/ prefix mean?

2. Getting a 404 when calling a third-party endpoint — what should you check first?

3. A third-party gateway errors because it does not support the stop parameter. What is the fastest fix?

4. Regarding cost control strategies, which practice is NOT recommended?

🛠️ Hands-On Exercises

1. Migrate all crews from previous chapters to an env-variable-based DeepSeek configuration; extract a shared get_llms() factory function returning cheap/smart tiers.
2. Use response_format to write a sentiment script: batch-output Sentiment object lists for 5 reviews and stat the distribution.
3. Set stream=True and observe the verbose log changes; compare deepseek-chat vs deepseek-reasoner token cost for the same task using crew.usage_metrics.

Model layer sorted. Next, the tooling system: equip agents with real search, scraping, and other capabilities.