Chapter 7 · LLM and Prompt Engineering Overview

Chapter goals: Understand the core principle of Chain-of-Thought (CoT), master Manual CoT, Few-shot CoT, and Zero-shot CoT usage, and know when CoT delivers the best results.

7.1 What Is Chain-of-Thought

Chain-of-Thought (CoT) is a prompt engineering technique that guides the model to think step by step by adding reasoning steps between the question and the answer, thereby improving performance on complex tasks.

Traditional prompts often go directly from question to answer:

Question: Xiao Ming has 5 apples, eats 2, then buys 3 more. How many does he have now?
Answer: 6

Chain-of-thought shows the intermediate reasoning:

Question: Xiao Ming has 5 apples, eats 2, then buys 3 more. How many does he have now?
Thinking:
- Started with 5 apples
- Ate 2, remaining: 5 - 2 = 3
- Bought 3 more, now: 3 + 3 = 6
Answer: 6

Google's team first systematically demonstrated the power of CoT in their 2022 paper: for complex tasks requiring multi-step computation — arithmetic, commonsense reasoning, and symbolic推理 — CoT brings significant performance gains.

7.2 Manual Few-shot CoT

The simplest and most effective approach is to manually write a few examples containing reasoning steps (few-shot), letting the model mimic this "show your thinking" style.

prompt = """
Q: The cafeteria had 23 apples. If they used 20 to make lunch and bought 6 more, how many apples do they have?
A: They started with 23 apples, used 20 for lunch (23 - 20 = 3), then bought 6 more (3 + 6 = 9). The answer is 9.

Q: A retail store marks up a price of $80 by 50%. Then they offer a 20% discount. What is the final price?
A: Original price $80. Markup by 50%: $80 * 1.5 = $120. Discount of 20%: $120 * 0.8 = $96. The answer is $96.

Q: {question}
A:
"""

Key points:

• Examples should be specific: each example shows a clear reasoning chain
• Maintain diversity: examples cover different types of reasoning (arithmetic, logic, commonsense)
• Consistent format: use a fixed "Thinking:" prefix to help the model understand the pattern

7.3 Zero-Shot CoT

When you don't have enough examples, you can use Zero-shot CoT. The core trick is adding a trigger phrase after the question, such as:

"Let's think step by step."

# Traditional prompt (prone to errors)
prompt = "I went to the market and bought 10 apples. I gave 2 apples to my neighbor and 2 to the repairman. I then went and bought 5 more apples and ate 1. How many apples did I remain with?"

# Zero-shot CoT
prompt = """I went to the market and bought 10 apples. I gave 2 apples to my neighbor and 2 to the repairman. I then went and bought 5 more apples and ate 1. How many apples did I remain with?

Let's think step by step."""

Research shows that a simple "Let's think step by step" is enough to trigger the model's chain-reasoning capability, especially on larger models (e.g. GPT-3.5+).

7.4 Auto-CoT: Automatically Generating Reasoning Chains

For large-scale application scenarios, manually writing examples is costly. Auto-CoT proposes an automated approach:

1. Question clustering: split the question dataset into several clusters
2. Representative sampling: pick representative questions from each cluster
3. Automatic reasoning chain generation: use Zero-shot CoT to generate reasoning for each representative question
4. Build example set: use the generated reasoning chains as few-shot examples

# Auto-CoT example generation flow
def auto_cot_example(question):
    base_prompt = f"{question}\nLet's think step by step."
    response = llm.complete(base_prompt)
    # Extract the reasoning chain portion
    return extract_chain_of_thought(response)

This approach is particularly suitable for scenarios needing大量 examples, such as benchmark evaluation or prompt optimization in production systems.

7.5 Applicable Scenarios and Limitations of CoT

Applicable Scenarios

• Math computation and arithmetic reasoning
• Multi-step logical deduction
• Planning tasks requiring intermediate states
• Factual knowledge queries (reduces hallucination)

Not Applicable Scenarios

• Simple classification tasks (e.g. sentiment analysis)
• Creative writing (may constrain divergent thinking)
• Real-time conversations (adds latency)
• Small-scale models (can be counterproductive when capability is insufficient)

Practical Advice

# Decision tree for whether to use CoT
def should_use_cot(task_type, model_size):
    if task_type in ['math', 'logic', 'multi_step']:
        return model_size >= 10  # at least 10B parameters
    elif task_type in ['classification', 'summary']:
        return False
    return model_size >= 70  # only large models are worth trying

Chapter Summary

• CoT core idea: guide the model to think step by step by showing reasoning steps
• Manual CoT: hand-write few-shot examples containing thinking processes
• Zero-shot CoT: add "Let's think step by step" trigger phrase
• Auto-CoT: automatically generate reasoning chains for large-scale scenarios
• Applicability: works well on complex reasoning tasks, may be counterproductive on simple tasks

🧪 随堂测验

点击你认为正确的选项。答错时会展示正确答案与原因解析。

1. For which type of task does CoT have the most significant effect?

2. Which type of CoT does "Let's think step by step" belong to?

3. What is the first step of Auto-CoT?

4. Which model is most suitable for using CoT?

🛠️ Hands-on Practice

1. Create a Manual Few-shot CoT prompt with 3 examples to solve math word problems; test the effect difference across different models.
2. Compare Zero-shot CoT (adding "Let's think step by step") vs. traditional prompts on the same complex problem.
3. Try implementing a simple Auto-CoT flow: automatically generate reasoning chains for 5 questions and build a few-shot example set.

Next chapter: Basic Prompt Elements and Instruction Design