📚 AI 工程与智能体教程站

English

简体中文

English

简体中文

Chapter 4 · Text Classification ​

Chapter goals: Complete movie review sentiment classification using two technical approaches — representation models (task-specific models, embeddings + classifier, zero-shot) and generative models (encoder-decoder and ChatGPT) — and learn to evaluate each approach's performance.

This chapter corresponds to the official notebook for Chapter 4 of Hands-On Large Language Models (by Jay Alammar & Maarten Grootendorst), classifying text using both representative models and generative models.

4.1 [Optional] Installing Dependencies in Colab ​

If you are viewing this notebook on Google Colab (or any other cloud environment), you need to uncomment and run the code block below to install the dependencies for this chapter:

💡 NOTE: Running the examples in this chapter requires a GPU. In Google Colab, go to Runtime > Change runtime type > Hardware accelerator > GPU > GPU type > T4.

python
# %%capture
# !pip install "transformers==4.41.2" "sentence-transformers==3.0.1" openai
# !pip install -U datasets

4.2 Data ​

We use the Rotten Tomatoes movie review dataset — each review is labeled as positive or negative:

python
from datasets import load_dataset

# Load our data
data = load_dataset("rotten_tomatoes")
data
python
data["train"][0, -1]

Look at the first and last training samples to get a feel for the data shape: one text (review text) plus one label (0 = negative, 1 = positive).

4.3 Text Classification Using Representation Models ​

4.3.1 Using a Task-Specific Model ​

The most direct approach is to use a model already fine-tuned on a similar task. Here we choose cardiffnlp/twitter-roberta-base-sentiment-latest, a sentiment analysis model trained on Twitter data:

python
from transformers import pipeline

# Path to our HF model
model_path = "cardiffnlp/twitter-roberta-base-sentiment-latest"

# Load model into pipeline
pipe = pipeline(
    model=model_path,
    tokenizer=model_path,
    return_all_scores=True,
    device="cuda:0"
)
python
import numpy as np
from tqdm import tqdm
from transformers.pipelines.pt_utils import KeyDataset

# Run inference
y_pred = []
for output in tqdm(pipe(KeyDataset(data["test"], "text")), total=len(data["test"])):
    negative_score = output[0]["score"]
    positive_score = output[2]["score"]
    assignment = np.argmax([negative_score, positive_score])
    y_pred.append(assignment)

This model outputs three classes (negative / neutral / positive); our task is binary classification, so we only compare the negative and positive scores and take the larger one as the prediction.

Next, define the evaluation function and inspect overall performance:

python
from sklearn.metrics import classification_report

def evaluate_performance(y_true, y_pred):
    """Create and print the classification report"""
    performance = classification_report(
        y_true, y_pred,
        target_names=["Negative Review", "Positive Review"]
    )
    print(performance)
python
evaluate_performance(data["test"]["label"], y_pred)

4.3.2 Classification Using Embeddings ​

Supervised Classification ​

Instead of using a ready-made sentiment model, first convert the text to vectors using an embedding model, then train a standard classifier on the vectors:

python
from sentence_transformers import SentenceTransformer

# Load model
model = SentenceTransformer('sentence-transformers/all-mpnet-base-v2')

# Convert text to embeddings
train_embeddings = model.encode(data["train"]["text"], show_progress_bar=True)
test_embeddings = model.encode(data["test"]["text"], show_progress_bar=True)
python
train_embeddings.shape

all-mpnet-base-v2 outputs 768-dimensional embedding vectors.

python
from sklearn.linear_model import LogisticRegression

# Train a Logistic Regression on our train embeddings
clf = LogisticRegression(random_state=42)
clf.fit(train_embeddings, data["train"]["label"])
python
# Predict previously unseen instances
y_pred = clf.predict(test_embeddings)
evaluate_performance(data["test"]["label"], y_pred)

Tip!

What if we don't use a classifier at all? We can average the embeddings of all documents in each target class, then predict which class a document best matches using cosine similarity:

python
import numpy as np
import pandas as pd
from sklearn.metrics import classification_report
from sklearn.metrics.pairwise import cosine_similarity

# Average the embeddings of all documents in each target label
df = pd.DataFrame(np.hstack([train_embeddings, np.array(data["train"]["label"]).reshape(-1, 1)]))
averaged_target_embeddings = df.groupby(768).mean().values

# Find the best matching embeddings between evaluation documents and target embeddings
sim_matrix = cosine_similarity(test_embeddings, averaged_target_embeddings)
y_pred = np.argmax(sim_matrix, axis=1)

# Evaluate the model
evaluate_performance(data["test"]["label"], y_pred)

Zero-Shot Classification ​

The most fascinating aspect of the embedding approach: no training needed at all! Just write a description for each label, encode the descriptions into embeddings, and see which label description a document is closest to:

python
# Create embeddings for our labels
label_embeddings = model.encode(["A negative review",  "A positive review"])
python
from sklearn.metrics.pairwise import cosine_similarity

# Find the best matching label for each document
sim_matrix = cosine_similarity(test_embeddings, label_embeddings)
y_pred = np.argmax(sim_matrix, axis=1)
python
evaluate_performance(data["test"]["label"], y_pred)

Tip!

What happens if you swap in different descriptions? Try "A very negative movie review" and "A very positive movie review" and see how performance changes!

4.4 Text Classification Using Generative Models ​

4.4.1 Encoder-Decoder Models ​

Generative models take a different path: they generate the answer rather than outputting scores. Here we use the encoder-decoder google/flan-t5-small:

python
# Load our model
pipe = pipeline(
    "text2text-generation",
    model="google/flan-t5-small",
    device="cuda:0"
)
python
# Prepare our data
prompt = "Is the following sentence positive or negative? "
data = data.map(lambda example: {"t5": prompt + example['text']})
data
python
# Run inference
y_pred = []
for output in tqdm(pipe(KeyDataset(data["test"], "t5")), total=len(data["test"])):
    text = output[0]["generated_text"]
    y_pred.append(0 if text == "negative" else 1)
python
evaluate_performance(data["test"]["label"], y_pred)

Note the parsing logic: if the model generates exactly "negative", record 0; otherwise record 1 — generative model outputs are free-text, so such a mapping rule is needed.

4.4.2 Classification with ChatGPT ​

Finally, try the closed-source OpenAI API. First create the client:

python
import openai

# Create client
client = openai.OpenAI(api_key="YOUR_KEY_HERE")

Define a generation function that assembles the system prompt and user prompt before calling the API:

python
def chatgpt_generation(prompt, document, model="gpt-3.5-turbo-0125"):
    """Generate an output based on a prompt and an input document."""
    messages=[
        {
            "role": "system",
            "content": "You are a helpful assistant."
            },
        {
            "role": "user",
            "content":   prompt.replace("[DOCUMENT]", document)
            }
    ]
    chat_completion = client.chat.completions.create(
      messages=messages,
      model=model,
      temperature=0
    )
    return chat_completion.choices[0].message.content
python
# Define a prompt template as a base
prompt = """Predict whether the following document is a positive or negative movie review:

[DOCUMENT]

If it is positive return 1 and if it is negative return 0. Do not give any other answers.
"""

# Predict the target using GPT
document = "unpretentious , charming , quirky , original"
chatgpt_generation(prompt, document)

Next, run the OpenAI model across the entire evaluation set. But make sure you have enough credits before running — it will call the API once per record (1,066 records) across the test set:

python
# You can skip this if you want to save your (free) credits
predictions = [chatgpt_generation(prompt, doc) for doc in tqdm(data["test"]["text"])]
python
# Extract predictions
y_pred = [int(pred) for pred in predictions]

# Evaluate performance
evaluate_performance(data["test"]["label"], y_pred)

4.5 Chapter Summary ​

  • There are two approaches to text classification: representation models (embeddings/scores) and generative models (generate answer text);
  • Task-specific models (twitter-roberta-base-sentiment-latest) work out of the box, but they output three classes, so you need to merge them into a binary decision yourself;
  • Embeddings + LogisticRegression form a strong supervised baseline; averaging embeddings per class and using cosine similarity lets you skip the classifier entirely;
  • Zero-shot classification only requires writing descriptions for labels and encoding them into embeddings — no training needed; wording of the descriptions affects performance;
  • Encoder-decoder models like flan-t5 generate free text, so string-mapping rules are needed to convert outputs to labels; ChatGPT is flexible but billed per call.

🧪 随堂测验

点击你认为正确的选项。答错时会展示正确答案与原因解析。

1. What type of task does the rotten_tomatoes dataset used in this chapter represent?

2. cardiffnlp/twitter-roberta-base-sentiment-latest outputs three classes, but the task is binary — how does the code handle this?

3. How is zero-shot classification implemented in this chapter?

4. When using flan-t5 for classification, how does the code convert generated text into labels?

🛠️ Hands-on Practice ​

  1. Per the Tip in Section 4.3.2, swap the zero-shot label descriptions to "A very negative movie review" and "A very positive movie review", recompute the classification_report, compare F1 scores with the original descriptions, and summarize how description wording affects zero-shot classification.
  2. Swap the embedding model from all-mpnet-base-v2 to sentence-transformers/all-MiniLM-L6-v2; rerun the full "embeddings + LogisticRegression" pipeline; compare accuracy and inference latency on the test set across the two models.
  3. Use the prompt template from Section 4.4.2 to call chatgpt_generation on data["train"][0]; then construct 3 adversarial examples (e.g. sarcastic reviews); observe whether GPT-3.5's judgments are correct and record failure patterns.