Chapter 19 · Monitoring and Results Analysis

Chapter goals: Understand the meaning of p50/p95/p99 percentile metrics, master Locust integration with Prometheus/Grafana, and learn to systematically locate performance bottlenecks.

19.1 Understanding Response Time Percentiles

Average response time is the most misleading metric — it gets diluted by many fast requests, hiding the impact of a few slow ones. Production-grade performance analysis must focus on tail latency:

Response time distribution for 100 requests:

  p50 (median)  = 120ms   ← Half of users are faster than this
  p90           = 350ms   ← 90% of users are faster than this
  p95           = 500ms   ← 95% of users are faster than this (common SLA threshold)
  p99           = 1200ms  ← The tail 1% of users may wait 1.2 seconds!
  Average       = 180ms   ← Pulled down by many fast requests, masking the problem

Why averages lie

Suppose 95 requests take 50ms and 5 requests take 2000ms:

• Average = (95×50 + 5×2000) / 100 = 147ms ← Looks fine
• p95 = 2000ms ← Exposes the real problem!

If your SLA is "99% of requests complete within 500ms", the average tells you nothing.

Reading Locust's Built-in Statistics

Key columns in the Locust Web UI and CSV reports:

Column	Meaning	What to Watch
Average	Mean response time	Reference value
Median (50%)	50th percentile	Normal user experience
95%	p95	Common SLA threshold
98%	p98	High-requirement scenarios
99%	p99	Extreme tail latency
RPS	Requests per second	Throughput
Failures/s	Failure rate	Stability

19.2 Prometheus + Grafana Integration

Locust supports pushing statistics to Prometheus via event hooks for real-time monitoring and historical trend analysis.

Method 1: locust-exporter (Recommended)

locust-exporter is a standalone Prometheus exporter that pulls data directly from the Locust master API:

# docker-compose.yml — one-command full monitoring stack
version: "3.8"
services:
  locust:
    image: locustio/locust
    ports:
      - "8089:8089"
    volumes:
      - ./locustfile.py:/mnt/locust/locustfile.py
    command: -f /mnt/locust/locustfile.py --master

  prometheus:
    image: prom/prometheus
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml

  grafana:
    image: grafana/grafana
    ports:
      - "3000:3000"

# prometheus.yml
scrape_configs:
  - job_name: "locust"
    static_configs:
      - targets: ["locust:8089"]
    metrics_path: /export/prometheus

Visit http://localhost:9090 for Prometheus UI; http://localhost:3000 for Grafana. Import the official Locust dashboard JSON to visualize RPS, response times, and failure rates in real time.

Method 2: Custom Event Hook

from locust import events
import requests

@events.request.add_listener
def send_to_prometheus(request_type, name, response_time, response_length, exception, context, **kwargs):
    if exception:
        return  # Skip failures for clean metrics
    requests.post("http://prometheus:9090/api/v1/write", json={
        "metric": "locust_request_duration_seconds",
        "value": [response_time / 1000,]
    })

19.3 Systematic Bottleneck Location

When a test fails SLA, follow this diagnostic flow:

1. Check p95/p99 — are tail latencies high while median is OK?
   ├── Yes → Look for specific slow endpoints in the stats table
   └── No  → System-wide slowdown; check infrastructure

2. Check RPS trend — did it flatten while users kept increasing?
   ├── Yes → Target system saturated; check CPU/memory/disk on target
   └── No  → Load-test machine may be the bottleneck

3. Check error rate — are failures concentrated on specific endpoints?
   ├── Yes → Application logic bug; check logs
   └── No  → Infrastructure issue; check network, DNS, load balancer

4. Check database — slow queries, connection pool exhaustion?
   ├── Yes → Add indexes, tune queries, increase pool size
   └── No  → Check external service dependencies (third-party APIs)

19.4 Chapter Summary

• Averages lie; always report p50, p95, p99 separately;
• Prometheus + Grafana with locust-exporter gives real-time monitoring;
• Diagnostic flow: percentiles → RPS trend → error concentration → DB → external deps;
• SLA should be defined on percentiles (e.g., p95 < 500ms), never on averages.

🧪 随堂测验

点击你认为正确的选项。答错时会展示正确答案与原因解析。

1. Why is the average response time a misleading metric?

2. What does locust-exporter do?

3. If p50 is 100ms but p99 is 5000ms, where should you investigate first?

4. What SLA metric should you use to define performance requirements?

🛠️ Hands-On Exercises

1. Set up Prometheus + Grafana with locust-exporter; visualize RPS and p95 response time during a live test.
2. Run a test with one deliberately slow endpoint (e.g., time.sleep(5)); analyze the percentile spread in the stats table.
3. Write an alerting rule in Prometheus: fire when p95 > 500ms for more than 2 minutes.

Proceed to Chapter 20: Capstone — Full-Stack Load Testing.