> ## Documentation Index
> Fetch the complete documentation index at: https://mintlify.com/QwenLM/Qwen/llms.txt
> Use this file to discover all available pages before exploring further.

# Performance Benchmarks

> Detailed performance analysis and optimization guidelines for quantized Qwen models

## Overview

This page provides comprehensive performance benchmarks for Qwen models across different quantization methods, model sizes, and hardware configurations. All measurements are from production testing on real hardware.

<Note>
  **Test Environment:** A100-SXM4-80G GPU, PyTorch 2.0.1, CUDA 11.8, Flash-Attention 2 (except where noted)
</Note>

## Memory and Speed Comparison

### Qwen-1.8B

<Tabs>
  <Tab title="Memory & Speed">
    | Precision | GPU Memory | Speed (tokens/s) | vs BF16 Memory | vs BF16 Speed |
    | --------- | ---------- | ---------------- | -------------- | ------------- |
    | BF16      | 4.23 GB    | 54.09            | -              | -             |
    | Int8      | 3.48 GB    | 55.56            | -18%           | +2.7%         |
    | Int4      | 2.91 GB    | 71.07            | -31%           | +31.4%        |
  </Tab>

  <Tab title="Quality">
    | Precision | MMLU | C-Eval | GSM8K | HumanEval |
    | --------- | ---- | ------ | ----- | --------- |
    | BF16      | 43.3 | 55.6   | 33.7  | 26.2      |
    | Int8      | 43.1 | 55.8   | 33.0  | 27.4      |
    | Int4      | 42.9 | 52.8   | 31.2  | 25.0      |
  </Tab>

  <Tab title="Use Case">
    **Best For:**

    * Edge devices and mobile deployment
    * Low-latency applications
    * Resource-constrained environments

    **Recommendation:** Int4 provides excellent speed gains with minimal quality loss for most tasks.
  </Tab>
</Tabs>

### Qwen-7B

<Tabs>
  <Tab title="Memory & Speed">
    | Precision | GPU Memory | Speed (tokens/s) | vs BF16 Memory | vs BF16 Speed |
    | --------- | ---------- | ---------------- | -------------- | ------------- |
    | BF16      | 16.99 GB   | 40.93            | -              | -             |
    | Int8      | 11.20 GB   | 37.47            | -34%           | -8.5%         |
    | Int4      | 8.21 GB    | 50.09            | -52%           | +22.4%        |
  </Tab>

  <Tab title="Quality">
    | Precision | MMLU | C-Eval | GSM8K | HumanEval |
    | --------- | ---- | ------ | ----- | --------- |
    | BF16      | 55.8 | 59.7   | 50.3  | 37.2      |
    | Int8      | 55.4 | 59.4   | 48.3  | 34.8      |
    | Int4      | 55.1 | 59.2   | 49.7  | 29.9      |
  </Tab>

  <Tab title="Use Case">
    **Best For:**

    * Production deployments
    * Consumer GPUs (RTX 3090, RTX 4090)
    * Balanced performance/quality

    **Recommendation:**

    * **Int8** for quality-critical applications (\< 1% accuracy loss)
    * **Int4** for memory-constrained deployments (fits on 16GB GPUs)
  </Tab>
</Tabs>

### Qwen-14B

<Tabs>
  <Tab title="Memory & Speed">
    | Precision | GPU Memory | Speed (tokens/s) | vs BF16 Memory | vs BF16 Speed |
    | --------- | ---------- | ---------------- | -------------- | ------------- |
    | BF16      | 30.15 GB   | 32.22            | -              | -             |
    | Int8      | 18.81 GB   | 29.28            | -38%           | -9.1%         |
    | Int4      | 13.01 GB   | 38.72            | -57%           | +20.2%        |
  </Tab>

  <Tab title="Quality">
    | Precision | MMLU | C-Eval | GSM8K | HumanEval |
    | --------- | ---- | ------ | ----- | --------- |
    | BF16      | 64.6 | 69.8   | 60.1  | 43.9      |
    | Int8      | 63.6 | 68.6   | 60.0  | 48.2      |
    | Int4      | 63.3 | 69.0   | 59.8  | 45.7      |
  </Tab>

  <Tab title="Use Case">
    **Best For:**

    * High-quality production systems
    * 24GB GPUs (RTX 3090/4090 with Int4)
    * Research and evaluation

    **Recommendation:**

    * **Int8** for A100/H100 deployments prioritizing quality
    * **Int4** for consumer GPUs (fits on 24GB)
  </Tab>
</Tabs>

### Qwen-72B

<Tabs>
  <Tab title="Memory & Speed">
    | Precision   | GPU Memory         | Speed (tokens/s) | vs BF16 Memory | vs BF16 Speed |
    | ----------- | ------------------ | ---------------- | -------------- | ------------- |
    | BF16        | 144.69 GB (2xA100) | 8.48             | -              | -             |
    | Int8        | 81.27 GB (2xA100)  | 9.05             | -44%           | +6.7%         |
    | Int4        | 48.86 GB           | 11.32            | -66%           | +33.5%        |
    | BF16 + vLLM | 2xA100             | 17.60            | -              | +107.5%       |
  </Tab>

  <Tab title="Quality">
    | Precision | MMLU | C-Eval | GSM8K | HumanEval |
    | --------- | ---- | ------ | ----- | --------- |
    | BF16      | 74.4 | 80.1   | 76.4  | 64.6      |
    | Int8      | 73.5 | 80.1   | 73.5  | 62.2      |
    | Int4      | 73.4 | 80.1   | 75.3  | 61.6      |
  </Tab>

  <Tab title="Use Case">
    **Best For:**

    * State-of-the-art performance
    * Enterprise deployments
    * High-throughput serving

    **Recommendation:**

    * **Int4** to fit on single A100-80GB
    * **vLLM** for maximum throughput in production
    * **Int8** for best quality on multi-GPU setups
  </Tab>
</Tabs>

<Tip>
  **Key Finding:** Int4 quantization provides the best overall value, offering 50-66% memory reduction with 20-33% speed improvement and minimal quality loss (\< 2% on most benchmarks).
</Tip>

## KV Cache Quantization Impact

### Batch Size Scaling

Performance of Qwen-7B (BF16) generating 1024 tokens:

| Batch Size | Without KV Cache | With KV Cache | Savings | Batch Size Increase |
| ---------- | ---------------- | ------------- | ------- | ------------------- |
| 1          | 16.3 GB          | 15.5 GB       | 0.8 GB  | -                   |
| 4          | 24.1 GB          | 17.2 GB       | 6.9 GB  | -                   |
| 16         | 31.7 GB          | 22.3 GB       | 9.4 GB  | -                   |
| 32         | 48.7 GB          | 30.2 GB       | 18.5 GB | -                   |
| 64         | **OOM**          | 48.2 GB       | -       | **2x enabled**      |
| 100        | **OOM**          | 72.4 GB       | -       | **3x enabled**      |

<Note>
  KV cache quantization enables 2-3x larger batch sizes, dramatically improving throughput for multi-user serving scenarios.
</Note>

### Sequence Length Scaling

Performance of Qwen-7B (BF16) with batch size 1:

| Sequence Length | Without KV Cache | With KV Cache | Savings | % Reduction |
| --------------- | ---------------- | ------------- | ------- | ----------- |
| 512             | 15.2 GB          | 15.0 GB       | 0.2 GB  | 1%          |
| 1024            | 16.3 GB          | 15.5 GB       | 0.8 GB  | 5%          |
| 2048            | 17.6 GB          | 15.8 GB       | 1.8 GB  | 10%         |
| 4096            | 19.5 GB          | 16.6 GB       | 2.9 GB  | 15%         |
| 8192            | 23.2 GB          | 17.6 GB       | 5.6 GB  | 24%         |

<Tip>
  Memory savings scale with sequence length. For 8K+ token generation, KV cache quantization reduces memory by 20-30%.
</Tip>

## Combined Quantization

Optimal memory efficiency combines GPTQ weight quantization with KV cache quantization:

### Qwen-7B Memory Breakdown

| Configuration   | GPU Memory | vs BF16 | Notes                  |
| --------------- | ---------- | ------- | ---------------------- |
| BF16 baseline   | 16.99 GB   | -       | No quantization        |
| BF16 + KV cache | 15.5 GB    | -9%     | KV cache only          |
| Int8            | 11.20 GB   | -34%    | GPTQ only              |
| Int8 + KV cache | \~10.5 GB  | -38%    | Combined               |
| Int4            | 8.21 GB    | -52%    | GPTQ only              |
| Int4 + KV cache | \~7.5 GB   | -56%    | **Maximum efficiency** |

<Tip>
  For production deployments with high batch sizes or long sequences, combining Int4 GPTQ with KV cache quantization provides optimal memory efficiency.
</Tip>

## Fine-tuning Memory Requirements

Memory usage for fine-tuning Qwen-7B with different methods (single A100-80GB):

| Method             | 256 tokens    | 512 tokens    | 1024 tokens   | 2048 tokens   | 4096 tokens    | 8192 tokens    |
| ------------------ | ------------- | ------------- | ------------- | ------------- | -------------- | -------------- |
| **LoRA**           | 20.1GB / 1.2s | 20.4GB / 1.5s | 21.5GB / 2.8s | 23.8GB / 5.2s | 29.7GB / 10.1s | 36.6GB / 21.3s |
| **LoRA (emb)**     | 33.7GB / 1.4s | 34.1GB / 1.6s | 35.2GB / 2.9s | 35.1GB / 5.3s | 39.2GB / 10.3s | 48.5GB / 21.7s |
| **Q-LoRA**         | 11.5GB / 3.0s | 11.5GB / 3.0s | 12.3GB / 3.5s | 13.9GB / 7.0s | 16.9GB / 11.6s | 23.5GB / 22.3s |
| **Full-parameter** | 43.5GB / 2.1s | 43.5GB / 2.2s | 43.5GB / 2.2s | 43.5GB / 2.3s | 47.1GB / 2.8s  | 48.3GB / 5.6s  |

<Note>
  **LoRA (emb)** includes trainable embedding and output layers, required when introducing new special tokens. Q-LoRA provides the best memory efficiency for fine-tuning.
</Note>

## Hardware Recommendations

### GPU Selection Guide

<Tabs>
  <Tab title="Consumer GPUs">
    <CardGroup cols={2}>
      <Card title="RTX 3090 / RTX 4090 (24GB)">
        **Recommended Models:**

        * Qwen-7B-Chat-Int4 (8.2GB) ✅
        * Qwen-14B-Chat-Int4 (13GB) ✅
        * Qwen-7B-Chat-Int8 (11.2GB) ✅

        **Not Recommended:**

        * Qwen-14B BF16 (30GB) ❌
        * Qwen-72B any variant ❌

        **Tip:** Use Int4 + KV cache for optimal batch processing
      </Card>

      <Card title="RTX 3060 / RTX 3070 (8-12GB)">
        **Recommended Models:**

        * Qwen-1.8B-Chat-Int4 (2.9GB) ✅
        * Qwen-7B-Chat-Int4 (8.2GB) ✅ (tight fit)

        **Not Recommended:**

        * Qwen-7B-Chat-Int8 (11.2GB) ❌
        * Qwen-14B any variant ❌

        **Tip:** Stick to smaller models or use CPU offloading
      </Card>
    </CardGroup>
  </Tab>

  <Tab title="Data Center GPUs">
    <CardGroup cols={2}>
      <Card title="A100-80GB / H100-80GB">
        **Recommended Models:**

        * Qwen-72B-Chat-Int4 (48.9GB) ✅
        * Qwen-14B-Chat BF16 (30GB) ✅
        * Qwen-7B-Chat BF16 (17GB) ✅

        **With vLLM:**

        * Qwen-72B-Chat BF16 (2xGPU) ✅

        **Tip:** Use BF16 or Int8 for quality, vLLM for throughput
      </Card>

      <Card title="A100-40GB">
        **Recommended Models:**

        * Qwen-14B-Chat-Int8 (18.8GB) ✅
        * Qwen-7B-Chat BF16 (17GB) ✅
        * Qwen-72B-Chat-Int4 with offloading

        **Not Recommended:**

        * Qwen-72B-Chat BF16 single GPU ❌

        **Tip:** Use tensor parallelism for Qwen-72B
      </Card>
    </CardGroup>
  </Tab>

  <Tab title="Multi-GPU">
    <CardGroup cols={2}>
      <Card title="2x A100-80GB">
        **Optimal Setup:**

        * Qwen-72B-Chat BF16 (144GB total)
        * vLLM for 2x throughput (17.6 tok/s)
        * Tensor parallelism for distribution

        **Use Cases:**

        * Production serving
        * High-quality inference
        * Research evaluations
      </Card>

      <Card title="4x A100-80GB">
        **Optimal Setup:**

        * Qwen-72B-Chat BF16 with LoRA fine-tuning
        * DeepSpeed ZeRO 3 for training
        * Maximum throughput serving

        **Use Cases:**

        * Enterprise deployments
        * Fine-tuning large models
        * Serving at scale
      </Card>
    </CardGroup>
  </Tab>
</Tabs>

## Optimization Strategies

### For Memory Efficiency

<Steps>
  <Step title="Start with Int4 GPTQ">
    Int4 provides 50-66% memory reduction with minimal quality loss

    ```python theme={null}
    model = AutoModelForCausalLM.from_pretrained(
        "Qwen/Qwen-7B-Chat-Int4",
        device_map="auto",
        trust_remote_code=True
    )
    ```
  </Step>

  <Step title="Enable KV cache quantization for batch/long sequences">
    Add KV cache quantization for larger batches or longer sequences

    ```python theme={null}
    model = AutoModelForCausalLM.from_pretrained(
        "Qwen/Qwen-7B-Chat-Int4",
        device_map="auto",
        trust_remote_code=True,
        use_cache_quantization=True,
        use_cache_kernel=True,
        use_flash_attn=False
    )
    ```
  </Step>

  <Step title="Profile your workload">
    Measure actual memory usage and adjust batch size/sequence length accordingly
  </Step>
</Steps>

### For Speed

<Steps>
  <Step title="Use Int4 for best speed">
    Int4 provides 20-33% speed improvement over BF16
  </Step>

  <Step title="Enable Flash Attention (if not using KV cache)">
    Flash Attention 2 improves speed by 30-40% for long sequences

    ```python theme={null}
    model = AutoModelForCausalLM.from_pretrained(
        "Qwen/Qwen-7B-Chat",
        device_map="auto",
        trust_remote_code=True,
        use_flash_attn=True
    )
    ```
  </Step>

  <Step title="Use vLLM for production serving">
    vLLM provides 2x throughput improvement with PagedAttention
  </Step>

  <Step title="Optimize batch size">
    Find the sweet spot between throughput and latency for your hardware
  </Step>
</Steps>

### For Quality

<Steps>
  <Step title="Use BF16 baseline for critical applications">
    No quantization for maximum quality
  </Step>

  <Step title="Use Int8 for quality-sensitive production">
    Less than 1% accuracy drop on most benchmarks
  </Step>

  <Step title="Evaluate Int4 on your domain">
    Int4 works well for most tasks but may degrade code generation (HumanEval)
  </Step>

  <Step title="Fine-tune after quantization if needed">
    Q-LoRA fine-tuning can recover some quality loss
  </Step>
</Steps>

## Common Bottlenecks

<AccordionGroup>
  <Accordion title="OOM during inference">
    **Symptoms:** CUDA out of memory errors

    **Solutions:**

    1. Use Int4 quantization (52-66% memory reduction)
    2. Enable KV cache quantization
    3. Reduce batch size or sequence length
    4. Use gradient checkpointing (training)
    5. Consider smaller model variant
  </Accordion>

  <Accordion title="Slow generation speed">
    **Symptoms:** Slow tokens/second, high latency

    **Solutions:**

    1. Use Int4 quantization (20-33% faster)
    2. Enable Flash Attention 2 (if not using KV cache)
    3. Use vLLM for batch inference
    4. Increase batch size to amortize overhead
    5. Check for CPU-GPU bottlenecks
  </Accordion>

  <Accordion title="Quality degradation">
    **Symptoms:** Poor outputs, hallucinations, reduced accuracy

    **Solutions:**

    1. Switch from Int4 to Int8 or BF16
    2. Use more calibration data during quantization
    3. Fine-tune after quantization (Q-LoRA)
    4. Evaluate on domain-specific benchmarks
    5. Consider task-specific quantization
  </Accordion>

  <Accordion title="Multi-GPU inefficiency">
    **Symptoms:** Low GPU utilization, poor scaling

    **Solutions:**

    1. Use tensor parallelism (vLLM, DeepSpeed)
    2. Increase batch size per GPU
    3. Use pipeline parallelism for very large models
    4. Check network bandwidth between GPUs
    5. Avoid DeepSpeed ZeRO 3 for multi-node (slow)
  </Accordion>
</AccordionGroup>

## Benchmark Methodology

### Inference Benchmarks

<Info>
  **Setup:**

  * **Hardware:** A100-SXM4-80G GPU (single GPU unless noted)
  * **Software:** PyTorch 2.0.1, CUDA 11.8, Flash-Attention 2
  * **Task:** Generate 2048 tokens from short prompt
  * **Metric:** Average tokens/second (includes prompt processing)
  * **Script:** [profile.py](https://qianwen-res.oss-cn-beijing.aliyuncs.com/profile.py)
</Info>

### Quality Benchmarks

<Info>
  **Benchmarks:**

  * **MMLU:** 5-shot multiple-choice questions (57 subjects)
  * **C-Eval:** 5-shot Chinese evaluation (52 subjects)
  * **GSM8K:** 8-shot grade school math problems
  * **HumanEval:** 0-shot Python code generation

  **Evaluation:** Best scores from official reports and [OpenCompass](https://opencompass.org.cn/leaderboard-llm)
</Info>

### Memory Benchmarks

<Info>
  **Measurement:**

  * Peak GPU memory during generation (via `torch.cuda.max_memory_allocated()`)
  * Includes model weights, KV cache, and temporary buffers
  * Single-batch inference unless otherwise noted
</Info>

## Next Steps

<CardGroup cols={2}>
  <Card title="GPTQ Quantization" icon="microchip" href="/quantization/gptq">
    Implement GPTQ quantization based on these benchmarks
  </Card>

  <Card title="KV Cache Quantization" icon="database" href="/quantization/kv-cache">
    Enable KV cache quantization for your use case
  </Card>

  <Card title="Deployment Guide" icon="rocket" href="/deployment">
    Deploy optimized models in production
  </Card>

  <Card title="Fine-tuning" icon="sliders" href="/finetuning">
    Fine-tune quantized models for your domain
  </Card>
</CardGroup>