> ## Documentation Index
> Fetch the complete documentation index at: https://mintlify.com/QwenLM/Qwen/llms.txt
> Use this file to discover all available pages before exploring further.

# Data Preparation

> Learn how to prepare and format training data for fine-tuning Qwen models

Proper data preparation is critical for successful fine-tuning. This guide covers the data format, preprocessing, and best practices for creating high-quality training datasets.

## Data Format

Qwen uses a standardized JSON conversation format that follows the ChatML structure. Each training sample consists of a conversation with multiple turns between user and assistant.

### Basic Structure

```json theme={null}
[
  {
    "id": "unique_identifier",
    "conversations": [
      {
        "from": "user",
        "value": "User message here"
      },
      {
        "from": "assistant",
        "value": "Assistant response here"
      }
    ]
  }
]
```

### Field Descriptions

<ParamField path="id" type="string" required>
  A unique identifier for each conversation. Used for logging and debugging.
</ParamField>

<ParamField path="conversations" type="array" required>
  An array of conversation turns. Must contain at least one user-assistant pair.
</ParamField>

<ParamField path="conversations[].from" type="string" required>
  The role of the speaker. Must be either `"user"` or `"assistant"`.
</ParamField>

<ParamField path="conversations[].value" type="string" required>
  The actual message content. Can contain multi-line text, code, or any UTF-8 content.
</ParamField>

## Multi-turn Conversations

You can include multiple user-assistant exchanges in a single conversation:

```json theme={null}
[
  {
    "id": "multi_turn_example",
    "conversations": [
      {
        "from": "user",
        "value": "What is the capital of France?"
      },
      {
        "from": "assistant",
        "value": "The capital of France is Paris."
      },
      {
        "from": "user",
        "value": "What is its population?"
      },
      {
        "from": "assistant",
        "value": "Paris has a population of approximately 2.2 million people within the city limits, and over 12 million in the metropolitan area."
      }
    ]
  }
]
```

<Tip>
  Multi-turn conversations help the model learn context tracking and coherent dialogue flow.
</Tip>

## ChatML Format Processing

The training script automatically converts your JSON data into ChatML format during preprocessing. Understanding this helps you prepare better training data.

### Automatic Template Application

Your JSON conversations are transformed into this structure:

```text theme={null}
<|im_start|>system
You are a helpful assistant.<|im_end|>
<|im_start|>user
User message here<|im_end|>
<|im_start|>assistant
Assistant response here<|im_end|>
```

### System Message

The default system message is `"You are a helpful assistant."` This is applied automatically by the preprocessing function in `finetune.py:125-176`.

<Note>
  The system message is currently hardcoded in the training script. To customize it, modify the `system_message` parameter in the `preprocess()` function.
</Note>

## Token Masking Strategy

The training script implements intelligent token masking to focus learning on assistant responses:

### What Gets Masked (Not Trained)

* System message content
* User message content
* Special tokens (`<|im_start|>`, role names)
* Padding tokens

### What Gets Trained

* **Only the assistant's responses** (excluding special tokens)

This is implemented using `IGNORE_TOKEN_ID` in the preprocessing function:

```python theme={null}
if role == '<|im_start|>user':
    _target = [im_start] + [IGNORE_TOKEN_ID] * (len(_input_id)-3) + [im_end] + nl_tokens
elif role == '<|im_start|>assistant':
    _target = [im_start] + [IGNORE_TOKEN_ID] * len(tokenizer(role).input_ids) + \
        _input_id[len(tokenizer(role).input_ids)+1:-2] + [im_end] + nl_tokens
```

<Tip>
  This masking strategy improves training efficiency by focusing gradient updates on the assistant's outputs.
</Tip>

## Creating Training Data

### Step-by-Step Example

<Steps>
  <Step title="Define Your Task">
    Identify what you want the model to learn. Examples:

    * Customer service responses
    * Code generation in a specific framework
    * Domain-specific question answering
    * Style transfer or tone adaptation
  </Step>

  <Step title="Collect Raw Data">
    Gather examples of high-quality interactions:

    ```python theme={null}
    # Example: Converting Q&A pairs to conversation format
    qa_pairs = [
        {"question": "How do I install Python?", "answer": "You can download..."},
        {"question": "What is a virtual environment?", "answer": "A virtual environment..."}
    ]
    ```
  </Step>

  <Step title="Format as JSON">
    Convert to the required conversation structure:

    ```python theme={null}
    import json

    training_data = []
    for idx, qa in enumerate(qa_pairs):
        training_data.append({
            "id": f"qa_{idx}",
            "conversations": [
                {"from": "user", "value": qa["question"]},
                {"from": "assistant", "value": qa["answer"]}
            ]
        })

    with open("training_data.json", "w", encoding="utf-8") as f:
        json.dump(training_data, f, ensure_ascii=False, indent=2)
    ```
  </Step>

  <Step title="Validate Format">
    Verify your data is correctly formatted:

    ```python theme={null}
    # Validation script
    with open("training_data.json", "r") as f:
        data = json.load(f)

    for item in data:
        assert "id" in item, "Missing id field"
        assert "conversations" in item, "Missing conversations field"
        for conv in item["conversations"]:
            assert conv["from"] in ["user", "assistant"], "Invalid role"
            assert "value" in conv, "Missing value field"

    print(f"✓ Validated {len(data)} conversations")
    ```
  </Step>
</Steps>

## Data Quality Guidelines

### Best Practices

<AccordionGroup>
  <Accordion title="Conversation Quality">
    * **Consistent tone**: Maintain consistent assistant personality
    * **Factual accuracy**: Verify all information is correct
    * **Natural flow**: Ensure conversations feel natural, not scripted
    * **Diverse examples**: Cover various phrasings and edge cases
  </Accordion>

  <Accordion title="Length Considerations">
    * **Typical range**: 50-500 tokens per response
    * **Max length**: Configure with `--model_max_length` (default: 8192)
    * **Avoid truncation**: Ensure important conversations fit within max length
    * **Balance**: Mix short and long responses
  </Accordion>

  <Accordion title="Dataset Size">
    * **Minimum**: 100-500 examples for domain adaptation
    * **Recommended**: 1,000-10,000 examples for robust fine-tuning
    * **Large-scale**: 10,000+ examples for significant behavior changes
    * **Quality > Quantity**: 100 high-quality examples beat 1,000 poor ones
  </Accordion>

  <Accordion title="Diversity">
    * **Topic coverage**: Include all relevant domains
    * **Query variations**: Multiple ways to ask the same thing
    * **Response styles**: Different levels of detail when appropriate
    * **Edge cases**: Include challenging or unusual scenarios
  </Accordion>
</AccordionGroup>

## Special Use Cases

### Code Generation

````json theme={null}
[
  {
    "id": "code_example_1",
    "conversations": [
      {
        "from": "user",
        "value": "Write a Python function to calculate fibonacci numbers"
      },
      {
        "from": "assistant",
        "value": "Here's a Python function to calculate Fibonacci numbers:\n\n```python\ndef fibonacci(n):\n    if n <= 1:\n        return n\n    return fibonacci(n-1) + fibonacci(n-2)\n```\n\nFor better performance, here's an iterative version:\n\n```python\ndef fibonacci_iterative(n):\n    if n <= 1:\n        return n\n    a, b = 0, 1\n    for _ in range(2, n + 1):\n        a, b = b, a + b\n    return b\n```"
      }
    ]
  }
]
````

### Multilingual Data

```json theme={null}
[
  {
    "id": "multilingual_1",
    "conversations": [
      {
        "from": "user",
        "value": "你好，请介绍一下自己"
      },
      {
        "from": "assistant",
        "value": "你好！我是通义千问，是阿里云开发的大型语言模型。我可以回答问题、进行对话、协助创作等。"
      },
      {
        "from": "user",
        "value": "Can you also speak English?"
      },
      {
        "from": "assistant",
        "value": "Yes, I can communicate in English as well. I'm a multilingual AI assistant developed by Alibaba Cloud, capable of understanding and responding in multiple languages including Chinese and English."
      }
    ]
  }
]
```

<Note>
  Qwen models are pretrained on multilingual data with focus on Chinese and English. They can handle code-switching naturally.
</Note>

## Data Preprocessing Options

### Lazy Preprocessing

The training script supports two preprocessing modes:

<CodeGroup>
  ```bash Lazy Mode (Recommended) theme={null}
  # Preprocesses data on-the-fly during training
  python finetune.py \
    --data_path data.json \
    --lazy_preprocess True
  ```

  ```bash Eager Mode theme={null}
  # Preprocesses all data before training starts
  python finetune.py \
    --data_path data.json \
    --lazy_preprocess False
  ```
</CodeGroup>

**Lazy preprocessing** (`LazySupervisedDataset`):

* Lower initial memory usage
* Faster training startup
* Recommended for large datasets
* Caches processed samples

**Eager preprocessing** (`SupervisedDataset`):

* All data processed upfront
* Slightly faster iteration
* Better for smaller datasets

## Validation Data

You can optionally provide evaluation data to monitor training progress:

```bash theme={null}
python finetune.py \
  --data_path train.json \
  --eval_data_path eval.json \
  --evaluation_strategy "steps" \
  --eval_steps 500
```

Your `eval.json` should follow the same format as training data.

<Warning>
  Evaluation during training increases overall training time. For production fine-tuning, consider evaluating only at the end or at sparse intervals.
</Warning>

## Complete Data Preparation Script

Here's a production-ready script for converting various formats to Qwen training format:

```python theme={null}
import json
import argparse
from pathlib import Path

def convert_to_qwen_format(input_file, output_file, format_type="qa"):
    """Convert various formats to Qwen training format.
    
    Args:
        input_file: Path to input data
        output_file: Path to save formatted data
        format_type: Type of input format (qa, chat, instruction)
    """
    with open(input_file, 'r', encoding='utf-8') as f:
        raw_data = json.load(f)
    
    formatted_data = []
    
    if format_type == "qa":
        # Question-answer pairs
        for idx, item in enumerate(raw_data):
            formatted_data.append({
                "id": f"qa_{idx}",
                "conversations": [
                    {"from": "user", "value": item["question"]},
                    {"from": "assistant", "value": item["answer"]}
                ]
            })
    
    elif format_type == "instruction":
        # Instruction-following format
        for idx, item in enumerate(raw_data):
            user_msg = item.get("instruction", "")
            if "input" in item and item["input"]:
                user_msg += f"\n\n{item['input']}"
            
            formatted_data.append({
                "id": f"inst_{idx}",
                "conversations": [
                    {"from": "user", "value": user_msg},
                    {"from": "assistant", "value": item["output"]}
                ]
            })
    
    elif format_type == "chat":
        # Already in conversation format, just validate
        formatted_data = raw_data
    
    # Validate and save
    for item in formatted_data:
        assert "id" in item and "conversations" in item
        for conv in item["conversations"]:
            assert conv["from"] in ["user", "assistant"]
            assert "value" in conv
    
    with open(output_file, 'w', encoding='utf-8') as f:
        json.dump(formatted_data, f, ensure_ascii=False, indent=2)
    
    print(f"✓ Converted {len(formatted_data)} conversations to {output_file}")

if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("--input", required=True, help="Input data file")
    parser.add_argument("--output", required=True, help="Output data file")
    parser.add_argument("--format", choices=["qa", "chat", "instruction"], 
                       default="qa", help="Input format type")
    args = parser.parse_args()
    
    convert_to_qwen_format(args.input, args.output, args.format)
```

Usage:

```bash theme={null}
python convert_data.py \
  --input raw_qa.json \
  --output training_data.json \
  --format qa
```

## Next Steps

<CardGroup cols={2}>
  <Card title="Full-Parameter Tuning" icon="server" href="/finetuning/full-parameter">
    Start training with full-parameter fine-tuning
  </Card>

  <Card title="LoRA Fine-tuning" icon="puzzle-piece" href="/finetuning/lora">
    Efficient training with LoRA adapters
  </Card>
</CardGroup>