> ## Documentation Index
> Fetch the complete documentation index at: https://mintlify.com/QwenLM/Qwen/llms.txt
> Use this file to discover all available pages before exploring further.
# Training Arguments
> Configure fine-tuning parameters for Qwen models
## Overview
Qwen fine-tuning uses Hugging Face Transformers' `TrainingArguments` with additional custom arguments for model loading, data processing, and LoRA training.
## Argument Classes
Four argument classes configure different aspects of training:
```python theme={null}
from transformers import HfArgumentParser
from dataclasses import dataclass, field
parser = HfArgumentParser((
ModelArguments,
DataArguments,
TrainingArguments,
LoraArguments
))
model_args, data_args, training_args, lora_args = parser.parse_args_into_dataclasses()
```
## ModelArguments
Specify which model to fine-tune.
Hugging Face model ID or local path to model checkpoint:
```bash theme={null}
--model_name_or_path Qwen/Qwen-7B-Chat
```
## DataArguments
Configure training and evaluation data.
Path to training data JSON file:
```bash theme={null}
--data_path ./data/train.json
```
Path to evaluation data JSON file (optional):
```bash theme={null}
--eval_data_path ./data/eval.json
```
Use lazy data loading to reduce memory usage:
```bash theme={null}
--lazy_preprocess
```
Enable for very large datasets that don't fit in memory.
## TrainingArguments
Extends standard Hugging Face `TrainingArguments` with Qwen-specific options.
### Core Training Parameters
Directory for saving model checkpoints and outputs:
```bash theme={null}
--output_dir ./output/qwen-finetuned
```
Number of training epochs:
```bash theme={null}
--num_train_epochs 5
```
Batch size per GPU during training:
```bash theme={null}
--per_device_train_batch_size 4
```
Batch size per GPU during evaluation:
```bash theme={null}
--per_device_eval_batch_size 8
```
Number of steps to accumulate gradients before updating:
```bash theme={null}
--gradient_accumulation_steps 4
```
Effective batch size = batch\_size × gradient\_accumulation\_steps × num\_gpus
### Learning Rate
Initial learning rate:
```bash theme={null}
--learning_rate 1e-4
```
Learning rate schedule:
* `linear`: Linear decay
* `cosine`: Cosine annealing
* `constant`: No decay
```bash theme={null}
--lr_scheduler_type cosine
```
Number of warmup steps:
```bash theme={null}
--warmup_steps 100
```
Warmup ratio (alternative to warmup\_steps):
```bash theme={null}
--warmup_ratio 0.1
```
### Optimization
Optimizer to use:
* `adamw_torch`: PyTorch AdamW
* `adamw_hf`: Hugging Face AdamW
* `adafactor`: Adafactor (memory efficient)
```bash theme={null}
--optim adamw_torch
```
Weight decay coefficient:
```bash theme={null}
--weight_decay 0.01
```
Adam beta1 parameter
Adam beta2 parameter
Maximum gradient norm for clipping:
```bash theme={null}
--max_grad_norm 1.0
```
### Model Configuration
Maximum sequence length (input + output):
```bash theme={null}
--model_max_length 4096
```
Enable LoRA fine-tuning:
```bash theme={null}
--use_lora
```
Directory for caching downloaded models:
```bash theme={null}
--cache_dir ./cache
```
### Checkpointing
When to save checkpoints:
* `steps`: Every `save_steps`
* `epoch`: Every epoch
* `no`: No saving
```bash theme={null}
--save_strategy steps
```
Save checkpoint every N steps:
```bash theme={null}
--save_steps 200
```
Maximum number of checkpoints to keep:
```bash theme={null}
--save_total_limit 3
```
### Evaluation
When to run evaluation:
* `steps`: Every `eval_steps`
* `epoch`: Every epoch
* `no`: No evaluation
```bash theme={null}
--evaluation_strategy steps
```
Evaluate every N steps:
```bash theme={null}
--eval_steps 100
```
### Logging
Log metrics every N steps:
```bash theme={null}
--logging_steps 10
```
TensorBoard log directory:
```bash theme={null}
--logging_dir ./logs
```
Reporting integrations:
* `tensorboard`
* `wandb`
* `none`
```bash theme={null}
--report_to tensorboard
```
### Performance
Use FP16 mixed precision:
```bash theme={null}
--fp16
```
Use BF16 mixed precision (recommended for modern GPUs):
```bash theme={null}
--bf16
```
Enable gradient checkpointing to reduce memory:
```bash theme={null}
--gradient_checkpointing
```
Path to DeepSpeed config file:
```bash theme={null}
--deepspeed ./ds_config.json
```
### Distributed Training
Local rank for distributed training (set automatically)
Find unused parameters in DDP:
```bash theme={null}
--ddp_find_unused_parameters
```
## Complete Example
```bash theme={null}
python finetune.py \
--model_name_or_path Qwen/Qwen-7B \
--data_path ./data/train.json \
--eval_data_path ./data/eval.json \
--output_dir ./output/qwen-finetuned \
--num_train_epochs 3 \
--per_device_train_batch_size 2 \
--per_device_eval_batch_size 2 \
--gradient_accumulation_steps 8 \
--evaluation_strategy steps \
--eval_steps 100 \
--save_strategy steps \
--save_steps 200 \
--save_total_limit 3 \
--learning_rate 1e-4 \
--weight_decay 0.01 \
--warmup_ratio 0.03 \
--lr_scheduler_type cosine \
--logging_steps 10 \
--model_max_length 2048 \
--gradient_checkpointing \
--bf16 \
--deepspeed ds_config.json
```