Qwen-2.5 Family Transformer Models

• Qwen-2.5 family are compact, open-source, decoder-only Transformer models from Alibaba, engineered for creative and conversational AI.
• They feature three variants (0.5B, 1.5B, and 3B parameters) with scalable architectures using FlashAttention v2 and gradient checkpointing for efficiency.
• Pretrained on 1.2 trillion tokens and fine-tuned with Direct Preference Optimization, these models excel in dialogue generation and real-time deployment on consumer-grade hardware.

The Qwen-2.5 family comprises compact, open-source, decoder-only Transformer models, developed and released by Alibaba Group in September 2024. Serving as successors to the original Qwen models, the Qwen-2.5 series is specifically engineered for high-quality generation in creative and conversational domains, with strong emphasis on resource-efficient scaling. Unlike the Qwen-1.8B, Qwen-7B, and Qwen-14B models described in earlier technical reports (Bai et al., 2023), Qwen-2.5 introduces three distinct variants (0.5B, 1.5B, and 3B parameters) with shared architecture, differentiated by layer count, hidden dimension, and attention head allocation. The series is characterized by its robust performance on tasks such as realistic movie dialogue generation and its applicability to real-time deployment on consumer-grade hardware (Gupta, 22 Feb 2025).

1. Architectural Specifications

The Qwen-2.5 family adheres to a unified decoder-only Transformer design, implementing layer normalization, GELU activations, and rotary positional embeddings. Each model variant is distinguished through scaling hyperparameters that systematically increase model capacity:

Variant	Layers (L)	Hidden Dim (H)	Attention Heads (A)	Total Params (P)
Qwen-2.5-0.5B	24	1,024	16	0.5 × 10^9
Qwen-2.5-1.5B	24	2,048	32	1.5 × 10^9
Qwen-2.5-3B	32	2,560	32	3.0 × 10^9

The approximate parameter count follows $P ≃ 12L H^2$ for decoder-only Transformers. The architecture incorporates FlashAttention v2 for efficiency, layer norm at every attention and MLP block, and supports gradient checkpointing and mixed precision operations. No Qwen-2.5 variant appears in the 2023 Qwen technical report, and layer counts, dimensions, and scaling specifics are solely reported for the new 0.5B, 1.5B, and 3B models (Gupta, 22 Feb 2025).

2. Pretraining Methodology

All Qwen-2.5 models are pretrained on approximately 1.2 trillion tokens drawn from diverse sources including web data, code, and conversational corpora. The pretraining protocol utilizes AdamW optimization ($\beta_1=0.9$, $\beta_2=0.95$, $\epsilon=10^{-6}$), linear warmup to a peak learning rate of $1 \times 10^{-4}$ over 10k steps, followed by cosine decay. Training is performed with a batch size of 512 sequences × 2,048 tokens, for approximately 200k steps. The pretraining regimen is designed to accelerate convergence and enhance generalization across both creative and dialogue generation benchmarks.

3. Fine-Tuning Strategies

Subsequent fine-tuning focuses on movie dialogue generation, leveraging the Cornell Movie-Dialog Corpus—comprising approximately 220k prompt-response pairs. Preprocessing includes sliding-window extraction and truncation to a maximum of 512 tokens per sequence, with an 80/20 train/test split, reserving 20% of the training set for validation. Optimization employs AdamW ($\text{LR}=2 \times 10^{-5}$, weight decay=0.01), with batch sizes of four packed sequences per step, yielding an effective batch size of roughly 2k tokens. Fine-tuning steps: $\sim$5k for 0.5B, $\sim$8k for 1.5B, and $\sim$10k for 3B, conducted on a single NVIDIA RTX 3060 Ti (8GB VRAM).

To maximize efficiency and model performance under hardware constraints, several mechanisms are employed:

• 4-bit quantization reduces model memory footprint.
• QLoRA adapters enable low-rank adaptation via frozen base weights.
• FlashAttention v2 accelerates attention computation.
• Gradient accumulation distributes updates over multiple micro-batches.
• NEFTune introduces Gaussian noise into embeddings for enhanced regularization.

4. Direct Preference Optimization (DPO) for Alignment

Qwen-2.5 models are further improved using Direct Preference Optimization (DPO), leveraging 10k prompt-pair responses scored by GPT-4o as AI feedback. DPO updates directly maximize the likelihood of preferred completions without an explicit reward modeling step. Manual tuning of the DPO hyperparameter β preceded convergence, requiring approximately 3k additional optimization steps. This alignment process ensures crafted outputs are more closely attuned to human-assessed preference criteria.

5. Computational Efficiency and Scaling

Deployment and training strategies are explicitly designed for maximal efficiency on low-memory hardware. By adhering to a staged approach—in which initial experimentation occurs with the smallest variant—subsequent scaling leverages quantization, QLoRA, and FlashAttention to maintain VRAM utilization below 7.5 GB across all variants. Key memory-saving techniques include:

• 4-bit quantization: Transforms 32-bit weights to 4 bits (via scale and zero-point), facilitating large models on consumer GPUs.
• QLoRA adapters: Freeze the full weight matrix $W \in \mathbb{R}^{H \times H}$ and learn Delta $W$ through low-rank factors $U,V \in \mathbb{R}^{H \times r}$, $r \ll H$.
• Gradient checkpointing: Activations periodically saved; otherwise recomputed during backpropagation, reducing memory pressure.

Sample efficiency is improved through staged model growth, attention kernel acceleration, and layer-wise checkpointing.

6. Empirical Benchmarks and Performance

Automatic and human-centric evaluation spans perplexity, BLEU score, G-Eval criteria, and head-to-head preference testing:

Metric	Q-0.5B	Q-1.5B	Q-3B	Llama 3.2 0.7B	Gemma 3B
Perplexity	3.45	2.62	2.10	3.20	2.50
BLEU	15.2	18.7	21.3	17.8	20.1

G-Eval scoring (0–1 scale) across coherence, consistency, fluency, and relevance demonstrates improvement from base to DPO-aligned Q-3B (e.g., coherence: 0.48 → 0.67; fluency: 0.50 → 0.68). Human preference evaluations indicate win rates of 11% (base Q-3B), 37% (fine-tuned Q-3B), and 52% (DPO Q-3B) on 100 prompt pairs. A plausible implication is that alignment via DPO yields measurable gains in output quality beyond supervised fine-tuning.

7. Deployment Practices and Model Use Cases

Qwen-2.5 variants are tailored to different resource profiles and creative demands:

• Qwen-2.5-0.5B: Suitable for 4GB VRAM, permits real-time chat and rapid prototyping, albeit with lower relevance and coherence.
• Qwen-2.5-1.5B: Optimizes quality/latency trade-off on 8GB VRAM, recommended for creative outputs.
• Qwen-2.5-3B: Delivers superior generation in the small-model domain post-DPO, deployable when VRAM and latency constraints permit.

Best practices encompass mandatory use of 4-bit quantization and QLoRA adapters for any >1B parameter fine-tuning on ≤8GB VRAM, application of FlashAttention for batch size maximization, DPO alignment (β≈0.5) for preference-driven generation, and NEFTune regularization to support generalization in text synthesis tasks beyond dialogue (Gupta, 22 Feb 2025).

These models stand as exemplars for lightweight, open-source generative pipelines in conversational and creative AI, directly addressing practical deployment challenges in resource-constrained environments.