Qwen2.5-1.5B Transformer Model

• Qwen2.5-1.5B is a dense, decoder-only Transformer model with 1.5B parameters designed for efficient language understanding and deterministic reasoning.
• It features 28 Transformer layers with SwiGLU activation, rotary positional encodings, and advanced quantization techniques for long-context inference.
• Post-training employs supervised fine-tuning and RLHF to enhance performance in tasks such as math and code synthesis while ensuring low latency for edge deployments.

Qwen2.5-1.5B is a dense, decoder-only Transformer model within the Qwen2.5 foundation model suite, designed to deliver broad language understanding, reasoning, and task-following capabilities under strict computational and memory constraints. With approximately 1.5 billion parameters, Qwen2.5-1.5B targets edge and enterprise deployments requiring strong deterministic reasoning, low inference latency, and efficiency in both general-purpose and domain-specialized applications (Qwen et al., 2024).

1. Model Architecture and Training Paradigm

Qwen2.5-1.5B's architecture consists of 28 Transformer decoder layers, each with 12 grouped query attention heads and 2 key/value heads (GQA), facilitating efficient KV-cache utilization during long-context inference. The model employs SwiGLU activation, pre-normalization with RMSNorm, tied input/output embeddings, and rotary positional encodings (RoPE) augmented with QKV bias and ABF frequency scaling. The pre-training sequence length is 32,768 tokens, with effective generation up to 8,192 tokens in post-training variants. The tokenizer is a byte-level BPE with a 151,643-token vocabulary (Qwen et al., 2024).

The pre-training dataset comprises 18 trillion tokens, including high-quality web text, mathematics, code, academic/scientific material, and a tailored blend of upsampled science/technology with downsampled social content. Training follows the autoregressive maximum-likelihood objective and applies empirical Chinchilla-optimal scaling for learning rate and batch size (Qwen et al., 2024).

The model is distributed in several quantized forms (bfloat16, 8b, 4b), supporting efficient inference on a wide range of hardware, from cloud clusters to consumer-level GPUs and CPUs. Quantization to 4b achieves ∼4× memory savings with minor degradation in task performance (Qwen et al., 2024).

2. Post-Training: Supervised and Reinforcement Learning Alignment

Post-training for Qwen2.5-1.5B involves a two-stage alignment process: large-scale supervised fine-tuning (SFT) and reinforcement learning from human (and reward-model) preferences (RLHF). SFT draws from >1M curated instruction–response pairs spanning long-form generation, mathematics (including chain-of-thought traces from Qwen2.5-Math), code synthesis (validated by static analyzers), and complex structured tasks. The SFT stage uses two epochs over the curated set, with a per-sequence length of 32,768 tokens, cosine learning rate decay from $7 \times 10^{-6}$ to $7 \times 10^{-7}$, and weight decay of 0.1 (Qwen et al., 2024).

Offline RL employs Direct Preference Optimization (DPO) over ∼150,000 preference pairs; online RL applies Group Relative Policy Optimization (GRPO) using a reward model trained on human-annotated and automatically labeled prompt–response pairs under criteria such as truthfulness, helpfulness, conciseness, and harmlessness. Rollout batches consist of 2,048 episodes with 8 responses per query (Qwen et al., 2024).

Instruction-tuned and alignment variants are denoted Qwen2.5-1.5B-Instruct (general task-following) and Qwen2.5-Math-1.5B or Qwen2.5-Math-Instruct-1.5B (math-specialized), each following the shared architectural recipe but with domain-adapted post-training datasets and objectives (Yang et al., 2024, Zheng et al., 2 Jun 2025).

3. Specialized Distillation, Domain Adaptation, and Internationalization

Distillation Practices

DistilQwen2.5-1.5B applies a two-stage knowledge distillation process involving multi-agent black-box teachers (e.g., Qwen-max, GPT-4o) and logit-level fusion with a white-box teacher (such as Qwen2.5-14B). Black-box KD augments and filters instruction–response data using a team of expansion, rewriting, selection, and verification agents, producing ∼3M high-quality pairs. White-box fusion compresses teacher logits into the student, storing and aligning only top-K=10 logit vectors per token. This pipeline nearly doubles scores on strict instruction-following tasks such as IFEval and AlpacaEval over the original checkpoint (Wang et al., 21 Apr 2025).

Domain Adaptation: Mathematics

Qwen2.5-Math-Instruct-1.5B employs a three-phased, self-improvement pipeline: (1) pre-training on synthetic and scraped math corpora (including data from Qwen2-Math-72B-Instruct); (2) iterative supervised fine-tuning (SFT) with chain-of-thought and tool-integrated reasoning (TIR) datasets, leveraging a reward model for rejection sampling; and (3) RL via GRPO, with combined reward signals from a scalar-valued reward model and a sparse verifier. Inference leverages a reward model for best-of-N sampling, significantly boosting math task accuracy (Yang et al., 2024, Zheng et al., 2 Jun 2025).

Internationalization

For Portuguese, the Amadeus-Verbo-Qwen2.5-1.5B adapts the base and instruction-tuned checkpoints via full-parameter SFT on ∼80k curated instruction–response pairs in Brazilian Portuguese. This process preserves architectural identity with the base Qwen2.5-1.5B, applying language-specific prompts and task mixes; results show parity on most tasks with the original, with gains becoming pronounced at ≥7B parameter scales (Cruz-Castañeda et al., 20 May 2025).

4. Empirical Benchmarking and Task-Specific Performance

Qwen2.5-1.5B substantially outperforms its predecessors (Qwen2-1.5B) and peers (Gemma2-2.6B, Llama-3.2-1B) across numerous reasoning, math, and code tasks. Notably, instruction tuning and RLHF nearly triple MATH benchmark accuracy (25.3%→55.2%), improve HumanEval coding from 42%→62%, and drive MMLU-Pro from 23%→32% (Qwen et al., 2024). The math-specialized Qwen2.5-Math-Instruct-1.5B achieves state-of-the-art results for its scale—e.g., GSM8K at 94.1% (RM@8 sampling), MATH (4-shot CoT) at 83.9%, and strong zero-shot/few-shot performance on math competitions (Yang et al., 2024).

In retrieval and query rewriting (TongSearch-QR), RL-tuned Qwen2.5-1.5B-Instruct achieves nDCG@10=24.6 on BRIGHT, outperforming prompt-only and dense retriever baselines by large margins, and yielding an order-of-magnitude better cost-efficiency compared to large LLMs such as GPT-4o and QwQ-32B (Qin et al., 13 Jun 2025).

For AI-native edge deployment, Qwen2.5-1.5B is empirically located at the "stability transition": deterministic pass@1 accuracy jumps from 0.373 (Llama-3.2-1B) to 0.531 and the instability gap $\Delta_5$ shrinks from 0.356 to 0.138. Edge Score (accuracy per ms×GB) is 56.4 (×10⁴), over three times better than 1B peers, confirming Qwen2.5-1.5B as the minimal reliable model for low-latency 6G reasoning (Ferrag et al., 2 Mar 2026).

5. Efficient RL, Scalability, and Optimization Techniques

Qwen2.5-1.5B supports a spectrum of reinforcement learning (RL) and algorithmic efficiency strategies:

• GRESO (GRPO with Efficient Selective Rollout): By filtering zero-variance (uninformative) prompts based on temporal consistency in reward traces, GRESO yields a 2.3–2.4× reduction in RL rollout cost and up to 2.0× reduction in total wall-clock training time, with no tradeoff in model accuracy (<0.3% deviation). Components include probabilistic pre-rollout filtering, self-adjusting exploration rates for easy/hard prompts, and adaptive batch sizing (Zheng et al., 2 Jun 2025).
• Thinker Framework: Inspired by Dual Process Theory, Qwen2.5-1.5B is successfully trained in a fast-slow stagewise QA regime. Fast Thinking achieves 25.18% accuracy using 1,000 tokens (with minimal loss vs. unconstrained baseline at much higher token budget); the full four-stage Thinker framework raises accuracy to 27.33% (+6.7% relative), and improves efficiency in both short- and long-form reasoning (Chung et al., 27 May 2025).
• LaPha (Latent Poincaré Shaping): LaPha augments Qwen2.5-Math-1.5B with a Poincaré embedding head, dense potential-based shaping rewards, and value-based search guiding. On MATH-500, base accuracy rises from 66.0%→88.2% with value-head–guided search, and similar gains are observed on AIME’24 and AIME’25. Poincaré geometry preserves reward informativeness for long and branched reasoning trajectories (Xia et al., 10 Feb 2026).

6. Robustness, Bias, and Limitations

Empirical audit of Qwen2.5-1.5B reveals scale-sensitive susceptibilities and deployment caveats:

• Positional Bias: In financial binary decision tasks, Qwen2.5-Instruct-1.5B exhibits extreme primacy bias ($r \approx 0.87$ on 10 categories), with median 60–80 percentage point boosts favoring the first-listed option. This bias emerges mechanistically in mid-to-late layers (layers 12–24) and is dominated by a small set of "universal bias heads." Model scaling and prompt engineering partially attenuate—but do not eliminate—this effect; domain-specific mitigation requires randomized orderings and targeted regularization of implicated attention heads (Dimino et al., 25 Aug 2025).
• Language Adaptation: For low-resource languages (e.g., Portuguese), single-epoch SFT of Qwen2.5-1.5B yields performance parity rather than improvement in few-shot tasks; meaningful improvement requires larger models ($\geq$7B) or data/PEFT augmentation (Cruz-Castañeda et al., 20 May 2025).
• RL Reward Modeling: For RL-fine-tuned retrieval and reasoning tasks, dense similarity-based rewards (e.g., from fixed embedding models, as in TongSearch-QR) tolerate noisy supervision and scale robustly. However, reliance on a single relevance model introduces brittleness to scorer imperfections (Qin et al., 13 Jun 2025).

7. Comparative Summary and Deployment Considerations

Qwen2.5-1.5B exemplifies the best accuracy–efficiency balance at sub-2B scale: it delivers leading deterministic accuracy and stability for safety-critical single-shot reasoning, supports high-throughput, quantization-friendly inference, and adapts readily to RL, distillation, and domain-specific workflows. A plausible implication is that, for resource-constrained use cases—edge AI, 6G controllers, lightweight chat assistants—Qwen2.5-1.5B sets the practical lower bound for viable semantic reasoning without incurring stochastic output collapse or prohibitive deployment cost (Qwen et al., 2024, Ferrag et al., 2 Mar 2026). Further scaling yields incremental rather than multiplicative gains, and known limitations (such as positional bias on decision tasks) should be directly addressed in responsible deployments.