---
title: 'Qwen2.5-Plus: Sparse MoE LLM'
url: https://www.emergentmind.com/topics/qwen2-5-plus
type: topic
---

# Qwen2.5-Plus: Sparse MoE LLM

Qwen2.5-Plus is a proprietary high-capacity Mixture-of-Experts (MoE) large language model (LLM) variant within the Qwen2.5 family, offered as a hosted API via Alibaba Cloud Model Studio. It extends the open-weight Qwen2.5 models (ranging from 0.5B to 72B parameters) by introducing sparse MoE layers, enabling efficient inference and competitive performance relative to closed-weight counterparts such as GPT-4o, while maintaining a significantly lower total inference cost. Qwen2.5-Plus incorporates advances in data scaling, training methodology, and model architecture, and serves as the foundation for several specialized domain models in mathematics, coding, and multimodal tasks [2412.15115].

## 1. Architecture and MoE Mechanism

Qwen2.5-Plus adopts a decoder-only Transformer backbone, closely following the Qwen2.5-72B model, with the following main specifications:

- 80 Transformer decoder layers
- Multi-head self-attention: 64 query heads and 8 key/value heads per layer
- Rotary positional embeddings (RoPE) with pre-normalization and RMSNorm
- SwiGLU activation in feed-forward networks (FFNs)
- Maximum context length: 131,072 tokens (generation up to 8,192 tokens)

The critical architectural distinction is the incorporation of sparse MoE feed-forward sublayers in place of dense FFNs. Specifically:

- Each layer’s two-layer FFN, $FFN(x)=W_2\,\phi(W_1x+b_1)+b_2$, is substituted with an MoE sublayer containing $E\approx32$ FFN “experts,” denoted $\{E_i(x)\}$.
- A gating network $g:\mathbb{R}^d\to\mathbb{R}^E$ outputs logits per token; gating probabilities $p(x)=\mathrm{softmax}(g(x))$ are used for expert selection.
- Top-$K$ routing ($K=2$) is performed, resulting in $\mathrm{MoE}(x)=\sum_{i\in \mathrm{Top}-K(p(x))} p_i(x) \cdot E_i(x)$.
- Experts are shared across layers (“shared-expert routing”); token-level segmentation allows each token to access a unique subset of experts, optimizing memory overhead.

The result is an increase in total parameter count to approximately 120 billion (with about 32 billion parameters activated per token), while retaining a hidden dimension 4× the embedding size (≈32,768). The attention, embedding, and normalization subcomponents are identical to dense Qwen2.5-72B.

## 2. Pre-training Regimen

Qwen2.5-Plus is pretrained on 18 trillion tokens of high-quality, domain-balanced data—a substantial increase from the 7 trillion tokens used in Qwen2. The training corpus includes:

- Filtered web crawl (with e-commerce and social domains down-sampled; science/academia up-sampled)
- Code and math corpora (from Qwen2.5-Coder and Qwen2.5-Math)
- Synthetic examples generated by Qwen2-72B-Instruct and Qwen2-Math-72B-Instruct, rigorously filtered via reward models

A byte-level BPE tokenizer (vocab size 151,643) with language-agnostic control tokens is applied in preprocessing. The training objective is the standard autoregressive next-token prediction,
$$
L_{NTP} = -\sum_{t=1}^T \log P_\theta(w_t|w_{<t}),
$$
with no auxiliary distillation or contrastive losses. Context lengths during pre-training progress in two stages: starting at 4,096 and increasing to 32,768 tokens (Qwen2.5-Plus maintains this maximum). RoPE frequencies are increased to extend positional fidelity for long contexts, moving the base from 10,000 to 1,000,000 via ABF.

## 3. Supervised Fine-tuning and Reinforcement Learning

The post-training phase combines large-scale supervised fine-tuning and multi-stage reinforcement learning:

- **Supervised fine-tuning (SFT):** Over 1 million high-quality instruction-response pairs, encompassing long-sequence generation (up to 8,192 tokens), mathematical reasoning, code synthesis and execution validation, structured data (JSON, tabular), and cross-lingual robustness. SFT uses two epochs, a sequence length of 32,768, linear learning rate decay ($7 \times 10^{-6} \to 7 \times 10^{-7}$), weight decay (0.1), and gradient clipping (norm 1.0).
- **Offline RL (Direct Preference Optimization):** 150,000 positive/negative pairs built from execution feedback and human review in math and code, optimized for 1 epoch ($\mathrm{LR}=7\times10^{-7}$) with an Online Merging Optimizer.
- **Online RL (Group Relative Policy Optimization):** A reward model trained on a mixture of proprietary and public annotated instances (truthfulness, helpfulness, harmlessness, debiasing). Policy is optimized via GRPO, a PPO-variant:
$$
L_{PPO}(\theta) = \mathbb{E}_t\left[\min(r_t(\theta)\hat{A}_t, \mathrm{clip}(r_t(\theta),1-\epsilon,1+\epsilon)\hat{A}_t)\right],
$$
where $\epsilon\approx 0.1$, $r_t(\theta)=\pi_\theta(a_t|s_t)/\pi_{old}(a_t|s_t)$.

## 4. Empirical Performance and Cost-effectiveness

Qwen2.5-Plus demonstrates strong results on a wide set of automatic benchmarks, spanning English and Chinese comprehension, mathematics, coding, and reasoning. Comparative results on selected benchmarks:

**Table: English Automatic Evaluation (%)**

| Model          | IF    | Know. | Compr. | Coding | Math  | Reasoning |
|:--------------|:------|:------|:-------|:-------|:------|:----------|
| GPT-4o (2024-08-06) | 83.28 | 68.08 | 76.51  | 58.05  | 52.36 | 66.45  |
| GPT-4o (2024-11-20) | 80.06 | 65.25 | 79.07  | 60.19  | 49.74 | 67.07  |
| Claude3.5-sonnet (Oct) | 84.22 | 74.61 | 79.02  | 67.17  | 48.67 | 70.20  |
| Qwen2.5-Plus   | 83.18 | 68.41 | 79.35  | 59.58  | 62.52 | 66.92  |

**Table: Select 70B+ Instruct Evaluation (%)**

| Model          | MMLU-Pro | MATH | GSM8K | HumanEval | IFEval | MTbench |
|:--------------|:---------|:-----|:------|:----------|:-------|:--------|
| Llama-3.1-70B | 66.4     | 68.0 | 95.1  | 80.5      | 83.6   | 8.79    |
| Llama-3.1-405B| 73.3     | 73.8 | 96.8  | 89.0      | 86.0   | 9.08    |
| Qwen2.5-72B   | 71.1     | 83.1 | 95.8  | 86.6      | 84.1   | 9.35    |
| Qwen2.5-Plus  | 72.5     | 84.7 | 96.0  | 87.8      | 86.3   | 9.30    |

Qwen2.5-Plus achieves near-parity with GPT-4o in both English and Chinese, often outperforming on mathematics and code synthesis benchmarks (e.g., GSM8K: 93.0% vs. GPT-4o 87.4%; HumanEval: 59.1% vs. 58.0%). Cost analysis on Alibaba Cloud Model Studio pricing highlights approximately double the cost-effectiveness relative to GPT-4o ($0.12/1,000 tokens vs. $0.25/1,000 tokens), with 2–3× token-throughput at comparable quality.

## 5. Inference, Quantization, and Deployment

MoE variants, including Qwen2.5-Plus, are exclusively available as hosted inference APIs; open-weight Qwen2.5-Instruct models up to 72B parameters are available with INT4 and FP8 quantized versions, but Qwen2.5-Plus quantization is not publicly released.

- **Deployment footprint:** Inference on a 4×A100 (80 GB) node requires approximately 96 GB bf16 weights and 24 GB KV cache (supporting up to 128k context), with per-token peak activation of ≈20 GB due to MoE sparse routing.
- **Mixed-precision:** Quality is maintained when hot-swapping to 16-bit compute for experts, facilitating mixed-precision acceleration.
- **Public Access:** No open-weight or quantized versions of the Qwen2.5-Plus MoE variant are published to date.

## 6. Specialized Variants and Application Domains

Qwen2.5-Plus underpins several domain-targeted and multimodal extensions:

- **Qwen2.5-Math:** Chain-of-thought math reasoning.
- **Qwen2.5-Coder:** Code generation in 40+ languages, execution-validated.
- **QwQ:** Scientific question-answering.
- **Qwen-VL:** Vision-language modeling via multimodal extension.

Specific use cases emphasize complex multi-step math problem solving, competitive code synthesis, long-document summarization and Q&A up to 128k context, and high-precision bilingual instruction following.

## 7. Contextual Significance and Prospects

Qwen2.5-Plus represents an effective integration of MoE architectures into large-scale instruction-aligned LLMs, matching closed-weight SOTA models at substantially reduced compute and monetary cost. Its modular design, efficient scaling via sparse expert routing, and robust post-training protocol position it as a foundation for a new generation of LLM-powered agent, reasoning, and multimodal solutions [2412.15115]. As of its release, the absence of open weights or quantization for Qwen2.5-Plus distinguishes its deployment model from the rest of the Qwen2.5 open-weight track. A plausible implication is that continued progress in MoE architectures and their deployment could further expand the feasible compute-accuracy frontier for industrial-scale LLMs.

Source: https://www.emergentmind.com/topics/qwen2-5-plus