---
title: Qwen3-30B & Qwen3-235B MoE Models
url: https://www.emergentmind.com/topics/qwen3-30b-and-qwen3-235b-models
type: topic
---

# Qwen3-30B & Qwen3-235B MoE Models

Qwen3-30B and Qwen3-235B are large-scale, Mixture-of-Experts (MoE) Transformer language models that represent the flagship open-source offerings in the Qwen3 series. These models are architected for efficiency, multilingual breadth, and dual-mode operation—combining explicit chain-of-thought (“thinking”) reasoning with direct response (“non-thinking”) generation under a unified control mechanism. Both models expand the MoE paradigm to 128 experts per layer, support extended context windows (up to 128K tokens in some variants), and introduce mode-switching and thinking-budget controls to balance latency and reasoning quality. They set state-of-the-art benchmarks in multilingual, reasoning, code, and agentic inference tasks within the open-source LLM ecosystem, while enabling substantial inference cost reductions compared to dense-model counterparts [2505.09388][2507.19766][2510.26732][2511.02237].

## 1. Model Architectures and Mixture-of-Experts Design

Qwen3-30B-A3B and Qwen3-235B-A22B employ a decoder-only Transformer structure with all feed-forward sublayers in every Transformer block replaced by a MoE sublayer. Each MoE layer contains 128 experts; per token, the top-8 experts by routing score are selected. This high-degree sparsity enables large parameter counts with active parameter footprints much lower than dense models of similar effective size.

**Architectural Details**:

| Model           | Layers | Attention Heads | Experts / Layer | Active Experts | MoE Dim per Expert | Total Params | Context Window |
|-----------------|--------|-----------------|-----------------|---------------|--------------------|--------------|---------------|
| Qwen3-30B-A3B   | 48     | 32Q / 4KV       | 128             | 8             | 2048/768 SwiGLU    | ≈ 30 B       | 128 K         |
| Qwen3-235B-A22B | 94/96  | 64Q / 4KV       | 128             | 8             | 4096/1536 SwiGLU   | ≈ 235 B      | 128 K         |

Key MoE Implementation Features:

- **Routing / Gating**: Linear map $g = W x$ followed by softmax, then top-8 gating per token—i.e., for token representation $x$, select $k=8$ experts $S$ with highest $p_i = \text{softmax}(g)_i$ and compute $moe(x) = \sum_{i\in S} \frac{p_i}{\sum_{j\in S} p_j} E_i(x)$, with $E_i$ a two-layer SwiGLU expert.
- **Global Load-Balancing**: Global-batch regularization $L_{balance} = \sum_i \text{Load}_i^2$ where $\text{Load}_i$ is the batch fraction routed to expert $i$, ensuring uniform expert utilization [2505.09388].
- **Adapters**: The “A3B” and “A22B” suffixes denote lightweight adapter modules (≈3B and ≈22B trainable parameters, respectively) inserted in each block. These adapters allow efficient context extension (≥128K tokens) and facilitate fine-tuning or reinforcement learning without modifying all core weights [2507.19766].

*This approach (Editors' term: “full-depth blockwise MoE”) provides strong parameter scaling with manageable hardware demand, as only 8/128 experts are used per token per layer, dramatically reducing active memory and FLOPs.*

## 2. Unified Reasoning and Latency Modes

Qwen3 introduces an explicit *mode control* that enables unified “thinking” (multi-step reasoning, e.g., chain-of-thought) and “non-thinking” (direct answer) behavior within a single model.

- **Chat-Template Mode Flags**: User messages are tagged with `/think` (reasoning) or `/no_think` (non-reasoning). Model outputs are correspondingly wrapped with `<think>...</think>`, with the final answer following [2505.09388].
- **Thinking Budgets**: User-supplied budgets $B$ (token count inside `<think>` block) restrict the number of reasoning tokens. Once $B$ is exceeded, the model forcibly emits the answer. Empirically, performance on reasoning tasks (math/coding/STEM) scales smoothly with $B$ (e.g., $B \in \{2K, 4K, 8K, 16K\}$), allowing direct control over latency–quality trade-offs.
- **Supervised Fine-Tuning**: During SFT (“Stage 3”), samples are mixed between thinking and non-thinking modes using these template flags, leading to robust in-context switching and hybrid operation.

*This dual-mode operation eliminates the need for separate chat and reasoning models, and the thinking budget mechanism affords fine-grained runtime controllability of cost versus quality.*

## 3. Evaluation Metrics, Benchmarks, and Reasoning Performance

The Qwen3-30B and Qwen3-235B models have been evaluated extensively under both controlled reasoning benchmarks and downstream task settings.

**Summary of Core Results**:

| Model           | AIME’24/’25 Reasoning (think) | Coding (LiveCodeBench, think) | MMLU Redux (think) | Multilingual (MT-AIME2024, think, 55 langs) |
|-----------------|-------------------------------|-------------------------------|--------------------|---------------------------------------------|
| Qwen3-235B-A22B | 85.7% / 81.5%                 | 70.7%                         | 92.7%              | 80.8%                                      |
| Qwen3-30B-A3B   | n/a  / n/a                    | 62.6%                         | 89.5%              | n/a                                        |

- For final-answer (“step-by-step to answer”) correctness, Qwen3-235B-A22B leads its 30B counterpart by ~4–7% absolute across most reasoning and coding benchmarks [2505.09388].
- In infrastructure-agnostic benchmarks (19/79-problem sets, across multiple domains), Qwen3-235B achieves average final-score similarity 0.529 (19-problem) and 0.487 (79-problem) versus Qwen3-30B’s 0.514 and 0.477 respectively. Notably, Qwen3-30B achieves higher *step-accuracy* (average semantic similarity of reasoning steps) at scale: 0.513 vs 0.488, reflecting more coherent reasoning chains in some settings [2510.26732].
- Both models exhibit exceptional *consistency* (run-to-run stddev 0.013–0.017), three times better than most peer models, making them suited for applications requiring stable behavior [2510.26732].
- Step-accuracy and final correctness are only weakly correlated ($r \approx 0.095$), indicating more steps does not guarantee correctness—a feature relevant for transparency and audit scenarios.

**Parameter Efficiency Paradox**: The marginal returns to increased parameter count diminish sharply above 70B; e.g., the 14B dense Phi-4-mini can outperform larger sparse MoE models on some reasoning tasks [2510.26732].

## 4. Efficiency, Latency, and Expert Routing Innovations

The MoE design in Qwen3-30B/235B reduces computational load by activating only a small expert subset per token. Additional efficiency gains are realized via opportunistic, batch-aware expert routing (OEA) [2511.02237]:

- **Memory-Bound Latency Model**: For batch size $B=16$ and $k=8$ active experts per token, the expected number of distinct experts $T(B)$ is
  $$
  T(B) = N \cdot [1 - (1 - k/N)^B]
  $$
  For $N=128$, $k=8$, $B=16$: $T\approx 82$.
- **Roofline Latency**: Each expert’s fetch latency dominates: $MoE$-$latency = b\cdot T + a\cdot(Bk)$, with $b$ for weight load and $a$ per-token compute.
- **Batch-Aware Routing (OEA)**: Inference-time “piggyback” routing assigns baseline experts per token ($k_0 \leq k$), then opportunistically fills the remainder among already-selected experts across the batch, reducing distinct experts fetched per batch without retraining.
- **Empirical Gains** (at $B=16$):
  - Qwen3-30B: OEA reduces MoE layer latency by 39% (from 175.7 μs to 106.8 μs at $k_0=3$).
  - Qwen3-235B: 15% reduction (from 119.4 μs to 101.4 μs at $k_0=5$).
  - No statistically significant loss in main benchmarks (AIME, GPQA, LiveCodeBench, MATH500) up to moderate $k_0$ pruning. Quality declines only with aggressive reduction of active experts per token.

Additional impact stems from the sparse MoE structure yielding $\sim3\times$–$4\times$ lower FLOPs and memory costs per token compared to dense models of equal size, and up to $\sim2\times$ higher throughput on fixed hardware in production deployments [2505.09388][2511.02237].

## 5. Reinforcement Learning for Ultra-Long Reasoning

Qwen3-30B-A3B and Qwen3-235B-A22B support fine-tuning with reinforcement learning for ultra-long reasoning chains (UloRL) [2507.19766]:

- **Segmented RL Rollouts**: Rather than generating 128K-token episodes in full, RL training divides outputs into $M$ segments, updating on completed segments in parallel for 2.06× speedup (e.g., 1240s → 601s per step at $M=4$).
- **Dynamic Masking of Mastered Positive Tokens (DMMPTs)**: When the model reliably predicts positive tokens ($p_\theta(t|c)\geq\tau$, $\tau=0.99$) and entropy falls below $\sigma=0.2$, these tokens are dynamically masked to prevent entropy collapse during RL fine-tuning.
- **Results**: UloRL with DMMPTs elevates Qwen3-30B-A3B from 70.9% to 82.8% (AIME-2025), and with token window extension (Yarn) to 140K reaches 85.1%, surpassing even Qwen3-235B-A22B at 81.5%. For ultra-long tasks, adapters and segment-level RL drive more effective learning than scale alone.
- **Implications**: Adapter-tuning allows for outsized performance improvements without retraining the full parameter core, and entropy stabilization prevents premature overfitting of positive token spans. However, large memory requirements (>128K context) and extra verifier models are needed.

## 6. Multilingual Generalization

Qwen3-235B-A22B is pre-trained on 36 trillion tokens in 119 languages/dialects (Qwen2.5: 29), supporting broad zero-shot and cross-lingual capacity [2505.09388]. Key outcomes include:

- MMMLU (14 languages): 86.7% (base), 84.3% (post-fine-tune)
- MGSM (8-language math CoT): 83.53%
- MT-AIME2024 (55 langs): 80.8% (post-training, thinking mode)
- INCLUDE (44-language regional): 73.46%
- Polymath (18 langs): 54.7%
- Benchmarks such as Flores-101 and Belebele (122 variants) confirm strong Indo-European and broader international performance.

This indicates high-quality multilingual pretraining and beneficial knowledge transfer from high-resource to low-resource languages.

## 7. Limitations and Recommended Use Cases

Both Qwen3-30B and Qwen3-235B, while setting open-source state-of-the-art in multi-domain “thinking” tasks, exhibit several nuanced characteristics:

- **Consistency**: Outstanding stability (stddev 0.013–0.017) across runs and domains, superior to nearly all contemporaries [2510.26732].
- **Transparency**: Weak link between step-accuracy and final correctness ($r\approx0.095$); not optimal for settings demanding highly interpretable chain-of-thoughts (e.g., education, audits)—whereas models like DeepSeek-R1 (step-accuracy 0.716) have an edge.
- **Resource Footprint**: Ultra-long output RL and large context operation require extensive GPU memory and runtime verifier models.
- **Scaling Law**: Additional parameter count above 70B yields diminished returns; careful data curation and instruction tuning dominate at high scale.

**Practical Suitability**: These models suit production environments demanding high repeatability and latency-tunable reasoning, less so scenarios prioritizing stepwise explainability or ultra-interpretable proof chains [2510.26732][2507.19766].

---

Collectively, Qwen3-30B-A3B and Qwen3-235B-A22B advance the MoE LLM paradigm by combining unified reasoning/direct modes, expert-activating sparsity, robust multilingual coverage, adapter-driven extensibility, and batch-optimized inference—all with consistent, reproducible outcomes across infrastructure and benchmarks.

Source: https://www.emergentmind.com/topics/qwen3-30b-and-qwen3-235b-models