Qwen3-32B: Dense Decoder-Only LLM

• The paper introduces a 32B parameter dense decoder-only LLM that integrates chain-of-thought and non-thinking modes to enable flexible reasoning and rapid response.
• The model utilizes a 64-layer Transformer with SwiGLU blocks and RoPE, supporting efficient multilingual processing across 119 languages for tasks like code synthesis and mathematical reasoning.
• Dual inference modes, controlled by prompt flags and a token-based thinking budget, allow users to balance latency and accuracy for multi-step reasoning tasks.

Qwen3-32B is a dense, decoder-only LLM comprising approximately 32 billion parameters. An element of the Qwen3 family, it is designed to advance multilingual and multi-modal reasoning, balancing performance, scalability, and efficient inference. The architecture integrates chain-of-thought and rapid response (“thinking” and “non-thinking” modes) within a single network, enabled by prompt control flags and a user-controllable thinking budget. The model employs a 64-layer Transformer with SwiGLU feed-forward blocks and supports dynamic adaptation to diverse use cases from mathematical reasoning to code synthesis, in 119 languages and dialects (Yang et al., 14 May 2025).

1. Model Configuration

Qwen3-32B follows the standard decoder-only Transformer architecture, with parameters and subcomponent sizes shown below.

Dimension	Value	Description
Layers ($L$)	64	Number of Transformer decoder blocks
Hidden size ($d_\text{model}$)	4096	Per-token embedding/hidden state dimensionality
Attention heads ($H$)	64	Number of parallel self-attention heads
Head dimension ($d_k$)	64	Each head's key, query, value dimension
FFN inner dim ($d_\text{ff}$)	11,008	Inner dimension of feed-forward subnetwork
Vocabulary size ($|V|$)	151,669	Size of token vocabulary

Total parameters are approximately 32B, with major allocations as follows: token embeddings and output unembedding ($0.62$B each, tied or separate), self-attention and feed-forward weights ($4.3$B and $5.8$B per layer, respectively), and minor contributions (<1%) from normalization parameters. The model does not employ learned position embeddings, instead relying on RoPE (rotary positional encoding) with negligible parameter count (Yang et al., 14 May 2025).

2. Transformer Block and Dense Architecture

Each decoder block consists of a pre-normalization RMSNorm, multi-head self-attention, and a two-layer feed-forward (“SwiGLU”) unit, both wrapped in residual connections. The functional mapping for an input $x \in \mathbb{R}^{b \times T \times d}$ (batch $d_\text{model}$0, sequence length $d_\text{model}$1) is:

1. $d_\text{model}$2
2. $d_\text{model}$3
3. $d_\text{model}$4 (attention residual)
4. $d_\text{model}$5
5. $d_\text{model}$6
6. $d_\text{model}$7 (FF residual)

Within each self-attention layer:

• Query, Key, Value: $d_\text{model}$8, $d_\text{model}$9, $H$0
• Output: $H$1
• Multi-head: concatenation across $H$2 heads, projected by $H$3

The SwiGLU feed-forward network operates as:

$H$4

where $H$5, with $H$6 split into two halves of size $H$7. No Mixture-of-Experts (MoE) routing is used in Qwen3-32B; all layers are fully dense (Yang et al., 14 May 2025).

3. Inference Modes: Thinking and Non-Thinking

Qwen3-32B supports dual inference modalities managed through prompt-level control:

• Thinking mode requests extended reasoning chains via the /think control token, instructing the model to emit a > ...</think> block that contains stepwise reasoning before the answer. > > - Non-thinking mode is activated with /no_think, instructing the model to skip or minimize the reasoning block, emitting an empty <think></think> and proceeding directly to the answer. > > A user-controlled “thinking budget” ($H$8), specified in tokens, hard-caps the length of the reasoning segment. Upon reaching $H$9 tokens, an abort message is inserted—“Considering the limited time … based on the thinking directly now.”—and answer generation continues. This budget mechanism requires no additional parameters. There is no published closed-form for $d_k$0 as a function of input complexity.

This dual-mode capability allows dynamic tradeoffs between latency and reasoning depth. Increasing $d_k$1 typically improves performance on multi-hop tasks (e.g., AIME, LiveCodeBench) but increases inference latency proportionally (Yang et al., 14 May 2025).

4. Training Methodology

Pre-training is performed on approximately 36 trillion tokens from 119 languages and dialects in three progressive stages:

1. General corpus: ~30T tokens, sequence length 4096.
2. Reasoning-focused: ~5T tokens emphasizing high-quality STEM and code data, sequence length 4096, accelerated learning rate decay.
3. Long-context: Hundreds of billions of tokens, sequence length 32,768, employing YARN and DCA techniques to ensure stability and effective long-sequence modeling.

Optimization uses AdamW, with weight decay and a learning rate schedule involving warmup, peaking, and stage-specific (cosine or linear) decay as predicted by scaling laws. The final batch size per GPU also follows parameter-scaling guidelines, with a global batch size of 1–2M tokens.

Post-training comprises four distinct stages:

• Long-CoT cold-start supervised fine-tuning (SFT)
• Reinforcement learning with reasoning-specific GRPO, leveraging approximately 4000 query-verifier pairs
• Supervised fusion of thinking and non-thinking modalities
• General RL on more than 20 reward-aligned tasks (instruction, formatting, agentic outputs, RAG, etc.) (Yang et al., 14 May 2025)

5. Inference-Time Controls and Implications

At inference, only the most recent control flag (/think or /no_think) embedded in the instruction prompt determines mode. The model autonomously parses the flag and routes text generation accordingly. Thinking budget enforcement is implemented as a strict token cap within the <think> region; exceeding this budget yields an abort message and triggers immediate answer emission.

Latency-performance tradeoff is user-managed: higher $d_k$2 yields improved multi-step accuracy but increases response time. No architectural changes or extra parameters are needed to realize these dual behaviors or the budget mechanism, implying a flexible and lightweight deployment pathway for a wide range of interactive applications (Yang et al., 14 May 2025).

6. Comparative Position and Multilingual Capability

Qwen3-32B achieves state-of-the-art or competitive results versus both larger proprietary and open-source dense and MoE LLMs on benchmarks encompassing code generation, mathematical reasoning, and agentic tasks. Compared to its predecessor, Qwen2.5, Qwen3-32B increases supported languages from 29 to 119, substantially expanding cross-lingual generation and comprehension.

Community access under Apache 2.0 licensing is emphasized to enable reproducibility and further research extension. The approach of leveraging knowledge from flagship models to train smaller-scale ones suggests improved data and computational efficiency without sacrificing competitive utility (Yang et al., 14 May 2025).