Qwen3-8B/14B LLM Innovations

• Qwen3-8B and Qwen3-14B are dense Transformer-based LLMs that employ innovations like Grouped Query Attention and SwiGLU to enhance long-context reasoning.
• They are trained on a diverse corpus of 36 trillion tokens across 119 languages, optimizing performance in STEM, code, and multilingual benchmarks through a three-stage schedule.
• The models support dynamic reasoning modes with user-controlled thinking budgets, balancing inference cost and latency for a variety of deployment scenarios.

Qwen3-8B and Qwen3-14B are dense LLMs developed within the Qwen3 family, targeting advanced performance, resource efficiency, and extensive multilingual support. Both models leverage architectural and training innovations derived from flagship Qwen3 models, offering dynamic reasoning capabilities through an integrated thinking vs. non-thinking mode paradigm, while enabling broad deployment from edge to server applications. They exhibit strong results across STEM, code, and multilingual benchmarks, and are released under the Apache 2.0 license for maximum reproducibility and research accessibility (Yang et al., 14 May 2025).

1. Architectural Configuration

Qwen3-8B and Qwen3-14B implement dense Transformer architectures, eschewing Mixture-of-Expert (MoE) structures for streamlined inference. Qwen3-8B has 36 layers with 32 query-heads and 8 KV-heads, and Qwen3-14B uses 40 layers with 40 query-heads and 8 KV-heads. Both models forego tied input/output embeddings, increasing effective parameterization. They support context lengths up to 128K tokens, enabling long-context modeling.

Key sub-layer mechanisms include Grouped Query Attention (GQA), replacing standard multi-head attention; SwiGLU nonlinearity in the feed-forward blocks; and rotary positional embeddings (RoPE) extended via Automatic Base Frequency (ABF) up to a positional index of 1M, augmented by Dual Chunk Attention (DCA) and YARN for inference at ≥32K tokens. All pre-normalization uses RMSNorm (no QKV biases), with QK-Norm applied for attention training stability. Fine-grained, instance-level data mixtures are tailored for each scale using learned scaling laws. Untied embeddings are used for all 8B+ models, improving model capacity (Yang et al., 14 May 2025).

2. Training Pipeline and Pretraining Data

The pretraining corpus comprises approximately 36 trillion tokens across 119 languages, encompassing web text, books, multilingual code (from GitHub/The Stack), STEM and math domains, long-form PDF extraction, and synthetic data (Qwen2.5-Math, Qwen2.5-Coder).

Training proceeds through a three-stage schedule:

1. General stage: ≈30T tokens, sequence length 4,096.
2. Reasoning stage: ≈5T high-quality STEM and code tokens, accelerated learning rate decay, sequence length 4,096.
3. Long-context stage: Hundreds of billions of tokens with sequence length 32,768, allocated as 75% of batches at [16,384–32,768] and 25% at [4,096–16,384].

Optimization employs AdamW with decoupled weight decay; learning rates and batch sizes are scale-optimized but undisclosed. RoPE base frequency modulation uses ABF without further dense-model-specific hyperparameter changes (Yang et al., 14 May 2025).

3. Dynamic Reasoning Modes and User-Controlled Budgets

A unified checkpoint supports both "thinking" (complex, multi-step reasoning) and "non-thinking" (rapid, context-based response) modes, selectable by template flags: "/think" for thinking mode (the default), and "/no_think" for non-thinking mode. Model outputs use a structured response format: > … <response>…, with the <think> block omitted for non-thinking mode.

Users can specify a maximum number of thinking tokens ($B_t$), with the model transitioning to output upon $B_t$ tokens consumed in the <think> segment. There is no closed-form for $B_t$; it is a user-specified integer (typical: 4096–8192 tokens) (Yang et al., 14 May 2025).

In thinking mode, inference cost and latency increase linearly with $B_t$ (Compute$(B_t) \simeq$ BaseCompute $+\ \alpha \cdot B_t$), yielding smooth task performance gains, particularly on reasoning-intensive benchmarks. Non-thinking mode minimizes FLOPs per token for applications with real-time latency constraints.

4. Evaluation Results Across Benchmarks

Base Pretrained Performance (Zero/Few-Shot):

• Qwen3-8B (Base): MMLU-Redux 76.17, GSM8K 89.84, EvalPlus 67.65, MultiPL-E 58.75.
• Qwen3-14B (Base): MMLU-Redux 79.88, GSM8K 92.49, EvalPlus 72.23, MultiPL-E 61.69.

Instruction-Tuned Performance:

• Qwen3-8B (Thinking): AIME'24 76.0, AutoLogi 89.1.
• Qwen3-8B (Non-Thinking): BFCL v3 60.2%, LiveCodeBench 22.8%.
• Qwen3-14B (Thinking): MMLU-Redux 88.6%, BFCL v3 70.4.
• Qwen3-14B (Non-Thinking): INCLUDE 71.7, CodeForces 1200 Elo.

Multilingual Evaluations:

Qwen3-8B (Thinking) achieves 76.02% on Multi-IF (8 languages), Qwen3-14B surpasses Qwen2.5-14B on INCLUDE (64.55 vs 60.26) and MMMLU (79.69 vs 78.34) (Yang et al., 14 May 2025).

5. Resource Consumption and Computational Efficiency

FLOPs per token and peak memory for the dense models are not explicitly reported. The design complexity is approximately $2 \times L \times d_{\text{model}}^2$ per token with additional SwiGLU overhead. Empirically, doubling $B_t$ results in proportionate latency increase and incremental accuracy gains on reasoning benchmarks. The base-compute plus per-token linear model applies for inference cost estimation (Yang et al., 14 May 2025).

6. Deployment and Application Considerations

Qwen3-8B targets edge devices, client-side deployments, and real-time chatbots with moderate reasoning demand, offering strong STEM and code performance at low resource overhead. Qwen3-14B is suited for mid-tier server inference, multilingual customer service, and workflows requiring moderate multi-step reasoning without the resource demands of 32B+ scale architectures.

Thinking budget selection recommendations:

• Non-thinking mode for low-complexity Q&A (lowest latency).
• $B_t = 2$K–4K for medium complexity (e.g., brief math/code).
• $B_t \geq 8$K for high-complexity reasoning (multi-step proofs, code debugging).
• For strict latency requirements, halving $B_t$0 reduces performance by ~3–5% (Yang et al., 14 May 2025).

7. Comparison with Prior Qwen and Other Model Series

Relative to Qwen2.5, Qwen3 models—especially Qwen3-8B and Qwen3-14B—expand multilingual support (from 29 to 119 languages and dialects) and introduce scalable reasoning through thinking/non-thinking integration and user-controlled computational budgeting. Performance gains are observed across established code, mathematics, and agent benchmarks versus Llama-3, Gemma-3, DeepSeek, and other contemporary dense and MoE baselines.

Earlier Qwen models, such as Qwen-8B and Qwen-14B, employed similar layerwise architectures but featured shorter context lengths (e.g., 32–40 layers, 128–40K context), tied embeddings in certain variants, and RoPE without ABF-based extension. The training corpus was substantially smaller (~3T tokens), and multilingual support less extensive (Bai et al., 2023).

Qwen3-8B and Qwen3-14B represent the state of the art among compact dense LLMs for code, reasoning, and multilingual generation, benefiting from innovations in architecture, specialization, and adaptive computational control (Yang et al., 14 May 2025, Bai et al., 2023).