Qwen Language Model Suite Overview

• Qwen Language Model Suite is a family of open-weight, multimodal models featuring both dense and MoE architectures for advanced NLP, vision, and audio processing.
• Successive generations from Qwen-1 to Qwen-3 incorporate innovations like GQA, SwiGLU, dual-chunk context, and unified multi-modal reasoning.
• The suite leverages large-scale pretraining and instruction tuning, enabling flexible, efficient deployment across diverse languages and applications.

The Qwen LLM Suite constitutes a comprehensive family of open-weight large-scale models designed to deliver state-of-the-art performance across natural language processing, vision-language, and audio-language tasks. Spanning multiple versions and generations, Qwen’s suite includes dense and mixture-of-experts (MoE) architectures, multimodal extensions, and specialized variants for reasoning, code, and math. Operating with a focus on modularity, multilingual support, and efficiency, the Qwen suite has played a central role in advancing open-source large model capabilities and democratizing access for both research and production environments (Li et al., 8 Jan 2026, Yang et al., 14 May 2025, Bai et al., 19 Feb 2025, Qwen et al., 2024, Yang et al., 2024, Bai et al., 2023, Chu et al., 2023, Bai et al., 2023, Cruz-Castañeda et al., 20 May 2025).

1. Suite Overview and Version Evolution

The Qwen suite originated with the launch of Qwen-1 (2023), introducing base and chat-aligned models with standard decoder-only Transformer architectures, achieving competitive accuracy across a variety of language, code, and reasoning benchmarks (Bai et al., 2023). Qwen-2 expanded the model scales up to 72B, incorporating enhancements such as Grouped Query Attention (GQA), SwiGLU activations, dual-chunk long-context support (up to 128K tokens), and the introduction of a Mixture-of-Experts (MoE) variant (Qwen2-57B-A14B) (Yang et al., 2024). Qwen2.5 further scaled data, introduced intricate supervised/RL post-training, and delivered specialized offshoots (Qwen2.5-Math, Qwen2.5-Coder, QwQ, Qwen2.5-VL, and Qwen-Audio), consistently reaching or surpassing proprietary closed-weight models in efficiency and accuracy, especially with new Turbo and Plus (MoE) hosted options (Qwen et al., 2024). Qwen3, the latest release, unified multi-modal and multi-lingual reasoning in a single architecture with a new “thinking mode” for chain-of-thought and an adaptive thinking budget for efficient inference (Yang et al., 14 May 2025).

Table 1. Qwen Model Suite, Scale and Key Innovations

Generation	Sizes (Dense/MoE)	Max Context	Key Innovations	Multimodal Extensions
Qwen (2023)	1.8B / 7B / 14B	16K	SFT + RLHF, basic tool use	Qwen-VL (vision), Code/Math
Qwen2	0.5B–72B / 57B-A14B	128K	GQA, SwiGLU, YARN, MoE	Qwen2-VL, Qwen-Audio
Qwen2.5	0.5B–72B / Turbo/Plus	128K–1M	18T tokens, advanced RLHF, MoE	Qwen2.5-VL, Qwen2.5-Audio
Qwen3	0.6B–32B / 30B-A3B/235B-A22B	128K	Unified thinking modes, 119 langs	Qwen3-VL, Qwen3-VL-Embedding

2. Core Model Architecture and Pretraining

All major Qwen variants adopt a decoder-only Transformer backbone, integrating architectural advancements per generation. Base models employ SwiGLU activations, Grouped Query Attention (GQA) for reduced KV-cache size, pre-norm RMSNorm, rotary positional encodings (RoPE), and large BPE vocabularies (≈151k tokens) (Yang et al., 2024, Qwen et al., 2024). MoE models utilize fine-grained expert segmentation, activating a subset of MLP experts per token with global load-balancing for compute efficiency (Yang et al., 14 May 2025).

Pretraining is performed via next-token prediction on large-scale, multi-domain, and highly multilingual web, code, and math corpora. Data scale increased from 3T (Qwen) to 7T tokens (Qwen2), 18T (Qwen2.5), and 36T (Qwen3), with instance-level filtering, synthetic multi-task data, and progressive curriculum for robust cross-lingual and domain adaptation (Qwen et al., 2024, Yang et al., 14 May 2025).

3. Instruction Tuning, Alignment, and Specialized Variants

Instruction tuning is implemented via large-scale supervised fine-tuning (SFT) over diverse instruction–response datasets (1M+ pairs in Qwen2.5), followed by multi-stage RLHF with Direct Preference Optimization (DPO) and Group Relative Policy Optimization (GRPO) (Qwen et al., 2024). Safety, system prompts, code-based validation, and preference alignment are explicitly targeted. The suite includes:

• Qwen-Chat/Qwen2-Chat/Qwen2.5-72B-Instruct: RLHF-aligned assistants that outperform other open-source models of similar scale, competitive with GPT-3.5 and, at larger scales, approach GPT-4o's performance on preference and reasoning tasks.
• Code-Qwen/Math-Qwen/Qwen2.5-Coder/Qwen2.5-Math: Specialized models trained and SFTed on curated code/math corpora, achieving state-of-the-art or near-state-of-the-art results on HumanEval, MBPP, GSM8K, MATH, and TheoremQA (Bai et al., 2023, Qwen et al., 2024).
• Multilingual Expansion: Qwen3 is trained across 119 languages/dialects, delivering high accuracy even in low- and mid-resource languages, with benchmarks such as Belebele (over 90.7 on Indo-European/Uralic families at 32B scale) (Yang et al., 14 May 2025).

4. Multimodal and Universal Models

Vision-Language: Qwen-VL and successors (Qwen2.5-VL, Qwen3-VL) pair the LLM backbone with native ViTs, advanced positional embeddings, dynamic resolution tokenization, and multimodal fusion, supporting structured document OCR, object grounding, temporal localization in video, and visual agent tasks. Instruction tuning and MME/DPO alignment allow seamless vision-language Q&A, diagram comprehension, and long-video analysis (Bai et al., 2023, Bai et al., 19 Feb 2025).

Audio-Language: Qwen-Audio and Qwen-Audio-Chat integrate a large-scale Whisper-v2 Transformer encoder with the LLM, using a unified tag-conditioned, multi-task pretraining and interoperable cross-modal interfaces. These models demonstrate superior performance in ASR (2.0/4.2% WER LibriSpeech), audio captioning, speaker ID, music QA, and multi-turn dialogue, all without task-specific tuning (Chu et al., 2023).

Multimodal Retrieval and Ranking: Qwen3-VL-Embedding and Qwen3-VL-Reranker offer dual-tower (bi-encoder) embedding and cross-encoder reranking for high-precision search over text, images, document images, and video. Multi-stage training entails contrastive pretraining, multi-task adjustment, relevance distillation, and Matryoshka Representation Learning (MRL) to allow dynamic embedding dimension trade-offs. State-of-the-art retrieval is demonstrated: Qwen3-VL-Embedding-8B achieves 77.8 MMEB-V2, outperforming E5-V, GME, Seed, and closed-source IFM-TTE (Li et al., 8 Jan 2026).

5. Training Regimes, Scaling Laws, and Engineering

Qwen2.5 and Qwen3 employ aggressive data scaling, context extension strategies (YARN, dual-chunk, ABF-RoPE), and domain balancing for robust long-context reasoning and memory retention (Qwen et al., 2024, Yang et al., 14 May 2025). Instruction tuning hyperparameters are determined by scaling-law fits ($\mu_\text{opt} \propto N^{-0.2}D^{-0.3}$), and quantization to INT4/8 is officially supported, enabling deployment on resource-constrained hardware with <1% degradation (Qwen et al., 2024). MoE models achieve server-tier quality at 1/5–1/10 activated parameter cost by routing tokens efficiently, exploiting sparse and dense expert blending (Yang et al., 14 May 2025).

6. Benchmarks, Comparative Performance, and Applications

Qwen2.5-72B and Qwen3-235B-A22B achieve state-of-the-art results across general NLP understanding (MMLU up to 86.3), code generation (LiveCodeBench v5: 70.7 vs DeepSeek-R1 64.3), math (GSM8K 91.5), tool use, alignment benchmarks (Arena-Hard >9.3), and long-context tasks (100% accuracy on 1M-token passkey retrieval with Qwen2.5-Turbo). The Qwen2.5-VL-72B matches and in some document QA tasks outperforms GPT-4o and Claude 3.5 Sonnet (OCRBench_v2_en: 61.5% vs GPT-4o 46.5%, Claude 45.2%) (Qwen et al., 2024, Bai et al., 19 Feb 2025, Yang et al., 14 May 2025).

Qwen-Audio surpasses SALMONN, Paraformer, and CLAP on automatic speech recognition, audio captioning, and classification (Chu et al., 2023). Qwen3-VL-Embedding-8B ranks first on MMEB-V2 with an overall score of 77.8, exceeding prior open and closed-source retrieval models (Li et al., 8 Jan 2026).

Prominent application domains include agent systems, code review, customer support, document understanding, visual Q&A, and multi-lingual chat. For instance, Qwen2.5-derived Amadeus-Verbo demonstrates streamlined creation of Portuguese-specialized large models with minimal cost by fine-tuning on 600k instruction pairs (Cruz-Castañeda et al., 20 May 2025).

7. Open Science, Model Release, and Ecosystem Integration

A core tenet of the Qwen initiative is open release: All model weights, tokenizers, and code across generations—spanning dense, MoE, and multimodal variants—are accessible under Apache 2.0 or compatible licenses via major repositories (Hugging Face, ModelScope, GitHub) (Yang et al., 14 May 2025, Qwen et al., 2024, Yang et al., 2024, Bai et al., 2023). Documentation, quantization scripts, and deployment guides enable reproducibility and fine-tuning on domain- or language-specific data.

Qwen’s unified API and checkpoint management facilitate seamless orchestration of language, vision, and audio models, supporting modular ensemble and cross-modal pipelines in production settings (Chu et al., 2023). Consistent architectural frameworks across generations and modes allow both horizontal and vertical stacking of improvements, ensuring upward compatibility and research extensibility.

---

References:

• (Li et al., 8 Jan 2026) Qwen3-VL-Embedding and Qwen3-VL-Reranker: A Unified Framework for State-of-the-Art Multimodal Retrieval and Ranking
• (Yang et al., 14 May 2025) Qwen3 Technical Report
• (Bai et al., 19 Feb 2025) Qwen2.5-VL Technical Report
• (Qwen et al., 2024) Qwen2.5 Technical Report
• (Yang et al., 2024) Qwen2 Technical Report
• (Chu et al., 2023) Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-LLMs
• (Bai et al., 2023) Qwen-VL: A Versatile Vision-LLM for Understanding, Localization, Text Reading, and Beyond
• (Bai et al., 2023) Qwen Technical Report
• (Cruz-Castañeda et al., 20 May 2025) Amadeus-Verbo Technical Report: The powerful Qwen2.5 family models trained in Portuguese