--- title: 'Qwen3.5-Omni: Next-Gen Omnimodal LLM' url: https://www.emergentmind.com/topics/qwen3-5-omni type: topic --- # Qwen3.5-Omni: Next-Gen Omnimodal LLM Qwen3.5-Omni is a next-generation omnimodal large language model that unifies text, vision, audio, and audio-visual understanding and generation within a single end-to-end agent. Scaling to hundreds of billions of parameters and supporting a context window of up to 256,000 tokens (approximately 10 hours of audio or 400 seconds of 720P video at 1 FPS), Qwen3.5-Omni is designed for robust, temporally precise multimodal modeling. The model achieves state-of-the-art (SOTA) results over 215 audio and audio-visual subtasks, surpassing Gemini-3.1 Pro on key benchmarks and demonstrating strong emergent capabilities such as omnimodal code generation [2604.15804]. ## 1. Technical Architecture and Design Qwen3.5-Omni implements a two-stage Thinker–Talker architecture built atop a Hybrid-Attention Mixture-of-Experts (MoE) Transformer. The Thinker stage ingests all incoming modalities—including text, images, audio, and silent video—via modality-specific encoders: a Byte Pair Encoding (BPE) tokenizer for text, a vision encoder inherited from Qwen3.5-VL for images, and a custom “Audio Transformer” (AuT) for audio. All inputs are temporally interleaved with explicit textual timestamps (e.g., “[Time 12.3 s]”) to maintain robust temporal modeling across very long context windows, obviating the need for sparse RoPE-based indexings. Within each Thinker layer, dense multi-headed self-attention alternates with a sparse Mixture-of-Experts feed-forward block, while Gated Delta Net (GDN) modules accelerate KV-cache input/output operations during long-sequence inference. The Talker consumes Thinker’s hidden representations and text emissions to autoregressively predict multi-codebook speech codec tokens. Its decoder stack includes a Multi-Token Prediction (MTP) head for 40 ms frame-level residual codebooks and a streaming, causal convolutional network (“Code2Wav”) for waveform synthesis at sub-5 ms latency. The ARIA (Adaptive Rate Interleave Alignment) module enforces tight alignment between text and speech generation. ### Key Architectural Hyperparameters | Parameter | Value/Characteristic | |-------------------------|---------------------------------| | Total parameters | O(10^11)–O(10^12) (hundreds of billions) | | Hidden size | ≈12,288 | | Attention heads/layer | 96 | | Experts per MoE layer | 128 (top-2 active) | | Layers (Thinker/Talker) | 60 / 16 | | Max context window | 256K tokens (≈10 h audio or 400 s video) | [2604.15804] ## 2. Training Corpus and Pretraining Protocol Pretraining proceeds in three distinct stages: 1. **Encoder Alignment:** The LLM is frozen, while image and audio encoders are aligned to the representation space using lightweight adapters. 2. **General Multimodal Pretraining:** All model parameters are unfrozen; the model is exposed to approximately 4 trillion tokens, with a sequence length of 32,000 tokens. The training data includes: - 0.92T pure text tokens. - 1.99T audio-text pairs, drawn from over 100 million hours of audio-visual data. - 0.95T image-text tokens. - 0.14T video-text and 0.29T video-audio pairs. 3. **Long-Context Adaptation:** The context window is expanded to 256K, with a higher fraction of long audio and video sequences to ensure stable, long-horizon modeling. A single unified autoregressive cross-entropy objective is applied across modalities. Formally, for modality $m$ and paired sequence $(x, y)_m$, the loss is \[ \ell_m = -\sum_{t=1}^{|y|} \log p(y_t | y_{