DeepSeek-VL2: MoE Vision-Language Model

• The paper presents DeepSeek-VL2's innovative Mixture-of-Experts framework that integrates dynamic tiling vision encoding with Multi-Head Latent Attention, achieving 8–16× efficiency improvements in memory and compute.
• It employs a three-stage pipeline and offers scalable variants (Tiny, Small, Base) to robustly address diverse tasks such as VQA, OCR, and visual grounding with high-resolution input.
• The model demonstrates competitive or superior benchmark performance while optimizing compute through sparse expert routing and advanced latent-attended memory mechanisms.

DeepSeek-VL2 is a Mixture-of-Experts (MoE) open-source vision-LLM family designed for high-throughput, high-resolution multimodal understanding. It introduces a dynamic tiling vision encoding module and leverages DeepSeekMoE LLMs equipped with Multi-Head Latent Attention (MLA) to address computational efficiency and scaling for both vision and language streams. DeepSeek-VL2 demonstrates state-of-the-art or competitive results on a broad spectrum of vision-language tasks, including visual question answering (VQA), optical character recognition (OCR), chart and document interpretation, and visual grounding, with notable strength in structured multimodal input and large-context reasoning. The suite comprises three variants (Tiny, Small, Base), with activated parameter counts of 1.0B, 2.8B, and 4.5B respectively, and emphasizes efficient model scaling as well as high performance per compute (Wu et al., 2024).

1. Model Architecture and Core Innovations

DeepSeek-VL2 is architected as a three-stage pipeline composed of (1) dynamic tiling vision encoding, (2) a vision-language adaptor, and (3) an MoE transformer language core augmented with MLA. The pipeline accepts images of arbitrary resolution and aspect ratio, segmenting them via dynamic tiling to minimize padding and maximize visual information per token. Tiles (local patches and a global thumbnail, all 384×384) are processed by a SigLIP encoder to produce grid-based patch embeddings, which are further condensed using a pixel-shuffle operation to regulate token counts.

The language backbone employs a decoder-only transformer derived from DeepSeekMoE, where sparse expert routing selects the top-6 experts from a pool of 64 (Tiny, Small) or 72 (Base) per token. Gating is handled by softmax or sigmoid functions depending on the variant, with expert bias correction implemented for the Base variant to enhance load balancing. The Multi-Head Latent Attention mechanism compresses the Key-Value cache into a fixed number (r=512) of learned latent vectors, greatly reducing both memory and compute costs in long-sequence inference. Attention for the current query is computed over this latent set, achieving 8–16× memory and compute savings compared to standard dense attention (Wu et al., 2024).

2. Data Regime and Training Procedures

Training is conducted across three distinct stages:

• Vision-Language Alignment: The SigLIP encoder and visual adaptor MLP are “warmed up” using 1.2M ShareGPT4V samples, with the LLM parameters held frozen.
• Multimodal Pretraining: The corpus combines approximately 70% image-text pairs (WIT, WikiHow, OBELICS, Wanjuan, OCR corpora, PubTabNet, FinTabNet, and more) with 30% text-only data, encompassing ≈800B multimodal and ≈300B text-only tokens.
• Supervised Instruction Fine-Tuning: 20B mixed multimodal and text instruction pairs are used, including VQA, OCR, document understanding, code generation, visual and ground dialogue.

Optimization is via next-token cross-entropy loss, augmented in the Base variant with an expert-load balancing auxiliary loss. Hyperparameters are staged: learning rates are initially 5.4×10⁻⁴, decaying or held constant depending on phase; vision module learning rate is scaled ×0.1 compared to the language core. Batch sizes and token sequence lengths scale with model size (up to 4,096 tokens per sequence). Training is distributed across up to 42 A100 GPUs with tensor, pipeline, and expert parallelism (Wu et al., 2024).

3. Model Variants and Capacity

DeepSeek-VL2 is released in three parameterized variants, differing in both total and activated parameter counts due to MoE routing:

Variant	LLM Total / Activated	SigLIP	Total Activated	#Experts (E)	Top-K
Tiny	3B / 0.57B	0.4B	1.0B	64	6
Small	16B / 2.4B	0.4B	2.8B	64	6
Base	27B / 4.1B	0.4B	4.5B	72	6

The design allows activation of only a sparse subset of experts per token, which reduces compute footprint while maintaining the underlying capacity for complex reasoning and memorization. Expert routing utilizes softmax-gated selection in Tiny and Small, with the Base variant employing sigmoid gating and explicit bias correction for expert-load balancing (Wu et al., 2024).

4. Inference Efficiency and Scaling Properties

Inference in DeepSeek-VL2 benefits from both architectural and algorithmic efficiencies:

• Sparse MoE Routing: Only 6 experts per token are evaluated, significantly lowering cost compared to dense models.
• Multi-Head Latent Attention: Reduces KV-cache storage and compute from sequence length $T_{past}$ to a fixed 512, which for long sequences results in approximately 8× to 16× reduction in GPU memory and attention computation.
• Empirical throughput is reported as more than double that of a comparably sized dense competitor (e.g., InternVL2-2B) for the Small variant, without accuracy trade-off.
• The smallest variant is deployable on a single A100 GPU with 10 GB memory, while the Base variant fits on an 80 GB card, enabling research and production use without distributed hardware (Wu et al., 2024).

Disabling dynamic tiling results in accuracy drops of 2–4 points on extreme-aspect data (InfoVQA). Turning off MLA (falling back to standard dense cache) increases per-token latency by 25–35%, demonstrating the practical gains of these modules (Wu et al., 2024).

5. Downstream Task Performance

DeepSeek-VL2 achieves competitive or superior results relative to contemporary dense and MoE vision-LLMs across a range of evaluation benchmarks:

OCR & Structured Document Understanding

Model	Params	DocVQA	ChartQA	InfoVQA	TextVQA	OCRBench
InternVL2-1B	0.9B	81.7	72.9	50.9	70.5	754
DeepSeek-VL2-Tiny	1.0B	88.9	81.0	66.1	80.7	809
Qwen2-VL-2B	2.2B	90.1	73.5	65.5	79.7	794
DeepSeek-VL2-Small	2.8B	92.3	84.5	75.8	83.4	834
DeepSeek-VL2 (Base)	4.5B	93.3	86.0	78.1	84.2	811

Multitask Reasoning and Visual QA

On MMStar, MMMU, MMBench, and related tasks, DeepSeek-VL2-Small and Base outperform most open-source peers of equal or higher activated parameter count. On RefCOCO visual grounding datasets, DeepSeek-VL2 (Base) achieves 95.1 (val), 96.7 (testA), and 92.7 (testB) on RefCOCO, setting a strong benchmark for parameter efficiency (Wu et al., 2024).

6. Applications, Strengths, and Limitations

DeepSeek-VL2 excels in scenarios that require high-resolution multimodal perception (e.g., document OCR, chart reading, visual dialogues) and can reliably process contextually complex, structured visual inputs. It robustly supports multi-image dialogue, bilingual OCR, and visual grounding with strong parametric efficiency. Application domains include scientific document analysis, robotic perception, web agent grounding, and assistive chat systems.

Current limitations include restricted context to “a few images” per prompt and occasional performance degradation on extreme object blur or rare classes. Multi-step reasoning, while improved, is still cited as an area for continued development (Wu et al., 2024). The model is publicly released, facilitating benchmarking and research reproducibility.

7. Outlook and Research Implications

The DeepSeek-VL2 series advances vision-language modeling via its synthesis of dynamic visual tiling and next-generation MoE transformers with latent-attended memory. It provides a foundation for further work in scaling open-source multimodal models, balancing computational efficiency with task accuracy. Potential research directions include expanding prompt engineering for more sophisticated reasoning, integrating spatially specialized attention modules for detailed visual localization, and scaling the number of images per multimodal context. DeepSeek-VL2’s framework sets a precedent for future models targeting fine-grained vision-language understanding, especially in applications demanding both throughput and high visual fidelity (Wu et al., 2024).