Qianfan-VL Model Series

• Qianfan-VL model series is a collection of large-scale vision-language models featuring domain-enhanced pretraining, a modular cross-modal architecture, and specialized OCR and retrieval variants.
• They employ a progressive multi-stage training pipeline with dynamic image tiling and synthetic data curation to excel in document intelligence, mathematical reasoning, and other multimodal challenges.
• Developed by Baidu, these models scale from edge-deployment to cloud-based systems, delivering high efficiency and competitive benchmark performance across various multimodal tasks.

The Qianfan-VL model series is a collection of large-scale, universal vision-LLMs distinguished by domain-enhanced pretraining, modular cross-modal architecture, and state-of-the-art performance in both generic and highly specialized multimodal tasks. Developed by Baidu, Qianfan-VL leverages progressive multi-stage training, high-precision synthetic data curation, and scalable infrastructure to support a wide spectrum of applications spanning document intelligence, OCR, mathematical reasoning, and enterprise-grade retrieval. The series comprises several notable variants, including the base Qianfan-VL family, the Qianfan-OCR document intelligence specialist, and embedded/retriever extensions such as Qwen3-VL-Embedding and Qwen3-VL-Reranker, unified by common design paradigms and shared technical underpinnings (Dong et al., 19 Sep 2025, Dong et al., 11 Mar 2026, Li et al., 8 Jan 2026).

1. Model Architecture and Variants

Qianfan-VL adopts a modular vision-language framework, seamlessly interconnecting a high-capacity vision encoder, a cross-modal adapter, and a transformer-based language decoder. The architecture scales from 3B (edge deployment) to 70B (cloud-based complex reasoning):

Variant	Language Backbone	Vision Encoder (InternViT/Qianfan-ViT)	Adapter	Max Context	Target
Qianfan-VL-3B	Qwen2.5-3B	InternViT (300M params)	4096→2048	32K	Edge/real-time
Qianfan-VL-8B	Llama 3.1-8B	InternViT (300M params)	4096→4096	32K	Servers
Qianfan-VL-70B	Llama 3.1-70B	InternViT (300M params)	4096→8192	32K	Cloud
Qianfan-OCR-4B	Qwen3-4B	Qianfan-ViT (24L, 1024H, 16HD)	1024→2560	32K–131K	Document/OCR

Each model exploits dynamic image tiling (448×448 patches, 256 tokens/tile, up to 12–16 tiles), with cross-modal adapters facilitating information flow between the vision features and the LM embedding space. Grouped-Query Attention and RMSNorm enhance memory efficiency and stability. Tokenization strategies incorporate BPE, special prompt tokens (e.g., > ), and explicit coordinate vocabularies for layout representations (Dong et al., 19 Sep 2025, Dong et al., 11 Mar 2026).

2. Progressive Domain-Enhanced Training

The Qianfan-VL series employs a four-stage, domain-enhancement-centric training pipeline:

1. Stage 1 (Cross-modal Alignment): Only the cross-modal adapter is updated, enabling stable initial bridging between frozen vision and language encoders on synthetic image-caption/VQA data.
2. Stage 2 (General Knowledge Injection): Full-parameter tuning on a dataset mixture of captioning, OCR, KIE, and video (scaling to trillions of tokens), injecting broad multimodal knowledge.
3. Stage 3 (Domain Enhancement): Weighted focus (e.g., λ=0.7) on domain-specific data (document understanding, OCR, math), interleaved with general knowledge to maximize specialization without catastrophic forgetting.
4. Stage 4 (Instruction Tuning): Multi-turn, edge-case, and complex CoT-instruction data refine reasoning and prompt-following.

Synthetic data generation is organized into six pipelines (Document OCR, Mathematics, Charts, Tables, Formulas, Scene OCR), incorporating noise simulation, multi-format renderings, chart/table synthesis, character-level validation, and strong rejection sampling for quality assurance. A mini-batch mixing strategy combines domain and general data with per-batch loss mixing (Dong et al., 19 Sep 2025).

3. Specialized Mechanisms: Qianfan-OCR and "Layout-as-Thought"

Qianfan-OCR represents a paradigm shift for document intelligence, introducing the Layout-as-Thought mechanism:

• Motivation: End-to-end OCR typically loses explicit bounding box and element-type outputs available in vision-first pipeline systems.
• Layout-as-Thought ("thinking" phase): By prepending a <think> token, the decoder emits structured layout representations (bounding boxes, types, briefs) as explicit token sequences before subsequent text generation. During training, a cross-entropy loss is computed over coordinate predictions ($p_{\text{coord}(i)}$ via softmax over 1,000 bins) and class predictions (25 layout types):

$$\mathcal{L}_{\text{layout}} = \sum_{i\in\{x_{\min},y_{\min},x_{\max},y_{\max}\}} \mathrm{CE}\bigl(p_{\text{coord}(i)}, b_i\bigr) + \mathrm{CE}\bigl(p_{\text{type}}, t\bigr)$$

• Functional Impact: Enables retrieval of structural grounding, improved recognition in complex multi-element or irregular layouts, and flexible, prompt-driven behavior as required by downstream tasks.
• Output Example: The model generates <layout>…</layout> sections containing {<box>…</box>, <label>…</label>, <brief>…</brief>} per document element, conditioned on high-entropy triggers (Dong et al., 11 Mar 2026).

4. Task Scope and Prompting Strategies

The Qianfan-VL models support extended, instruction-aware prompting for a variety of multimodal tasks:

• OCR and Document Parsing: Image-to-Markdown conversion, paragraph/heading extraction.
• Table Extraction: Markdown-formatted table recovery.
• Chart Understanding: Chart QA, e.g., reporting peak values.
• Document QA: Direct answering on PDF or scene document inputs, e.g., theorem citation.
• Key Information Extraction: Structured JSON outputs (e.g., invoice fields).
• Retrieval and Ranking (via Qwen3-VL-Embedding and Reranker): Bi-encoder dense retrieval, cross-encoder relevance scoring for multimodal pairs, Matryoshka Representation Learning for variable-dimension embeddings, and quantization-aware training (Li et al., 8 Jan 2026).

The Qianfan-OCR pipeline is optimized for one-pass, image-to-structured conversion with Markdown/HTML outputs, leveraging OTSL for table layouts and robust instruction tuning across prompt styles (Dong et al., 11 Mar 2026).

5. Empirical Performance and Benchmark Analysis

Qianfan-VL models achieve state-of-the-art results across general, domain-specific, and retrieval-focused benchmarks:

Benchmark	Qianfan-OCR-4B	Qwen3-VL-4B	Qianfan-VL-8B/70B	External SOTA
OmniDocBench v1.5	93.12	—	—	94.50 (pipe)	(Dong et al., 11 Mar 2026)
OlmOCR Bench	79.8	—	—	80.0 (pipe)	(Dong et al., 11 Mar 2026)
OCRBench	880	873	854/873	881	(Dong et al., 19 Sep 2025, Dong et al., 11 Mar 2026)
DocVQA	92.8	94.9	93.54/94.75	94.75	(Dong et al., 19 Sep 2025, Dong et al., 11 Mar 2026)
ChartQA	88.1	83.3	87.72/89.60	85.76	(Dong et al., 19 Sep 2025, Dong et al., 11 Mar 2026)
Key Info Ex. (mean 5sets)	87.9	83.5	—	79.2–84.2	(Dong et al., 11 Mar 2026)
MMEB-V2 (Qwen3-VL-Emb-8B)	—	—	—	77.8	(Li et al., 8 Jan 2026)

Performance exceeds or matches contemporary pipeline and multimodal baselines on image, video, and document retrieval (MMEB-V2, JinaVDR, ViDoRe-v3), with measured improvements on specialized tasks directly attributable to the domain enhancement and layout reasoning modules. Loss of spatial context in pipeline systems leads to task failures (CharXiv QA, ChartBench) that Qianfan-OCR avoids (>85% retention) (Dong et al., 11 Mar 2026).

6. Scalability, Infrastructure, and Deployment

Qianfan-VL models are trained at extreme scale on the Baidu Kunlun P800 chip cluster, with >5,000 chips orchestrated for three-dimensional parallelism (data, tensor, pipeline) and up to 32K context tokens. Key techniques include sequence parallelism, dynamic batching, selective recompute, and comm-computation fusion for >90% scaling efficiency. Hardware-level optimizations (separate comm/GEMM units, AllReduce, multi-stream fusion) reduce communication overhead, enabling multi-trillion-token training (Dong et al., 19 Sep 2025).

Deployment tracks according to model size and target application: 3B for edge/real-time OCR, 8B for server multimodal QA, and 70B for cloud CoT/long-context workloads. Modular retraining at Stages 3–4 allows rapid domain adaptation from existing checkpoints.

7. Series Integration and Model Differentiation

The Qianfan-VL series is unified at the architectural and training levels but exhibits clear specialization among variants:

• Qianfan-OCR: Only model in the series to emit explicit, tokenized layout dumps and deliver end-to-end OCR/understanding with competitive speed and accuracy. Dedicated OCR-centric data mixtures throughout later training stages.
• Qwen3-VL-Embedding/Reranker: Optimized for high-precision multimodal retrieval, supporting flexible input modalities (text, image, video), variable-dimension embeddings (MRL), and quantization-aware deployment.
• General Qianfan-VL-8B/70B: Broader image-text-document reasoning, strong math/logic CoT, mathematical reasoning, and robust hallucination resistance.

A plausible implication is that the Qianfan-VL methodology—progressive domain enhancement, carefully balanced scale, and explicit structural reasoning—sets a template for future universal vision-LLM development, offering both infrastructure-level and algorithmic best practices for large-scale, cross-modal intelligence (Dong et al., 19 Sep 2025, Dong et al., 11 Mar 2026, Li et al., 8 Jan 2026).