--- title: 'Qianfan-OCR: Unified Vision-Language Model' url: https://www.emergentmind.com/topics/qianfan-ocr type: topic --- # Qianfan-OCR: Unified Vision-Language Model Qianfan-OCR is a 4-billion-parameter end-to-end vision-language model that unifies optical character recognition (OCR), document parsing, explicit layout analysis, and advanced document understanding within a single neural architecture. Developed by Baidu and released in 2026, it supports direct image-to-Markdown conversion and a wide spectrum of prompt-driven document tasks such as table extraction, chart analysis, document-level question answering, and key information extraction (KIE). Qianfan-OCR achieves state-of-the-art or near state-of-the-art performance across multiple structured OCR, document intelligence, and multi-language benchmarks, while introducing novel architectural and training methodologies, notably the Layout-as-Thought mechanism for explicit layout representation [2603.13398]. ## 1. Unified Model Architecture and Training Pipeline Qianfan-OCR features a modular, unified VLM pipeline comprising a high-resolution Vision Transformer (Qianfan-ViT), a cross-modal MLP adapter, and a Qwen3-4B language modeling backbone. The full model contains 4B parameters, with 3.6B assigned to non-embedding parameters. - **Vision Encoder (Qianfan-ViT):** Inputs of arbitrary resolution are partitioned into up to 16 tiles of 448×448 pixels. Each tile is processed by a 24-layer Vision Transformer (hidden size 1024, 16 heads) with patch size 14×14 (yielding 256 tokens per tile). This AnyResolution tiling enables practical handling of full document pages. - **Cross-Modal Adapter:** A two-layer MLP with GELU activation (mapping 1024-dim visual features to 2560-dim language token space). Initially trained for alignment, it is subsequently fine-tuned end-to-end with the rest of the system. - **Language Model Backbone (Qwen3-4B):** Based on 36 transformer layers (hidden size 2560), equipped with Grouped-Query Attention (GQA: 32 query heads, 8 key-value heads) and RMSNorm normalization, supporting input contexts up to 32K (extendable to 131K) tokens. **Multi-stage Curriculum:** Training proceeds in four distinct stages, each with targeted objectives: 1. **Cross-Modal Alignment:** Adapter-only training on generic image-caption pairs and simple OCR instances (50B tokens, LR 1e-3). 2. **Foundational OCR:** Full model training over diverse OCR data (2T tokens)—document OCR (45%), scene OCR (25%), captions, handwriting, formulas, tables, and multilingual scenarios. 3. **Domain-Specific Enhancement:** Full model is further trained (800B tokens) on complex tables, formula recognition, chart understanding, KIE, multilingual and complex document data; 30% of batches remain general VLM tasks to prevent catastrophic forgetting. 4. **Instruction Tuning:** The model is tuned on millions of prompt-engineered tasks, spanning parsing, table extraction, QA, layout analysis, and KIE, across multi-page and multi-lingual instances. AdamW is used throughout (β₁=0.9, β₂=0.95, weight decay 0.05, cosine LR schedule), with batch sizes and learning rates specified for each stage. Ablation of these stages (on sibling Qianfan-VL-8B) demonstrates that all contribute additively to final OCR accuracy [2603.13398, 2509.18189]. ## 2. Layout-as-Thought Mechanism End-to-end OCR models typically lose explicit access to document layout (e.g., bounding boxes, reading order, structural tags) previously available in multi-stage pipeline systems. Qianfan-OCR introduces the *Layout-as-Thought* mechanism—an intervention that enables two-phase response generation in presence of a special prompt token ``. - **Operational Flow:** Upon the `` trigger, the model first emits a block-structured layout description: ``` [[x₁,y₁,x₂,y₂]] Introduction ... ``` Here, coordinates are normalized to [0, 999] and encoded via dedicated `` tokens. After layout emission, the model produces the downstream output (OCR, structured Markdown, or QA result). - **Technical Realization:** The process can be schematized as: ```python def generate_with_thought(image, prompt): vis = vision_encoder(image) tokens = adapter(vis) + tokenize(prompt) thought = lm.generate(tokens, until="") out = lm.generate(tokens + tokenize(thought), until end-of-response) return thought, out ``` - **Impact:** This mechanism restores bounding box, type, and reading order information on demand, which is crucial for document post-processing and downstream tasks that require spatial grounding. On OmniDocBench v1.5, Layout-as-Thought increases table extraction metrics (Table^TEDs +0.19, Table^TEDss +0.18) and yields substantial gains on high-entropy (complex) layouts while imposing mild overhead or slight performance regression on simple homogeneous pages. ## 3. Prompt-Driven End-to-End OCR and Structured Output Qianfan-OCR supports fully prompt-conditioned operation, enabling seamless mapping from document image to highly structured output formats, notably Markdown. - **Workflow:** A user supplies a document image and a natural or instruction-style prompt (e.g., "Convert to Markdown," "Extract tables," "What date is listed in the contract?"). The model then emits a Markdown, table, or JSON format output in a single pass—or opt-in two-phase with layout block if `` is used. - **Output Capabilities:** The system natively emits: - Markdown with headings, paragraphs, fenced LaTeX (formula), Markdown/HTML tables, and image placeholders. - Structured table extractions, e.g.: ``` | Column A | Column B | Column C | |----------|----------|----------| | 1.23 | foo | bar | ``` - Natural language responses for chart or document QA. - JSON-encoded KIE results. This enables a unified solution for a variety of enterprise and research document intelligence workloads, superseding the need for separate OCR, parsing, and reasoning stages. ## 4. Quantitative Benchmark Performance Qianfan-OCR is evaluated on standard OCR and document understanding benchmarks, both as an end-to-end system and compared to multi-stage pipelines and large generalist VLMs. | Model | OmniDocBench v1.5 Overall | OCRBench | OlmOCR Bench | CCOCR Overall | DocVQA | ChartQA | KIE Mean | |----------------------|--------------------------:|---------:|-------------:|--------------:|-------:|--------:|---------:| | Qianfan-OCR | 93.12 | **880** | **79.8** | **79.3** | 92.8 | 88.1 | **87.9** | | Qwen3-VL-4B | n/a | 873 | 79.2 | 76.5 | 94.9 | 83.3 | 83.5 | | PaddleOCR-VL 1.5 | 94.50 | n/a | 80.0 | n/a | n/a | n/a | n/a | | Gemini-3.1-Pro | n/a | n/a | n/a | n/a | n/a | n/a | 79.2 | | Seed-2.0 | n/a | n/a | n/a | n/a | n/a | n/a | 78.0 | - On OmniDocBench v1.5, Qianfan-OCR leads among end-to-end models (93.12) and is competitive with specialist pipelines [2603.13398]. - On OlmOCR Bench, Qianfan-OCR outperforms Qwen3-VL-4B (79.8 vs. 79.2). - For general OCR (OCRBench/CCOCR), Qianfan-OCR shows the highest published numbers among VLMs at comparable scale. - On document understanding (DocVQA, ChartQA, CharXiv), two-stage pipelines collapse, particularly on spatially complex QA, while Qianfan-OCR maintains robust performance. - For KIE, Qianfan-OCR achieves the highest mean score across public benchmarks, exceeding Gemini-3.1-Pro, Seed-2.0, and Qwen3-VL-235B. **Throughput:** Qianfan-OCR achieves 1.024 pages/sec on an NVIDIA A100 (W8A8 quantized), half the speed of traditional pipelines at native precision, but achieves parity with quantized inference. ## 5. Experimental Analysis and Ablations - **Training Stage Ablation:** On the Qianfan-VL-8B model, each stage of the curriculum (alignment, foundational, domain-enhancement, instruction) contributes distinct, additive improvements to OCR accuracy. - **Layout-as-Thought Ablation:** Activation of `` slightly reduces overall OmniDocBench score (92.64 from 93.12), but specifically boosts table metrics and benefits high-entropy page layouts. - **Failure Modes:** Layout-as-Thought adds unnecessary computational cost and may marginally reduce performance on simple, single-column or homogeneous layouts. At present, layout reasoning is rigid (supervised structure), not adaptive to downstream goals. Its extensions to KIE and QA tasks remain unquantified. ## 6. Context, Innovations, and Extensions Qianfan-OCR is the first end-to-end 4B-parameter VLM to match or exceed specialist OCR pipelines on structured parsing tasks, with unified support for layout recovery and downstream QA/analytics within a single neural solution. Its design draws on domain-enhancement strategies developed for the larger Qianfan-VL family [2509.18189], demonstrating the enduring value of staged curriculum learning and high-precision synthetic data for OCR, as well as architectural scalability (from edge to cloud deployments). **Notable architectural elements and potential improvements:** - The Layout-as-Thought mechanism is conceptual analog to chain-of-thought prompting in text-based LMs, adapting it to layout grounding. Current implementation is deterministic and fixed-form; future directions include reinforcement learning for task-adaptive or free-form structure generation. - Further gain is anticipated from model/data scaling, advanced distillation/pruning for lighter variants, and extension to video, 3D text, or highly stylized and cursive scripts. - Applications include automated industrial processing (KYC, invoice/contract review), academic publishing (PDF-to-Markdown/LaTeX), grading or exam analysis, and chart analytics. Qianfan-OCR’s public availability on Baidu AI Cloud Qianfan platform facilitates wide adoption in research and enterprise. Its demonstrated ability to unify high-performance OCR, layout analysis, and advanced document understanding under a single architecture establishes a new methodological foundation for document AI [2603.13398, 2509.18189]. Source: https://www.emergentmind.com/topics/qianfan-ocr