MonkeyOCRv2: A Visual-Text Foundation Model for Document AI

Abstract: Mainstream visual encoders are pretrained on natural images and cannot be effectively applied to document images without document-oriented adaptation, as dense text and fine-grained character strokes demand character-level visual perception. We present MonkeyOCRv2, a visual-text pretrained model for document AI. First, we construct MonkeyDoc v2, to our knowledge the largest document-image pretraining corpus, comprising 113 million images spanning 17 languages. Second, we propose a pretraining strategy that jointly learns image-to-text generation and pixel-level document reconstruction: the former aligns visual representations with textual content, while the latter preserves character strokes and layout details. Extensive experiments are conducted on five representative document analysis tasks, including text recognition, formula recognition, text detection, document tampering detection, and overlapping text segmentation. Replacing the original encoders with MonkeyOCRv2 consistently improves performance across all five tasks. Finally, we validate its effectiveness as the vision encoder of multimodal LLMs on the more challenging tasks of document parsing and document understanding. Kept frozen and paired with a lightweight LLM, it yields a 0.7B document parsing model that sets a new open-source state-of-the-art on MDPBench, a recent benchmark spanning digital-born and photographed documents across 17 languages, surpassing the previous best 3B dots.mocr by 2.8% absolute with a vision encoder roughly 11$\times$ smaller. The frozen encoder also powers a document understanding model that outperforms counterparts built on CLIP, DINO, and SAM across eight benchmarks under identical training settings. These results suggest that document-oriented visual pretraining can serve as a foundation for document intelligence in its own right.

• The paper presents MonkeyOCRv2, a document-native visual-text encoder pretrained on a 113M image corpus to bridge the gap between natural image models and document-specific tasks.
• It employs a joint optimization of autoregressive text generation and pixel-wise reconstruction to capture fine-grained text details and complex layouts.
• MonkeyOCRv2 demonstrates consistent performance gains over larger baselines in text recognition, document parsing, and tampering detection across multiple languages.

MonkeyOCRv2: A Document-Native Visual-Text Foundation Model for Document AI

Introduction and Motivation

Current state-of-the-art vision foundation models (VFMs) such as CLIP, DINO, and SAM are predominantly pretrained on natural image corpora with objectives centered around high-level semantic abstraction, global alignment, or object boundaries. This paradigm results in a representational mismatch when these models are deployed in document AI scenarios, where dense text, intricate character-level cues, and complex layouts are central. Such local details are fundamentally important for knowledge extraction, document digitization, and the broader document intelligence pipeline. To address this domain gap, the paper presents MonkeyOCRv2—a family of visual-text encoders designed and pretrained explicitly for document intelligence via joint optimization of image-to-text generation and pixel-wise document reconstruction, thus yielding document-native representations transferable across diverse document AI tasks.

Figure 1: Overview of MonkeyOCRv2; by joint learning of text generation and pixel-level reconstruction, the model overcomes the representational mismatch of natural image-pretrained VFMs, providing transferable document-native visual embeddings.

Pretraining Methodology and Corpus Construction

The backbone of MonkeyOCRv2 is the MonkeyDoc v2 corpus, the largest document-oriented visual-text pretraining dataset to date, comprising 113 million document images in 17 languages. The dataset encompasses diverse document typologies—scientific papers, handwritten notes, forms, legal contracts, financial reports—distributed among 8M page-level and 105M cropped-element samples, covering both real (54%) and synthesized (46%) instances. Data annotation leverages multi-expert OCR systems and LLM-mediated filtering to enhance annotation quality and logical ordering, while extensive text and formula synthesis extend coverage to rare glyphs and complex layouts.

Figure 2: MonkeyOCRv2 is pretrained on a large-scale corpus of multilingual and multi-type document images to ensure robust generalization.

Pretraining is realized with an encoder-decoder architecture: the encoder (ViT-Small/Base or ViTAEv2-Small) feeds into both a vision decoder (for image reconstruction) and a text decoder (for autoregressive text generation). The loss function couples cross-entropy for image-to-text with a pixelwise (and optionally, gradient- or distance-aware) reconstruction objective, encouraging the encoder to preserve fine-grained visual cues and layout regularities even when linguistic context is weak or occluded.

Downstream Task Transferability

Text Recognition and Formula Recognition

Replacing the visual encoders of state-of-the-art text recognizers (e.g., CRNN, PARSeq) with MonkeyOCRv2 yields consistent and substantive absolute accuracy gains (up to 16% for CRNN on the challenging Union14M text benchmark; 3.3% for PARSeq). On saturated English and Chinese benchmarks, only marginal improvements are observed—a reflection of ceiling effects. Critically, for formula recognition with UniMERNet, MonkeyOCRv2 improves performance on both CDM (character-level) and ExpRate (exact matching), enabling smaller-capacity models to outperform larger baselines, especially for handwritten and complex formulas.

Text Detection, Segmentation, and Tampering Detection

Integration with modern detectors (DBNet, PSENet, DPText-DETR) shows consistent gains on curved and multi-oriented scene text benchmarks, exceeding both ImageNet and text detection-specific oCLIP initialization. In document tampering detection and overlapping text segmentation, MonkeyOCRv2-as-backbones yield significant performance improvements, especially for fine-grained manipulation localization and robust segmentation under occlusion and cross-domain generalization.

Figure 3: MonkeyOCRv2 achieves robust improvements for text detection on several benchmarks, outperforming baselines irrespective of detector architecture.

Document Parsing and Understanding

A fixed (frozen) MonkeyOCRv2 encoder, paired with lightweight LLMs, is evaluated for multilingual document parsing (MDPBench, 17 languages) and open-domain document understanding (VQA-style). On MDPBench, the 0.1B-parameter MonkeyOCRv2 vision encoder establishes a new open-source state-of-the-art, surpassing dots.mocr (1.2B) by 2.8% and PaddleOCR-VL-1.6 by 8.3%, while being up to 11x smaller. On OmniDocBench, MonkeyOCRv2 outperforms much larger generalist VLMs such as Qwen3-VL-235B and GPT-5.2, despite a minimal decoding pipeline and absence of specialized post-training.

Figure 4: MonkeyOCRv2 outperforms larger counterparts on MDPBench and yields consistent absolute performance gains across seven document analysis tasks.

In controlled document understanding (VQA) tasks, holding the LLM and pipeline constant, MonkeyOCRv2 outperforms all major VFMs—CLIP, DINO, SAM, SigLIP 2, OpenVision, oCLIP, DiT—by wide margins. The ablation shows that joint pixel-level reconstruction is required to close the domain gap: removing reconstruction increases reliance on language priors and reduces robustness, especially for scrambled or occluded inputs. An explicit evaluation on CHAOS-Bench (perturbed/hallucinated documents) demonstrates a 3.7-point improvement over the strongest prior open-source model.

Figure 5: Left: scrambled text recognition accuracy across model and resolution; Right: the accuracy gap between normal and scrambled text diminishes with reconstruction supervision, indicating reduced dependence on language priors.

Qualitative Analysis

Visualization of parsing results on Arabic and Chinese documents demonstrates that MonkeyOCRv2-Parsing maintains correct language flow, reading order, and structure, avoiding hallucination and misordering seen in strong baselines but generalist vision backbones.

Figure 6: Parsing an Arabic document: MonkeyOCRv2 preserves right-to-left reading order and avoids hallucinations present in other models.

Figure 7: Parsing a photographed Chinese instruction manual: MonkeyOCRv2 maintains accurate table structure and content, outperforming reference models.

On fine-grained document understanding (e.g., text-rich infographics, forms), MonkeyOCRv2-based models recover dense text, numerals, and tabular details accurately, while generalist vision models struggle with subtle character distinctions, logical ordering, and rare glyphs.

Figure 8: In document understanding, MonkeyOCRv2 exhibits superior recognition of fine-grained text over leading VFMs.

Theoretical and Practical Implications

The findings from the paper present a strong claim: general-purpose VFMs pretrained on natural images, even with large-scale data and advanced architectures, cannot, by themselves, realize fine-grained, robust document intelligence. Document-native pretraining—particularly when pixel-level reconstruction objectives are incorporated—is necessary to preserve character, stroke, and layout features critical for text-heavy visual reasoning.

Practically, the results imply that compact document-specialized vision encoders can supplant or outperform much larger generalist backbones in document parsing, recognition, and understanding pipelines. This reduces computational requirements and democratizes high-quality document AI for lower-resource deployments or languages with limited prior annotation.

Theoretically, the combination of generative and reconstruction-based supervision challenges the primacy of global semantic alignment for visual tasks, advocating instead for tasks and losses that reflect the local visual statistics and granularity of the target domain.

Limitations and Prospects

The paper notes that, while the frozen encoder achieves state-of-the-art results on discriminative benchmarks like MDPBench, further post-training and architectural refinement are required to outperform the latest task-specialized parsers on saturated benchmarks such as OmniDocBench. MonkeyOCRv2 maintains a minimal, decoder-agnostic transfer protocol, and future work should address transformer/decoder design, optimization schedules, and broadening coverage to low-resource and historical scripts.

Conclusion

MonkeyOCRv2 advances the state of document intelligence by introducing a document-native visual-text encoder family, operating on a 113M image corpus with joint text and pixel-level supervision. Across multilingual parsing, text/formula recognition, detection, segmentation, tampering detection, and open-domain document understanding, MonkeyOCRv2 provides consistent and substantial performance gains. The research shows, with detailed controlled experiments and ablations, that domain-specific objectives are essential for robust document AI, and that such encoders can be both compact and highly transferable. The release of MonkeyDoc v2 and pretrained models establishes crucial infrastructure for future advancements in truly multimodal document intelligence.

---