MonkeyOCRv2: Document AI Pretraining

• The paper introduces MonkeyOCRv2 as a document-specific visual encoder that employs dual objectives—text generation and pixel-level reconstruction—to preserve fine character details and layout.
• MonkeyOCRv2 is pretrained on MonkeyDoc v2, the largest multilingual document-image corpus, covering diverse sources and 17 languages for broad applicability.
• The model outperforms traditional backbones in OCR, formula recognition, detection, and parsing tasks while offering size-efficiency and flexible downstream integration.

Searching arXiv for the cited MonkeyOCR and MonkeyOCRv2 papers to ground the article with up-to-date metadata. MonkeyOCRv2 is a visual-text pretrained model for document AI that is designed for document images rather than natural-image semantics. Its central premise is that document perception requires character-level visual fidelity—fine strokes, punctuation, superscripts and subscripts, dense layouts, and reading-order-sensitive structure—and therefore cannot be reduced to the objectives typically used for mainstream visual encoders pretrained on natural images. The model is pretrained from scratch on MonkeyDoc v2, a 113 million-image corpus spanning 17 languages, with a joint image-to-text generation and pixel-level document reconstruction objective, and is evaluated both as a backbone replacement for specialized document-analysis systems and as a frozen vision encoder for lightweight multimodal LLMs (Liu et al., 13 Jul 2026).

1. Lineage and relation to MonkeyOCR

MonkeyOCRv2 is related to, but not identical with, the earlier MonkeyOCR system. The 2025 MonkeyOCR paper introduced a document parsing framework based on a Structure-Recognition-Relation triplet paradigm, or SRR, which decomposes parsing into three questions: “Where is it?”, “What is it?”, and “How is it organized?”. In that formulation, a YOLO-based document layout detector predicts semantic regions, a unified large multimodal model recognizes cropped blocks with type-specific prompts, and a reading-order module reconstructs the final sequence. That system was trained and evaluated with MonkeyDoc, a bilingual dataset containing 3.9 million instances spanning over ten document types, and it reported an average 5.1% improvement over MinerU, including +15.0% on formulas and +8.6% on tables (Li et al., 5 Jun 2025).

MonkeyOCRv2 shifts the primary object of study from a parser to a document-oriented visual foundation model. The 2026 paper positions it as a “standalone encoder intended as a backbone substitution,” while also showing that it can be kept frozen and paired with lightweight LLMs for parsing and understanding (Liu et al., 13 Jul 2026). A common misconception is to treat MonkeyOCRv2 as merely a second iteration of the SRR parser. The publication record described here does not support that reading: the 2025 paper explicitly does not mention a version labeled MonkeyOCRv2, whereas the 2026 paper establishes MonkeyOCRv2 as a distinct document-native encoder family.

This distinction matters methodologically. MonkeyOCR emphasized a structured document workflow; MonkeyOCRv2 emphasizes pretraining representations that preserve document-specific visual evidence. A plausible implication is that MonkeyOCRv2 generalizes across a broader class of downstream systems because its unit of reuse is the encoder rather than the entire parser.

2. MonkeyDoc v2 and multilingual pretraining data

MonkeyDoc v2 is described as “to our knowledge the largest document-image pretraining corpus,” comprising 113 million images across 17 languages: Simplified Chinese, Traditional Chinese, English, Arabic, German, Spanish, French, Hindi, Indonesian, Italian, Japanese, Korean, Dutch, Portuguese, Russian, Thai, and Vietnamese (Liu et al., 13 Jul 2026). The corpus contains 8M page-level images and 105M cropped document elements, with 61M samples (54%) marked as real and 52M samples (46%) marked as synthetic.

Its document coverage is intentionally broad. The corpus includes printed papers, scanned books, handwritten notes, newspapers, magazines, financial reports, rendered documents, tables, formulas, layouts, and multi-type document regions. The data engine is organized into three modules: Expert Model Labeling, Multilingual Corpus-Based Data Synthesis, and Data Filtering. Real documents are labeled with a multi-expert agreement pipeline, while synthetic OCR data are rendered from multilingual LLM corpora across the same 17 languages. Table data are generated from real templates and synthetic table structures, and formula data are collected from arXiv papers, yielding approximately 0.8M formula samples.

The paper also gives several concrete curation details. Raw data are drawn from official training splits of FinePDFs, MonkeyDoc, Union14M, UniMER-1M, CDLA, D4LA, DocGenome, DocLayNet, M6Doc, SVRD, TabRecSet, COCOTextV2, HierText, DSTextV2, LSVT, OpenImagesV5Text, ReCTS, TextOCR, and MTWI, together with large-scale collections of newspapers, handwritten notes, and presentation slides. After filtering, about 0.9M of the initial 1.2M pages from challenging sources are retained.

The scale of MonkeyDoc v2 is not presented as an end in itself. It is used to support the claim that document encoders need multilingual and structurally diverse supervision at both page and element levels. This suggests that the corpus is intended to close a gap left by prior document-oriented encoders that either covered narrower data distributions, were publicly limited, or lacked the combination of scale and fine-grained supervision targeted here.

3. Pretraining formulation and document-specific objectives

MonkeyOCRv2 uses a three-part pretraining framework composed of a vision encoder $E_v$, a vision decoder $D_v$, and a text decoder $D_t$ (Liu et al., 13 Jul 2026). For an input image $I \in \mathbb{R}^{H \times W \times 3}$, the encoder produces

$\mathbf{z} = E_v(I),$

and the vision decoder reconstructs

$\hat{I} = D_v(\mathbf{z}).$

The pretraining objective combines image-to-text generation with reconstruction:

$$\mathcal{L}_{\mathrm{pretrain}} = \mathcal{L}_{\mathrm{text}} + \lambda \mathcal{L}_{\mathrm{rec}}.$$

Here, $\mathcal{L}_{\mathrm{text}}$ is the standard autoregressive cross-entropy for text prediction from visual tokens with task-specific prompts. The reconstruction branch is intended to prevent the encoder from discarding the fine-grained details that document images encode in strokes, glyph shapes, punctuation, and layout.

The default reconstruction loss is pixel MSE. The paper also defines a structure-aware reconstruction variant built from a Sobel-gradient magnitude, a soft edge map, a truncated distance-to-edge initialization, an iterative distance-map update, and a structure-matching loss. The reported hyperparameters are $\alpha = 0.5$, $\beta = 0.25$, $D_v$0, $D_v$1, and $D_v$2. However, the reported results use the MSE-only objective by default, that is, $D_v$3; the structure-aware objective is evaluated only in document understanding experiments.

The conceptual argument is that the two objectives are complementary. Text generation aligns visual representations with textual content, while reconstruction preserves visible evidence that may be weakly coupled to language priors. The paper explicitly frames this as a move away from “global semantic abstraction” toward “character-level visual fidelity.”

4. Encoder variants and optimization regime

Three encoder variants are trained from scratch under a common recipe (Liu et al., 13 Jul 2026).

Variant	Backbone	Parameters
MonkeyOCRv2-S	ViT-Small	28M
MonkeyOCRv2-B	ViT-Base	113M
MonkeyOCRv2-AS	ViTAEv2-Small	21M

The ViTAEv2-Small variant is preferred for tasks that need stronger multi-scale inductive bias, specifically detection, segmentation, and tampering localization. Patch size is 14 for the S and B variants and 16 for the AS variant. The maximum input pixels are 1,003,520 for S and B, and 1,802,240 for AS. Training uses dynamic resolution, 64 NVIDIA A800 GPUs, a peak learning rate of $D_v$4, and batch size 256.

The model family is intended to be modular rather than monolithic. In downstream settings, the encoder is either swapped into existing task-specific architectures or kept frozen and connected to a lightweight LLM through an MLP projector. That design choice is important for the paper’s broader claim: document-oriented pretraining can function as a reusable substrate for document AI rather than as a component tied to a single end-to-end system.

5. Downstream transfer across document AI tasks

The paper evaluates MonkeyOCRv2 on seven tasks and reports consistent gains when the original vision backbones are replaced (Liu et al., 13 Jul 2026). In text recognition, swapping MonkeyOCRv2-S into CRNN raises overall accuracy from 58.7% to 67.3%, while PARSeq rises from 82.2% to 84.3%. On Union14M-Benchmark, CRNN gains 16.0% absolute average accuracy and PARSeq gains 3.3%; PARSeq plus MonkeyOCRv2 reaches 87.6% on Union14M, exceeding the previous best SVTRv2 at 86.1%. The appendix also reports average gains on common benchmarks: CRNN improves from 90.2 to 92.5 and PARSeq from 96.4 to 96.8.

In formula recognition, UniMERNet-T with MonkeyOCRv2-S improves on OmniDocBench 1.6 by CDM +0.9 and ExpRate +3.9, on MathWriting by CDM +5.2 and ExpRate +3.3, and on UniMER-Test by ExpRate +9.3 for Complex Printed Expressions and +7.6 for Handwritten Expressions. The paper additionally states that the 110M MonkeyOCRv2-T variant surpasses the 325M UniMERNet-B.

In text detection, replacing the backbone with MonkeyOCRv2-AS improves DBNet, PSENet, and DPText-DETR across ICDAR2015, ArT, Total-Text, and CTW1500. Representative examples include DBNet on ICDAR2015 improving from 85.0 to 88.5 F-measure, DPText-DETR on ArT improving by 3.1%, DBNet on Total-Text improving by 3.3%, and PSENet and DPText-DETR on CTW1500 improving by 3.7% and 3.2%. The model also outperforms both the original ImageNet-pretrained encoder and oCLIP. In document tampering detection, FFDN with MonkeyOCRv2-AS reaches 87.4 IoU / 93.3 F1 on DocTamper-Test, with an 11.3% absolute F1 gain over the baseline, and obtains best F1 scores of 88.9 on DocTamper-FCD and 80.4 on DocTamper-SCD. In overlapping text segmentation, Mask2Former improves mIoU$D_v$5 from 70.3 to 76.6 and MOTS from 72.6 to 76.9, with overlap-region IoU also improving.

The document parsing setting uses a frozen MonkeyOCRv2-S or MonkeyOCRv2-B encoder, an MLP projector, and Qwen3-0.6B. The parsing procedure is: predict coordinates and categories of document elements in reading order, crop each element, recognize content with prompts, and assemble the structured output. Training is two-stage, with the encoder frozen throughout: first the MLP projector is trained alone, then the MLP and LLM are trained jointly. On MDPBench, which spans digital-born and photographed documents across 17 languages, MonkeyOCRv2-B-Parsing reaches 83.3 overall, surpassing the previous best dots.mocr at 80.5 by 2.8% absolute. The paper also states that the vision encoder is only 0.1B, roughly 11$D_v$6 smaller than dots.mocr’s 1.2B vision encoder, and that MonkeyOCRv2-B-Parsing outperforms PaddleOCR-VL-1.6 by 8.3%. On OmniDocBench 1.6, the parsing model is described as competitive but still behind the strongest specialized parsers.

For document understanding, the paper uses a controlled VLM setup analogous to LLaVA: a frozen vision encoder, an MLP projector, and Qwen3-1.7B, with all encoders trained under identical data, settings, and decoding procedures. Across DocVQA, InfoVQA, DeepForm, KLC, WTQ, ChartQA, DT-VQA, and OCRBench, MonkeyOCRv2-B reaches 57.2 overall, ahead of OpenVision-B at 44.0 and RADIOv2.5-B at 37.5, and far ahead of CLIP-B at 16.0, SigLIP 2-B at 24.9, DINOv3-B at 16.1, and SAM-B at 25.2.

6. Analytical results and size-efficiency

The paper includes several analyses intended to explain why the pretraining objectives transfer effectively (Liu et al., 13 Jul 2026). In document understanding, the reconstruction ablation reports 50.7 for the MonkeyOCRv2-S baseline, 51.7 with MSE-only reconstruction, 55.9 with structure-aware reconstruction, and 57.2 for MonkeyOCRv2-B*. Within the scope of that experiment, reconstruction improves results, and the structure-aware variant improves them further.

The scrambled-text analysis uses the accuracy gap between semantically coherent text and scrambled text as a proxy for reliance on linguistic context. The reported outcome is that reconstruction improves scrambled-text recognition substantially, especially at low resolution, narrows the semantic–scrambled gap, and helps the encoder remain useful when language priors are weak. The paper explicitly notes that this metric is an operational proxy rather than a perfect hallucination measure. On CHAOS-Bench, reconstruction is also associated with improved faithfulness to visually perturbed text; the reported page-average recall on visually modified words is 17.9 for MonkeyOCRv2-B-Parsing, and MonkeyOCRv2-S-Parsing improves by +2.6 over its baseline without reconstruction.

A further analytical point concerns efficiency at the encoder level. MonkeyOCRv2-B-Parsing attains 83.3 on MDPBench with a 0.1B vision encoder, while dots.mocr reaches 80.5 with a 1.2B vision encoder. The paper uses this comparison to support the “$D_v$7 smaller encoder” claim. The appendix also reports average visual token counts in the controlled understanding setup: CLIP uses 196 tokens, SigLIP 2, RADIO, and DINOv3 use 825, SAM and SAM2 use 4096, DiT uses 196, oCLIP uses 1024, and MonkeyOCRv2-S/B uses 1082. This suggests that the paper’s efficiency argument is not merely about token minimization, but about the quality of document-native visual features at a given model scale.

7. Limitations, reproducibility, and significance

The paper states several limitations directly (Liu et al., 13 Jul 2026). First, the parsing model uses a minimal frozen-encoder setup and does not include the progressive post-training pipeline used by some specialized parsers, which leaves a gap on saturated benchmarks such as OmniDocBench. Second, the parsing architecture is autoregressive and sequential at the layout stage, so it prioritizes accuracy and structure over speed. Third, the study does not fully isolate the separate effects of decoder design or reconstruction-weight scheduling. Fourth, MonkeyDoc v2 is multilingual but remains skewed toward high-resource scripts; low-resource and historical writing systems are left as future work.

The reproducibility claims are comparatively explicit. Code and data will be released at https://github.com/Yuliang-Liu/MonkeyOCRv2. The paper also states that all downstream validation and test splits are excluded from pretraining, downstream training, checkpoint selection, prompt selection, and hyperparameter tuning.

Taken together, MonkeyOCRv2 defines a document-native alternative to generic visual backbones. Its distinguishing properties are the scale and multilinguality of MonkeyDoc v2, the combined text-generation and reconstruction objective, and the demonstration that a frozen encoder can support lightweight document parsing and document understanding systems while remaining effective as a drop-in replacement for OCR, formula recognition, detection, segmentation, and tampering models. The paper’s concluding interpretation is that document-oriented visual pretraining can serve as a foundation for document intelligence in its own right (Liu et al., 13 Jul 2026).