---
title: 'MonkeyDoc v2: Document Pretraining Corpus'
url: https://www.emergentmind.com/topics/monkeydoc-v2
type: topic
---

# MonkeyDoc v2: Document Pretraining Corpus

Searching arXiv for the cited MonkeyOCRv2 paper and a few closely related document-AI baselines to ground the article.
MonkeyDoc v2 is a multilingual, multi-type document-image pretraining corpus introduced as the sole pretraining corpus for the MonkeyOCRv2 visual encoder family in “MonkeyOCRv2: A Visual-Text Foundation Model for Document AI” [2607.11562]. It contains **113 million** visual–text samples spanning **17 languages**, with each sample defined as either a full document page or a cropped document element paired with textual supervision. Its stated purpose is to teach a vision encoder to resolve dense text, fine-grained character strokes, and document layouts at character level rather than relying on visual representations inherited from natural-image pretraining. Within MonkeyOCRv2, MonkeyDoc v2 underwrites a dual-objective pretraining regime that jointly optimizes image-to-text generation and pixel-level document reconstruction, thereby supplying both semantic labels and document-native structural variation for downstream OCR, detection, parsing, and document understanding tasks [2607.11562].

## 1. Definition, scope, and role

MonkeyDoc v2 is defined as a multilingual, document-image pretraining corpus containing **113 million** visual–text samples, where a sample is either a full page or a cropped document element such as a text line, table, or formula paired with supervision [2607.11562]. The corpus is described as the sole pretraining corpus for MonkeyOCRv2 encoders, and its role is explicitly tied to a dual training objective: **image-to-text generation** for alignment between visual tokens and textual content, and **pixel-level document reconstruction** for preserving stroke-level and layout information [2607.11562].

This design positions MonkeyDoc v2 as a document-native alternative to mainstream visual pretraining regimes based on natural images. The motivating claim is that mainstream visual encoders are pretrained on natural images and “cannot be effectively applied to document images without document-oriented adaptation,” because documents require perception of dense text and fine character strokes rather than coarse object-level semantics [2607.11562]. In that framing, MonkeyDoc v2 is not merely a benchmark collection but a foundation corpus for learning representations specialized for document intelligence.

A plausible implication is that MonkeyDoc v2 is intended to collapse several previously separated supervision regimes—OCR labels, layout annotations, table and formula content, multilingual script coverage, and synthetic edge-case generation—into a single pretraining substrate. The paper’s downstream results are presented as evidence that such unified, document-oriented pretraining can function “as a foundation for document intelligence in its own right” [2607.11562].

## 2. Scale, language coverage, and document composition

MonkeyDoc v2 comprises **113M** total samples across **17** languages, counting Simplified and Traditional Chinese separately [2607.11562]. Its granularity is explicitly split into **8M** page-level images and **105M** cropped document elements, with the term “sample” defined to include either kind of unit [2607.11562]. This mixed granularity is central to the corpus design: page-level samples preserve global layout and reading order, whereas cropped elements concentrate supervision on local regions such as lines, words, table cells, and formulas.

The 17 languages are listed as Simplified Chinese, Traditional Chinese, English, Arabic, German, Spanish, French, Hindi, Indonesian, Italian, Japanese, Korean, Dutch, Portuguese, Russian, Thai, and Vietnamese [2607.11562]. The language distribution is described as dominated by **English (~19M samples)** and **Chinese (~13M Simplified + Traditional)**, followed in descending scale by Dutch, Portuguese, Russian, French, Indonesian, Spanish, Hindi, German, Japanese, Italian, Thai, Korean, Arabic, and Vietnamese [2607.11562]. The corpus is therefore strongly English–Chinese in mass while still covering Latin, CJK, Arabic, Devanagari, Thai, and Vietnamese scripts.

MonkeyDoc v2 covers a broad spectrum of document types, including printed papers and scientific articles, scanned books, handwritten notes, newspapers and magazines, financial reports, rendered and digital-born documents such as PDF pages and slides, visually rich documents such as forms, tables, and infographics, scene or “in-the-wild” text, and both LaTeX-rendered and real-world formulas and tables [2607.11562]. Representative examples are reported to include PDFs, invoices, forms, slides, manuals, receipts, web pages, and photographed documents across scripts [2607.11562].

The corpus deliberately mixes real and synthetic data. The stated split is **61M real samples (54%)** and **52M synthetic samples (46%)** [2607.11562]. Real data is used to preserve realistic layouts, scanning and photographic noise, and document artifacts, while synthetic data is used to expand multilingual coverage, cover rare characters and low-frequency symbols, and generate diverse tables and formulas with controlled structures [2607.11562]. This suggests a corpus design in which realism and coverage are treated as complementary rather than competing objectives.

## 3. Data sources and construction pipeline

MonkeyDoc v2 is built through what the paper terms a “data engine” with three core modules: **Expert Model Labeling**, **Multilingual Corpus-Based Data Synthesis**, and **Data Filtering** [2607.11562]. Real-world documents are collected from the official training splits of many public datasets, including FinePDFs, MonkeyDoc (v1), Union14M, UniMER‑1M, CDLA, D4LA, DocGenome, DocLayNet, M6Doc, SVRD, TabRecSet, COCOTextV2, HierText, DSTextV2, LSVT, OpenImagesV5Text, ReCTS, TextOCR, and MTWI, along with large collections of newspapers, handwritten notes, and presentation slides gathered by the authors [2607.11562]. Synthetic data is generated from multilingual LLM corpora, Unicode character sets per language, and formulas crawled from arXiv, producing approximately **0.8M formula image–LaTeX pairs** [2607.11562].

For real documents, annotation is performed using a multi-expert agreement pipeline rather than a single OCR model. The process first applies a layout detection model to segment each page into text blocks, tables, formulas, and “other” regions; then runs several complementary OCR or expert recognizers on each cropped element; then computes pairwise similarity among predictions; and finally chooses the prediction with the highest average agreement across experts as the label [2607.11562]. This produces bounding boxes and region types from layout detection, per-region text transcription from multi-expert OCR agreement, and reading order for page-level data [2607.11562].

Synthetic generation proceeds by sampling multilingual text from LLM corpora, rendering it with diverse fonts, styles, and resolutions, constructing full character sets per language to generate random strings that cover rare glyphs, generating table templates populated with multilingual text, and rendering formulas collected from arXiv [2607.11562]. The paper emphasizes that synthetic data comes with perfect textual labels and controlled structural patterns, complementing the noise and variability of real documents.

Quality control is implemented through two LLM-based filters. The first is a **layout completeness check**: all detected regions are masked with white blocks, the masked page is fed into a strong document-oriented VLM, and if the model still reads residual text, the page is discarded because the detector likely missed regions [2607.11562]. The second is a **reading-order consistency** check: recognized text is concatenated according to annotated reading order, an LLM evaluates whether the sequence is logically consistent and free of reading-order errors, and inconsistent samples are discarded [2607.11562]. From an initial **1.2M** pages drawn from particularly challenging sources, approximately **0.9M** are retained after these filters [2607.11562].

The paper states that all validation and test splits of downstream benchmarks are excluded from pretraining, fine-tuning, and hyperparameter search to avoid contamination [2607.11562]. It does not discuss deduplication explicitly, and it does not provide a detailed legal or ethical analysis, though it notes that the listed datasets are public research datasets, that only official training splits are used, and that synthetic data is generated from publicly available corpora and Unicode tables [2607.11562].

## 4. Pretraining objectives and representation learning function

MonkeyDoc v2 is the exclusive training corpus for MonkeyOCRv2 encoder variants including ViT-S, ViT-B, and ViTAEv2-S [2607.11562]. The pretraining architecture contains a vision encoder \(E_v\), a vision decoder \(D_v\) used for reconstruction, and a text decoder \(D_t\) used for generation. Given an input image \(I \in \mathbb{R}^{H\times W \times 3}\), the encoder produces visual tokens \(\mathbf{z} = E_v(I)\), the vision decoder reconstructs \(\hat{I} = D_v(\mathbf{z})\), and the text decoder generates supervised text autoregressively from \(\mathbf{z}\) plus task prompts [2607.11562].

The default pixel-level reconstruction loss is given as
\[
\mathcal{L}_{\mathrm{pix} = \frac{1}{3HW}\left\|\hat{\mathbf{I} - \mathbf{I}\right\|_2^2
\]
and is used to force the encoder to preserve enough information for the decoder to rebuild the input document image, including small glyphs and layout details [2607.11562]. A structure-aware variant used in document understanding experiments augments this with edge- and distance-to-edge consistency through soft edge maps \(\mathcal{E}(\mathbf{I})\), truncated distance-to-edge maps \(\mathcal{D}(\mathbf{I})\), and a structure-matching loss
\[
\mathcal{L}_{\mathrm{struct} = \frac{1}{HW}\|\mathcal{D}(\hat{\mathbf{I}) - \mathcal{D}(\mathbf{I})\|_1 + \beta\frac{1}{HW}\|\mathcal{E}(\hat{\mathbf{I}) - \mathcal{E}(\mathbf{I})\|_1
\]
with full reconstruction objective
\[
\mathcal{L}_{\mathrm{rec} = \mathcal{L}_{\mathrm{pix} + \alpha \mathcal{L}_{\mathrm{struct}
\]
where \(\alpha = 0.5\) and \(\beta = 0.25\) [2607.11562].

The text generation objective is standard autoregressive cross-entropy, denoted \(\mathcal{L}_{\mathrm{text}}\), over page-level reading order, line-level content, formulas, table cell text, and related supervised text targets [2607.11562]. The overall pretraining objective is
\[
\mathcal{L}_{\mathrm{pretrain} = \mathcal{L}_{\mathrm{text} + \lambda \mathcal{L}_{\mathrm{rec}
\]
with \(\lambda = 1.0\) fixed [2607.11562]. In the authors’ interpretation, \(\mathcal{L}_{\mathrm{text}}\) aligns visual tokens with semantic text content, while \(\mathcal{L}_{\mathrm{rec}}\) ensures retention of fine-grained visual evidence such as strokes, punctuation, and layout structures that a purely semantic objective might discard [2607.11562].

To support character-level perception, the pretraining regime uses dynamic resolution training with variable-length tokenization, following the design pattern of Qwen2.5-VL, so that the number of visual tokens adapts to input resolution [2607.11562]. Patch sizes are \(14\times14\) for MonkeyOCRv2-S/B with maximum pixel budget approximately **1,003,520**, and \(16\times16\) for MonkeyOCRv2-AS with maximum pixel budget approximately **1,802,240** [2607.11562]. The combination of **8M** page-level samples and **105M** crops is presented as a mechanism for simultaneously preserving global structure and exposing tiny text at high effective resolution [2607.11562].

## 5. Covered document phenomena and multilingual challenges

MonkeyDoc v2 is designed to cover the principal challenges of document vision: dense text and small fonts; complex layouts including multi-column text, sidebars, headers, footers, and mixed figure/table regions; multilingual and mixed-script documents; formulas and structured expressions; tables and spreadsheets with merged cells and complex borders; overlapping or occluded text; handwriting; and photographed pages with distortion, blur, and reflections [2607.11562].

The multilingual design is explicitly tied to script coverage and character inventory completeness. The dataset includes Latin-based languages, CJK scripts, Arabic, Devanagari for Hindi, Thai, and Vietnamese, and it deliberately extracts full character sets per language and generates random character strings to ensure coverage of rare characters and diacritics [2607.11562]. The paper identifies Arabic contextual forms, Thai vowel signs, Vietnamese diacritics, and CJK ideographs as cases where such coverage is especially important [2607.11562].

The reported downstream multilingual parsing results on MDPBench are **84.2** for Latin languages and **82.1** for non-Latin languages using MonkeyOCRv2-B-Parsing, with the paper remarking that the gap is small despite heavy-tail languages [2607.11562]. Qualitative examples are described as showing improved parsing of Arabic right-to-left multi-column documents and Chinese photographed manuals relative to PaddleOCR-VL-1.6 and dots.mocr [2607.11562]. This suggests that MonkeyDoc v2’s multilingual balance, while uneven in sample counts, is sufficient to support broad cross-script transfer.

A common misconception would be to treat MonkeyDoc v2 simply as a larger OCR dataset. The corpus is described instead as multi-type and layout-rich, spanning page-level reading order, local element recognition, formulas, tables, photographed documents, handwriting, and scene text [2607.11562]. Its function is therefore not restricted to transcription accuracy but extends to layout-sensitive and visually subtle tasks such as tampering detection and overlapping text segmentation.

## 6. Empirical impact and comparison with earlier corpora

The paper attributes a broad set of downstream gains to encoders pretrained solely on MonkeyDoc v2 [2607.11562]. In text recognition, replacing standard backbones with MonkeyOCRv2-S raises CRNN overall accuracy across Union14M benchmark, a Chinese benchmark, and occluded scene text from **58.7%** to **67.3%**, and raises PARSeq overall accuracy from **82.2%** to **84.3%**; on Union14M-Benchmark, PARSeq improves from **86.4%** to **87.6%**, surpassing prior SOTA SVTRv2 at **86.1%** [2607.11562]. In formula recognition, UniMERNet-T with MonkeyOCRv2-S improves OmniDocBench 1.6 formula-subset CDM from **89.9** to **90.8** and ExpRate from **57.2** to **61.1**, MathWriting CDM from **65.6** to **70.8**, and MathWriting ExpRate from **12.9** to **16.2** [2607.11562].

In text detection, replacing ImageNet-pretrained backbones or oCLIP with MonkeyOCRv2-AS yields consistent F-measure gains across ICDAR 2015, ArT, Total-Text, and CTW1500; one cited example is DBNet on ICDAR2015 improving from **85.0** to **88.5** [2607.11562]. In document tampering detection, swapping ConvNeXt-V2 or DeepSolo-backbone to MonkeyOCRv2-AS inside FFDN on DocTamper raises IoU from **69.4** to **87.4** and F1 from **82.0** to **93.3** on DocTamper-Test [2607.11562]. In overlapping text segmentation on the MOT dataset, Mask2Former mIoU\(_\text{Text}\) increases from **70.3** to **76.6**, and MOTS mIoU\(_\text{Text}\) from **72.6** to **76.9** [2607.11562].

The most prominent system-level result is in document parsing. MonkeyOCRv2-B-Parsing, a **0.7B** model with **0.1B** vision encoder, attains **83.3** on MDPBench, surpassing dots.mocr (**80.5**) and PaddleOCR-VL-1.6 (**75.0**) while using a vision encoder “about **11× smaller**” than the previous best open-source dots.mocr [2607.11562]. In controlled document understanding experiments, a frozen MonkeyOCRv2-B encoder paired with Qwen3-1.7B yields an average **57.2** across eight document VQA benchmarks, outperforming CLIP, SigLIP2, DINOv3, SAM/SAM2, oCLIP, DiT, OpenVision-B, and RADIOv2.5-B under identical training settings [2607.11562].

The paper compares MonkeyDoc v2 with earlier document-oriented corpora. It reports: oCLIP at **450K** scene-text images in **2** languages; DiG at **35.6M** scene-text images in **1** language; DiT (IIT-CDIP) at **42M** scanned documents in **1** language with no fine-grained text annotations; Donut at **13M** scanned or synthetic documents in **4** languages; Pix2Struct at **80M** web screenshots in **1** language; and MonkeyDoc v2 at **113M** multi-type documents in **17** languages with dense text annotations [2607.11562]. On that basis, MonkeyDoc v2 is described as larger, more diverse in type and script, and more richly annotated than prior open document corpora known to the authors [2607.11562].

Relative to MonkeyDoc (v1), the paper states that MonkeyDoc v2 expands from a smaller, more limited corpus to **113M samples**, broadens language coverage from primarily Chinese and English to **17 languages**, expands document types to include scanned and digital documents, handwritten notes, newspapers, and slides, and shifts from mainly recognition and spotting data to a unified corpus supporting layout, recognition, tables, formulas, and document parsing [2607.11562]. This suggests a transition from a task-oriented corpus toward a foundation-style pretraining resource.

## 7. Availability, reconstruction, and broader significance

The paper states that “Code and data will be released at https://github.com/Yuliang-Liu/MonkeyOCRv2” [2607.11562]. It also notes that “data” in practice may mean training recipes for synthetic generation, lists of public datasets and usage, and possibly partial or derived annotations, since some underlying corpora have their own licenses and re-hosting the full aggregate may be constrained [2607.11562]. The source enumeration and pipeline description are sufficiently explicit that practitioners can reconstruct a similar corpus by reusing the same public training splits, implementing a layout plus multi-expert OCR labeling pipeline, synthesizing multilingual text and tables from LLM corpora and Unicode sets, and applying layout-completeness and reading-order-consistency checks [2607.11562].

The paper advises respecting the licenses of underlying datasets, excluding validation and test splits of planned evaluation benchmarks, and being mindful of bias in OCR models and LLM-based filters if replicating the expert-labeling pipeline [2607.11562]. In practical terms, it also suggests that the easiest route to benefit from MonkeyDoc v2 is to use MonkeyOCRv2’s pretrained encoders or the MonkeyOCRv2-Parsing model as drop-in backbones or frozen VLM vision modules [2607.11562].

In the broader document-AI landscape, MonkeyDoc v2 is presented as an answer to a longstanding dependency on natural-image encoders such as CLIP, DINO, SAM, and related families for document tasks [2607.11562]. The central claim is that dense, multilingual, document-native supervision can produce a visual encoder specialized for reading documents rather than objects and scenes. The reported gains across recognition, formula understanding, detection, tampering detection, overlapping text segmentation, parsing, and document understanding are used to support that claim [2607.11562].

A plausible implication is that MonkeyDoc v2 marks a shift in document AI from downstream adaptation of generic vision backbones toward corpus-scale pretraining explicitly optimized for document statistics. On the evidence presented, its significance lies less in any single benchmark result than in establishing a corpus-and-objective template for document-native foundation models [2607.11562].

Source: https://www.emergentmind.com/topics/monkeydoc-v2