dots.mocr: Multimodal OCR Framework

• dots.mocr is a document parsing framework that generalizes traditional OCR by treating text and graphical elements as equal parsing targets.
• It employs an encoder-decoder transformer with a high-resolution vision encoder and autoregressive LLM to output serializable representations like SVG.
• The framework achieves state-of-the-art performance on document and SVG reconstruction benchmarks, enabling robust downstream multimodal reasoning.

dots.mocr is a document parsing framework, introduced in "Multimodal OCR: Parse Anything from Documents," which generalizes traditional Optical Character Recognition (OCR) to treat textual and graphical content (such as charts, tables, diagrams, and icons) as equally important parsing targets. The dots.mocr paradigm enables end-to-end extraction of both text and structured graphics from documents, representing both as serializable, code-level outputs (e.g., SVG for graphics), thus supporting faithful document reconstruction and downstream multimodal reasoning tasks (Zheng et al., 13 Mar 2026).

1. Conceptual Foundations of Multimodal OCR

Conventional OCR focuses primarily on text detection and recognition, often relegating graphical or non-textual elements to image crops or ignoring them entirely. The dots.mocr approach extends OCR to "Multimodal OCR" (MOCR), where all visually salient components—textual, tabular, and graphical—are parsed into uniform structured representations. In this paradigm, graphics are not post-hoc addenda but first-class parsing targets: charts, diagrams, icons, scientific figures, and tables are reconstructed as programmatic code (e.g., SVG), preserving original semantics and enabling reusability, editability, and richer downstream supervision (Zheng et al., 13 Mar 2026).

The canonical output representation is an ordered sequence of triplets $(\mathcal B_k, c_k, p_k)$, where $\mathcal B_k$ is the bounding box, $c_k$ is the semantic category (e.g., 'Paragraph', 'Table', 'Chart', 'SVG'), and $p_k$ is the content payload (plain text, LaTeX, CSV, or SVG code).

2. Model Architecture and Output Representation

dots.mocr employs an encoder–decoder transformer architecture comprising three major modules:

• High-Resolution Vision Encoder: A 1.2B-parameter CNN–Transformer, ingesting full-page raster images (up to 11M pixels), extracting a dense grid of visual tokens $\mathbf V \in \mathbb{R}^{N \times d}$ that retain fine typography and low-level chart primitives.
• Multimodal Connector: Cross-attention layers bridging the visual tokens and the LLM, projecting visual context into the LLM space via multi-head attention $$\mathrm{Attention}(Q,K,V) = \operatorname{softmax}(QK^\top/\sqrt{d})V$$.
• Autoregressive LLM Decoder: A version of Qwen2.5-1.5B conditioned on visual context, responsible for sequentially generating blocks of text or code, with bounding boxes and semantic types serialized in order, encapsulating both page layout and reading sequence.

Each output is serialized as a block, for example:

<block bbox=(x0,y0,x1,y1) type=Chart> <svg>...</svg> </block>

This unified, stream-based output allows reconstruction into downstream machine-readable formats (e.g., JSON) and preserves all document elements for rendering, querying, and knowledge grounding (Zheng et al., 13 Mar 2026).

3. Training Pipeline and Data Engine

dots.mocr is trained via a staged pretraining process founded on a large-scale heterogeneous data engine:

• Vision-Language Interface Pretraining: Initial alignment using synthetic OCR and generic vision tasks (image classification, object detection) to stabilize cross-attention.
• Mixed Page-Parsing Pretraining: Incorporates tens of millions of PDF pages (auto-labeled by dots.ocr (Li et al., 2 Dec 2025)), hundreds of millions of rendered web pages with DOM-aligned markup, and general vision datasets. Text parsing is emphasized via scheduled mix ratios.
• MOCR-Specific Pretraining: Focuses on document-level OCR signals and image→SVG pairs, with ∼5M curated SVG examples. Curriculum increases resolution (1–10 MP) and output sequence length (up to $T \approx 4000$ tokens).
• Instruction Tuning: Supervised fine-tuning on ∼1M high-quality render-consistent examples, applying SVG canonicalization (viewBox normalization, numeric precision reduction).

Data Sources:

• PDF documents stratified by language and layout complexity, auto-labeled by dots.ocr (∼30M).
• Rendered HTML/web page–DOM pairs (∼100M).
• Native and deduplicated SVG assets (∼5M).
• Public vision and OCR datasets (ImageNet, COCO, scene-text datasets) (Zheng et al., 13 Mar 2026).

Losses are standard autoregressive cross-entropy over tokens; when geometry is discretized, a Smooth-$L_1$ regression loss may be added.

4. Empirical Evaluation and Comparative Performance

Document Parsing:

On the olmOCR-Bench, dots.mocr achieves an overall score of 83.9 ± 0.9, establishing a new state-of-the-art among open models. On the OCR Arena Elo leaderboard, dots.mocr ranks second to Gemini 3 Pro, outperforming all prior open-source document parsers and dots.ocr:

Model	olmOCR-Bench	OmniDocBench1.5	XDocParse	Average
Gemini 3 Pro	1180.4	1128.0	1323.7	1210.7
dots.mocr	1104.4	1059.0	1210.7	1124.7
dots.ocr	1041.1	1027.2	1190.3	1086.2

Structured Graphics:

On image→SVG benchmarks, dots.mocr (and the SVG-optimized variant dots.mocr-svg) achieves superior ISVGEN reconstruction scores, outperforming Gemini 3 Pro on most datasets:

Method	UniSVG〈Low〉	UniSVG〈High〉	UniSVG〈Overall〉	ChartMimic	Design2Code	GenExam	SciGen	ChemDraw
dots.mocr-svg	0.860	0.931	0.902	0.905	0.834	0.800	0.797	0.901
Gemini 3 Pro	0.563	0.850	0.735	0.788	0.760	0.756	0.783	0.839

General Visual QA:

dots.mocr is competitive with larger VLMs on tasks such as CharXiv descriptive reasoning, InfoVQA, DocVQA, and ChartQA (Zheng et al., 13 Mar 2026).

5. Advantages, Failure Modes, and Limitations

Advantages

• By promoting graphics to first-class targets, dots.mocr enables end-to-end reconstruction of all document elements—including charts, icons, diagrams—as programmatic code rather than as intractable image crops.
• Preserves semantic hierarchies, supports downstream workflows like search, QA, and translation, and unlocks image↔SVG corpora valuable for large-scale multimodal pretraining.
• Provides unified handling of text and graphics in a single 3B-parameter model, simplifying deployment and inference (Zheng et al., 13 Mar 2026).

Limitations and Observed Failure Cases

• Ground-truth SVG supervision is rare, introducing sparse or ambiguous graphics annotation; this can result in hallucinated or mislocalized vector code, especially for complex elements.
• Geometric precision is imperfect for fine structures (e.g., chemical bonds), occasionally yielding slight misalignment.
• Separate inference passes are currently required for page-level text and region-level SVG parsing; fusion into a truly simultaneous parser remains an open direction.

Extensions and Future Directions

Proposed extensions include targeting additional programmatic formats (TikZ, D3.js, CAD, SMILES), integrating reward-based fine-tuning with render-based metrics, and advancing toward unified decoding of text and SVG in a single inference pass (Zheng et al., 13 Mar 2026).

6. Relationship to Related Paradigms

While traditional document OCR architectures focus exclusively on text (e.g., two-stage text box detection and recognition), and leading pipelines such as dots.ocr achieve strong multilingual and relational parsing (Li et al., 2 Dec 2025), dots.mocr generalizes document understanding to all page modalities. It leverages thoroughly automated and multilingual labeled data engines built upon the dots.ocr framework for training, but diverges by enforcing multimodal serialization and structured output encompassing both text and vector graphics.

Comparative studies show that when in-the-wild text detection models are adapted to standard document OCR, they outperform legacy OCR engines, but these approaches still restrict outputs to textual content (Olejniczak et al., 2022, Zheng et al., 13 Mar 2026). The MOCR paradigm, by integrating programmatic reconstruction of graphical elements, bridges the gap between document parsing and high-fidelity, modality-agnostic document understanding.

7. Applications and Impact

dots.mocr facilitates:

• Construction of massive image↔SVG corpora for programmatic VLM and MOCR pretraining.
• Document understanding tasks, yielding machine-editable JSON+SVG representations from scanned documents and web content.
• Interactive AI assistants capable of reasoning about chart semantics, reconstructing web and UI layouts, and supporting automated web design and scientific content digitization (Zheng et al., 13 Mar 2026).

A plausible implication is broader adoption of standardized output formats (JSON plus SVG) for document intelligence pipelines and increased emphasis on end-to-end multimodal pretraining. The model code and pretrained weights for dots.mocr are available at https://github.com/rednote-hilab/dots.mocr.