Multimodal OCR: Parse Anything from Documents

Abstract: We present Multimodal OCR (MOCR), a document parsing paradigm that jointly parses text and graphics into unified textual representations. Unlike conventional OCR systems that focus on text recognition and leave graphical regions as cropped pixels, our method, termed dots.mocr, treats visual elements such as charts, diagrams, tables, and icons as first-class parsing targets, enabling systems to parse documents while preserving semantic relationships across elements. It offers several advantages: (1) it reconstructs both text and graphics as structured outputs, enabling more faithful document reconstruction; (2) it supports end-to-end training over heterogeneous document elements, allowing models to exploit semantic relations between textual and visual components; and (3) it converts previously discarded graphics into reusable code-level supervision, unlocking multimodal supervision embedded in existing documents. To make this paradigm practical at scale, we build a comprehensive data engine from PDFs, rendered webpages, and native SVG assets, and train a compact 3B-parameter model through staged pretraining and supervised fine-tuning. We evaluate dots.mocr from two perspectives: document parsing and structured graphics parsing. On document parsing benchmarks, it ranks second only to Gemini 3 Pro on our OCR Arena Elo leaderboard, surpasses existing open-source document parsing systems, and sets a new state of the art of 83.9 on olmOCR Bench. On structured graphics parsing, dots.mocr achieves higher reconstruction quality than Gemini 3 Pro across image-to-SVG benchmarks, demonstrating strong performance on charts, UI layouts, scientific figures, and chemical diagrams. These results show a scalable path toward building large-scale image-to-code corpora for multimodal pretraining. Code and models are publicly available at https://github.com/rednote-hilab/dots.mocr.

• The paper introduces a unified OCR system that treats text and graphics as first-class outputs, generating structured code such as SVG.
• It leverages a high-resolution vision encoder and a structured language decoder to achieve state-of-the-art performance on diverse document parsing benchmarks.
• Key implications include scalable multimodal pretraining, enhanced document intelligence, and improved downstream reasoning over complex layouts.

Multimodal OCR: A Unified Paradigm for Parsing Text and Graphics in Documents

Introduction and Motivation

The paper "Multimodal OCR: Parse Anything from Documents" (2603.13032) presents Multimodal OCR (MOCR), an end-to-end document parsing paradigm that goes beyond conventional OCR. Traditional OCR approaches prioritize text extraction, relegating graphics such as charts, diagrams, tables, and other non-textual elements to coarse pixel crops, thus losing fine-grained semantic and geometric structure. MOCR instead treats textual and graphical elements as first-class parsing targets and produces structured, machine-interpretable code—primarily SVG—for all semantically meaningful components.

Figure 1: Overview of MOCR: parses all elements on a document page into unified, ordered textual representations for faithful full-page reconstruction.

This paradigm shift is motivated by the growing prevalence of vision-LLMs (VLMs) now capable of generating structured, executable visual representations. These advances offer the possibility of reconstructing not just the linguistic, but also the visual and logical content of real-world documents, and thus providing a scalable source for multimodal pretraining supervision.

Comparison with Traditional OCR Pipelines

Traditional OCR systems are predominantly text-centric pipelines. They perform layout analysis, text detection, and text recognition, followed by optional reading order and structure prediction. Non-textual graphical content is either cropped as raster images or simply ignored, resulting in a lossy pipeline for downstream machine reasoning over visual documents. This design fundamentally restricts the extractable information and structured supervision—both necessary for advanced document intelligence and multimodal pretraining.

MOCR, by contrast, unifies the parsing space: all parseable components, including text, tables, layout, mathematical formulas, schematic diagrams, UI screenshots, icons, and statistical charts, are transformed into unified, ordered serializations. Graphics are encoded as SVG, supporting render-and-reuse, semantic composability, and precise element recovery.

Figure 2: MOCR parses graphics into structured code (e.g., SVG) compared to text-only OCR that discards non-textual regions as pixels.

MOCR Model Architecture and Data Engine

The dots.mocr system embodies the MOCR paradigm via three primary components:

• High-resolution Vision Encoder: A 1.2B-parameter model ingesting up to ~11M pixel images, capturing long-range dependencies, dense layout, and fine-grained geometric features necessary for both text and geometry parsing.
• Multimodal Connector: Bridges the visual encoder and the structured autoregressive language decoder.
• Structured Language Decoder: Qwen2.5-1.5B, enabling autoregressive generation of long, highly-structured outputs (including SVG programs, table markup, LaTeX, etc.).

The data engine underpinning MOCR is constructed from a broad mix of sources: PDFs with dense layout and multilinguality, webpage renderings with aligned DOM/SVG labels, native SVG assets (icons, charts, UI components), and general-purpose vision-language corpora to enhance robustness and downstream applicability. Key pretraining operations include render-based normalization (to address program aliasing), sampling for layout and content diversity, and staged curriculum learning to stabilize multi-task generalization.

Task Definition and Learning Formulation

MOCR is formulated as a sequence generation problem, where each structured element is predicted with localization ($\mathcal{B}_k$), semantic type ($c_k$), and a type-specific serialization payload ($p_k$):

$$\mathbf{S} = \big[(\mathcal{B}_1, c_1, p_1), \ldots, (\mathcal{B}_K, c_K, p_K)\big]$$

Textual payloads include plain text, tables (as Markdown or custom markup), and formulas (as LaTeX). Visual elements are parsed as executable SVG code wherever possible; if programmatic description is infeasible, they may be preserved as raster. This serialization is autoregressively decoded to maximize global and local structure inference while maintaining machine interpretability.

Benchmarking and Strong Numerical Results

MOCR and its variant dots.mocr-svg are evaluated on comprehensive document parsing and structured graphic parsing benchmarks. Dots.mocr sets the state-of-the-art on olmOCR-Bench with a score of 83.9, outperforming all open-source methods and trailing only Gemini 3 Pro. On the newly proposed Elo-based document parsing leaderboards—which employ VLM-judged pairwise comparisons to overcome the brittleness of rule-based metrics—dots.mocr achieves the strongest rating among academic and open-source models.

Figure 3: Comparative performance across document/image parsing benchmarks (text and graphics). Dots.mocr-svg consistently surpasses open-source and many proprietary systems.

On the structured graphic parsing tasks (UniSVG, ChartMimic, Design2Code, SciGen, ChemDraw, etc.), dots.mocr-svg achieves higher ISVGEN reconstruction scores than Gemini 3 Pro on all reported datasets, with improvements especially pronounced for vector graphics, scientific diagrams, and chemical structure parsing. These results substantiate the assertion that MOCR outperforms even the strongest proprietary VLMs in multimodal parsing, particularly for structure-sensitive visual elements.

Qualitative Results and Generalization

Qualitative analysis demonstrates robust handling of heterogeneous documents, including multilingual pages, dense multi-column layouts, mathematical formulas, scanned historical documents, and handwritten content.

Figure 4: Document-level structural parsing illustrating fine-grained layout partitioning and content categorization across diverse formats.

The model generalizes to non-traditional OCR benchmarks, achieving effective reading order preservation and cross-domain visual recognition for web UI screenshots, scene text, icons, charts, and complex scientific illustrations.

Figure 5: Parsing on web screenshots and real-world images, displaying MOCR's ability to generalize layout and structure extraction.

SVG program output decoding for icons, statistical graphics, and cross-domain scientific illustrations is both semantically and geometrically faithful, facilitating downstream editing, composability, and supervision.

Figure 6: dots.mocr-svg generating concise SVG code for icons, preserving geometry, grouping, and style for direct vector reconstruction.

Figure 7: SVG-based recovery of statistical charts, including axes, legends, and encodings, as structured executable representations.

Figure 8: MOCR parses complex, domain-specific illustrations, generalizing beyond charts to arbitrary structured graphics.

Evaluation Protocol and Broader Implications

To address the limitations of strict rule-based evaluation—which penalizes surface-form variations in structured parsing—the OCR Arena framework leverages an LLM-as-a-judge protocol. Pairwise model outputs are symmetrically and impartially judged by a large VLM (Gemini 3 Flash), and Elo rating aggregation provides interpretable, statistically robust cross-model rankings.

This approach is necessary as document parsing evolves towards emitting programmatic, non-unique, and semantically equivalent outputs, which are not adequately captured by edit distances or string-matching metrics.

Practical and Theoretical Implications

MOCR marks a transition from text-centric NLP resource construction to holistic multimodal supervision. By converting existing document graphics into structured program supervision, this paradigm enables the extraction of millions of image-code (and image-code-text) pairs from raw document corpora without additional annotation. These paired resources can drive large-scale, controllable, and richly supervised pretraining for future VLMs, grounding linguistic and visual reasoning in executable structure.

The paradigm is extensible: while SVG is adopted as the canonical structured visual representation, MOCR is agnostic to programmatic target. Future instantiations could involve TikZ (for scientific illustration rendering), D3.js (for web visualizations), domain-specific CAD, or chemical notations. The capacity for parsing full webpages, complex layouts, and diverse graphical elements further broadens the pool of harvestable data for downstream fine-tuned multimodal models.

Conclusion

MOCR introduces a scalable, unified approach to document parsing, integrating both text and graphical modalities into structured, reusable outputs. By closing the gap between pixel-level graphics handling and semantic, program-level reconstruction, MOCR substantially expands the extractable supervision from large-scale document corpora. This has direct implications for multimodal foundation model pretraining, semantic document search, and downstream vision–language understanding tasks. As document representations and LMMs continue to converge, the paradigm established by MOCR provides a foundation for further developments in executable vision, fine-grained structure-grounded reasoning, and machine-native knowledge extraction.