Vision-Centric OCR

• Vision-centric OCR is a paradigm that converts complex visual images—such as charts, web pages, and diagrams—into rich, structured, and executable code.
• Dataset engineering involves comprehensive collection, rigorous cleaning, and self-annotation to create multimodal pairs that accurately capture both textual and visual semantics.
• Unified transformer-based architectures with a frozen visual encoder and autoregressive decoder, trained via SFT and RL, enable precise, code-oriented reconstruction of complex visuals.

Vision-centric Optical Character Recognition (OCR) is a paradigm that treats OCR not merely as transcribing character sequences from images, but as parsing visually information-dense images—including charts, web pages, diagrams, scientific plots, and other non-trivial composites—into rich, structured, and often executable representations (e.g., HTML, SVG, Python/Matplotlib, LaTeX/TikZ). By leveraging unified vision-language architectures, end-to-end learning, and code-oriented outputs, vision-centric OCR subsumes and extends text-centric approaches, enabling programmatic reconstruction of both textual and semantic visual structure. This approach is increasingly central to modern multimodal document understanding, data visualization workflows, web engineering, and scientific publishing (Zhong et al., 29 Jan 2026).

1. Vision-Centric OCR: Scope and Distinction

Vision-centric OCR targets visual scenes that encode information not just in text, but in layout, object structure, graphics, and interactive components. Unlike conventional (text-centric) OCR—whose sole output is a character or word stream—vision-centric OCR produces interpretable, executable code suitable for rendering or further programmatic manipulation (e.g., DOM trees for web pages, chart scripts for visualizations, geometric diagrams, SVGs, circuits, chemical molecules). Vision-centric OCR thus bridges the semantic gap between pixels and code-level representations, critical for domains where the visual arrangement carries domain-specific meaning (charts: axes, bars, colors; web: forms, nav bars; plots: geometric constructions, color-coding) (Zhong et al., 29 Jan 2026).

In classical OCR pipelines, recognition is disconnected from layout or graphical elements; vision-centric models handle text and graphics jointly, learning from the co-occurrence and spatial arrangement of elements.

2. Vision-Centric Dataset Engineering

Construction of vision-centric datasets involves multi-stage engineering tailored to domains where visual information dominates. The pipeline (as in OCRVerse (Zhong et al., 29 Jan 2026)) typically comprises:

1. Comprehensive Data Collection: Sourcing domain-specific corpora (e.g., charts from ChartMimic/MCD/MSRL, web layouts from Web2M/Web2Code, SVGs from UniSVG, geometry diagrams from DaTikZ-v3/Cosyn-400k, molecules from ChemDraw collections).
2. Cleaning and Filtering: Enforcing annotation consistency (e.g., valid HTML for webpages, syntactically correct TikZ/LaTeX, valency rules for molecules), eliminating corrupt/incomplete data.
3. Self-annotation/Bootstrapping: Training compact, domain-specific models to annotate unlabeled data (chart-to-code, image-to-HTML/SVG/TikZ), dramatically scaling dataset size while preserving quality.
4. Multimodal Pairing: For each image, annotating with executable code that, upon re-render, faithfully reconstructs the underlying visual semantics—enabling rigorous fidelity evaluation through SSIM, LPIPS, CLIPSim, and code-execution metrics.

A balanced dataset requires hundreds of thousands of samples per domain to avoid overfitting to trivial patterns and to cover layout and graphical diversity (Zhong et al., 29 Jan 2026).

3. Unified Model Architecture

Vision-centric OCR is anchored in transformer-based vision-LLMs with the following structure:

• Frozen Visual Encoder: Typically a ViT or Swin Transformer, mapping the image (or rasterized page/screenshot) into a dense grid of visual embeddings $$\mathbf{H} = f_{\mathrm{enc}}(x) \in \mathbb{R}^{N \times d}$$. The encoder is pretrained for general vision tasks and remains frozen or lightly adapted.
• Cross-modal Adapter: A learnable layer (e.g., cross-attention) that projects visual features into the key/value spaces of the transformer decoder. This adapter can mediate between high-resolution images and code-oriented outputs without explicit cropping or modality fusion.
• Autoregressive Decoder: A large transformer stack that generates a token sequence representing executable output (HTML, SVG, LaTeX, Python/Matplotlib, etc.). At each timestep, the decoder attends simultaneously to prior outputs and visual embeddings, optimizing for code correctness and visual fidelity.

Formally, decoding at time $t$ uses

$\mathrm{CrossAttn}(Q, K, V) = \softmax\left(\frac{Q K^T}{\sqrt{d}}\right) V,$

where $Q = W_Q h_t$, $K = W_K \mathbf{H}$, $V = W_V \mathbf{H}$, and output probability

$$P_\theta(y_t|x, y_{<t}) = \mathrm{softmax}(W_\mathrm{out} h_t).$$

All visual and adapter weights are typically kept frozen during fine-tuning, with only the decoder updated for output domain adaptation (Zhong et al., 29 Jan 2026).

4. Two-Stage SFT–RL Training Paradigm

Vision-centric OCR models employ a dual-phase training regimen for cross-domain adaptation:

1. Supervised Fine-Tuning (SFT): Mixed-batch training over all targeted domains (text-centric and vision-centric), minimizing next-token cross-entropy

$$\mathcal{L}_{\mathrm{SFT}}(\theta) = -\mathbb{E}_{(x, y)} \sum_{t=1}^T \log P_\theta(y_t | x, y_{<t}).$$

This stage ensures initial domain coverage and output-format adaptation.

1. Domain-Specific Reinforcement Learning (RL): Fine-tuning with custom reward functions reflecting desired output attributes:
   • For text-centric tasks: edit-distance, BLEU, Tree Edit Distance Similarity (TEDS-S) on tables.
   • For vision-centric tasks: visual fidelity rewards (DINOv2 cosine similarity, LPIPS), code execution success (charts, plots), format-alignment (HTML, SVG, LaTeX syntax).
   • Group Relative Policy Optimization (GRPO) is used: for each input, a batch of $G$ outputs is sampled, normalized advantages computed, and the clipped PPO-style objective minimized: $$\mathcal{L}_{\mathrm{RL}}(\theta) = \mathbb{E}\left[\frac{1}{G} \sum_{i=1}^G \min(\rho_i A_i, \mathrm{clip}(\rho_i, 1-\epsilon, 1+\epsilon)A_i)\right].$$ RL enables the model to learn output format flexibly and to align code outputs not just with textual content but with visual/structural makeup (Zhong et al., 29 Jan 2026).

5. Benchmarking and Evaluation Protocols

Vision-centric OCR models are assessed on standard and domain-specific benchmarks using metrics capturing both recognition and layout fidelity:

• ChartMimic: Chart-to-code accuracy (execution success rate, CodeBLEU, ElementIoU, GPT-4o-derived high-level scores).
• UniSVG: SSIM, LPIPS, CLIP similarity for SVG graphic reconstruction.
• Design2Code: Low-level HTML element-matching, high-level CLIP similarity for web page code generation.
• Image2Struct, ChemDraw: Plot and molecule code reconstructions, measured by rendering success and structure similarity.

OCRVerse (4B) achieves leading results—execution rate 84.8% (ChartMimic), composite SSIM-LPIPS 76.3 (UniSVG), web element-matching 85.7 (Design2Code)—matching or surpassing much larger models (Qwen3-VL-8B, InternVL3-8B, GPT-5) with substantially fewer parameters (Zhong et al., 29 Jan 2026).

6. Advantages, Limitations, and Prospective Directions

Advantages:

• Holistic Parsing: Unified handling of both text-centric and vision-centric (code-oriented) OCR, enabling pipeline-free programmatic reconstruction of diverse visual domains.
• Parameter Efficiency: End-to-end models attain state-of-the-art accuracy with $4\times$–$18\times$ fewer parameters versus frontier models.
• Cross-domain Fusion: Supervised fine-tuning and domain-specific RL enable simultaneous adaptation to heterogeneous visual outputs with minimal domain conflict.

Limitations:

• Limited Layout Awareness: Most models lack explicit region-level layout modules. This introduces fidelity gaps in reading-order-sensitive or ultra-dense page layouts.
• Domain Coverage: Current scope omits emerging types (floor plans, interactive flows, medical diagrams), and some error modes remain in complex multi-modal documents.
• Visual Hallucination: As in DeepSeek-OCR (Liang et al., 7 Jan 2026), high-compression vision token budgets can induce over-reliance on language priors, leading to hallucinated outputs when context length or corruption is high.

Future Work:

• Integrating hybrid layout attention, hierarchical decoding for structural trees, multi-modal feedback loops, richer domain coverage (UI flows, blueprints, map code).
• Developing evaluation protocols measuring prior-agnostic visual grounding and segmentation-aware structure parsing.

7. Synthesis of Vision-Centric OCR and Downstream Applications

Vision-centric OCR now supports critical real-world workloads:

• Automated chart, plot, and diagram reconstruction for scientific literature mining.
• Web page reverse engineering for UI testing and information extraction.
• Chemistry and geometry code conversion for structured data repositories.
• Document holotyping ("document holograms")—raster input → structured semantic output with provenance traceability, suitable for active-learning and human verification.

This increasingly vision-grounded, code-oriented paradigm signals the convergence of document intelligence, multimodal retrieval, and visual data science under unified, transformer-based architectures (Zhong et al., 29 Jan 2026). The ongoing shift from sequence-only transcription toward holistic, semantic rendering of information-dense images marks a critical inflection point in the evolution of OCR.