DianJin-OCR-R1: Reasoning-Enhanced OCR

• DianJin-OCR-R1 is a reasoning-enhanced OCR framework built on Qwen2.5-VL-7B-Instruct that interleaves model predictions with expert tool outputs for improved document parsing.
• It employs a structured four-stage process (<think>, <tool>, <rethink>, <answer>) to mitigate hallucinations and enhance reliability in seal, table, and formula recognition.
• Empirical results show significant gains on benchmarks like ReST and OmniDocBench through combined supervised and reinforcement fine-tuning strategies.

Searching arXiv for the primary paper and closely related OCR pipeline papers. Using arXiv search to verify the paper records and related work. DianJin-OCR-R1 is a reasoning-enhanced OCR framework built on a large vision-LLM (LVLM), specifically Qwen2.5-VL-7B-Instruct, for document parsing tasks such as seal recognition, table recognition, and formula recognition. Its defining characteristic is a reasoning-and-tool interleaved inference and training paradigm: the model first performs its own recognition, then incorporates outputs from specialized external OCR tools, reflects on discrepancies, re-checks the image, and finally emits a corrected structured result. The framework is designed to reduce hallucinations that are typical of generative LVLMs and to narrow or surpass the gap between general-purpose LVLMs and expert OCR systems on domain-specific benchmarks such as ReST and OmniDocBench (Chen et al., 18 Aug 2025).

1. Definition and Core Paradigm

DianJin-OCR-R1 is a post-training framework rather than a new base architecture. The underlying LVLM remains Qwen2.5-VL-7B-Instruct, while the contribution lies in training the model to generate an explicit multi-stage reasoning trace with tool integration (Chen et al., 18 Aug 2025).

The framework formalizes OCR as a structured reasoning process with four tagged components: `,<tool></tool>,<rethink></rethink>, and<answer></answer>`. In the first stage, the model performs its own OCR-style recognition. In the second, it receives outputs from expert OCR tools. In the third, it reflects on disagreements between its own recognition and tool outputs while re-inspecting the image. In the fourth, it generates the final answer in the task-specific target format, such as seal text, table HTML, or formula LaTeX (Chen et al., 18 Aug 2025).

This design addresses two limitations emphasized in the underlying work. First, generative LVLMs can hallucinate words or phrases that are not present in the image. Second, general-purpose LVLMs are often weaker than expert OCR systems on fine-grained document parsing tasks. DianJin-OCR-R1 treats expert tools as grounding references during reasoning, using their outputs to mitigate hallucinations and improve fidelity to the visual input (Chen et al., 18 Aug 2025).

A plausible implication is that the framework recasts OCR from a single-pass sequence generation problem into a tool-grounded verification problem. That interpretation is consistent with broader evidence that OCR pipelines benefit substantially from explicit pre-processing, task decomposition, and robustness-oriented evaluation rather than relying on a single end-to-end recognition step alone (Goswami et al., 6 Nov 2025).

2. Model Architecture and Reasoning-and-Tool Interleaving

The base model is Qwen2.5-VL-7B-Instruct. DianJin-OCR-R1 does not modify the encoder-decoder architecture; instead, it changes the training target and inference behavior through supervised fine-tuning and reinforcement fine-tuning on reasoning traces (Chen et al., 18 Aug 2025).

Operationally, the framework proceeds in two turns during data construction. Given an image $i$ and task instruction $q$, the LVLM first produces its own recognition result. External OCR tools are then invoked on the same image. Their outputs are serialized into <tool> blocks and fed back together with the original image and instruction. The model then produces a reasoning trace containing its initial recognition, the tool results, a reflective comparison, and a final corrected answer (Chen et al., 18 Aug 2025).

The output sequence can be summarized as a concatenation of four semantically distinct segments:

$$y = y_{\text{think}} \; y_{\text{tool}} \; y_{\text{rethink}} \; y_{\text{answer}}.$$

The supervised objective is to maximize the likelihood of the full reasoning chain and answer:

$$\mathcal{L}_{\text{SFT}} = - \sum_{(i,q,r,o) \in R} \log P_\theta(r,o \mid i,q).$$

Here, $r$ denotes the reasoning trace and $o$ the final answer (Chen et al., 18 Aug 2025).

The tool outputs are plain-text serializations of expert predictions. For seal recognition they are strings; for table recognition they are HTML fragments; for formula recognition they are LaTeX expressions. The model is trained to interpret these not as authoritative labels but as references to be compared against its own perception of the image (Chen et al., 18 Aug 2025).

This interleaving is distinct from merely prompting a VLM with auxiliary context. In ablation, simply providing tool outputs helps, but training the model specifically on reasoning traces with tools yields substantially larger gains, indicating that tool use is learned as a structured behavior rather than treated as static side information (Chen et al., 18 Aug 2025).

3. Expert OCR Tools and Task Scope

DianJin-OCR-R1 is evaluated on three OCR subdomains: seal recognition, table recognition, and formula recognition. Each task uses different expert tools, reflecting the claim that expert models are tailored for specific OCR tasks and are typically less prone to hallucination than large general-purpose LVLMs (Chen et al., 18 Aug 2025).

For seal recognition, the framework uses PP-StructureV3, specifically PP-OCRv4 server seal detection and recognition, together with Qwen-VL-OCR. For table recognition, it uses PP-StructureV3 components including PP-LCNet_x1.0_table_cls, SLANeXt_wired, and SLANeXt_wireless, and also MonkeyOCR-3B in the reasoning chain. For formula recognition, it uses PP-StructureV3 with PP-FormulaNet_plus-M, along with MonkeyOCR-3B; GOT and PP-FormulaNet_plus-S appear in tool-choice ablations (Chen et al., 18 Aug 2025).

The final answer format depends on task type. Seal recognition yields a title string. Table recognition yields HTML with structure-preserving attributes such as colspan and rowspan. Formula recognition yields LaTeX. This task-conditioned output formalism aligns DianJin-OCR-R1 with document parsing rather than plain text-line OCR (Chen et al., 18 Aug 2025).

The system’s design is consistent with a broader pattern in OCR research: specialized upstream or auxiliary modules can materially improve downstream recognition quality. Explicit orientation correction improves OCR substantially on English and Indic documents (Goswami et al., 6 Nov 2025); localization-free chunked recognition improves receipt-level OCR when standard instance-level OCR fails on whole pages (Zhang et al., 2022); and lightweight detection-plus-rectification can maximize end OCR quality even when geometric localization is only “good enough” rather than perfect (Wojcik et al., 7 Sep 2025). DianJin-OCR-R1 extends this modular logic into reasoning itself, with expert OCR outputs functioning as intermediate references rather than merely pre-processing signals.

4. Training Procedure and Data Construction

The reasoning datasets are built from three source corpora. For seal recognition, the source is the ReST train set with 5,000 images, from which 1,024 filtered reasoning samples are retained. For table recognition, the sources are in-house datasets and TabRecSet, with 38.1K tables in total, and again 1,024 reasoning samples are retained. For formula recognition, the source is UniMER-1M, and 1,024 reasoning samples are retained (Chen et al., 18 Aug 2025).

Each reasoning sample has the form $(i_i, q_i, r_i, o_i)$, where $i_i$ is the image, $q_i$ the task instruction, $r_i$ the reasoning chain, and $q$0 the final answer. The reasoning traces are generated using Qwen-VL-Max with carefully designed two-turn prompts and then filtered by task-specific correctness thresholds (Chen et al., 18 Aug 2025).

The filtering criteria are task specific. For seal recognition, the final output must exactly equal the ground truth. For table recognition, the output must exceed the model’s own initial recognition in TEDS and also exceed a threshold of $q$1. For formula recognition, the final output must achieve $q$2 and outperform the initial recognition (Chen et al., 18 Aug 2025).

The training pipeline has two stages.

First, supervised fine-tuning is performed with the loss

$q$3

The implementation uses 1 node with 8 NVIDIA A100 GPUs, DeepSpeed Zero-3, learning rate $q$4, sequence length 16K, bf16 precision, 2 epochs, and gradient accumulation of 4 steps (Chen et al., 18 Aug 2025).

Second, reinforcement fine-tuning is performed with Group Relative Policy Optimization. The model receives a format reward and an accuracy reward. The format reward is $q$5 only when the output exactly uses <think>, <tool>, <rethink>, and <answer> with no extra content; otherwise it is $q$6. The accuracy reward is task-specific: exact match for seals, TEDS for tables, and $q$7 for formulas (Chen et al., 18 Aug 2025).

This two-stage design places DianJin-OCR-R1 in the same general family as reasoning-first systems that combine supervised reasoning traces with RL-based optimization, but its reward design is specialized for OCR outputs rather than generic textual reasoning (Chen et al., 18 Aug 2025).

5. Empirical Performance on ReST and OmniDocBench

The framework is evaluated on three benchmarks: the ReST test set for seal recognition, the OmniDocBench table subset, and the OmniDocBench formula subset (Chen et al., 18 Aug 2025).

On ReST seal recognition, the baseline Qwen2.5-VL-7B-Instruct achieves an accuracy of $q$8. Expert baselines include PP-StructureV3 at $q$9, Qwen-VL-OCR at $$y = y_{\text{think}} \; y_{\text{tool}} \; y_{\text{rethink}} \; y_{\text{answer}}.$$0, and InternVL3-8B at $$y = y_{\text{think}} \; y_{\text{tool}} \; y_{\text{rethink}} \; y_{\text{answer}}.$$1. DianJin-OCR-R1 improves this substantially: the supervised fine-tuned version reaches $$y = y_{\text{think}} \; y_{\text{tool}} \; y_{\text{rethink}} \; y_{\text{answer}}.$$2, and the reinforcement-fine-tuned version reaches $$y = y_{\text{think}} \; y_{\text{tool}} \; y_{\text{rethink}} \; y_{\text{answer}}.$$3 (Chen et al., 18 Aug 2025).

On OmniDocBench tables, Qwen2.5-VL-7B-Instruct achieves TEDS $$y = y_{\text{think}} \; y_{\text{tool}} \; y_{\text{rethink}} \; y_{\text{answer}}.$$4, STEDS $$y = y_{\text{think}} \; y_{\text{tool}} \; y_{\text{rethink}} \; y_{\text{answer}}.$$5, and NED $$y = y_{\text{think}} \; y_{\text{tool}} \; y_{\text{rethink}} \; y_{\text{answer}}.$$6. PP-StructureV3 achieves TEDS $$y = y_{\text{think}} \; y_{\text{tool}} \; y_{\text{rethink}} \; y_{\text{answer}}.$$7, RapidTable $$y = y_{\text{think}} \; y_{\text{tool}} \; y_{\text{rethink}} \; y_{\text{answer}}.$$8, MonkeyOCR-3B $$y = y_{\text{think}} \; y_{\text{tool}} \; y_{\text{rethink}} \; y_{\text{answer}}.$$9, GOT $$\mathcal{L}_{\text{SFT}} = - \sum_{(i,q,r,o) \in R} \log P_\theta(r,o \mid i,q).$$0, and InternVL3-8B $$\mathcal{L}_{\text{SFT}} = - \sum_{(i,q,r,o) \in R} \log P_\theta(r,o \mid i,q).$$1. DianJin-OCR-R1 improves to TEDS $$\mathcal{L}_{\text{SFT}} = - \sum_{(i,q,r,o) \in R} \log P_\theta(r,o \mid i,q).$$2, STEDS $$\mathcal{L}_{\text{SFT}} = - \sum_{(i,q,r,o) \in R} \log P_\theta(r,o \mid i,q).$$3, and NED $$\mathcal{L}_{\text{SFT}} = - \sum_{(i,q,r,o) \in R} \log P_\theta(r,o \mid i,q).$$4 after supervised fine-tuning, and further to TEDS $$\mathcal{L}_{\text{SFT}} = - \sum_{(i,q,r,o) \in R} \log P_\theta(r,o \mid i,q).$$5, STEDS $$\mathcal{L}_{\text{SFT}} = - \sum_{(i,q,r,o) \in R} \log P_\theta(r,o \mid i,q).$$6, and NED $$\mathcal{L}_{\text{SFT}} = - \sum_{(i,q,r,o) \in R} \log P_\theta(r,o \mid i,q).$$7 after reinforcement fine-tuning (Chen et al., 18 Aug 2025).

On OmniDocBench formulas, Qwen2.5-VL-7B-Instruct achieves CDM $$\mathcal{L}_{\text{SFT}} = - \sum_{(i,q,r,o) \in R} \log P_\theta(r,o \mid i,q).$$8, ER $$\mathcal{L}_{\text{SFT}} = - \sum_{(i,q,r,o) \in R} \log P_\theta(r,o \mid i,q).$$9, and NED $r$0. PP-StructureV3 achieves CDM $r$1, ER $r$2, and NED $r$3; UniMERNet-B achieves CDM $r$4; MonkeyOCR-3B achieves CDM $r$5, ER $r$6, and NED $r$7. DianJin-OCR-R1 reaches CDM $r$8, ER $r$9, and NED $o$0 after supervised fine-tuning, and CDM $o$1, ER $o$2, and NED $o$3 after reinforcement fine-tuning (Chen et al., 18 Aug 2025).

These results support two bounded conclusions. First, reasoning-and-tool interleaving improves a general-purpose LVLM over its non-reasoning counterpart. Second, on the reported benchmarks, the resulting framework exceeds the standalone expert baselines included in the evaluation (Chen et al., 18 Aug 2025).

6. Hallucination Mitigation, Ablations, and Relation to OCR Pipeline Design

A central claim of DianJin-OCR-R1 is hallucination mitigation. The motivating example in the seal task shows Qwen2.5-VL hallucinating a plausible but absent phrase, whereas PP-Structure produces character-level mistakes without inventing large new content. The framework is intended to reduce the former failure mode by forcing comparison against expert OCR outputs during <rethink> (Chen et al., 18 Aug 2025).

The paper does not define a standalone hallucination-rate metric. Instead, hallucination reduction is inferred from improvements in task metrics such as accuracy, TEDS, STEDS, CDM, ER, and NED. This suggests that hallucination is treated as one component of overall OCR error rather than isolated as a separate evaluation dimension (Chen et al., 18 Aug 2025).

Ablation studies clarify which elements matter. Simply giving tool outputs to the baseline improves performance: seal accuracy rises from $o$4 to $o$5, table TEDS from $o$6 to $o$7, and formula CDM from $o$8 to $o$9. However, supervised fine-tuning with reasoning and tools raises these further to $(i_i, q_i, r_i, o_i)$0, $(i_i, q_i, r_i, o_i)$1, and $(i_i, q_i, r_i, o_i)$2 respectively. Reinforcement fine-tuning with tools yields the best overall results, including seal accuracy $(i_i, q_i, r_i, o_i)$3, table TEDS $(i_i, q_i, r_i, o_i)$4, and formula CDM $(i_i, q_i, r_i, o_i)$5 (Chen et al., 18 Aug 2025).

Tool-choice ablations show that stronger expert tools improve the final model more. In formula recognition, using GOT plus PP-FormulaNet_plus-S yields DianJin-OCR-R1 performance of CDM $(i_i, q_i, r_i, o_i)$6, ER $(i_i, q_i, r_i, o_i)$7, and NED $(i_i, q_i, r_i, o_i)$8, whereas using MonkeyOCR-3B plus PP-FormulaNet_plus-M yields CDM $(i_i, q_i, r_i, o_i)$9, ER $i_i$0, and NED $i_i$1 (Chen et al., 18 Aug 2025). This suggests that the framework’s headroom depends materially on expert-tool quality.

From a systems perspective, DianJin-OCR-R1 fits into a broader OCR pipeline literature that emphasizes modularity. Orientation handling can restore OCR performance nearly to upright baselines and produce large relative improvements under rotation (Goswami et al., 6 Nov 2025). Chunk-based localization-free recognition can improve full-page receipt OCR without explicit detectors (Zhang et al., 2022). Lightweight rectification can optimize end OCR quality under real-time constraints (Wojcik et al., 7 Sep 2025). Multimodal post-correction can push historical-document transcription below $i_i$2 CER without fine-tuning or image pre-processing (Greif et al., 1 Apr 2025). DianJin-OCR-R1 can be understood as transferring that modular philosophy into the reasoning stage: tool outputs become part of the model’s internal evidence chain rather than merely upstream preprocessing artifacts.

A plausible implication is that the framework may benefit further from upstream robustness modules. Since OCR-oriented VLMs remain vulnerable to perturbations, especially on structure-sensitive tasks such as tables and charts, explicit handling of orientation, rectification, and visual corruption could improve the quality of both the model’s own initial recognition and the expert tools it consults (Cheng et al., 24 Jun 2026).

7. Limitations, Positioning, and Future Directions

DianJin-OCR-R1 is evaluated only on three task families: seal titles, tables, and formulas. The reasoning datasets are also relatively small, with 1,024 retained reasoning samples per task. Tool integration is implemented through offline pre-called outputs inserted into prompts rather than dynamic learned tool selection during inference (Chen et al., 18 Aug 2025).

The model still has nontrivial residual error. Even after reinforcement fine-tuning, seal accuracy is $i_i$3, table TEDS is $i_i$4, and formula ER is $i_i$5, indicating remaining failures on complex seals, table structures, and mathematical expressions (Chen et al., 18 Aug 2025). The paper also notes that if expert tools fail systematically, the LVLM may be misled rather than corrected.

In the broader landscape of OCR research, DianJin-OCR-R1 occupies a specific position. It is neither a pure OCR engine in the classical sense nor a generic tool-augmented multimodal assistant. Instead, it is an OCR-specialized reasoning framework in which expert tools serve as structured references within a supervised and RL-optimized reasoning trace (Chen et al., 18 Aug 2025).

Several future directions follow directly from the reported design. One is extending the framework beyond seal, table, and formula recognition to richer document parsing tasks. Another is replacing static tool injection with dynamic tool selection and learned orchestration. A third is enlarging the tool ecosystem to include layout analysis, key-value extraction, chart parsing, or multilingual OCR components (Chen et al., 18 Aug 2025).

A plausible implication is that future versions could integrate robustness-aware components more explicitly. OCR robustness studies show that high clean accuracy does not guarantee strong retention under perturbation, and that charts and tables are especially fragile under blur, elastic deformation, color shift, and occlusion (Cheng et al., 24 Jun 2026). Since DianJin-OCR-R1 already excels on structured outputs such as HTML tables and LaTeX formulas, robustness-oriented evaluation under such perturbations would be a natural extension of its current benchmark profile.