---
title: 'DianJin-OCR-R1: Reasoning-Enhanced OCR'
url: https://www.emergentmind.com/topics/dianjin-ocr-r1
type: topic
---

# DianJin-OCR-R1: Reasoning-Enhanced OCR

Searching arXiv for the primary paper and closely related OCR pipeline papers.
Using arXiv search to verify the paper records and related work.
DianJin-OCR-R1 is a reasoning-enhanced OCR framework built on a large vision-language model (LVLM), specifically Qwen2.5-VL-7B-Instruct, for document parsing tasks such as seal recognition, table recognition, and formula recognition. Its defining characteristic is a reasoning-and-tool interleaved inference and training paradigm: the model first performs its own recognition, then incorporates outputs from specialized external OCR tools, reflects on discrepancies, re-checks the image, and finally emits a corrected structured result. The framework is designed to reduce hallucinations that are typical of generative LVLMs and to narrow or surpass the gap between general-purpose LVLMs and expert OCR systems on domain-specific benchmarks such as ReST and OmniDocBench [2508.13238].

## 1. Definition and Core Paradigm

DianJin-OCR-R1 is a post-training framework rather than a new base architecture. The underlying LVLM remains Qwen2.5-VL-7B-Instruct, while the contribution lies in training the model to generate an explicit multi-stage reasoning trace with tool integration [2508.13238].

The framework formalizes OCR as a structured reasoning process with four tagged components: `<think></think>`, `<tool></tool>`, `<rethink></rethink>`, and `<answer></answer>`. In the first stage, the model performs its own OCR-style recognition. In the second, it receives outputs from expert OCR tools. In the third, it reflects on disagreements between its own recognition and tool outputs while re-inspecting the image. In the fourth, it generates the final answer in the task-specific target format, such as seal text, table HTML, or formula LaTeX [2508.13238].

This design addresses two limitations emphasized in the underlying work. First, generative LVLMs can hallucinate words or phrases that are not present in the image. Second, general-purpose LVLMs are often weaker than expert OCR systems on fine-grained document parsing tasks. DianJin-OCR-R1 treats expert tools as grounding references during reasoning, using their outputs to mitigate hallucinations and improve fidelity to the visual input [2508.13238].

A plausible implication is that the framework recasts OCR from a single-pass sequence generation problem into a tool-grounded verification problem. That interpretation is consistent with broader evidence that OCR pipelines benefit substantially from explicit pre-processing, task decomposition, and robustness-oriented evaluation rather than relying on a single end-to-end recognition step alone [2511.04161].

## 2. Model Architecture and Reasoning-and-Tool Interleaving

The base model is Qwen2.5-VL-7B-Instruct. DianJin-OCR-R1 does not modify the encoder-decoder architecture; instead, it changes the training target and inference behavior through supervised fine-tuning and reinforcement fine-tuning on reasoning traces [2508.13238].

Operationally, the framework proceeds in two turns during data construction. Given an image $i$ and task instruction $q$, the LVLM first produces its own recognition result. External OCR tools are then invoked on the same image. Their outputs are serialized into `<tool>` blocks and fed back together with the original image and instruction. The model then produces a reasoning trace containing its initial recognition, the tool results, a reflective comparison, and a final corrected answer [2508.13238].

The output sequence can be summarized as a concatenation of four semantically distinct segments:
$$
y = y_{\text{think}} \; y_{\text{tool}} \; y_{\text{rethink}} \; y_{\text{answer}}.
$$
The supervised objective is to maximize the likelihood of the full reasoning chain and answer:
$$
\mathcal{L}_{\text{SFT}} = - \sum_{(i,q,r,o) \in R} \log P_\theta(r,o \mid i,q).
$$
Here, $r$ denotes the reasoning trace and $o$ the final answer [2508.13238].

The tool outputs are plain-text serializations of expert predictions. For seal recognition they are strings; for table recognition they are HTML fragments; for formula recognition they are LaTeX expressions. The model is trained to interpret these not as authoritative labels but as references to be compared against its own perception of the image [2508.13238].

This interleaving is distinct from merely prompting a VLM with auxiliary context. In ablation, simply providing tool outputs helps, but training the model specifically on reasoning traces with tools yields substantially larger gains, indicating that tool use is learned as a structured behavior rather than treated as static side information [2508.13238].

## 3. Expert OCR Tools and Task Scope

DianJin-OCR-R1 is evaluated on three OCR subdomains: seal recognition, table recognition, and formula recognition. Each task uses different expert tools, reflecting the claim that expert models are tailored for specific OCR tasks and are typically less prone to hallucination than large general-purpose LVLMs [2508.13238].

For seal recognition, the framework uses PP-StructureV3, specifically PP-OCRv4 server seal detection and recognition, together with Qwen-VL-OCR. For table recognition, it uses PP-StructureV3 components including PP-LCNet_x1.0_table_cls, SLANeXt_wired, and SLANeXt_wireless, and also MonkeyOCR-3B in the reasoning chain. For formula recognition, it uses PP-StructureV3 with PP-FormulaNet_plus-M, along with MonkeyOCR-3B; GOT and PP-FormulaNet_plus-S appear in tool-choice ablations [2508.13238].

The final answer format depends on task type. Seal recognition yields a title string. Table recognition yields HTML with structure-preserving attributes such as `colspan` and `rowspan`. Formula recognition yields LaTeX. This task-conditioned output formalism aligns DianJin-OCR-R1 with document parsing rather than plain text-line OCR [2508.13238].

The system’s design is consistent with a broader pattern in OCR research: specialized upstream or auxiliary modules can materially improve downstream recognition quality. Explicit orientation correction improves OCR substantially on English and Indic documents [2511.04161]; localization-free chunked recognition improves receipt-level OCR when standard instance-level OCR fails on whole pages [2212.05525]; and lightweight detection-plus-rectification can maximize end OCR quality even when geometric localization is only “good enough” rather than perfect [2509.06246]. DianJin-OCR-R1 extends this modular logic into reasoning itself, with expert OCR outputs functioning as intermediate references rather than merely pre-processing signals.

## 4. Training Procedure and Data Construction

The reasoning datasets are built from three source corpora. For seal recognition, the source is the ReST train set with 5,000 images, from which 1,024 filtered reasoning samples are retained. For table recognition, the sources are in-house datasets and TabRecSet, with 38.1K tables in total, and again 1,024 reasoning samples are retained. For formula recognition, the source is UniMER-1M, and 1,024 reasoning samples are retained [2508.13238].

Each reasoning sample has the form $(i_i, q_i, r_i, o_i)$, where $i_i$ is the image, $q_i$ the task instruction, $r_i$ the reasoning chain, and $o_i$ the final answer. The reasoning traces are generated using Qwen-VL-Max with carefully designed two-turn prompts and then filtered by task-specific correctness thresholds [2508.13238].

The filtering criteria are task specific. For seal recognition, the final output must exactly equal the ground truth. For table recognition, the output must exceed the model’s own initial recognition in TEDS and also exceed a threshold of $0.98$. For formula recognition, the final output must achieve $1\text{-NED} \le 0.015$ and outperform the initial recognition [2508.13238].

The training pipeline has two stages.

First, supervised fine-tuning is performed with the loss
$$
\mathcal{L}_{\text{SFT}} = - \mathbb{E}_{(i,q,r,o) \sim R} \log P_\theta(r,o \mid i,q).
$$
The implementation uses 1 node with 8 NVIDIA A100 GPUs, DeepSpeed Zero-3, learning rate $1.0 \times 10^{-5}$, sequence length 16K, bf16 precision, 2 epochs, and gradient accumulation of 4 steps [2508.13238].

Second, reinforcement fine-tuning is performed with Group Relative Policy Optimization. The model receives a format reward and an accuracy reward. The format reward is $1.0$ only when the output exactly uses `<think>`, `<tool>`, `<rethink>`, and `<answer>` with no extra content; otherwise it is $0.0$. The accuracy reward is task-specific: exact match for seals, TEDS for tables, and $\text{CDM}(o,a) + 0.5 \cdot \mathbf{1}[\text{CDM}(o,a)=1.0]$ for formulas [2508.13238].

This two-stage design places DianJin-OCR-R1 in the same general family as reasoning-first systems that combine supervised reasoning traces with RL-based optimization, but its reward design is specialized for OCR outputs rather than generic textual reasoning [2508.13238].

## 5. Empirical Performance on ReST and OmniDocBench

The framework is evaluated on three benchmarks: the ReST test set for seal recognition, the OmniDocBench table subset, and the OmniDocBench formula subset [2508.13238].

On ReST seal recognition, the baseline Qwen2.5-VL-7B-Instruct achieves an accuracy of $0.527$. Expert baselines include PP-StructureV3 at $0.600$, Qwen-VL-OCR at $0.472$, and InternVL3-8B at $0.475$. DianJin-OCR-R1 improves this substantially: the supervised fine-tuned version reaches $0.722$, and the reinforcement-fine-tuned version reaches $0.766$ [2508.13238].

On OmniDocBench tables, Qwen2.5-VL-7B-Instruct achieves TEDS $0.853$, STEDS $0.904$, and NED $0.107$. PP-StructureV3 achieves TEDS $0.818$, RapidTable $0.832$, MonkeyOCR-3B $0.845$, GOT $0.787$, and InternVL3-8B $0.812$. DianJin-OCR-R1 improves to TEDS $0.895$, STEDS $0.935$, and NED $0.080$ after supervised fine-tuning, and further to TEDS $0.901$, STEDS $0.939$, and NED $0.072$ after reinforcement fine-tuning [2508.13238].

On OmniDocBench formulas, Qwen2.5-VL-7B-Instruct achieves CDM $0.915$, ER $0.642$, and NED $0.243$. PP-StructureV3 achieves CDM $0.966$, ER $0.730$, and NED $0.198$; UniMERNet-B achieves CDM $0.932$; MonkeyOCR-3B achieves CDM $0.973$, ER $0.767$, and NED $0.196$. DianJin-OCR-R1 reaches CDM $0.977$, ER $0.770$, and NED $0.188$ after supervised fine-tuning, and CDM $0.976$, ER $0.787$, and NED $0.179$ after reinforcement fine-tuning [2508.13238].

These results support two bounded conclusions. First, reasoning-and-tool interleaving improves a general-purpose LVLM over its non-reasoning counterpart. Second, on the reported benchmarks, the resulting framework exceeds the standalone expert baselines included in the evaluation [2508.13238].

## 6. Hallucination Mitigation, Ablations, and Relation to OCR Pipeline Design

A central claim of DianJin-OCR-R1 is hallucination mitigation. The motivating example in the seal task shows Qwen2.5-VL hallucinating a plausible but absent phrase, whereas PP-Structure produces character-level mistakes without inventing large new content. The framework is intended to reduce the former failure mode by forcing comparison against expert OCR outputs during `<rethink>` [2508.13238].

The paper does not define a standalone hallucination-rate metric. Instead, hallucination reduction is inferred from improvements in task metrics such as accuracy, TEDS, STEDS, CDM, ER, and NED. This suggests that hallucination is treated as one component of overall OCR error rather than isolated as a separate evaluation dimension [2508.13238].

Ablation studies clarify which elements matter. Simply giving tool outputs to the baseline improves performance: seal accuracy rises from $0.527$ to $0.560$, table TEDS from $0.853$ to $0.865$, and formula CDM from $0.915$ to $0.958$. However, supervised fine-tuning with reasoning and tools raises these further to $0.722$, $0.895$, and $0.977$ respectively. Reinforcement fine-tuning with tools yields the best overall results, including seal accuracy $0.766$, table TEDS $0.901$, and formula CDM $0.976$ [2508.13238].

Tool-choice ablations show that stronger expert tools improve the final model more. In formula recognition, using GOT plus PP-FormulaNet_plus-S yields DianJin-OCR-R1 performance of CDM $0.969$, ER $0.722$, and NED $0.227$, whereas using MonkeyOCR-3B plus PP-FormulaNet_plus-M yields CDM $0.979$, ER $0.770$, and NED $0.189$ [2508.13238]. This suggests that the framework’s headroom depends materially on expert-tool quality.

From a systems perspective, DianJin-OCR-R1 fits into a broader OCR pipeline literature that emphasizes modularity. Orientation handling can restore OCR performance nearly to upright baselines and produce large relative improvements under rotation [2511.04161]. Chunk-based localization-free recognition can improve full-page receipt OCR without explicit detectors [2212.05525]. Lightweight rectification can optimize end OCR quality under real-time constraints [2509.06246]. Multimodal post-correction can push historical-document transcription below $1\%$ CER without fine-tuning or image pre-processing [2504.00414]. DianJin-OCR-R1 can be understood as transferring that modular philosophy into the reasoning stage: tool outputs become part of the model’s internal evidence chain rather than merely upstream preprocessing artifacts.

A plausible implication is that the framework may benefit further from upstream robustness modules. Since OCR-oriented VLMs remain vulnerable to perturbations, especially on structure-sensitive tasks such as tables and charts, explicit handling of orientation, rectification, and visual corruption could improve the quality of both the model’s own initial recognition and the expert tools it consults [2606.26041].

## 7. Limitations, Positioning, and Future Directions

DianJin-OCR-R1 is evaluated only on three task families: seal titles, tables, and formulas. The reasoning datasets are also relatively small, with 1,024 retained reasoning samples per task. Tool integration is implemented through offline pre-called outputs inserted into prompts rather than dynamic learned tool selection during inference [2508.13238].

The model still has nontrivial residual error. Even after reinforcement fine-tuning, seal accuracy is $0.766$, table TEDS is $0.901$, and formula ER is $0.787$, indicating remaining failures on complex seals, table structures, and mathematical expressions [2508.13238]. The paper also notes that if expert tools fail systematically, the LVLM may be misled rather than corrected.

In the broader landscape of OCR research, DianJin-OCR-R1 occupies a specific position. It is neither a pure OCR engine in the classical sense nor a generic tool-augmented multimodal assistant. Instead, it is an OCR-specialized reasoning framework in which expert tools serve as structured references within a supervised and RL-optimized reasoning trace [2508.13238].

Several future directions follow directly from the reported design. One is extending the framework beyond seal, table, and formula recognition to richer document parsing tasks. Another is replacing static tool injection with dynamic tool selection and learned orchestration. A third is enlarging the tool ecosystem to include layout analysis, key-value extraction, chart parsing, or multilingual OCR components [2508.13238].

A plausible implication is that future versions could integrate robustness-aware components more explicitly. OCR robustness studies show that high clean accuracy does not guarantee strong retention under perturbation, and that charts and tables are especially fragile under blur, elastic deformation, color shift, and occlusion [2606.26041]. Since DianJin-OCR-R1 already excels on structured outputs such as HTML tables and LaTeX formulas, robustness-oriented evaluation under such perturbations would be a natural extension of its current benchmark profile.

Source: https://www.emergentmind.com/topics/dianjin-ocr-r1