- The paper introduces a two-stage fine-tuning regime combining parameter-efficient LoRA and selective full-parameter SFT to adapt DeepSeek-OCR-2 for high-precision molecular recognition.
- The model employs a hybrid architecture—with a visual tokenizer, LM-as-vision-encoder, and autoregressive decoder—that preserves both global graph topology and bond-level details, outperforming zero-shot baselines.
- Despite robust performance on synthetic and real-world benchmarks, the approach lags behind image-to-graph models, underscoring the need for hybrid strategies to achieve precise SMILES serialization.
Fine-tuning DeepSeek-OCR-2 for Molecular Structure Recognition: Technical Overview and Implications
Introduction and Motivation
Optical Chemical Structure Recognition (OCSR) is pivotal for automating the extraction of chemical data from scientific literature, specifically by converting 2D molecular diagrams into machine-readable formats such as SMILES strings. While Vision-LLMs (VLMs) have delivered substantial gains in general OCR applications, their direct adaptation to high-precision molecular structure recognition has been problematic, especially when exploiting full-parameter supervised fine-tuning (SFT). This work presents a systematic adaptation of DeepSeek-OCR-2 for the domain of molecular optical recognition, introducing a progressive two-stage transfer protocol and benchmarking against established OCSR architectures.
Methodological Contributions
Dataset and Training Corpus
The proposed MolSeek-OCR leverages a mixed-domain training regime to enhance both domain coverage and visual robustness. The pipeline utilizes large-scale synthetic datasets (PubChem) rendered in diverse visual styles (MolScribe, ChemDraw) and a real-world corpus (USPTO-MOL) to expose the model to representative data distributions, including typical patent-specific drawing artifacts. The two-stage supervised fine-tuning process comprises:
- Stage 1: Parameter-efficient LoRA SFT: By deploying Low-Rank Adaptation (LoRA) modules on not only the last-stage decoder but also primary attention, feed-forward, and vision-language projection layers, the approach targets both the text generation and cross-modal alignment subspaces. This allows gradient flow along critical axes responsible for mapping visual structure to chemical serialization.
- Stage 2: Progressive Full-parameter SFT: Continuing from the LoRA-adapted weights, the protocol freezes the lowest-level visual tokenizer and input embedding, optimizing only higher-level visual and language branches with split learning rates. This selective adaptation reflects a hypothesis that lower-level representations stabilize quickly, while high-level cross-modal alignment and sequence modeling require domain-specific tuning for molecular OCR.
Model Architecture
DeepSeek-OCR-2 features a three-component backbone: a visual tokenizer, an LM-as-vision-encoder, and an autoregressive decoder, with a compression interface linking the visual and language streams. This architecture is inherently favorable for image-to-SMILES conversion, as it enables both global graph topology preservation and extraction of granular bond-level features. Token-level supervision is limited strictly to the textual SMILES output, which is essential for enforcing strict sequence-level fidelity.
MolSeek-OCR outperforms zero-shot VLM baselines and achieves exact matching accuracy metrics on par with leading image-to-sequence models such as DECIMER across synthetic (ChemDraw, Indigo) and real-world (USPTO, Staker, ACS) benchmarks, including perturbed test conditions. This establishes the effectiveness of the progressive adaptation route not only for in-domain data but also for visually heterogeneous and artifact-laden inputs.
However, MolSeek-OCR consistently lags behind state-of-the-art image-to-graph models, notably MolScribe, in exact matching accuracy. These results highlight the persistent limitations of autoregressive string generation relative to direct graph construction in achieving high-fidelity molecular reconstruction—especially critical in scenarios requiring unambiguous SMILES serialization.
Reinforcement-style Post-training and Data Curation
Attempts to utilize reinforcement-style fine-tuning strategies, including Group Sequence Policy Optimization (GSPO) and Representation Finetuning (ReFT), did not yield further improvements. Despite leveraging chemistry-aware reward structures designed to encourage chemical validity and stereochemical accuracy, these approaches often decreased sequence-level exact-match performance. This degradation is attributed to a divergence between graph-equivalence and the precise string-level outputs demanded by OCSR tasks—reinforcement, while enhancing structural validity, fails to enforce lexical fidelity requisite for SMILES.
Implications and Future Directions
The study reinforces the view that conservative, progressive adaptation of OCR VLMs via parameter-efficient and staged SFT can yield robust molecular recognition with minimal catastrophic forgetting and stable convergence properties. Nevertheless, explicit graph-centric modeling remains necessary for maximal OCSR performance due to the syntactic brittleness and ambiguity of SMILES representations.
For future development in AI-based chemical document understanding, several trajectories are apparent:
- Hybrid Vision-Language and Graph Generation: Bridging image-conditioned sequence models with graph-generative components (e.g., via multitask objectives or hybrid decoders) might align sequence- and graph-level fidelity.
- Canonicalization-aware Objectives: Directly incorporating chemical canonicalization constraints into model objectives could mitigate exact-match failures without sacrificing underlying graph fidelity.
- Scaling and Multi-modal Pretraining: Leveraging ever-larger vision-language and chemical corpora, along with further advancements in multi-modal VLM pretraining, promises incremental improvements in both coverage and domain generalization.
- Active Data Selection and Hard Example Mining: Focused curation strategies tailored to the failure modes of sequence decoders (e.g., complex stereochemistry, rare bond patterns) may act as crucial scaffolds for further progress.
Conclusion
The adaptation of DeepSeek-OCR-2 to molecular structure recognition, realized through a two-stage fine-tuning paradigm culminating in the MolSeek-OCR model, achieves competitive exact-match accuracy within the image-to-sequence paradigm. The research empirically underscores the current limitations of VLM-based OCSR relative to image-to-graph strategies and highlights the challenges of reinforcement-style sequence improvement for strict chemical serialization tasks. As these methods continue to evolve, substantial gains will likely be derived from innovations at the intersection of sequential and graph-based molecular representations, as well as from further scaling and domain-specialized pretraining.
