Decoder Pre-Training with only Text for Scene Text Recognition (2408.05706v1)

Abstract: Scene text recognition (STR) pre-training methods have achieved remarkable progress, primarily relying on synthetic datasets. However, the domain gap between synthetic and real images poses a challenge in acquiring feature representations that align well with images on real scenes, thereby limiting the performance of these methods. We note that vision-LLMs like CLIP, pre-trained on extensive real image-text pairs, effectively align images and text in a unified embedding space, suggesting the potential to derive the representations of real images from text alone. Building upon this premise, we introduce a novel method named Decoder Pre-training with only text for STR (DPTR). DPTR treats text embeddings produced by the CLIP text encoder as pseudo visual embeddings and uses them to pre-train the decoder. An Offline Randomized Perturbation (ORP) strategy is introduced. It enriches the diversity of text embeddings by incorporating natural image embeddings extracted from the CLIP image encoder, effectively directing the decoder to acquire the potential representations of real images. In addition, we introduce a Feature Merge Unit (FMU) that guides the extracted visual embeddings focusing on the character foreground within the text image, thereby enabling the pre-trained decoder to work more efficiently and accurately. Extensive experiments across various STR decoders and language recognition tasks underscore the broad applicability and remarkable performance of DPTR, providing a novel insight for STR pre-training. Code is available at https://github.com/Topdu/OpenOCR

Decoder Pre-Training with only Text for Scene Text Recognition: Insights and Implications

The paper "Decoder Pre-Training with only Text for Scene Text Recognition" by Zhao et al. introduces a novel method for scene text recognition (STR) by leveraging vision-LLMs, specifically CLIP, to bridge the domain gap between synthetic and real image datasets. Scene text recognition has traditionally been challenging due to the complexities of natural environments, such as varying backgrounds, fonts, and imaging conditions. The prevalent reliance on synthetic datasets for pre-training has not been entirely successful in aligning feature representations of synthetic data with real-world images.

Core Contributions

1. Decoder Pre-Training with Text: The authors propose an innovative pre-training method named DPTR (Decoder Pre-training with Text for STR). DPTR utilizes text embeddings produced by CLIP's text encoder as pseudo-visual embeddings for the STR decoder pre-training, bypassing the need for synthetic image data. This approach leverages the alignment capabilities within CLIP's embedding space to enhance STR performance.
2. Introduction of Offline Random Perturbation (ORP): To mitigate the risk of overfitting due to the fixed nature of text embeddings, the paper proposes ORP, which infuses natural image features as background noise into text embeddings. This strategy enriches the diversity of inputs during pre-training, enhancing the model's robustness and preventing overfitting.
3. Feature Merge Unit (FMU): The paper introduces FMU, which employs a cross-attention mechanism to focus the model's attention on character foregrounds in images. FMU effectively filters out background noise, thus ensuring that the embeddings passed to the decoder are pertinent to STR tasks.

Detailed Evaluation and Results

The authors conducted extensive experiments involving various STR decoders and tasks, including English, Chinese, and multilingual text recognition. The deployment of DPTR resulted in notable performance gains across different scenarios and datasets. Specific highlight includes achieving state-of-the-art results with PARSeq, improving accuracy by substantial margins on both synthetic and real datasets.

The paper presents detailed ablation studies, showcasing the effectiveness of individual components like ORP and FMU. For instance, it demonstrates that a small noise ratio in ORP significantly improves decoder training by preventing overfitting without disrupting the core alignment of text embeddings with real-image embeddings. This result is evidenced by improved focus on text characters in attention maps when using DPTR-trained models compared to those pre-trained on synthetic images.

Theoretical and Practical Implications

The introduction of DPTR signifies a shift in pre-training strategies for STR by effectively utilizing LLMs trained on large-scale text-image pairings. This approach not only addresses the domain gap challenge but also emphasizes the potential of text-only training paradigms when aligned with robust vision-LLMs like CLIP.

Practically, this method can reduce dependency on large-scale labeled real text images, which are challenging and costly to obtain, particularly for languages other than English. It offers a scalable solution with broad applicability across various languages and text recognition tasks, providing a template for future research in optical character recognition and related fields.

Speculation on Future Developments

Given the paper’s success with leveraging CLIP, further exploration could investigate deeper integration of other large vision-LLMs, potentially enhancing cross-domain and cross-language STR applications. Additionally, expanding the scope to diversified datasets might further validate the generalizability and robustness of DPTR, possibly leading to improved models capable of seamlessly transitioning from synthetic pre-training to real-world applications.

Overall, this paper represents an important step in enhancing STR with innovative pre-training techniques, bridging gaps previously encountered due to domain discrepancies between synthetic and real datasets. The findings and methodologies outlined pave the way for continued advancements in the domain of robust, efficient scene text recognition.