Text-Aware Image Restoration (TAIR)

Text-Aware Image Restoration (TAIR) is an emerging paradigm in low-level vision that mandates not only the recovery of visual scene content from degraded images but also the preservation and faithful restoration of embedded textual information. This task has gained importance due to the rising prevalence of images containing critical textual cues (e.g., street signs, documents, storefronts) and the shortcomings of conventional restoration methods, which often hallucinate, blur, or alter text regions even as perceptual quality improves.

1. Definition and Motivations

Text-Aware Image Restoration is defined as the restoration of degraded images where both scene and text fidelity are jointly optimized. Unlike traditional methods that focus solely on visual realism or general naturalness, TAIR tasks require that alphabetic, numeric, or logographic characters remain fully readable and correctly rendered after restoration. This addresses a phenomenon termed text-image hallucination, where restored images may contain plausible but incorrect or invented text, leading to incorrect or unusable outcomes in applications such as document digitization, AR navigation, or automatic text extraction.

TAIR responds to several pressing challenges:

• Conventional diffusion-based models and other generative frameworks often hallucinate text during denoising, especially in heavily degraded regions.
• Text content presents discrete, semantic information that cannot be addressed adequately by losses or architectures that treat it as generic image detail.
• There has been a lack of large-scale, high-quality benchmarks with dense text annotations suitable for this style of evaluation and training.

2. SA-Text Benchmark: Dataset for TAIR

The SA-Text benchmark is a large-scale, high-complexity scene text dataset, curated to support text-aware restoration research at scale. It comprises over 100,000 512×512 natural scene images with dense, diverse, and complex text instances annotated at both the localization (polygonal bounding boxes) and recognition (transcription) levels.

Key features of SA-Text include:

• Curation pipeline: Text detection via state-of-the-art spotters, iterated crop and re-detect for enhanced recall, dual vision-LLM (VLM) transcriptions for robustness, and VLM-based image quality filtering to ensure annotated regions are crisp and unambiguous.
• Scale and diversity: SA-Text is larger than prior benchmarks (e.g., ICDAR, Total-Text), enabling robust evaluation under the TAIR paradigm.
• Complex rather than synthetic layouts, supporting evaluation of generalization to real-world conditions.

This resource enables, for the first time, the quantitative and qualitative measurement of the restoration task with respect to both general perceptual quality and textual preservation/accuracy.

3. TeReDiff: Multi-Task Diffusion Framework

TeReDiff (Text Restoration Diffusion) is a multi-task diffusion-based architecture specifically constructed for TAIR. Its key mechanisms include:

• Diffusion restoration backbone: Progressive denoising of degraded images using a U-Net style network (optionally with ControlNet modules), conditioned not only on image-level context but also on text-semantic information.
• Integrated text-spotting module: A transformer-based encoder-decoder receives internal (multi-scale) features from the diffusion network and predicts the precise locations (as polygons or boxes) and transcriptions of all text regions in the image. These features are more semantically rich than those available from standard backbones (e.g., ResNet), leading to higher recognition accuracy especially with limited supervision.
• Text-prompted diffusion: Recognized text is used to form prompts, which are then fed back into the denoising process at each diffusion step. This closes the loop between scene restoration and text recognition, biasing the model towards generating text regions that better match the predicted underlying content.
• Multi-task loss: The total loss is a combination of standard denoising objectives and text-spotting losses (e.g., region classification, bounding polygon regression, character-level recognition), ensuring that both image quality and text fidelity are improved during joint optimization.

Mathematically, core losses include: $$\mathcal{L}_\text{diff} = \mathbb{E}_{z_0, t, p_t, c_t, \epsilon \sim \mathcal{N}(0, 1)} \left[ \left\| \epsilon - \epsilon_\theta(z_t, t, p_t, c_t) \right\|_2^2 \right]$$ for diffusion restoration, and complex transformer-based detection and character losses for spotting.

4. Text-Spotting Module and Prompt Feedback

The integrated text-spotting module serves two purposes:

• Text-centric feature extraction: It parses multi-scale diffusion features to predict both the spatial layout (polygons/boxes) and the textual content (characters/words) in the image.
• Prompt feedback: The set of recognized text strings is dynamically used to form conditioning prompts for subsequent denoising steps. This mechanism guides the diffusion process away from hallucinated or ambiguous text patterns, and towards semantically faithful restoration.

Stage-wise training is employed:

• Initial training of the diffusion model;
• Freezing of the diffusion backbone and training of the text-spotting head with the extracted features;
• Joint fine-tuning for optimal synergy.

Empirically, this not only improves overall text recognition accuracy but also directly suppresses the formation of text-image hallucination artifacts that plague prior generative approaches.

5. Experimental Evaluation

Extensive experiments are conducted on both the SA-Text benchmark and real-world text images. Key findings:

• Superior detection and recognition: TeReDiff achieves the highest F1-scores for both text detection and end-to-end text recognition (E2E F1, no lexicon), outperforming all evaluated baselines—including state-of-the-art diffusion and GAN-based general and dedicated text restoration models.
• Resistance to hallucination: Unlike DiffBIR, FaithDiff, SeeSR, and Real-ESRGAN, which often degrade text or hallucinate plausible but incorrect glyphs under severe degradation, TeReDiff preserves and restores the actual textual information, as substantiated by both OCR accuracy and qualitative assessment.
• Ablation studies confirm that feedback via text-spotting prompts and joint training are essential for maximal gains in text fidelity; using ground-truth texts as prompts defines an upper bound.

Representative quantitative results (e.g., SA-Text Level 2, ABCNet v2):

• TeReDiff: Detection F1 = 67.10, E2E F1 (no lexicon) = 24.42
• Next best (DiffBIR): Detection F1 = 62.03, E2E F1 = 19.60

The same pattern is seen across spotting architectures and under more severe degradations.

6. Technical Innovations and Open Research Directions

Major contributions of TAIR, as instantiated in TeReDiff, include

• Formal definition and large-scale benchmarking for the joint restoration and text fidelity task.
• Demonstration that "text-prompted" diffusion, with feedback from a text-spotting head, substantially improves character recovery and scene faithfulness.
• Empirical evidence that multi-task generative models can outperform both dedicated text SR (super-resolution) models and general-purpose restoration baselines on complex, real-world text images.

Remaining challenges include: robustness to small or extremely degraded text, highly cluttered backgrounds, scaling models to more varied and realistic wild data, and engineering more sophisticated prompt architectures (possibly informed by user or deployment context).

The TAIR paradigm is relevant across a broad range of applications, including document digitization, AR scene understanding, forensic video analysis, and accessibility systems. Its methodology—closing the loop between semantic content and generative restoration—signals a trajectory towards more information-preserving, cross-modal visual computing.

Text-Aware Image Restoration, as advanced by the TeReDiff framework and the new SA-Text benchmark, marks a substantial evolution in restoration research, establishing a new standard for semantic fidelity in both academic and practical settings.