Scene Texts: Detection, Recognition & Applications

• Scene texts are textual elements in natural scenes characterized by diverse scripts, fonts, distortions, and environmental conditions.
• Advanced detection and segmentation methods, including transformer-based models, enable precise localization even in cluttered or curved layouts.
• Innovative synthesis, editing, and removal techniques facilitate realistic text manipulation and enhance applications in VQA, 3D scene parsing, and embodied AI.

Scene texts are textual elements encountered in natural images, spanning a vast array of appearances, layouts, and environmental conditions. These texts form an important subclass of image content, as they often convey semantic information critical for scene understanding, retrieval, navigation, and human–machine interaction. The field of scene text research encompasses detection, segmentation, recognition, synthesis, removal, and editing, addressing the rich visual and linguistic variability encountered in real environments.

1. Definition and Challenges

Scene texts are character or word sequences present in unconstrained, real-world images—distinct from document images—such as those captured from street views, product packaging, advertisements, or natural scenes. Typical characteristics are significant diversity in script, font, color, size, orientation, shape (straight, curved, or arbitrary), background clutter, illumination variation, occlusions, and partial visibility. Unlike digitally typeset text or scanned documents, scene texts are subject to variable camera pose, perspective distortion, physical surfaces (planar/non-planar), as well as environmental factors, leading to pronounced challenges in detection and recognition. State-of-the-art models persistently grapple with arbitrary-shaped text, severe curvature, heavy background clutter, and diverse writing styles (Lee et al., 2019).

2. Detection and Segmentation of Scene Texts

Detection localizes the extent of text regions, while segmentation aims for pixel-level delineation. Conventional detection frameworks either formulate the task as bounding box regression (quadrilateral or rotated rectangles), region proposal followed by text verification, or representing text as centerlines, polygons, or masks. Advanced models, such as the Aggregated Text Transformer (ATTR), employ multi-scale self-attention over image pyramids, using Transformer encoders to produce instance-level binary masks that are robust to dense layouts and curve variations (Zhou et al., 2022). Methods like the Multi-Perspective Feature Learning Network (MT) combine lightweight convolutional backbones with segmentation heads and auxiliary geometric tasks for high-efficiency detection at 50 FPS on benchmarks (Yang et al., 2021).

Segmentation methods, including EAFormer, introduce edge-aware Transformer architectures that explicitly extract and inject text edge maps via cross-attention, yielding improved accuracy along fine text boundaries and significant gains on benchmarks, especially after relabeling datasets with precise polygon masks (Yu et al., 2024).

Alternative annotation-efficient frameworks, such as scribble-supervised detection with weak labels, reduce human labeling costs by using centerline scribbles as proxies for full polygon boundaries, achieving performance on par with full supervision (Zhang et al., 2020).

3. Recognition in Arbitrary Conditions

Scene text recognition (STR) targets transcription of localized text into character strings, requiring resilience to shape, orientation, and style variations. Architectures such as the Temporal Convolutional Encoder (TCE) leverage dilated 1D convolutions to enlarge contextual receptive fields, combined with attention refinement within the CNN backbone, augmenting transcription accuracy and convergence (Du et al., 2019). Orientation-Independent STR frameworks introduce modules like the Character Image Reconstruction Network (CIRN) to explicitly disentangle content and orientation, delivering large recognition accuracy gains for vertical and rotated scripts (e.g., in Chinese) (Yu et al., 2023).

Models designed for font independence adopt multi-font glyph generation branches, enforcing the encoder to focus on essential character shapes while discarding stylistic variation, which is critical for recognizing scene texts rendered in novel or rare fonts (Wang et al., 2020).

Recognition in the presence of arbitrary-shaped or curved text is addressed by architectures that utilize 2D self-attention to model full spatial dependencies, such as self-attention text recognition networks, which show superior performance on irregular shape benchmarks (Lee et al., 2019).

4. Editing, Synthesis, and Removal

Manipulating scene texts involves generating, modifying, or excising text content in natural images. High-fidelity scene text synthesis models (e.g., DreamText) employ diffusion-based generative models with character-level embedding and attention mask guidance, integrating hybrid (discrete and continuous) optimization to enforce precise glyph placement, kerning, and text region focus (Wang et al., 2024). The joint training of text encoders on polystylistic corpora enables robust synthesis across font styles, outperforming multiple SOTA baselines on sequence accuracy and perceptual metrics.

Style and content editing frameworks, such as QuadNet, disentangle background, foreground style, and target content in a latent feature space, leveraging background inpainting, style encoders, and AdaIN-based style fusion for fine-grained manipulation of text instances (e.g., editing rotation, font, or color via deep semantic editing) (Su et al., 2023). Similarly, SwapText and related pipelines decouple foreground text and background completion, using geometric alignment and self-attention fusion to translate, replace, or transfer scene text while faithfully mimicking geometry and surrounding texture (Yang et al., 2020).

Selective removal tasks, exemplified by the SSTR framework, allow targeting and erasing specific words by conditioning U-Net modules with user-specified word indices (via FiLM layers), efficiently excising the designated text while conserving non-target regions and background with minimal collateral alteration (Mitani et al., 2023).

5. Synthetic Data and Training Paradigms

Scene text methods heavily leverage synthetic data for robust supervised training. Early 2D synthesis pipelines (SynthText) have been superseded by 3D virtual world frameworks (SynthText3D), which render texts as 3D meshes in photorealistic environments using physically-based engines (Unreal Engine 4), with full geometric, lighting, occlusion, and typography variability. 3D-based synthesis closes the domain gap and provides superior data for training detection and recognition models; 10K 3D synthetic images surpass the effectiveness of 800K 2D synthetic images in scene text detection benchmarks (Liao et al., 2019).

Advanced verisimilar synthesis methods combine semantic segmentations, background saliency, and adaptive appearance modeling to place text in only semantically appropriate and visually plausible regions, further boosting detection/recognition performance (Zhan et al., 2018).

6. Downstream Applications and Representations

Scene texts are critical for downstream vision–language applications, notably visual question answering (VQA), embodied AI, and 3D scene understanding. Frameworks such as TextBlockV2 eliminate dependency on precise character or word-level detection by clustering text into text blocks and employing pre-trained vision–language Transformers for detection-free recognition, leveraging LLMs' robustness to occlusion, incomplete detections, and context-dependent encoding (Lyu et al., 2024). In 3D scene parsing, scene texts (in the sense of "scene-level textual summaries") are produced by pipelines that map geometric and object-level information into relational paragraphs, interpreted by multimodal LLMs for planning, grounding, and QA over 3D scenes (Li et al., 20 Sep 2025).

In linguistically dense scripts (e.g., Vietnamese), phrase constitution with distributional attention is used to compose meaning over OCR tokens, linking scene text semantics with VQA in a linguistically principled manner (Nguyen et al., 2024).

7. Empirical Performance and Future Directions

Continuous benchmarking on datasets representing curved, multi-lingual, occluded, or low-light scene text has driven model advancements. SOTA detection and segmentation methods demonstrate F-measures exceeding 90% on major benchmarks (Zhou et al., 2022, Yang et al., 2021). Recognition models incorporating TCE, orientation disentanglement, and glyph generation achieve top-1 accuracy above 90% and strong robustness to heterogenous fonts and orientations (Du et al., 2019, Yu et al., 2023, Wang et al., 2020). Editing and synthesis architectures attain high FID/LPIPS scores and clean artifact-free insertion, editing, or removal of scene texts (Wang et al., 2024, Su et al., 2023).

Open problems include reducing dependency on full supervision through weak labels (scribbles, polygons), extending cross-modal reasoning with scene texts in complex vision–language task settings (3D VQA, embodied planning), and enhancing model generalization to rare scripts and real-world degradations (e.g., extremely low-light environments (Hsu et al., 2022)). Future directions also explore the integration of large multimodal LLMs for detection-free recognition and language-centric scene understanding.