Advances in Scene-Text Spotting

• Scene-text spotting is the task of simultaneously localizing and recognizing text in natural images, addressing challenges such as scale variation, occlusion, and irregular geometry.
• Recent research leverages unified transformer-based, segmentation, and autoregressive methods to improve the synergy between detection and recognition components.
• Innovative approaches incorporate linguistic priors, structured queries, and benchmarking on diverse datasets to advance robust and efficient end-to-end text spotters.

Searching arXiv for recent and foundational scene-text spotting papers relevant to the requested topic. Searching arXiv for unified, transformer-based, and language-aware scene-text spotting methods. Searching arXiv for dataset, context, video, and structured scene-text spotting works. Scene-text spotting is the end-to-end task of localizing text instances in natural images and recognizing their transcriptions simultaneously. In the literature, it is treated as a harder problem than either scene text detection or scene text recognition alone, because the system must recover both geometry and content under large variations in scale, font, color, illumination, orientation, and shape, while also handling blur, occlusion, perspective distortion, cluttered backgrounds, and dense layouts (Lyu et al., 2018). Recent work has broadened the task beyond word-level detect-then-recognize pipelines toward segmentation-based spotters, one-stage attention models, autoregressive sequence generators, language-aware decoders, query-conditioned systems, and video or structured-text variants (Kil et al., 2023).

1. Task definition and failure modes

Scene-text spotting in the wild consists of detecting and recognizing text appearing in images such as signboards, traffic signals, or brands in clothing or objects (Sabir et al., 2020). The technical difficulty is not merely that text is small or visually degraded. The representation itself is unstable: a rectangle is often a poor representation for curved text, arbitrary orientation breaks assumptions used by box-based detectors, and recognition quality is highly sensitive to localization quality (Lyu et al., 2018).

Several papers make the same diagnosis from different directions. Mask-based spotters emphasize that irregular or curved words are not well modeled by horizontal boxes or quadrilaterals, while transformer-based spotters note that annotation formats such as a point, bounding box, quadrilateral, polygon, or mask all encode different geometric trade-offs (Lyu et al., 2018, Kil et al., 2023). Dense scenes amplify these issues further. WordLenSpotter argues that long-tailed word-length distributions in dense images cause suboptimal detection recall for both short and long words, because standard spotters primarily model objectness and appearance rather than the spatial semantics of word length (Wang et al., 2023).

The scope of the task has also expanded. Vietnamese urban spotting is framed as particularly difficult because Vietnamese characters carry many diacritics and visually similar variants, and because urban scenes contain frequent occlusion, perspective distortion, clutter, low contrast, reflections, and variable lighting (Nguyen et al., 2024). Scene video text spotting adds temporal effects such as motion blur, camera shake, illumination changes, perspective changes, occlusion, and fast motion, making it strictly harder than spotting in static images (Cheng et al., 2021). Structured scene-text spotting introduces a query regular expression as part of the input, so the system must detect and recognize only instances that satisfy a user-specified pattern (Garcia-Bordils et al., 2023).

A recurring misconception is that scene-text spotting is simply text detection followed by OCR. Multiple papers argue instead that detection and recognition are mutually constraining tasks. Recognition quality can improve localization, and localization quality can improve recognition; much of the recent literature is concerned with how to formalize that synergy rather than treating it as an incidental by-product of a shared backbone (Huang et al., 2023, Huang et al., 2024).

2. Representations and formal problem formulations

A central axis of scene-text spotting research is the representation used for each text instance. Different formulations encode different assumptions about geometry, reading order, supervision cost, and decoder design.

Representation	Exemplar papers	Salient property
Mask / character map	Mask TextSpotter, MANGO	Shape-aware spotting without rigid boxes
Point / single point	SPTS v2, UNITS	Low annotation cost, sequence generation
Bounding box / quadrilateral / polygon	UNITS, ESTextSpotter	Flexible geometry for oriented and arbitrary-shaped text
Text block	TextBlockV2	Coarse detection with context-rich recognition

Mask TextSpotter reframes spotting as instance segmentation plus recognition. It predicts instance-aware text masks and character maps, allowing the detector to conform to rotated, perspective-distorted, curved, and irregular text lines more naturally than rigid boxes (Lyu et al., 2018). MANGO also rejects explicit RoI cropping, using position-aware mask attention so that different text instances in an image are allocated to different feature-map channels and decoded directly as sequences (2012.04350).

At the other end of the design space, SPTS v2 argues that a single point per text instance can be sufficient supervision for training a strong end-to-end spotter (Liu et al., 2023). In its formulation, each instance is represented as a sequence

$[x, y, t]$

where $x,y$ are discretized coordinates of the indicated point and $t$ is the transcription (Liu et al., 2023). UNITS generalizes this sequence view by prepending a detection-format token and then generating coordinate tokens and transcription tokens autoregressively. Its training loss is the standard cross-entropy objective

$$L = - \sum_{j=1}^{N} w_{j} \log P(\tilde{y}_{j} \mid I, y_{1:j-1}),$$

with prompt tokens masked out by setting their weights to $0$ (Kil et al., 2023). UNITS supports four detection formats—central point, bounding box, quadrilateral, and polygon—represented with $1$, $2$, $4$, and $16$ points, respectively, while coordinates are quantized into discrete bins with $n_{bins}=1000$ (Kil et al., 2023).

TextBlockV2 proposes a different relaxation: a text block rather than a word or character is the primary detected unit (Lyu et al., 2024). A text block is defined as a group of nearby and visually similar text instances, and the recognizer is a fine-tuned pre-trained LLM that transcribes the block crop (Lyu et al., 2024). This suggests a broader conceptual shift: precise word-level delineation is not always the only viable interface between localization and recognition.

3. Architectural paradigms

A major branch of the literature extends detection architectures into end-to-end spotters. Mask TextSpotter is explicitly inspired by Mask R-CNN: a backbone CNN extracts features, an RPN proposes candidate text regions, and RoI-based heads perform detection and recognition jointly (Lyu et al., 2018). Its recognition-related character maps are not merely auxiliary; removing the character-map subnet reduces detection performance by $x,y$0 on ICDAR2013 and $x,y$1 on ICDAR2015, which is direct evidence that recognition supervision can improve localization (Lyu et al., 2018). GLASS retains a detection-first structure but fuses global shared features with local high-resolution rotated word crops through a global-to-local attention mechanism; on TextOCR test, the baseline without GLASS reports $x,y$2 end-to-end Hmean, while GLASS reports $x,y$3 (Ronen et al., 2022).

Another branch aims to weaken or remove the dependence of recognition on precise RoI extraction. MANGO is a one-stage spotter that removes RoI operations entirely and decodes text from attention-weighted feature aggregation (2012.04350). SRSTS v2 decouples recognition from detection through a sampling module that predicts representative feature points around each positive anchor point, so recognition is no longer dependent on precise text boundaries while still being optimized collaboratively with detection (Wu et al., 2022). TextBlockV2 pushes this logic further by treating spotting as a coarse detector $x,y$4 followed by a PLM-based recognizer $x,y$5, explicitly asking whether machines can spot texts without precise detection and whether text blocks are an alternative to word- or character-level units (Lyu et al., 2024).

Transformer-based architectures have made the detector–recognizer relationship a primary design target. ESTextSpotter argues that prior transformer methods mostly rely on implicit synergy through shared queries or shared parameters, and therefore decomposes the shared query into task-aware detection and recognition queries with a vision-language communication module (Huang et al., 2023). SwinTextSpotter v2 similarly introduces Recognition Conversion and Recognition Alignment so that recognition loss can back-propagate into the detector and detector outputs can dynamically align recognition features; its Box Selection Schedule allows the detector to be reduced from six refinement stages to three without performance degradation (Huang et al., 2024).

Autoregressive sequence generation is a distinct line of work. SPTS v2 separates location assignment and text recognition into an Instance Assignment Decoder and a Parallel Recognition Decoder that share parameters, reducing the number of autoregressive loops from $x,y$6 to $x,y$7 in the representative setting $x,y$8, $x,y$9, and yielding a reported $t$0 speed-up over SPTS (Liu et al., 2023). UNITS treats spotting as a language-modeling problem over image-conditioned token sequences, and its starting-point prompting strategy allows the model to continue spotting beyond the number of instances it was trained to generate in one pass (Kil et al., 2023). LSGSpotter and IAST specialize this generative perspective to arbitrary reading order and inverse-like text by introducing start-point localization or explicit reading-order estimation before autoregressive decoding (Lyu et al., 2024, Zhang et al., 2024).

4. Linguistic, semantic, and contextual priors

A substantial body of work argues that visual evidence alone is insufficient for robust scene-text spotting. One early direction injects contextual semantics from the image. The visual-semantic extension of COCO-Text augments each text instance with object labels, scene labels, and captions, producing COCO-Text-V, IC17-V, and COCO-Pairs, and supporting both post-hoc OCR re-ranking and end-to-end multimodal training (Sabir et al., 2020). In its re-ranking experiments, semantic context yields modest but measurable gains: the CNN baseline rises from $t$1 to $t$2 with fine-tuned BERT on caption-based similarity, while the LSTM baseline rises from $t$3 to $t$4; a human upper bound of about $t$5 on a small re-ranking task indicates that the problem remains difficult (Sabir et al., 2020).

A second direction injects linguistic priors directly into the decoder. “Efficiently Leveraging Linguistic Priors for Scene Text Spotting” replaces one-hot character supervision with soft character distributions derived from CANINE and trains with KL divergence: $t$6 using threshold $t$7 to post-process the target distributions (Nguyen et al., 2024). The reported effect is not limited to recognition: detection H-mean improves from $t$8 on TotalText, $t$9 on ICDAR15, and $$L = - \sum_{j=1}^{N} w_{j} \log P(\tilde{y}_{j} \mid I, y_{1:j-1}),$$0 on CTW1500 (Nguyen et al., 2024). A3S adds a semantic embedding head to ABCNet v2 and aligns predicted embeddings with BERT embeddings through adversarial learning,

$$L = - \sum_{j=1}^{N} w_{j} \log P(\tilde{y}_{j} \mid I, y_{1:j-1}),$$1

with $$L = - \sum_{j=1}^{N} w_{j} \log P(\tilde{y}_{j} \mid I, y_{1:j-1}),$$2, $$L = - \sum_{j=1}^{N} w_{j} \log P(\tilde{y}_{j} \mid I, y_{1:j-1}),$$3, and $$L = - \sum_{j=1}^{N} w_{j} \log P(\tilde{y}_{j} \mid I, y_{1:j-1}),$$4; on CTW1500, the end-to-end score rises from $$L = - \sum_{j=1}^{N} w_{j} \log P(\tilde{y}_{j} \mid I, y_{1:j-1}),$$5 to $$L = - \sum_{j=1}^{N} w_{j} \log P(\tilde{y}_{j} \mid I, y_{1:j-1}),$$6 in the None setting and from $$L = - \sum_{j=1}^{N} w_{j} \log P(\tilde{y}_{j} \mid I, y_{1:j-1}),$$7 to $$L = - \sum_{j=1}^{N} w_{j} \log P(\tilde{y}_{j} \mid I, y_{1:j-1}),$$8 in the Full setting (Fujitake, 2023).

PLM-centric spotters push the same idea more aggressively. TextBlockV2 fine-tunes GPT2-Base or T5-Base for OCR on block crops, introduces the Unified Vision-Language Mask, and reports that GPT2 is particularly effective for multi-line, reversed, occluded, incomplete-detection, and multi-word cases (Lyu et al., 2024). TiCLS builds a scene-text-oriented character-level PLM and initializes a dedicated linguistic decoder from it; on ICDAR 2015 it reports $$L = - \sum_{j=1}^{N} w_{j} \log P(\tilde{y}_{j} \mid I, y_{1:j-1}),$$9, $0$0, $0$1, and $0$2 under Strong, Weak, Generic, and lexicon-free settings, and on Total-Text it reports $0$3 None and $0$4 Full (Jang et al., 3 Feb 2026). InstructOCR introduces human-language instructions via a learnable BERT text encoder and reports that instruction-based OCR pretraining improves downstream VQA by $0$5 on TextVQA and $0$6 on ST-VQA (Duan et al., 2024).

A third direction conditions spotting on structured prior knowledge. STEP defines structured scene-text spotting as finding only instances whose transcriptions match a query regular expression, encodes the regex as a fixed-length sequence of multi-hot vectors, and conditions the encoder, character decoder, and location decoder on that structure (Garcia-Bordils et al., 2023). This is a different use of language prior from generic word semantics: the query affects both where the model looks and what transcriptions it accepts.

5. Benchmarks, datasets, and evaluation protocols

The benchmarking ecosystem reflects the heterogeneity of the task. ICDAR 2013 is used as a horizontal-text benchmark with bounding boxes, ICDAR 2015 as a multi-oriented benchmark with quadrilateral annotations, Total-Text for arbitrary-shaped text, CTW-1500 for curved text, TextOCR for large-scale arbitrary-shaped text, and HierText for hierarchical natural-scene and document text (Kil et al., 2023). Some benchmark sizes are repeatedly reported: Total-Text has $0$7 training images and $0$8 test images, and ICDAR 2015 has $0$9 training and $1$0 test images (Fujitake, 2023). VinText contains $1$1 images and $1$2K text instances, split into $1$3 training images, $1$4 testing images, and $1$5 unseen test images (Nguyen et al., 2024). DSTD1500, introduced for dense spotting, contains $1$6 images and $1$7 English text instances (Wang et al., 2023). The ICDAR 2021 scene video text spotting competition dataset contains $1$8 video clips from $1$9 scenarios, split into $2$0 training, $2$1 validation, and $2$2 test videos (Cheng et al., 2021).

Evaluation is correspondingly diverse. Standard image-level detection uses precision, recall, and H-mean or F-measure, while end-to-end spotting is typically reported under lexicon settings such as Strong, Weak, Generic, None, or Full (Ronen et al., 2022). SPTS v2 introduces a point-based metric because point outputs do not naturally map to IoU-based box evaluation; the reported difference from conventional box or polygon metrics is at most about $2$3 on Total-Text and SCUT-CTW1500 (Liu et al., 2023). Context-based OCR re-ranking uses Accuracy and Mean Reciprocal Rank,

$2$4

to evaluate whether the correct transcription is promoted to the top of the candidate list (Sabir et al., 2020).

Video protocols add temporal constraints. The ICDAR 2021 competition defines Task 1 video text detection, Task 2 video text tracking, and Task 3 end-to-end video text spotting (Cheng et al., 2021). In addition to framewise metrics, the paper proposes sequence-level spotting using Recall$2$5, Precision$2$6, and F-score$2$7, with correctness requiring both a spatial-temporal localization constraint and exact transcription matching (Cheng et al., 2021). Structured spotting uses the same precision, recall, F-score, and average edit distance metrics, but the target text is conditioned on a query regex and may span spaces rather than word-level units (Garcia-Bordils et al., 2023).

These protocols matter because they embody different assumptions about output form. A detector-oriented H-mean, a point-based spotting metric, an MRR-style reranking score, and a sequence-level video score are not interchangeable; each privileges a particular interface between localization and recognition.

6. Specialized settings, limitations, and research directions

Several papers target regimes where conventional left-to-right, word-level spotting is systematically brittle. Inverse-like or arbitrary-reading-order text is one such regime. IAST introduces a Reading-Order Estimation Module and a Dynamic Sampling Module for mirrored, symmetrical, and retro-flexed text, reporting $2$8 None and $2$9 Full on Inverse-Text (Zhang et al., 2024). LSGSpotter instead predicts a start point for each instance and autoregressively decodes both character content and coordinate shift, reporting $4$0 lexicon-free and $4$1 with the full lexicon on InverseText; if ground-truth start points are substituted for predicted ones, Total-Text improves from $4$2 to $4$3, indicating that start-point localization remains a bottleneck (Lyu et al., 2024).

Dense, multilingual, and application-specific settings expose different weaknesses. WordLenSpotter injects character-count and aspect-ratio priors through a Spatial Length Predictor and a Length-aware Segmentation head; on DSTD1500 it reports $4$4 precision, $4$5 recall, and $4$6 F-score for detection, with end-to-end scores of $4$7 without lexicon and $4$8 with full lexicon (Wang et al., 2023). The Vietnamese urban ensemble paper shows that complementary detectors and recognizers can help—its best ensemble, combining DB++ + SPIN and SAST + ABINet, reaches Char$4$9 and F-measure $16$0—but also stresses that more models do not automatically improve results, because some pairings degrade performance through overlap conflicts and mismatched predictions (Nguyen et al., 2024).

The literature also converges on several limitations. Context datasets built with off-the-shelf models have noisy automatic annotations, and COCO-Text itself has detection recall issues (Sabir et al., 2020). Autoregressive spotters remain constrained by decoding length and runtime, even when prompting strategies mitigate the output-budget problem (Kil et al., 2023). Ensembling improves robustness at the cost of higher computational complexity (Nguyen et al., 2024). STEP’s regex encoding is limited to fixed-length strings and cannot directly express operators such as + or * in a single pass (Garcia-Bordils et al., 2023). TiCLS explicitly reports parameter overhead of about $16$1 relative to DeepSolo and a forward pass that is about $16$2 slower (Jang et al., 3 Feb 2026).

These developments suggest that scene-text spotting is no longer a single formulation but a family of closely related problems. The field now spans mask-based and point-based representations, block-level and word-level units, global-local fusion, explicit detector–recognizer communication, external linguistic priors, query-conditioned structured OCR, and video-level sequence evaluation. A plausible implication is that future progress will depend less on a universal geometry and more on choosing the right interface between localization, reading order, language prior, and evaluation protocol for the regime at hand.