Papers
Topics
Authors
Recent
Search
2000 character limit reached

Artificial Text Overlay Detection

Updated 14 July 2026
  • Artificial text overlay detection is the process of localizing, delineating, and extracting digitally superimposed text from images and videos using both geometric and semantic cues.
  • It employs specialized detection pipelines that integrate segmentation, multi-scale fusion, and boundary-aware decoding to manage stylized, curved, and temporally persistent text.
  • Future research aims to enhance multimodal robustness by mitigating text overlay-induced hallucinations and improving cross-domain performance in vision-language systems.

Searching arXiv for the cited papers and closely related work to ground the article. arxiv_search.query({"11search_query11 OR id:(Słucki et al., 2018) OR id:(Yang et al., 2021) OR id:(Kannao et al., 2016) OR id:(Yakun et al., 19 Apr 2026)11"," OR id:(Słucki et al., 2018) OR id:(Yang et al., 2021) OR id:(Kannao et al., 2016) OR id:(Yakun et al., 19 Apr 2026)11search_query11,"11sort_by11 arxiv_search.query({"11search_query11 text overlay detection\" OR 11all:\11 overlays\" OR 11all:\11 poster text detector\"","11max_results11 OR id:(Słucki et al., 2018) OR id:(Yang et al., 2021) OR id:(Kannao et al., 2016) OR id:(Yakun et al., 19 Apr 2026)11search_query11,"11sort_by11 Artificial text overlay detection concerns the localization, delineation, extraction, and, in some settings, semantic disambiguation of text that has been digitally superimposed on images or video frames. Typical instances include titles, taglines, credits, captions, lower-thirds, watermarks, and explanatory phrases rendered during editing rather than captured as part of the physical scene. The task differs from generic scene-text detection because overlays are often intentionally stylized, temporally persistent, horizontally structured, or embedded in editorial layouts; in movie posters and promotional creatives they may be curved, textured, shadowed, or irregular, whereas in broadcast and social video they are frequently optimized for readability on cluttered or fast-changing imagery (&&&11search_query11&&&, &&&11id:(Ning et al., 2024) OR id:(Słucki et al., 2018) OR id:(Yang et al., 2021) OR id:(Kannao et al., 2016) OR id:(Yakun et al., 19 Apr 2026)11&&&).

Artificial overlays are non-natural, digitally rendered text composited onto media during authoring or post-production. In social media videos, the relevant categories include titles, captions, callouts, lower-thirds, and explanatory phrases; captions and subtitles are treated as overlays, while burned-in text and on-screen graphics such as boxed captions and logos are also overlays in practice (&&&11id:(Ning et al., 2024) OR id:(Słucki et al., 2018) OR id:(Yang et al., 2021) OR id:(Kannao et al., 2016) OR id:(Yakun et al., 19 Apr 2026)11&&&). In image-centric settings, especially movie posters, overlays include titles, taglines, and credits that are integrated into promotional design rather than captured from a scene (&&&11search_query11&&&).

A recurring distinction is between overlays and scene text. Scene text is naturally present in the filmed scene and is affected by perspective, rotation, occlusion, motion, and environmental lighting, whereas overlays are editor-controlled and often obey stronger layout priors. One practical system does not introduce a separate classifier for overlays versus scene text; instead, it exploits characteristics common to overlays, notably horizontal orientation and temporal behavior, using a detector that only detects horizontal text and a post-processing stage that consolidates repeated content across frames (&&&11id:(Ning et al., 2024) OR id:(Słucki et al., 2018) OR id:(Yang et al., 2021) OR id:(Kannao et al., 2016) OR id:(Yakun et al., 19 Apr 2026)11&&&). By contrast, artistic overlays in posters are explicitly treated as arbitrary-shape text whose curvature, decorative structure, and textured rendering require segmentation-style localization and boundary-aware decoding (&&&11search_query11&&&).

Broadcast-news overlays introduce an additional notion of the “text band,” defined as a horizontally aligned rectangle enclosing one or more adjoining text regions that form a semantically coherent unit. Four conditions are specified for a text band: all text regions in the band have almost the same stroke width; no sharp changes in background as well as foreground color occur across regions; all regions share a common baseline; and there is no separator between adjoining text regions (&&&11sort_by11&&&). This formulation is narrower than arbitrary-shape text detection but captures a large class of editorial overlays used in news production.

A broader, downstream framing arises in multimodal reasoning. “Text Overlay-Induced Hallucination” (TOIH) denotes the failure mode in which a vision-LLM prioritizes misleading overlay text over contradictory visual evidence. This extends artificial text overlay detection from geometric localization into robustness against overlay semantics (&&&11relevance11&&&). This suggests that the topic spans both classical document-free text detection and modality disentanglement in multimodal systems.

11max_results11. Visual characteristics and problem difficulty

Artificial overlays exhibit domain-specific statistics that differ substantially across media types. In movie posters and related promotional graphics, stylized art fonts may be heavily embellished, warped, shadowed, or rendered with textures and gradients; text can be curved or irregular; sizes vary dramatically; and overlays often sit on cluttered, high-contrast backgrounds designed to draw attention (&&&11search_query11&&&). These properties challenge detectors that rely on strictly local cues or simplistic geometric priors.

Social-media overlays tend to be more regular but remain challenging. Reported characteristics include frequent placement near the bottom or center, integration with the background rather than solid bands, clean modern fonts, multiple text colors, and temporal persistence across several frames; the “final” version of an overlay often has the most characters because phrases may build up progressively by words or lines (&&&11id:(Ning et al., 2024) OR id:(Słucki et al., 2018) OR id:(Yang et al., 2021) OR id:(Kannao et al., 2016) OR id:(Yakun et al., 19 Apr 2026)11&&&). This temporal accumulation changes the extraction problem: redundancy reduction and temporal deduplication become as important as single-frame localization.

TV-news overlays are typically horizontally aligned bands with high contrast between foreground text and background, clutter-free local background, and high edge density in text regions. Their band boundaries manifest as strong horizontal and vertical transitions in edge projection profiles (&&&11sort_by11&&&). However, animation effects, scrolling elements, and variable channel-specific styles can disrupt single-frame detection and motivate explicit tracking.

The main failure modes recur across these domains. In artistic imagery, background textures, strong edges, gradients, and decorative motifs induce false positives unless feature fusion suppresses clutter and boundary decoding remains shape-aware (&&&11search_query11&&&). In video, small fonts, motion blur, curved or rotated text, animated overlays, and multilingual scripts remain difficult or are intentionally excluded by some systems (&&&11id:(Ning et al., 2024) OR id:(Słucki et al., 2018) OR id:(Yang et al., 2021) OR id:(Kannao et al., 2016) OR id:(Yakun et al., 19 Apr 2026)11&&&). In TV news, low-contrast overlays, translucent backgrounds, heavy animation, and non-horizontal layouts violate the assumptions that underpin band-based detection (&&&11sort_by11&&&). In multimodal reasoning, even correctly localized overlays can be harmful when their semantics contradict visual evidence, producing systematic answer shifts toward the overlaid text (&&&11relevance11&&&).

A common misconception is that overlay text is simply an easier version of scene text. The literature instead shows that overlays can be easier along some axes, such as perspective stability, but harder along others, including stylization, temporal redundancy, semantic interference, and the need to separate editorial typography from designed backgrounds.

11sort_by11. Segmentation-based detectors for stylized and arbitrary-shape overlays

A recent line of work treats artistic overlays as an arbitrary-shape text segmentation problem. The detector in “Artistic-style text detector and a new Movie-Poster dataset” consists of four main components: a feature extraction backbone, a feature enhancement network, a multi-scale feature fusion module, and a boundary discrimination module (&&&11search_query11&&&). A standard convolutional backbone extracts hierarchical features PRESERVED_PLACEHOLDER_11search_query11–PRESERVED_PLACEHOLDER_11id:(Ning et al., 2024) OR id:(Słucki et al., 2018) OR id:(Yang et al., 2021) OR id:(Kannao et al., 2016) OR id:(Yakun et al., 19 Apr 2026)11; lower levels provide detail for small glyphs and thin strokes, while higher levels provide semantics needed to distinguish text from decorative strokes and textures.

The feature enhancement network uses Criss-Cross Attention (CCA) to inject horizontal and vertical contextual information. For a feature map PRESERVED_PLACEHOLDER_11max_results11^ with query, key, and value projections PRESERVED_PLACEHOLDER_11sort_by11, attention is computed over the criss-cross set PRESERVED_PLACEHOLDER_11relevance11^ of each spatial location:

PRESERVED_PLACEHOLDER_11sort_order11^

PRESERVED_PLACEHOLDER_11descending11^

A practical implementation includes a learnable scaling PRESERVED_PLACEHOLDER_11search_query11^ and an output projection PRESERVED_PLACEHOLDER_11all:\11:

PRESERVED_PLACEHOLDER_11 OR all:\11^

Because the aggregation path is restricted to rows and columns, CCA injects long-range dependencies along dominant text directions without the cost of full non-local attention. In poster imagery, this improves robustness to curved or irregular text, mixed lighting, and cluttered backgrounds (&&&11search_query11&&&).

Multi-scale fusion is built on an FPN that constructs pyramid features PRESERVED_PLACEHOLDER_11id:(Ning et al., 2024) OR id:(Słucki et al., 2018) OR id:(Yang et al., 2021) OR id:(Kannao et al., 2016) OR id:(Yakun et al., 19 Apr 2026)11search_query11–PRESERVED_PLACEHOLDER_11id:(Ning et al., 2024) OR id:(Słucki et al., 2018) OR id:(Yang et al., 2021) OR id:(Kannao et al., 2016) OR id:(Yakun et al., 19 Apr 2026)11id:(Ning et al., 2024) OR id:(Słucki et al., 2018) OR id:(Yang et al., 2021) OR id:(Kannao et al., 2016) OR id:(Yakun et al., 19 Apr 2026)11^ from backbone outputs PRESERVED_PLACEHOLDER_11id:(Ning et al., 2024) OR id:(Słucki et al., 2018) OR id:(Yang et al., 2021) OR id:(Kannao et al., 2016) OR id:(Yakun et al., 19 Apr 2026)11max_results11–PRESERVED_PLACEHOLDER_11id:(Ning et al., 2024) OR id:(Słucki et al., 2018) OR id:(Yang et al., 2021) OR id:(Kannao et al., 2016) OR id:(Yakun et al., 19 Apr 2026)11sort_by11:

PRESERVED_PLACEHOLDER_11id:(Ning et al., 2024) OR id:(Słucki et al., 2018) OR id:(Yang et al., 2021) OR id:(Kannao et al., 2016) OR id:(Yakun et al., 19 Apr 2026)11relevance11^

PRESERVED_PLACEHOLDER_11id:(Ning et al., 2024) OR id:(Słucki et al., 2018) OR id:(Yang et al., 2021) OR id:(Kannao et al., 2016) OR id:(Yakun et al., 19 Apr 2026)11sort_order11^

Residual Dense Blocks (RDBs) are inserted at fusion points to suppress background noise while preserving fine text detail. Within an RDB, dense connectivity is defined by

PRESERVED_PLACEHOLDER_11id:(Ning et al., 2024) OR id:(Słucki et al., 2018) OR id:(Yang et al., 2021) OR id:(Kannao et al., 2016) OR id:(Yakun et al., 19 Apr 2026)11descending11^

and the block output is

PRESERVED_PLACEHOLDER_11id:(Ning et al., 2024) OR id:(Słucki et al., 2018) OR id:(Yang et al., 2021) OR id:(Kannao et al., 2016) OR id:(Yakun et al., 19 Apr 2026)11search_query11^

Dense concatenation reuses intermediate features across layers, which is described as effective at amplifying stroke-like patterns while diminishing background textures, gradients, and high-frequency clutter (&&&11search_query11&&&).

The final boundary discrimination module predicts a text kernel probability map PRESERVED_PLACEHOLDER_11id:(Ning et al., 2024) OR id:(Słucki et al., 2018) OR id:(Yang et al., 2021) OR id:(Kannao et al., 2016) OR id:(Yakun et al., 19 Apr 2026)11all:\11, a boundary probability map PRESERVED_PLACEHOLDER_11id:(Ning et al., 2024) OR id:(Słucki et al., 2018) OR id:(Yang et al., 2021) OR id:(Kannao et al., 2016) OR id:(Yakun et al., 19 Apr 2026)11 OR all:\11, and optional regression maps PRESERVED_PLACEHOLDER_11max_results11search_query11^ for offsets or distance fields. The training loss combines binary cross-entropy, Dice loss, and an optional regression loss:

PRESERVED_PLACEHOLDER_11max_results11id:(Ning et al., 2024) OR id:(Słucki et al., 2018) OR id:(Yang et al., 2021) OR id:(Kannao et al., 2016) OR id:(Yakun et al., 19 Apr 2026)11^

At inference, high-PRESERVED_PLACEHOLDER_11max_results11max_results11^ regions are expanded with boundary likelihoods PRESERVED_PLACEHOLDER_11max_results11sort_by11^ and refined with PRESERVED_PLACEHOLDER_11max_results11relevance11^ to produce polygons, reducing reliance on heuristic grouping or graph-based merging (&&&11search_query11&&&).

A related segmentation-based framework is Adaptive Shrink-Mask for Text Detection (ASMTD), which replaces fixed shrink-mask-dependent reconstruction with an Adaptive Shrink-Mask (ASM) and adds Super-pixel Window (SPW) supervision during training (&&&11max_results11&&&). The method contrasts fixed extension,

PRESERVED_PLACEHOLDER_11max_results11sort_order11^

with an adaptive offset

PRESERVED_PLACEHOLDER_11max_results11descending11^

where PRESERVED_PLACEHOLDER_11max_results11search_query11^ is the PRESERVED_PLACEHOLDER_11max_results11all:\11-th pixel value in the fused feature map PRESERVED_PLACEHOLDER_11max_results11 OR all:\11. Ground-truth shrink-masks are generated using the Vatti clipping algorithm with shrink offset

PRESERVED_PLACEHOLDER_11sort_by11search_query11^

The adaptive offset target is a scalar minimum-distance field:

PRESERVED_PLACEHOLDER_11sort_by11id:(Ning et al., 2024) OR id:(Słucki et al., 2018) OR id:(Yang et al., 2021) OR id:(Kannao et al., 2016) OR id:(Yakun et al., 19 Apr 2026)11^

This decouples contour reconstruction from the exact geometry of the predicted shrink-mask, improving robustness when masks deviate from ground truth (&&&11max_results11&&&).

SPW supervision introduces local contextual supervision over an anchor window PRESERVED_PLACEHOLDER_11sort_by11max_results11:

PRESERVED_PLACEHOLDER_11sort_by11sort_by11^

The total ASMTD loss is

PRESERVED_PLACEHOLDER_11sort_by11relevance11^

with PRESERVED_PLACEHOLDER_11sort_by11sort_order11, PRESERVED_PLACEHOLDER_11sort_by11descending11, and PRESERVED_PLACEHOLDER_11sort_by11search_query11^ (&&&11max_results11&&&). Reported results include PRESERVED_PLACEHOLDER_11sort_by11all:\11^ and PRESERVED_PLACEHOLDER_11sort_by11 OR all:\11^ on MSRA-TD11sort_order11search_query11search_query11, PRESERVED_PLACEHOLDER_11relevance11search_query11^ and PRESERVED_PLACEHOLDER_11relevance11id:(Ning et al., 2024) OR id:(Słucki et al., 2018) OR id:(Yang et al., 2021) OR id:(Kannao et al., 2016) OR id:(Yakun et al., 19 Apr 2026)11^ on Total-Text, and PRESERVED_PLACEHOLDER_11relevance11max_results11^ and PRESERVED_PLACEHOLDER_11relevance11sort_by11^ on CTW11id:(Ning et al., 2024) OR id:(Słucki et al., 2018) OR id:(Yang et al., 2021) OR id:(Kannao et al., 2016) OR id:(Yakun et al., 19 Apr 2026)11sort_order11search_query11search_query11, with ablations showing gains from both ASM and SPW (&&&11max_results11&&&).

Taken together, these detectors illustrate the current consensus for difficult overlays: robust global context modeling, noise suppression during multi-scale fusion, and boundary-aware decoding are central when artistic or arbitrary-shape text must be localized directly rather than approximated by rectangular boxes.

11relevance11. Video pipelines: keyframes, tracking, recognition, and temporal consolidation

Artificial text overlay extraction in video is often implemented as a staged pipeline rather than a monolithic detector. One social-media system uses four processing steps: keyframe extraction, text detection, text recognition, and post-processing (&&&11id:(Ning et al., 2024) OR id:(Słucki et al., 2018) OR id:(Yang et al., 2021) OR id:(Kannao et al., 2016) OR id:(Yakun et al., 19 Apr 2026)11&&&). Keyframes are obtained by extracting I-frames using ffmpeg, based on the observation that overlays tend to persist across frames and that processing only I-frames greatly reduces the number of frames processed without a significant loss of overlay information.

Text detection is performed with TextBoxes, an SSD-based word detector that outputs horizontal word-level bounding boxes with confidence scores over multiple convolutional feature maps, followed by non-maximum suppression. Final boxes are expanded by PRESERVED_PLACEHOLDER_11relevance11relevance11^ px on each side to compensate for tight NMS cuts that clip characters (&&&11id:(Ning et al., 2024) OR id:(Słucki et al., 2018) OR id:(Yang et al., 2021) OR id:(Kannao et al., 2016) OR id:(Yakun et al., 19 Apr 2026)11&&&). Recognition uses a CRNN with CTC loss; cropped detections are converted to grayscale, scaled to fixed height PRESERVED_PLACEHOLDER_11relevance11sort_order11^ px with variable width, and encoded as a feature sequence for two Bidirectional LSTM layers. The CTC formulation is given by

PRESERVED_PLACEHOLDER_11relevance11descending11^

where PRESERVED_PLACEHOLDER_11relevance11search_query11^ is an alignment path over the CTC alphabet and PRESERVED_PLACEHOLDER_11relevance11all:\11^ collapses repeats and removes blanks (&&&11id:(Ning et al., 2024) OR id:(Słucki et al., 2018) OR id:(Yang et al., 2021) OR id:(Kannao et al., 2016) OR id:(Yakun et al., 19 Apr 2026)11&&&).

Domain adaptation is handled through a synthetic overlay dataset of approximately PRESERVED_PLACEHOLDER_11relevance11 OR all:\11^ images. The corpus draws on transcripts from PRESERVED_PLACEHOLDER_11sort_order11search_query11^ social media videos and the top PRESERVED_PLACEHOLDER_11sort_order11id:(Ning et al., 2024) OR id:(Słucki et al., 2018) OR id:(Yang et al., 2021) OR id:(Kannao et al., 2016) OR id:(Yakun et al., 19 Apr 2026)11^ COCA words, with digits and special characters added; backgrounds are sampled from text-free regions of PRESERVED_PLACEHOLDER_11sort_order11max_results11^ randomly selected video frames; typography covers PRESERVED_PLACEHOLDER_11sort_order11sort_by11^ fonts across PRESERVED_PLACEHOLDER_11sort_order11relevance11^ families; and each word is rendered in PRESERVED_PLACEHOLDER_11sort_order11sort_order11^ samples with random font, size, and one of three predefined text colors, then resized to PRESERVED_PLACEHOLDER_11sort_order11descending11^ px (&&&11id:(Ning et al., 2024) OR id:(Słucki et al., 2018) OR id:(Yang et al., 2021) OR id:(Kannao et al., 2016) OR id:(Yakun et al., 19 Apr 2026)11&&&). Fine-tuning only the LSTM layers yields the best reported recognition accuracy, PRESERVED_PLACEHOLDER_11sort_order11search_query11, compared with PRESERVED_PLACEHOLDER_11sort_order11all:\11^ for a generic CRNN and PRESERVED_PLACEHOLDER_11sort_order11 OR all:\11^ for Tesseract. End-to-end on PRESERVED_PLACEHOLDER_11descending11search_query11^ words, the fine-tuned CRNN reports Precision PRESERVED_PLACEHOLDER_11descending11id:(Ning et al., 2024) OR id:(Słucki et al., 2018) OR id:(Yang et al., 2021) OR id:(Kannao et al., 2016) OR id:(Yakun et al., 19 Apr 2026)11, Recall PRESERVED_PLACEHOLDER_11descending11max_results11, F11id:(Ning et al., 2024) OR id:(Słucki et al., 2018) OR id:(Yang et al., 2021) OR id:(Kannao et al., 2016) OR id:(Yakun et al., 19 Apr 2026)11^ PRESERVED_PLACEHOLDER_11descending11sort_by11, and Similarity PRESERVED_PLACEHOLDER_11descending11relevance11^ (&&&11id:(Ning et al., 2024) OR id:(Słucki et al., 2018) OR id:(Yang et al., 2021) OR id:(Kannao et al., 2016) OR id:(Yakun et al., 19 Apr 2026)11&&&).

Temporal redundancy is reduced by sorting recognized phrases in reverse chronological order and comparing them by normalized Levenshtein distance. If two phrases fall below a threshold, they are treated as overlapping and the longer version is retained. The edit-distance recursion is

PRESERVED_PLACEHOLDER_11descending11sort_order11^

PRESERVED_PLACEHOLDER_11descending11descending11^

with normalization by PRESERVED_PLACEHOLDER_11descending11search_query11^ (&&&11id:(Ning et al., 2024) OR id:(Słucki et al., 2018) OR id:(Yang et al., 2021) OR id:(Kannao et al., 2016) OR id:(Yakun et al., 19 Apr 2026)11&&&). This stage addresses the fact that overlays often stabilize over time rather than appearing once.

A different video-oriented strategy is developed for TV-news broadcast overlays. The pipeline begins with a contrast enhancement stage over Scharr gradient magnitudes. After computing the normalized gradient magnitude PRESERVED_PLACEHOLDER_11descending11all:\11, the method defines

PRESERVED_PLACEHOLDER_11descending11 OR all:\11^

with the lowest non-suppressed gradient magnitude

PRESERVED_PLACEHOLDER_11search_query11search_query11^

The parameter PRESERVED_PLACEHOLDER_11search_query11id:(Ning et al., 2024) OR id:(Słucki et al., 2018) OR id:(Yang et al., 2021) OR id:(Kannao et al., 2016) OR id:(Yakun et al., 19 Apr 2026)11^ is auto-selected using Otsu’s threshold PRESERVED_PLACEHOLDER_11search_query11max_results11^ so that PRESERVED_PLACEHOLDER_11search_query11sort_by11, yielding

PRESERVED_PLACEHOLDER_11search_query11relevance11^

Histogram equalization of the stretched response produces the final edge map PRESERVED_PLACEHOLDER_11search_query11sort_order11^ (&&&11sort_by11&&&).

Localization then proceeds by parameter-free edge-density analysis through horizontal and vertical projection profile derivatives. The horizontal projection profile is

PRESERVED_PLACEHOLDER_11search_query11descending11^

and, within a candidate band PRESERVED_PLACEHOLDER_11search_query11search_query11, the vertical profile is

PRESERVED_PLACEHOLDER_11search_query11all:\11^

Connected component analysis in one dimension groups derivative peaks, and boundary evidence is derived from local means per cluster rather than from global density thresholds (&&&11sort_by11&&&). This yields band rectangles rather than word boxes, exploiting the horizontal alignment property of broadcast overlays.

Tracking is handled through overlap sets between previously tracked rectangles PRESERVED_PLACEHOLDER_11search_query11 OR all:\11^ and current detections PRESERVED_PLACEHOLDER_11all:\11search_query11, together with RCC-11sort_order11^ relations based on the fractional overlap

PRESERVED_PLACEHOLDER_11all:\11id:(Ning et al., 2024) OR id:(Słucki et al., 2018) OR id:(Yang et al., 2021) OR id:(Kannao et al., 2016) OR id:(Yakun et al., 19 Apr 2026)11^

The tracker explicitly handles unique correspondences, merges, splits, disappearances, and new entries, and uses color histogram checks to distinguish temporary detection failures from true exits (&&&11sort_by11&&&). Stable tracks are binarized and passed once to a domain-adapted Tesseract OCR system trained on PRESERVED_PLACEHOLDER_11all:\11max_results11^ fonts and a lexicon derived from approximately PRESERVED_PLACEHOLDER_11all:\11sort_by11^ web news articles and metadata. Reported OCR error rates improve from PRESERVED_PLACEHOLDER_11all:\11relevance11^ to PRESERVED_PLACEHOLDER_11all:\11sort_order11^ at the character level and from PRESERVED_PLACEHOLDER_11all:\11descending11^ to PRESERVED_PLACEHOLDER_11all:\11search_query11^ at the word level after dictionary corrections (&&&11sort_by11&&&).

These two video pipelines occupy different points in the design space. The social-media system prioritizes keyframe efficiency, word-box detection, and textual consolidation; the TV-news system prioritizes band geometry, explicit tracking, and domain-adapted OCR. This suggests that temporal overlay extraction is best understood as a joint detection-tracking-recognition problem rather than a framewise text detector alone.

11sort_order11. Datasets, benchmarks, and evaluation practice

Dataset design in this area reflects the heterogeneity of overlay styles. The Movie-Poster dataset was introduced to address the scarcity of artistic-style text data and targets promotional overlays characterized by serif, sans-serif, hand-drawn, decorative, and distressed fonts; gradient fills, drop-shadows, strokes, glow, and emboss; curved and irregular text such as arcs, circles, perspective warps, and freeform baselines; challenging backgrounds; and overlays on faces, objects, and effects (&&&11search_query11&&&). Annotations are provided as polygons and/or masks, with train/val/test splits designed for benchmarking overlay detection.

The social-media video study evaluates on data collected from PRESERVED_PLACEHOLDER_11all:\11all:\11^ Facebook videos from June 11max_results11search_query11id:(Ning et al., 2024) OR id:(Słucki et al., 2018) OR id:(Yang et al., 2021) OR id:(Kannao et al., 2016) OR id:(Yakun et al., 19 Apr 2026)11search_query11^ to January 11max_results11search_query11id:(Ning et al., 2024) OR id:(Słucki et al., 2018) OR id:(Yang et al., 2021) OR id:(Kannao et al., 2016) OR id:(Yakun et al., 19 Apr 2026)11all:\11. It uses a PRESERVED_PLACEHOLDER_11all:\11 OR all:\11-crop OCR test set and an end-to-end set of PRESERVED_PLACEHOLDER_11 OR all:\11search_query11^ frames containing PRESERVED_PLACEHOLDER_11 OR all:\11id:(Ning et al., 2024) OR id:(Słucki et al., 2018) OR id:(Yang et al., 2021) OR id:(Kannao et al., 2016) OR id:(Yakun et al., 19 Apr 2026)11^ total words, annotated with bounding boxes and transcriptions (&&&11id:(Ning et al., 2024) OR id:(Słucki et al., 2018) OR id:(Yang et al., 2021) OR id:(Kannao et al., 2016) OR id:(Yakun et al., 19 Apr 2026)11&&&). No public benchmark is used, which reflects the limited availability of overlay-specific video corpora at the time.

The TV-news broadcast study uses PRESERVED_PLACEHOLDER_11 OR all:\11max_results11^ annotated frames at PRESERVED_PLACEHOLDER_11 OR all:\11sort_by11^ for text-band detection and three PRESERVED_PLACEHOLDER_11 OR all:\11relevance11-hour videos from three Indian English news channels for tracking and OCR evaluation, while also reporting ablations on ICDAR 11max_results11search_query11search_query11sort_by11, ICDAR 11max_results11search_query11id:(Ning et al., 2024) OR id:(Słucki et al., 2018) OR id:(Yang et al., 2021) OR id:(Kannao et al., 2016) OR id:(Yakun et al., 19 Apr 2026)11id:(Ning et al., 2024) OR id:(Słucki et al., 2018) OR id:(Yang et al., 2021) OR id:(Kannao et al., 2016) OR id:(Yakun et al., 19 Apr 2026)11^ Born-digital, and ICDAR 11max_results11search_query11id:(Ning et al., 2024) OR id:(Słucki et al., 2018) OR id:(Yang et al., 2021) OR id:(Kannao et al., 2016) OR id:(Yakun et al., 19 Apr 2026)11sort_by11^ (&&&11sort_by11&&&). On its TV-news dataset, CE+PP-TB achieves Precision PRESERVED_PLACEHOLDER_11 OR all:\11sort_order11, Recall PRESERVED_PLACEHOLDER_11 OR all:\11descending11, F-measure PRESERVED_PLACEHOLDER_11 OR all:\11search_query11, and PRESERVED_PLACEHOLDER_11 OR all:\11all:\11^ s per frame, compared with PP-TB at Precision PRESERVED_PLACEHOLDER_11 OR all:\11 OR all:\11, Recall PRESERVED_PLACEHOLDER_11id:(Ning et al., 2024) OR id:(Słucki et al., 2018) OR id:(Yang et al., 2021) OR id:(Kannao et al., 2016) OR id:(Yakun et al., 19 Apr 2026)11search_query11search_query11, and F-measure PRESERVED_PLACEHOLDER_11id:(Ning et al., 2024) OR id:(Słucki et al., 2018) OR id:(Yang et al., 2021) OR id:(Kannao et al., 2016) OR id:(Yakun et al., 19 Apr 2026)11search_query11id:(Ning et al., 2024) OR id:(Słucki et al., 2018) OR id:(Yang et al., 2021) OR id:(Kannao et al., 2016) OR id:(Yakun et al., 19 Apr 2026)11^ (&&&11sort_by11&&&).

Standard detection metrics include Precision, Recall, F-measure/Hmean, Average Precision, and IoU (&&&11search_query11&&&). For ASMTD, the reported benchmark results are:

Ablation studies are central to evaluating overlay detectors because failure cases are strongly tied to architecture. Removing CCA reduces recall on curved or irregular text; replacing RDBs with plain convolutions increases false positives on textured backgrounds; omitting boundary discrimination necessitates complex post-processing and degrades contour accuracy; and restricting the pyramid to PRESERVED_PLACEHOLDER_11id:(Ning et al., 2024) OR id:(Słucki et al., 2018) OR id:(Yang et al., 2021) OR id:(Kannao et al., 2016) OR id:(Yakun et al., 19 Apr 2026)11id:(Ning et al., 2024) OR id:(Słucki et al., 2018) OR id:(Yang et al., 2021) OR id:(Kannao et al., 2016) OR id:(Yakun et al., 19 Apr 2026)11relevance11–PRESERVED_PLACEHOLDER_11id:(Ning et al., 2024) OR id:(Słucki et al., 2018) OR id:(Yang et al., 2021) OR id:(Kannao et al., 2016) OR id:(Yakun et al., 19 Apr 2026)11id:(Ning et al., 2024) OR id:(Słucki et al., 2018) OR id:(Yang et al., 2021) OR id:(Kannao et al., 2016) OR id:(Yakun et al., 19 Apr 2026)11sort_order11^ rather than PRESERVED_PLACEHOLDER_11id:(Ning et al., 2024) OR id:(Słucki et al., 2018) OR id:(Yang et al., 2021) OR id:(Kannao et al., 2016) OR id:(Yakun et al., 19 Apr 2026)11id:(Ning et al., 2024) OR id:(Słucki et al., 2018) OR id:(Yang et al., 2021) OR id:(Kannao et al., 2016) OR id:(Yakun et al., 19 Apr 2026)11descending11–PRESERVED_PLACEHOLDER_11id:(Ning et al., 2024) OR id:(Słucki et al., 2018) OR id:(Yang et al., 2021) OR id:(Kannao et al., 2016) OR id:(Yakun et al., 19 Apr 2026)11id:(Ning et al., 2024) OR id:(Słucki et al., 2018) OR id:(Yang et al., 2021) OR id:(Kannao et al., 2016) OR id:(Yakun et al., 19 Apr 2026)11search_query11^ affects small-text performance (&&&11search_query11&&&). In ASMTD, the baseline shrink-mask-only model reaches PRESERVED_PLACEHOLDER_11id:(Ning et al., 2024) OR id:(Słucki et al., 2018) OR id:(Yang et al., 2021) OR id:(Kannao et al., 2016) OR id:(Yakun et al., 19 Apr 2026)11id:(Ning et al., 2024) OR id:(Słucki et al., 2018) OR id:(Yang et al., 2021) OR id:(Kannao et al., 2016) OR id:(Yakun et al., 19 Apr 2026)11all:\11^ on MSRA-TD11sort_order11search_query11search_query11, adding ASM raises it to PRESERVED_PLACEHOLDER_11id:(Ning et al., 2024) OR id:(Słucki et al., 2018) OR id:(Yang et al., 2021) OR id:(Kannao et al., 2016) OR id:(Yakun et al., 19 Apr 2026)11id:(Ning et al., 2024) OR id:(Słucki et al., 2018) OR id:(Yang et al., 2021) OR id:(Kannao et al., 2016) OR id:(Yakun et al., 19 Apr 2026)11 OR all:\11, and adding SPW raises it to PRESERVED_PLACEHOLDER_11id:(Ning et al., 2024) OR id:(Słucki et al., 2018) OR id:(Yang et al., 2021) OR id:(Kannao et al., 2016) OR id:(Yakun et al., 19 Apr 2026)11max_results11search_query11^ without affecting inference speed (&&&11max_results11&&&).

A methodological pattern emerges across these datasets and metrics. Overlay-specific benchmarks do not merely substitute different images for scene-text data; they encode assumptions about typography, layout, persistence, and semantic role that materially affect both architecture and evaluation.

11descending11. Multimodal robustness, semantic interference, and future directions

Artificial text overlay detection has acquired a semantic dimension in the study of multimodal hallucination. VisualTextTrap formalizes Text Overlay-Induced Hallucination (TOIH) and constructs a benchmark of PRESERVED_PLACEHOLDER_11id:(Ning et al., 2024) OR id:(Słucki et al., 2018) OR id:(Yang et al., 2021) OR id:(Kannao et al., 2016) OR id:(Yakun et al., 19 Apr 2026)11max_results11id:(Ning et al., 2024) OR id:(Słucki et al., 2018) OR id:(Yang et al., 2021) OR id:(Kannao et al., 2016) OR id:(Yakun et al., 19 Apr 2026)11^ samples annotated across PRESERVED_PLACEHOLDER_11id:(Ning et al., 2024) OR id:(Słucki et al., 2018) OR id:(Yang et al., 2021) OR id:(Kannao et al., 2016) OR id:(Yakun et al., 19 Apr 2026)11max_results11max_results11^ fine-grained attributes within four dimensions—Temporal, Action, Object, and Spatial—with hallucination intensity quantified on a five-level scale PRESERVED_PLACEHOLDER_11id:(Ning et al., 2024) OR id:(Słucki et al., 2018) OR id:(Yang et al., 2021) OR id:(Kannao et al., 2016) OR id:(Yakun et al., 19 Apr 2026)11max_results11sort_by11–PRESERVED_PLACEHOLDER_11id:(Ning et al., 2024) OR id:(Słucki et al., 2018) OR id:(Yang et al., 2021) OR id:(Kannao et al., 2016) OR id:(Yakun et al., 19 Apr 2026)11max_results11relevance11^ (&&&11relevance11&&&). Three overlay conditions are defined: Text-Free (PRESERVED_PLACEHOLDER_11id:(Ning et al., 2024) OR id:(Słucki et al., 2018) OR id:(Yang et al., 2021) OR id:(Kannao et al., 2016) OR id:(Yakun et al., 19 Apr 2026)11max_results11sort_order11), Text-Congruent, and Text-Contradictory (PRESERVED_PLACEHOLDER_11id:(Ning et al., 2024) OR id:(Słucki et al., 2018) OR id:(Yang et al., 2021) OR id:(Kannao et al., 2016) OR id:(Yakun et al., 19 Apr 2026)11max_results11descending11).

For a multiple-choice video-QA instance PRESERVED_PLACEHOLDER_11id:(Ning et al., 2024) OR id:(Słucki et al., 2018) OR id:(Yang et al., 2021) OR id:(Kannao et al., 2016) OR id:(Yakun et al., 19 Apr 2026)11max_results11search_query11, TOIH is characterized by

PRESERVED_PLACEHOLDER_11id:(Ning et al., 2024) OR id:(Słucki et al., 2018) OR id:(Yang et al., 2021) OR id:(Kannao et al., 2016) OR id:(Yakun et al., 19 Apr 2026)11max_results11all:\11^

The per-sample confidence shifts are

PRESERVED_PLACEHOLDER_11id:(Ning et al., 2024) OR id:(Słucki et al., 2018) OR id:(Yang et al., 2021) OR id:(Kannao et al., 2016) OR id:(Yakun et al., 19 Apr 2026)11max_results11 OR all:\11^

PRESERVED_PLACEHOLDER_11id:(Ning et al., 2024) OR id:(Słucki et al., 2018) OR id:(Yang et al., 2021) OR id:(Kannao et al., 2016) OR id:(Yakun et al., 19 Apr 2026)11sort_by11search_query11^

The dominant regime is PRESERVED_PLACEHOLDER_11id:(Ning et al., 2024) OR id:(Słucki et al., 2018) OR id:(Yang et al., 2021) OR id:(Kannao et al., 2016) OR id:(Yakun et al., 19 Apr 2026)11sort_by11id:(Ning et al., 2024) OR id:(Słucki et al., 2018) OR id:(Yang et al., 2021) OR id:(Kannao et al., 2016) OR id:(Yakun et al., 19 Apr 2026)11^ and PRESERVED_PLACEHOLDER_11id:(Ning et al., 2024) OR id:(Słucki et al., 2018) OR id:(Yang et al., 2021) OR id:(Kannao et al., 2016) OR id:(Yakun et al., 19 Apr 2026)11sort_by11max_results11, meaning confidence mass moves from the correct answer to the overlay-induced option (&&&11relevance11&&&).

The benchmark introduces metrics that explicitly quantify overlay-induced failure. Hallucination Resistance Rate is

PRESERVED_PLACEHOLDER_11id:(Ning et al., 2024) OR id:(Słucki et al., 2018) OR id:(Yang et al., 2021) OR id:(Kannao et al., 2016) OR id:(Yakun et al., 19 Apr 2026)11sort_by11sort_by11^

Semantic Conflict Sensitivity Index is

PRESERVED_PLACEHOLDER_11id:(Ning et al., 2024) OR id:(Słucki et al., 2018) OR id:(Yang et al., 2021) OR id:(Kannao et al., 2016) OR id:(Yakun et al., 19 Apr 2026)11sort_by11relevance11^

Additional measures include Visual Yielding Rate, Hallucination Acceptance Rate, Interference Cost Ratio, Synergy Gain/Loss Index, Text-Induced Hallucination Rate, Textual Induction Bias, Weighted Hallucination Rate, Hallucination Surge Rate, and a Hallucination Resistance Curve per conflict level (&&&11relevance11&&&). These metrics move beyond localization accuracy by evaluating whether systems can resist semantically misleading overlays.

The proposed mitigation model, VTHM-MoE, is a Vision-Text Disentanglement framework with dual encoders:

PRESERVED_PLACEHOLDER_11id:(Ning et al., 2024) OR id:(Słucki et al., 2018) OR id:(Yang et al., 2021) OR id:(Kannao et al., 2016) OR id:(Yakun et al., 19 Apr 2026)11sort_by11sort_order11^

where PRESERVED_PLACEHOLDER_11id:(Ning et al., 2024) OR id:(Słucki et al., 2018) OR id:(Yang et al., 2021) OR id:(Kannao et al., 2016) OR id:(Yakun et al., 19 Apr 2026)11sort_by11descending11^ patches and PRESERVED_PLACEHOLDER_11id:(Ning et al., 2024) OR id:(Słucki et al., 2018) OR id:(Yang et al., 2021) OR id:(Kannao et al., 2016) OR id:(Yakun et al., 19 Apr 2026)11sort_by11search_query11. Query-guided patch selection uses

PRESERVED_PLACEHOLDER_11id:(Ning et al., 2024) OR id:(Słucki et al., 2018) OR id:(Yang et al., 2021) OR id:(Kannao et al., 2016) OR id:(Yakun et al., 19 Apr 2026)11sort_by11all:\11^

For each selected patch, the model forms a three-token representation

PRESERVED_PLACEHOLDER_11id:(Ning et al., 2024) OR id:(Słucki et al., 2018) OR id:(Yang et al., 2021) OR id:(Kannao et al., 2016) OR id:(Yakun et al., 19 Apr 2026)11sort_by11 OR all:\11^

Cross-modal consistency is measured by

PRESERVED_PLACEHOLDER_11id:(Ning et al., 2024) OR id:(Słucki et al., 2018) OR id:(Yang et al., 2021) OR id:(Kannao et al., 2016) OR id:(Yakun et al., 19 Apr 2026)11relevance11search_query11^

and routing logits are defined as

PRESERVED_PLACEHOLDER_11id:(Ning et al., 2024) OR id:(Słucki et al., 2018) OR id:(Yang et al., 2021) OR id:(Kannao et al., 2016) OR id:(Yakun et al., 19 Apr 2026)11relevance11id:(Ning et al., 2024) OR id:(Słucki et al., 2018) OR id:(Yang et al., 2021) OR id:(Kannao et al., 2016) OR id:(Yakun et al., 19 Apr 2026)11^

Four dimension-specialized experts—Temporal, Action, Object, and Spatial—are inserted at the output of the 11id:(Ning et al., 2024) OR id:(Słucki et al., 2018) OR id:(Yang et al., 2021) OR id:(Kannao et al., 2016) OR id:(Yakun et al., 19 Apr 2026)11descending11th transformer layer:

PRESERVED_PLACEHOLDER_11id:(Ning et al., 2024) OR id:(Słucki et al., 2018) OR id:(Yang et al., 2021) OR id:(Kannao et al., 2016) OR id:(Yakun et al., 19 Apr 2026)11relevance11max_results11^

The overall loss is

PRESERVED_PLACEHOLDER_11id:(Ning et al., 2024) OR id:(Słucki et al., 2018) OR id:(Yang et al., 2021) OR id:(Kannao et al., 2016) OR id:(Yakun et al., 19 Apr 2026)11relevance11sort_by11^

with PRESERVED_PLACEHOLDER_11id:(Ning et al., 2024) OR id:(Słucki et al., 2018) OR id:(Yang et al., 2021) OR id:(Kannao et al., 2016) OR id:(Yakun et al., 19 Apr 2026)11relevance11relevance11, PRESERVED_PLACEHOLDER_11id:(Ning et al., 2024) OR id:(Słucki et al., 2018) OR id:(Yang et al., 2021) OR id:(Kannao et al., 2016) OR id:(Yakun et al., 19 Apr 2026)11relevance11sort_order11, and PRESERVED_PLACEHOLDER_11id:(Ning et al., 2024) OR id:(Słucki et al., 2018) OR id:(Yang et al., 2021) OR id:(Kannao et al., 2016) OR id:(Yakun et al., 19 Apr 2026)11relevance11descending11^ (&&&11relevance11&&&).

Reported clean-benchmark accuracies for VTHM-MoE are PRESERVED_PLACEHOLDER_11id:(Ning et al., 2024) OR id:(Słucki et al., 2018) OR id:(Yang et al., 2021) OR id:(Kannao et al., 2016) OR id:(Yakun et al., 19 Apr 2026)11relevance11search_query11^ on LLaVA-Video, PRESERVED_PLACEHOLDER_11id:(Ning et al., 2024) OR id:(Słucki et al., 2018) OR id:(Yang et al., 2021) OR id:(Kannao et al., 2016) OR id:(Yakun et al., 19 Apr 2026)11relevance11all:\11^ on VideoMME, and PRESERVED_PLACEHOLDER_11id:(Ning et al., 2024) OR id:(Słucki et al., 2018) OR id:(Yang et al., 2021) OR id:(Kannao et al., 2016) OR id:(Yakun et al., 19 Apr 2026)11relevance11 OR all:\11^ on TemporalBench, while the model also improves resistance under contradictory overlays (&&&11relevance11&&&). The paper identifies temporal contradictions as the hardest and action contradictions as easier because short, localized motion facilitates visual grounding.

This multimodal line of work reframes artificial text overlay detection as part of a larger control problem: systems must first detect the presence and location of overlays, then determine whether the overlay is helpful, irrelevant, or contradictory. A plausible implication is that future overlay detectors will increasingly couple geometric localization with semantic calibration, especially in pipelines where OCR outputs are consumed by reasoning models.

Current limitations remain domain-dependent. Extreme artistic distortions and very low contrast text remain difficult in poster imagery; small fonts, curved or rotated text, heavy motion blur, and animated overlays remain difficult in social video; low-contrast overlays, translucent backgrounds, and scrolling tickers remain difficult in broadcast video; and OCR errors or missed detections can attenuate discrepancy signals in multimodal systems (&&&11search_query11&&&, &&&11id:(Ning et al., 2024) OR id:(Słucki et al., 2018) OR id:(Yang et al., 2021) OR id:(Kannao et al., 2016) OR id:(Yakun et al., 19 Apr 2026)11&&&, &&&11sort_by11&&&, &&&11relevance11&&&). Reported future directions include curved-text modeling with Bezier or Fourier representations, transformer backbones for richer global context, stronger OCR and text-region segmentation, temporal consistency models for video overlays, style-diverse synthetic generation, and conflict-aware pre-training for multimodal robustness (&&&11search_query11&&&, &&&11max_results11&&&, &&&11relevance11&&&).

Topic to Video (Beta)

No one has generated a video about this topic yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this topic yet.

Sign Up to Generate

Follow Topic

Get notified by email when new papers are published related to Artificial Text Overlay Detection.

Sign Up to Follow Topic by Email