---
title: Artificial Text Overlay Detection
url: https://www.emergentmind.com/topics/artificial-text-overlay-detection
type: topic
---

# Artificial Text Overlay Detection

Searching arXiv for the cited papers and closely related work to ground the article.
arxiv_search.query({"search_query":"id:2406.16307 OR id:1804.10687 OR id:2111.09560 OR id:1604.00470 OR id:2604.17375","max_results":10,"sort_by":"relevance","sort_order":"descending"})
arxiv_search.query({"search_query":"all:\"Artificial text overlay detection\" OR all:\"textual overlays\" OR all:\"movie poster text detector\"","max_results":10,"sort_by":"lastUpdatedDate","sort_order":"descending"})
Artificial text overlay detection concerns the localization, delineation, extraction, and, in some settings, semantic disambiguation of text that has been digitally superimposed on images or video frames. Typical instances include titles, taglines, credits, captions, lower-thirds, watermarks, and explanatory phrases rendered during editing rather than captured as part of the physical scene. The task differs from generic scene-text detection because overlays are often intentionally stylized, temporally persistent, horizontally structured, or embedded in editorial layouts; in movie posters and promotional creatives they may be curved, textured, shadowed, or irregular, whereas in broadcast and social video they are frequently optimized for readability on cluttered or fast-changing imagery [2406.16307][1804.10687].

## 1. Scope, definitions, and domain boundaries

Artificial overlays are non-natural, digitally rendered text composited onto media during authoring or post-production. In social media videos, the relevant categories include titles, captions, callouts, lower-thirds, and explanatory phrases; captions and subtitles are treated as overlays, while burned-in text and on-screen graphics such as boxed captions and logos are also overlays in practice [1804.10687]. In image-centric settings, especially movie posters, overlays include titles, taglines, and credits that are integrated into promotional design rather than captured from a scene [2406.16307].

A recurring distinction is between overlays and scene text. Scene text is naturally present in the filmed scene and is affected by perspective, rotation, occlusion, motion, and environmental lighting, whereas overlays are editor-controlled and often obey stronger layout priors. One practical system does not introduce a separate classifier for overlays versus scene text; instead, it exploits characteristics common to overlays, notably horizontal orientation and temporal behavior, using a detector that only detects horizontal text and a post-processing stage that consolidates repeated content across frames [1804.10687]. By contrast, artistic overlays in posters are explicitly treated as arbitrary-shape text whose curvature, decorative structure, and textured rendering require segmentation-style localization and boundary-aware decoding [2406.16307].

Broadcast-news overlays introduce an additional notion of the “text band,” defined as a horizontally aligned rectangle enclosing one or more adjoining text regions that form a semantically coherent unit. Four conditions are specified for a text band: all text regions in the band have almost the same stroke width; no sharp changes in background as well as foreground color occur across regions; all regions share a common baseline; and there is no separator between adjoining text regions [1604.00470]. This formulation is narrower than arbitrary-shape text detection but captures a large class of editorial overlays used in news production.

A broader, downstream framing arises in multimodal reasoning. “Text Overlay-Induced Hallucination” (TOIH) denotes the failure mode in which a vision-language model prioritizes misleading overlay text over contradictory visual evidence. This extends artificial text overlay detection from geometric localization into robustness against overlay semantics [2604.17375]. This suggests that the topic spans both classical document-free text detection and modality disentanglement in multimodal systems.

## 2. Visual characteristics and problem difficulty

Artificial overlays exhibit domain-specific statistics that differ substantially across media types. In movie posters and related promotional graphics, stylized art fonts may be heavily embellished, warped, shadowed, or rendered with textures and gradients; text can be curved or irregular; sizes vary dramatically; and overlays often sit on cluttered, high-contrast backgrounds designed to draw attention [2406.16307]. These properties challenge detectors that rely on strictly local cues or simplistic geometric priors.

Social-media overlays tend to be more regular but remain challenging. Reported characteristics include frequent placement near the bottom or center, integration with the background rather than solid bands, clean modern fonts, multiple text colors, and temporal persistence across several frames; the “final” version of an overlay often has the most characters because phrases may build up progressively by words or lines [1804.10687]. This temporal accumulation changes the extraction problem: redundancy reduction and temporal deduplication become as important as single-frame localization.

TV-news overlays are typically horizontally aligned bands with high contrast between foreground text and background, clutter-free local background, and high edge density in text regions. Their band boundaries manifest as strong horizontal and vertical transitions in edge projection profiles [1604.00470]. However, animation effects, scrolling elements, and variable channel-specific styles can disrupt single-frame detection and motivate explicit tracking.

The main failure modes recur across these domains. In artistic imagery, background textures, strong edges, gradients, and decorative motifs induce false positives unless feature fusion suppresses clutter and boundary decoding remains shape-aware [2406.16307]. In video, small fonts, motion blur, curved or rotated text, animated overlays, and multilingual scripts remain difficult or are intentionally excluded by some systems [1804.10687]. In TV news, low-contrast overlays, translucent backgrounds, heavy animation, and non-horizontal layouts violate the assumptions that underpin band-based detection [1604.00470]. In multimodal reasoning, even correctly localized overlays can be harmful when their semantics contradict visual evidence, producing systematic answer shifts toward the overlaid text [2604.17375].

A common misconception is that overlay text is simply an easier version of scene text. The literature instead shows that overlays can be easier along some axes, such as perspective stability, but harder along others, including stylization, temporal redundancy, semantic interference, and the need to separate editorial typography from designed backgrounds.

## 3. Segmentation-based detectors for stylized and arbitrary-shape overlays

A recent line of work treats artistic overlays as an arbitrary-shape text segmentation problem. The detector in “Artistic-style text detector and a new Movie-Poster dataset” consists of four main components: a feature extraction backbone, a feature enhancement network, a multi-scale feature fusion module, and a boundary discrimination module [2406.16307]. A standard convolutional backbone extracts hierarchical features $C2$–$C5$; lower levels provide detail for small glyphs and thin strokes, while higher levels provide semantics needed to distinguish text from decorative strokes and textures.

The feature enhancement network uses Criss-Cross Attention (CCA) to inject horizontal and vertical contextual information. For a feature map $X \in \mathbb{R}^{C \times H \times W}$ with query, key, and value projections $Q, K, V \in \mathbb{R}^{d \times H \times W}$, attention is computed over the criss-cross set $C(i)$ of each spatial location:
$$
s_{i,j} = \frac{Q_i^\top K_j}{\sqrt d}, \qquad
\alpha_{i,j} = \operatorname{softmax}_{j \in C(i)}(s_{i,j}),
$$
$$
Y_i = X_i + \sum_{j \in C(i)} \alpha_{i,j} V_j.
$$
A practical implementation includes a learnable scaling $\gamma$ and an output projection $W_o$:
$$
Y = X + \gamma \cdot W_o\!\left(\sum_{j \in C(i)} \alpha_{i,j} V_j\right).
$$
Because the aggregation path is restricted to rows and columns, CCA injects long-range dependencies along dominant text directions without the cost of full non-local attention. In poster imagery, this improves robustness to curved or irregular text, mixed lighting, and cluttered backgrounds [2406.16307].

Multi-scale fusion is built on an FPN that constructs pyramid features $P5$–$P2$ from backbone outputs $C5$–$C2$:
$$
P5 = \operatorname{Conv}_{1 \times 1}(C5),
$$
$$
P_l = \operatorname{Conv}_{3 \times 3}\!\big(\operatorname{RDB}(C_l + \operatorname{Up}_{2 \times}(P_{l+1}))\big), \quad l \in \{4,3,2\}.
$$
Residual Dense Blocks (RDBs) are inserted at fusion points to suppress background noise while preserving fine text detail. Within an RDB, dense connectivity is defined by
$$
y_l = H_l([x_0, x_1, \ldots, x_{l-1}]),
$$
and the block output is
$$
y_{\mathrm{RDB}} = F_c([x_0, x_1, \ldots, x_L]) + x_0.
$$
Dense concatenation reuses intermediate features across layers, which is described as effective at amplifying stroke-like patterns while diminishing background textures, gradients, and high-frequency clutter [2406.16307].

The final boundary discrimination module predicts a text kernel probability map $p_k$, a boundary probability map $p_b$, and optional regression maps $r$ for offsets or distance fields. The training loss combines binary cross-entropy, Dice loss, and an optional regression loss:
$$
\mathcal{L} = \lambda_k \big(\mathcal{L}_{\mathrm{BCE}}(p_k) + \mathcal{L}_{\mathrm{Dice}}(p_k)\big)
+ \lambda_b \big(\mathcal{L}_{\mathrm{BCE}}(p_b) + \mathcal{L}_{\mathrm{Dice}}(p_b)\big)
+ \lambda_r \mathcal{L}_{\mathrm{reg}}.
$$
At inference, high-$p_k$ regions are expanded with boundary likelihoods $p_b$ and refined with $r$ to produce polygons, reducing reliance on heuristic grouping or graph-based merging [2406.16307].

A related segmentation-based framework is Adaptive Shrink-Mask for Text Detection (ASMTD), which replaces fixed shrink-mask-dependent reconstruction with an Adaptive Shrink-Mask (ASM) and adds Super-pixel Window (SPW) supervision during training [2111.09560]. The method contrasts fixed extension,
$$
o_f = \frac{S_s}{L_s}\,\delta_t,
$$
with an adaptive offset
$$
o_a = w(f_i),
$$
where $f_i$ is the $i$-th pixel value in the fused feature map $h_f$. Ground-truth shrink-masks are generated using the Vatti clipping algorithm with shrink offset
$$
o_s = \frac{S_t}{L_t}\left(1-\delta_s^2\right), \qquad \delta_s = 0.4.
$$
The adaptive offset target is a scalar minimum-distance field:
$$
o_a(x) = \min_{m=1,\dots,M} \|p(x)-p_m\|_2^2.
$$
This decouples contour reconstruction from the exact geometry of the predicted shrink-mask, improving robustness when masks deviate from ground truth [2111.09560].

SPW supervision introduces local contextual supervision over an anchor window $A_v$:
$$
\mathrm{SPW}(A_v) = \frac{\sum_k S(A_v \cap A_{k,s})}{S(A_v)}.
$$
The total ASMTD loss is
$$
\mathcal{L} = \lambda_1 L_{sm} + \lambda_2 L_{o_a} + \lambda_3 L_{\mathrm{SPW}},
$$
with $\lambda_1 = 1$, $\lambda_2 = 0.25$, and $\lambda_3 = 0.25$ [2111.09560]. Reported results include $F=86.3$ and $FPS=62.5$ on MSRA-TD500, $F=86.1$ and $FPS=70.9$ on Total-Text, and $F=83.9$ and $FPS=72.1$ on CTW1500, with ablations showing gains from both ASM and SPW [2111.09560].

Taken together, these detectors illustrate the current consensus for difficult overlays: robust global context modeling, noise suppression during multi-scale fusion, and boundary-aware decoding are central when artistic or arbitrary-shape text must be localized directly rather than approximated by rectangular boxes.

## 4. Video pipelines: keyframes, tracking, recognition, and temporal consolidation

Artificial text overlay extraction in video is often implemented as a staged pipeline rather than a monolithic detector. One social-media system uses four processing steps: keyframe extraction, text detection, text recognition, and post-processing [1804.10687]. Keyframes are obtained by extracting I-frames using ffmpeg, based on the observation that overlays tend to persist across frames and that processing only I-frames greatly reduces the number of frames processed without a significant loss of overlay information.

Text detection is performed with TextBoxes, an SSD-based word detector that outputs horizontal word-level bounding boxes with confidence scores over multiple convolutional feature maps, followed by non-maximum suppression. Final boxes are expanded by $2$ px on each side to compensate for tight NMS cuts that clip characters [1804.10687]. Recognition uses a CRNN with CTC loss; cropped detections are converted to grayscale, scaled to fixed height $32$ px with variable width, and encoded as a feature sequence for two Bidirectional LSTM layers. The CTC formulation is given by
$$
p(l \mid x) = \sum_{\pi \in B^{-1}(l)} \prod_{t=1}^{T} y_{\pi_t}^{(t)},
$$
where $\pi$ is an alignment path over the CTC alphabet and $B$ collapses repeats and removes blanks [1804.10687].

Domain adaptation is handled through a synthetic overlay dataset of approximately $600{,}000$ images. The corpus draws on transcripts from $100$ social media videos and the top $5{,}000$ COCA words, with digits and special characters added; backgrounds are sampled from text-free regions of $50$ randomly selected video frames; typography covers $71$ fonts across $30$ families; and each word is rendered in $100$ samples with random font, size, and one of three predefined text colors, then resized to $100 \times 32$ px [1804.10687]. Fine-tuning only the LSTM layers yields the best reported recognition accuracy, $80.1\%$, compared with $75.7\%$ for a generic CRNN and $57.8\%$ for Tesseract. End-to-end on $1{,}128$ words, the fine-tuned CRNN reports Precision $0.45$, Recall $0.42$, F1 $0.432$, and Similarity $0.62$ [1804.10687].

Temporal redundancy is reduced by sorting recognized phrases in reverse chronological order and comparing them by normalized Levenshtein distance. If two phrases fall below a threshold, they are treated as overlapping and the longer version is retained. The edit-distance recursion is
$$
d_{i,0}=i,\qquad d_{0,j}=j,
$$
$$
d_{i,j} = \min\big(d_{i-1,j}+1,\ d_{i,j-1}+1,\ d_{i-1,j-1}+1[a_i \neq b_j]\big),
$$
with normalization by $\max(|a|,|b|)$ [1804.10687]. This stage addresses the fact that overlays often stabilize over time rather than appearing once.

A different video-oriented strategy is developed for TV-news broadcast overlays. The pipeline begins with a contrast enhancement stage over Scharr gradient magnitudes. After computing the normalized gradient magnitude $I_{nm} = I_m/g_{\max}$, the method defines
$$
\beta(x,y) = \alpha\big(I_{nm}(x,y)-0.5\big)+0.5,
$$
with the lowest non-suppressed gradient magnitude
$$
g_{ns} = \frac{(\alpha-1)g_{\max}}{2\alpha}.
$$
The parameter $\alpha$ is auto-selected using Otsu’s threshold $g_{otsu}$ so that $g_{otsu} = g_{ns}$, yielding
$$
\alpha = \frac{g_{\max}}{g_{\max}-2g_{otsu}}.
$$
Histogram equalization of the stretched response produces the final edge map $\Omega_{ce}$ [1604.00470].

Localization then proceeds by parameter-free edge-density analysis through horizontal and vertical projection profile derivatives. The horizontal projection profile is
$$
P_{hp}(y) = \sum_{x=1}^{w} \Omega(x,y),
$$
and, within a candidate band $[y_i,y_j]$, the vertical profile is
$$
P_{vp}(x) = \sum_{y=y_i}^{y_j} \Omega(x,y).
$$
Connected component analysis in one dimension groups derivative peaks, and boundary evidence is derived from local means per cluster rather than from global density thresholds [1604.00470]. This yields band rectangles rather than word boxes, exploiting the horizontal alignment property of broadcast overlays.

Tracking is handled through overlap sets between previously tracked rectangles $tR_i(\tau-1)$ and current detections $dR_j(\tau)$, together with RCC-5 relations based on the fractional overlap
$$
\gamma_{fo}(A,B) = \frac{|A \cap B|}{|A|}.
$$
The tracker explicitly handles unique correspondences, merges, splits, disappearances, and new entries, and uses color histogram checks to distinguish temporary detection failures from true exits [1604.00470]. Stable tracks are binarized and passed once to a domain-adapted Tesseract OCR system trained on $34$ fonts and a lexicon derived from approximately $1{,}300{,}000$ web news articles and metadata. Reported OCR error rates improve from $20.97\%$ to $4.99\%$ at the character level and from $56.94\%$ to $7.04\%$ at the word level after dictionary corrections [1604.00470].

These two video pipelines occupy different points in the design space. The social-media system prioritizes keyframe efficiency, word-box detection, and textual consolidation; the TV-news system prioritizes band geometry, explicit tracking, and domain-adapted OCR. This suggests that temporal overlay extraction is best understood as a joint detection-tracking-recognition problem rather than a framewise text detector alone.

## 5. Datasets, benchmarks, and evaluation practice

Dataset design in this area reflects the heterogeneity of overlay styles. The Movie-Poster dataset was introduced to address the scarcity of artistic-style text data and targets promotional overlays characterized by serif, sans-serif, hand-drawn, decorative, and distressed fonts; gradient fills, drop-shadows, strokes, glow, and emboss; curved and irregular text such as arcs, circles, perspective warps, and freeform baselines; challenging backgrounds; and overlays on faces, objects, and effects [2406.16307]. Annotations are provided as polygons and/or masks, with train/val/test splits designed for benchmarking overlay detection.

The social-media video study evaluates on data collected from $100$ Facebook videos from June 2017 to January 2018. It uses a $1{,}000$-crop OCR test set and an end-to-end set of $100$ frames containing $1{,}128$ total words, annotated with bounding boxes and transcriptions [1804.10687]. No public benchmark is used, which reflects the limited availability of overlay-specific video corpora at the time.

The TV-news broadcast study uses $150$ annotated frames at $720 \times 576$ for text-band detection and three $1$-hour videos from three Indian English news channels for tracking and OCR evaluation, while also reporting ablations on ICDAR 2003, ICDAR 2011 Born-digital, and ICDAR 2013 [1604.00470]. On its TV-news dataset, CE+PP-TB achieves Precision $0.7600$, Recall $0.8544$, F-measure $0.8045$, and $0.084$ s per frame, compared with PP-TB at Precision $0.5327$, Recall $0.6108$, and F-measure $0.5691$ [1604.00470].

Standard detection metrics include Precision, Recall, F-measure/Hmean, Average Precision, and IoU [2406.16307]. For ASMTD, the reported benchmark results are:
- MSRA-TD500: $P=89.8$, $R=83.1$, $F=86.3$, $FPS=62.5$
- Total-Text: $P=88.5$, $R=83.8$, $F=86.1$, $FPS=70.9$
- CTW1500: $P=87.8$, $R=80.3$, $F=83.9$, $FPS=72.1$ [2111.09560]

Ablation studies are central to evaluating overlay detectors because failure cases are strongly tied to architecture. Removing CCA reduces recall on curved or irregular text; replacing RDBs with plain convolutions increases false positives on textured backgrounds; omitting boundary discrimination necessitates complex post-processing and degrades contour accuracy; and restricting the pyramid to $P3$–$P5$ rather than $P2$–$P5$ affects small-text performance [2406.16307]. In ASMTD, the baseline shrink-mask-only model reaches $F=84.3$ on MSRA-TD500, adding ASM raises it to $85.6$, and adding SPW raises it to $86.3$ without affecting inference speed [2111.09560].

A methodological pattern emerges across these datasets and metrics. Overlay-specific benchmarks do not merely substitute different images for scene-text data; they encode assumptions about typography, layout, persistence, and semantic role that materially affect both architecture and evaluation.

## 6. Multimodal robustness, semantic interference, and future directions

Artificial text overlay detection has acquired a semantic dimension in the study of multimodal hallucination. VisualTextTrap formalizes Text Overlay-Induced Hallucination (TOIH) and constructs a benchmark of $6{,}057$ samples annotated across $88$ fine-grained attributes within four dimensions—Temporal, Action, Object, and Spatial—with hallucination intensity quantified on a five-level scale $L1$–$L5$ [2604.17375]. Three overlay conditions are defined: Text-Free ($V_{\text{free}}$), Text-Congruent, and Text-Contradictory ($V_{\text{contra}}$).

For a multiple-choice video-QA instance $(v,q,A)$, TOIH is characterized by
$$
P_\theta(a^* \mid v', q) \ll P_\theta(a^* \mid v, q), \qquad
P_\theta(a_d \mid v', q) > P_\theta(a_d \mid v, q).
$$
The per-sample confidence shifts are
$$
\Delta P_i(y_i) = P_i(y_i \mid V_{\text{contra}}) - P_i(y_i \mid V_{\text{free}}),
$$
$$
\Delta P_i(o_i) = P_i(o_i \mid V_{\text{contra}}) - P_i(o_i \mid V_{\text{free}}).
$$
The dominant regime is $\Delta P_i(o_i) > 0$ and $\Delta P_i(y_i) < 0$, meaning confidence mass moves from the correct answer to the overlay-induced option [2604.17375].

The benchmark introduces metrics that explicitly quantify overlay-induced failure. Hallucination Resistance Rate is
$$
\mathrm{HRR} = \mathbb{E}_{i \in V_{\text{contra}}} \mathbf{1}[C_i].
$$
Semantic Conflict Sensitivity Index is
$$
\mathrm{SCSI} = \frac{\mathbb{E}_{i \in V_{\text{contra}}} \mathrm{SCS}_i \cdot H_i}{\mathbb{E}_{i \in V_{\text{contra}}} H_i}.
$$
Additional measures include Visual Yielding Rate, Hallucination Acceptance Rate, Interference Cost Ratio, Synergy Gain/Loss Index, Text-Induced Hallucination Rate, Textual Induction Bias, Weighted Hallucination Rate, Hallucination Surge Rate, and a Hallucination Resistance Curve per conflict level [2604.17375]. These metrics move beyond localization accuracy by evaluating whether systems can resist semantically misleading overlays.

The proposed mitigation model, VTHM-MoE, is a Vision-Text Disentanglement framework with dual encoders:
$$
F_{\mathrm{vis}} = \mathcal{E}_{\mathrm{vis}}(V) \in \mathbb{R}^{N \times d}, \qquad
F_{\mathrm{ocr}} = \mathcal{E}_{\mathrm{ocr}}(V) \in \mathbb{R}^{N \times d},
$$
where $N=32$ patches and $d=4096$. Query-guided patch selection uses
$$
s = \mathrm{CrossAttn}(q_{\mathrm{vis}}, F_{\mathrm{vis}}) \in \mathbb{R}^N, \qquad
\mathcal{I}_K = \mathrm{TopK}(s).
$$
For each selected patch, the model forms a three-token representation
$$
T_k = \big[\hat{f}^{\mathrm{vis}}_k;\ \hat{f}^{\mathrm{ocr}}_k;\ \Delta_k\big], \qquad
\Delta_k = \hat{f}^{\mathrm{ocr}}_k - \hat{f}^{\mathrm{vis}}_k.
$$
Cross-modal consistency is measured by
$$
c_k = \cos(h^{\mathrm{vis}}_k, h^{\mathrm{ocr}}_k),
$$
and routing logits are defined as
$$
g = \mathrm{Gate}(h) + (1-\omega)\cdot \mathrm{softmax}(\mathrm{Cls}(h)), \qquad
\omega = 1 - c_k^2.
$$
Four dimension-specialized experts—Temporal, Action, Object, and Spatial—are inserted at the output of the 16th transformer layer:
$$
H^{(16)} \leftarrow H^{(16)} + \mathrm{MoE}(H^{(16)}).
$$
The overall loss is
$$
\mathcal{L} = \mathcal{L}_{\mathrm{lm}} + \lambda_{\mathrm{cls}}\mathcal{L}_{\mathrm{cls}} + \lambda_{\mathrm{sft}}\mathcal{L}_{\mathrm{sft}} + \lambda_{\mathrm{aux}}\mathcal{L}_{\mathrm{aux}},
$$
with $\lambda_{\mathrm{cls}} = 1.1$, $\lambda_{\mathrm{sft}} = 1.0$, and $\lambda_{\mathrm{aux}} = 0.01$ [2604.17375].

Reported clean-benchmark accuracies for VTHM-MoE are $77.7\%$ on LLaVA-Video, $61.4\%$ on VideoMME, and $53.1\%$ on TemporalBench, while the model also improves resistance under contradictory overlays [2604.17375]. The paper identifies temporal contradictions as the hardest and action contradictions as easier because short, localized motion facilitates visual grounding.

This multimodal line of work reframes artificial text overlay detection as part of a larger control problem: systems must first detect the presence and location of overlays, then determine whether the overlay is helpful, irrelevant, or contradictory. A plausible implication is that future overlay detectors will increasingly couple geometric localization with semantic calibration, especially in pipelines where OCR outputs are consumed by reasoning models.

Current limitations remain domain-dependent. Extreme artistic distortions and very low contrast text remain difficult in poster imagery; small fonts, curved or rotated text, heavy motion blur, and animated overlays remain difficult in social video; low-contrast overlays, translucent backgrounds, and scrolling tickers remain difficult in broadcast video; and OCR errors or missed detections can attenuate discrepancy signals in multimodal systems [2406.16307][1804.10687][1604.00470][2604.17375]. Reported future directions include curved-text modeling with Bezier or Fourier representations, transformer backbones for richer global context, stronger OCR and text-region segmentation, temporal consistency models for video overlays, style-diverse synthetic generation, and conflict-aware pre-training for multimodal robustness [2406.16307][2111.09560][2604.17375].

Source: https://www.emergentmind.com/topics/artificial-text-overlay-detection