WeText: Weakly Supervised Scene Text Detection

• The paper introduces WeText, which mitigates the need for extensive character-level annotations by bootstrapping from a small fully labeled set and mining additional data from weakly labeled corpora.
• It employs a unified SSD-based regression detector that streamlines detection and reduces error accumulation typical in multi-stage pipelines.
• Results demonstrate significant improvements on benchmarks like ICDAR 2013, with weak supervision boosting text-line F-scores to around 86.9%.

Searching arXiv for the primary WeText paper and closely related scene-text work for citation support. WeText is a weakly supervised scene text detection framework that addresses two persistent constraints in character-based scene text detection: the cost of obtaining large amounts of character-level annotations and the error accumulation typical of multi-stage pipelines. It is designed to train robust and accurate scene text detection models from a small fully annotated dataset together with larger unannotated or weakly annotated corpora, while using a unified regression-based character detector to reduce the proposal-generation and classification fragmentation of earlier systems (Tian et al., 2017).

1. Problem setting and motivation

Scene text detection aims to locate text in natural images, where text in the wild varies in font, size, style, orientation, perspective distortion, background clutter, lighting, language, and script structure (Tian et al., 2017). Within this setting, WeText adopts a character-based formulation. The paper argues that character-based detection is especially attractive for multilingual and multi-oriented text because it is not restricted to horizontal word boxes or explicit word boundaries.

The same formulation, however, exposes two practical weaknesses. First, character-based methods need many annotated character boxes, and these are expensive and time-consuming to create. Second, they are typically multi-stage systems that detect character candidates, classify them as text or non-text, and then group them into words or text lines. The paper identifies this decomposition as a source of error accumulation.

WeText is introduced as a response to both issues. Its central premise is that a “light” supervised model trained on a small fully annotated dataset can be used to search for additional character samples in much larger external collections, including either unannotated images or images with only word/text-line annotations. The framework therefore treats limited full supervision as a seed for broader data exploitation rather than as the sole training resource.

2. Supervision regimes and the bootstrapping framework

WeText contrasts three supervision regimes: fully supervised, semi-supervised, and weakly supervised (Tian et al., 2017). In the fully supervised setting, all training images have character-level annotations. In the experiments, the ICDAR 2013 training set provides only 229 fully annotated images, and these are used to train the initial detector. In the semi-supervised setting, a small labeled set $D$ is paired with a large unlabeled set $R$. In the weakly supervised setting, the same $D$ is paired with a large weakly labeled set $R'$, where the extra annotations are only at the word or text-line level, denoted as $G = \{g_1, g_2, \dots\}$.

The operational structure is a bootstrapping pipeline. A light supervised character detector is first trained on $D$. That detector is then applied to external data to mine additional character samples. The newly mined positives are merged with the original fully labeled set, and the detector is retrained. The paper studies this procedure in both semi-supervised and weakly supervised variants, and it also evaluates iterative refinement by repeating the mining-and-retraining cycle.

A recurrent misconception is that weak supervision in WeText is annotation-free. It is not. Semi-supervised learning uses no labels in the additional data, whereas weakly supervised learning uses coarse word/text-line boxes. Those higher-level annotations do not provide character boxes, but they do constrain the mining stage and make character discovery more reliable.

Another point of terminology is that the initial model is described as “light” supervised not because it is a small network, but because it is trained only from limited annotation. This distinction matters because the underlying detector remains a modern deep detection architecture rather than a deliberately lightweight model.

3. Unified scene character detector

A major component of WeText is the unified scene character detector, which replaces the traditional candidate-generation-plus-classification pipeline with a proposal-free regression-based detector adapted from SSD (Single Shot MultiBox Detector) (Tian et al., 2017). The paper frames this architectural choice as a direct response to error accumulation in conventional character-based methods.

The detector uses a VGG-16 backbone. The last fully connected layers are converted into convolution layers, and extra convolution layers are added for multi-scale prediction. Predictions are made using $3 \times 3$ filters over multiple feature maps. At each feature location and for each default anchor, the network predicts four box offsets and two class scores, corresponding to text versus background. Each anchor therefore outputs six values.

This unification matters because candidate generation and text/non-text classification are not treated as separate stages. The network directly predicts localization and confidence, which the paper argues improves robustness and efficiency. During inference, non-maximum suppression is applied with a Jaccard overlap threshold of $0.45$. Because characters are dense, the system keeps the top 1000 candidates before NMS and the top 500 detections after NMS.

The design is not presented as a new loss construction. Instead, the paper adopts SSD-style regression-based detection and concentrates its methodological novelty on how a strong detector can be improved through sample mining from weakly or unlabeled data.

4. Semi-supervised and weakly supervised sample mining

The mining stage is the mechanism by which WeText converts limited fully labeled supervision into a broader training signal (Tian et al., 2017). Given a labeled set $D$, an unlabeled set $R$, and an initial model $R$0, the semi-supervised variant applies $R$1 to $R$2 and obtains character candidates

$R$3

where $R$4 is a detected box and $R$5 is its confidence. A candidate is selected as positive when

$R$6

The paper notes a threshold trade-off: if $R$7 is too high, too few samples are mined; if $R$8 is too low, many false positives are included. It reports $R$9 to $D$0, with $D$1 used in experiments.

The weakly supervised variant uses the same initial detector but constrains candidate selection with word or text-line boxes $D$2. A candidate is selected only if its confidence exceeds a lower threshold $D$3 and its horizontal and vertical overlap ratios with a valid text region exceed $D$4 and $D$5, with $D$6. Because the weak annotations filter out many false positives outside text regions, the confidence threshold can be lower; the paper reports $D$7 and uses $D$8.

This asymmetry between $D$9 and $R'$0 is central. Weak supervision improves candidate mining in two ways: it reduces false positives and allows more true characters to be recovered at lower confidence. The paper therefore characterizes the mined weakly supervised data as larger, cleaner, and more diverse than the data mined in the purely semi-supervised setting.

After character detection, WeText groups detected characters into text lines using TextFlow, a graph-based min-cost flow grouping method. At this stage, the system uses a low confidence threshold of $R'$1, since the graph-based grouping removes many false positives.

5. Datasets, evaluation protocol, and empirical results

The experiments use four datasets: ICDAR 2013, FORU, COCO-Text, and SWT (Tian et al., 2017). ICDAR 2013 provides 229 training images with character annotations and 233 test images. FORU uses the English2k subset with 1162 images and 14,888 annotated characters, together with both character-level and word-level boxes. COCO-Text contributes 14,712 training images with at least one legible English text region and only word-level boxes. SWT contains 307 images and is described as challenging because of clutter, low contrast, and small text.

Evaluation uses precision, recall, and F-score, with character detection using IoU $R'$2 as the positive match criterion. On ICDAR 2013 character detection, the baseline trained only on the 229 fully annotated ICDAR images reaches recall 84.80, precision 61.44, and F-score 71.26. Semi-supervised retraining improves over this baseline, while weak supervision improves further and approaches full supervision.

Model	Character F-score	Text-line F-score
Baseline	71.26	82.3
FORU_Semi	73.05	83.4
FORU_Weakly	75.37	85.4
FORU_GT	78.02	86.3
COCO-Text_Semi	74.91	84.2
COCO-Text_Weakly	78.38	86.9

The text-line results on ICDAR 2013 are especially notable. The weakly supervised model using COCO-Text reaches an F-score of $R'$3, which the paper describes as state-of-the-art performance, despite using only 229 fully annotated images for the initial supervised training. The paper further states that compared to TextBoxes single-scale, WeText is better by about 6 F-score points, and it remains about 1 point better than multi-scale TextBoxes.

On SWT, the same pattern holds. The baseline reaches 53.9 F-score, while FORU_Weakly reaches 57.1, FORU_GT reaches 58.9, and COCO-Text_Weakly reaches 59.8. These results support the claim that weak supervision consistently outperforms semi-supervision and can approach the quality of full supervision when the weakly labeled corpus is large enough.

The method is also reported as efficient: character detection takes about 0.19 s/image, text line extraction about 0.13 s/image, and the full pipeline about 0.32 s/image on a Titan X GPU.

6. Iterative refinement, significance, and limitations

WeText includes an iterative learning analysis in which the detector mines positives, retrains, and then repeats the cycle (Tian et al., 2017). On FORU, the second round improves FORU_Semi_TL from 83.4% to 84.3% and FORU_Weakly_TL from 85.4% to 86.2%. After the third round, improvement saturates. This suggests that iterative bootstrapping is useful but exhibits diminishing returns as the detector approaches the quality obtainable under full supervision.

The framework’s significance lies in the interaction of three design choices: bootstrapping from a small clean seed, using weak labels to constrain sample mining, and adopting a proposal-free SSD-based character detector. The paper’s strongest empirical message is not merely that additional data help, but that weakly labeled word/text-line boxes provide enough structure to make mined character samples substantially more useful than those obtained from unlabeled data alone.

Several limitations are stated explicitly. The method still struggles with handwriting, ultra-low-contrast text, and some very difficult cases. Its performance depends on the availability of enough unlabeled or weakly labeled data. It is centered on character-level detection followed by graph grouping, so it still requires post-processing to produce final text lines. These caveats prevent weak supervision from being equivalent to full end-to-end text-line supervision, even when the numerical gap becomes small.

A broader interpretive point follows from the reported results. WeText shows that high-performing scene text detection does not require exhaustive character-level annotation across all training data, provided that a sufficiently strong initial detector and a disciplined mining strategy are available. In that sense, it reframes weak supervision not as a minor annotation shortcut but as a viable training regime for character-based scene text detection.