TextBoxes++: Robust Scene Text Detector

• TextBoxes++ is an end-to-end trainable scene text detector designed to accurately localize horizontal and oriented text in natural images.
• It employs a modified VGG-16 backbone with additional convolutional layers and elongated filters to capture diverse text shapes at multiple scales.
• By leveraging specialized default boxes and a cascaded non-maximum suppression strategy, it achieves state-of-the-art performance on various benchmark datasets.

TextBoxes++ is an end-to-end trainable, single-shot scene text detector designed to address the challenges unique to text detection in natural images, including arbitrary orientations, small text regions, and extreme aspect ratio variability. Building upon the SSD framework, TextBoxes++ directly predicts both horizontal and oriented boxes (quadrilaterals or rotated rectangles) for robust, high-speed text localization. Its architecture, specialized default boxes, loss functions, and post-processing pipeline are specifically adapted to the geometric complexity of scene text, achieving leading accuracy and efficiency on multiple public benchmarks (Liao et al., 2018).

1. Architectural Design

The backbone of TextBoxes++ is VGG-16, reconfigured by converting the two fully connected layers (fc6, fc7) into convolutional layers (conv6, conv7), and extending the network with four additional convolutional stages (conv8–conv11) that progressively decrease in spatial resolution. Six text-box prediction layers are appended to layers conv4_3, conv7, conv8_2, conv9_2, conv10_2, and conv11_2, each responsible for generating predictions at different scales.

Each feature map location predicts for a set of predefined horizontal "default boxes." For each default box, the network outputs a two-way text/non-text class score, regression offsets for the minimum enclosing horizontal rectangle, and offsets for either a quadrilateral (predicting four corner coordinates) or a rotated rectangle (top two edge points plus height).

The convolutional layers utilize elongated 3×5 rectangular filters on select layers to capture the highly elongated shapes typical of scene text. Additionally, dense vertical placement is implemented: vertical offsets are introduced by shifting default boxes by half a cell height in the vertical direction, improving coverage for tightly packed text lines.

2. Oriented-Box Parameterization and Regression

Default or "anchor" boxes are parameterized as $(x_0, y_0, w_0, h_0)$ in center-size form, with further representation as either four corners for quadrilaterals or two edge points plus height for rotated rectangles. The network regresses the following offsets:

• Horizontal Rectangle:
  • $\Delta x = (x - x_0)/w_0$
  • $\Delta y = (y - y_0)/h_0$
  • $\Delta w = \log(w/w_0)$
  • $\Delta h = \log(h/h_0)$
• Quadrilateral (for $n=1,\dots,4$):
  • $\Delta x^{(q)}_n = (x^{(q)}_n - x^{(q)}_{0n})/w_0$
  • $\Delta y^{(q)}_n = (y^{(q)}_n - y^{(q)}_{0n})/h_0$
• Rotated Rectangle (for $n=1,2$):
  • $\Delta x^{(r)}_n = (x^{(r)}_n - x^{(r)}_{0n})/w_0$
  • $\Delta x = (x - x_0)/w_0$0
  • $\Delta x = (x - x_0)/w_0$1

At inference, predicted offsets are decoded to yield the final oriented boxes by applying the inverse transforms to the default box parameters.

3. Default Boxes and Prediction Strategy

TextBoxes++ employs default boxes with aspect ratios $\Delta x = (x - x_0)/w_0$2 to encompass the exceptionally wide (long horizontal text) and tall (vertical text) instances in natural images. Default boxes are generated at six feature map layers with spatial strides of $\Delta x = (x - x_0)/w_0$3 pixels, enabling detection of text at varied scales. The inclusion of vertically offset default boxes (shifted by half the cell height) doubles the density along the vertical axis, improving recall for closely stacked lines.

Each location in the feature maps predicts for every assigned default box:

• Text/non-text class scores
• Horizontal rectangle offsets
• Oriented box offsets (quadrilateral or rotated rectangle)

4. Loss Function and Training Protocol

The network optimizes a multi-task loss:

$\Delta x = (x - x_0)/w_0$4

where $\Delta x = (x - x_0)/w_0$5 is the number of positive matches, $\Delta x = (x - x_0)/w_0$6 is the 2-way softmax classification loss, $\Delta x = (x - x_0)/w_0$7 is a Smooth $\Delta x = (x - x_0)/w_0$8 loss over all regression dimensions, and $\Delta x = (x - x_0)/w_0$9 balances the two terms.

A default box is considered positive if its IoU (Intersection-over-Union) with any ground-truth horizontal box is at least $\Delta y = (y - y_0)/h_0$0, or if it is the best match for a ground truth. Data augmentation includes random-crop with Jaccard-overlap or object-coverage constraints ($\Delta y = (y - y_0)/h_0$1), with thresholds chosen randomly. Training uses multi-scale crops: $\Delta y = (y - y_0)/h_0$2 for most iterations, $\Delta y = (y - y_0)/h_0$3 during later training, promoting robustness to variations in text size.

5. Inference and Post-Processing Pipeline

Inference is executed in a single forward pass, producing outputs at each prediction layer proportional to the number of default boxes and regression dimensions. The post-processing consists solely of cascaded non-maximum suppression (NMS):

1. Fast NMS is first applied to the minimum enclosing horizontal rectangles, discarding proposals with IoU $\Delta y = (y - y_0)/h_0$4.
2. Precise NMS is subsequently performed on oriented quadrilaterals (or rotated rectangles) using a lower threshold ($\Delta y = (y - y_0)/h_0$5).

No other post-processing is required. This streamlined approach maintains both high speed and localization accuracy.

6. Benchmark Results and Quantitative Performance

TextBoxes++ achieves leading results on several public datasets. Notable results include:

Dataset	Single-Scale F-measure	Multi-Scale F-measure	FPS
ICDAR 2015 (1024²)	0.817	0.829	11.6 / ~2.3
COCO-Text (768²)	0.559	0.587	19.8
ICDAR 2013	0.80	0.88	–
SVT	0.64 (spotting)	–	–

Direct comparison against EAST (Pyramid PVANET2×, F 0.782 @ 13.2 fps) and Shi et al. (F 0.750 @ 8.9 fps) on ICDAR 2015 demonstrates superior or at least competitive performance. Multi-scale evaluation further boosts F-measure at the expense of computational speed (Liao et al., 2018).

7. End-to-End Text Detection and Recognition

TextBoxes++ can be seamlessly integrated with a CRNN-based text recognizer, which produces a word recognition probability $\Delta y = (y - y_0)/h_0$6. The combined detection-recognition score is computed using the harmonic mean of the exponentiated detection and recognition confidences:

$\Delta y = (y - y_0)/h_0$7

where $\Delta y = (y - y_0)/h_0$8 is the detector confidence and $\Delta y = (y - y_0)/h_0$9 is the recognition score (maximum over a lexicon if provided, or $\Delta w = \log(w/w_0)$0 if not).

With this pipeline, TextBoxes++ achieves F-measures of $\Delta w = \log(w/w_0)$1 (spotting) and $\Delta w = \log(w/w_0)$2 (end-to-end) using a strong lexicon on ICDAR 2015, surpassing prior systems by at least $\Delta w = \log(w/w_0)$3 percentage points; on ICDAR 2013 (strong lexicon), scores are $\Delta w = \log(w/w_0)$4 (spotting) and $\Delta w = \log(w/w_0)$5 (end-to-end). Joint detection and recognition raises the detection F-score on ICDAR 2015 from $\Delta w = \log(w/w_0)$6 to approximately $\Delta w = \log(w/w_0)$7, demonstrating the benefit of synergy between detection and recognition modules.

TextBoxes++ demonstrates that a single-shot detection approach, when tailored for the special geometry of scene text and with appropriate post-processing, can provide state-of-the-art speed and accuracy for both multi-oriented and horizontal text benchmarks, and can be deployed in practical end-to-end text understanding systems (Liao et al., 2018).