---
title: 'TextOCR: Real-Image Scene Text Benchmark'
url: https://www.emergentmind.com/topics/textocr
type: topic
---

# TextOCR: Real-Image Scene Text Benchmark

TextOCR is a large-scale real-image scene text dataset and benchmark for arbitrary-shaped text detection and recognition, collected on images from the TextVQA dataset and annotated at word level with polygons and transcriptions [2105.05486]. It was introduced to support text detection, text recognition, end-to-end text spotting, and scene text based reasoning in tasks such as TextVQA and TextCaps, and it has subsequently been used not only as a benchmark for OCR systems but also as a training substrate for text spotting, motion-blur restoration, and text-aware image compression [2105.05486] [2511.06087] [2603.04115].

## 1. Origin and motivation

TextOCR was introduced to address three gaps identified in prior scene text research: the lack of large real-world, text-dense OCR datasets; insufficient support for arbitrary-shaped, high-density text; and the absence of ground-truth scene text annotations for TextVQA and TextCaps images [2105.05486]. Earlier resources such as Synth90k and SynthText were large but synthetic, while real datasets such as ICDAR variants, COCO-Text, and Total-Text were smaller, often collected for other purposes, or less suitable for arbitrary-shaped and text-dense scenes. TextOCR instead uses the images from TextVQA, which were selected because scene text is central to the semantics of the image, so that OCR can be evaluated in isolation and also integrated with downstream reasoning [2105.05486].

The benchmark is explicitly oriented toward arbitrary-shaped scene text detection and recognition on real images. It supports text detection, text recognition, end-to-end text spotting, and downstream reasoning in which recognized text is consumed by visual question answering and captioning systems [2105.05486]. In later work, this role broadens further: GLASS treats TextOCR as a stress test for large-scale end-to-end scene text spotting under broad scale variation and arbitrary orientations, and DeepSolo uses TextOCR as a large-scale pretraining source for Transformer-based text spotting [2208.03364] [2211.10772].

## 2. Dataset composition and annotation design

TextOCR contains 28,134 images and 903,069 non-empty words, with 1,320,064 labeled instances in total [2105.05486]. The split reported in the original benchmark is as follows.

| Split | Images | Non-empty words |
|---|---:|---:|
| Train | 21,749 | 714,186 |
| Validation | 3,153 | 108,386 |
| Test | 3,232 | 80,497 |

The dataset is unusually dense: it has 32.1 words per image on average, and more than 10% of images contain 100+ words [2105.05486]. Text instances are distributed across documents, posters, product packaging, screens, charts, street scenes, and other real photographs inherited from TextVQA and OpenImages, rather than synthetic renderings [2105.05486].

TextOCR is explicitly designed for arbitrary-shaped text. It includes 150,197 rotated words with rotation angle greater than \(20^\circ\), and 19,310 curved words where the polygon uses more than four points [2105.05486]. Annotation is polygon-based rather than box-only: each word region is represented by an ordered polygon, and the edge from the first to second point indicates the top-left reading direction. Horizontal or near-horizontal words are often annotated with rectangles, while curved or irregular words use multi-point polygons [2105.05486].

For recognition, only Latin-script words are transcribed. Non-Latin or illegible words are labeled with “.”, treated as an empty word [2105.05486]. This design makes the dataset predominantly English and Latin-script for recognition, while still preserving dense detection supervision over the whole image. The annotation interface supports arbitrary polygons, zoom and re-annotation, and case-sensitive transcription, and expert auditors reviewed annotations to enforce precise boundaries and minimal background inclusion [2105.05486].

## 3. Benchmark tasks and difficulty profile

TextOCR supports two principal OCR tasks. The first is word-level text recognition, where the input is a cropped word image and the metric is exact-match word accuracy. The second is end-to-end text spotting, where the model must both localize and transcribe text in the full image; evaluation follows ICDAR2015-style matching with polygonal or box overlap and reports Precision, Recall, and F-measure, with
\[
F = \frac{2 \cdot P \cdot R}{P + R}.
\]
The overlap term is expressed through IoU between predicted and ground-truth regions [2105.05486].

The dataset is difficult not merely because of scale, but because of scene structure. Heatmap analysis in the original benchmark shows that TextOCR text instances are more uniformly distributed across the image than COCO-Text, ICDAR15, and Total-Text [2105.05486]. GLASS characterizes TextOCR as a large-scale, arbitrary-shaped scene-text benchmark with natural scenes, broad scale variation from very small to very large text, arbitrary orientations, cluttered backgrounds, and lexicon-free evaluation, making it a more representative test of unconstrained scene text spotting [2208.03364].

This difficulty persists even for strong models. The original paper reports that current state-of-the-art OCR models trained only on synthetic or other real datasets perform poorly on TextOCR relative to more conventional benchmarks, and that even after TextOCR fine-tuning, TextOCR remains harder than ICDAR15 or Total-Text for the same architectures [2105.05486]. A plausible implication is that TextOCR probes several failure modes simultaneously: dense instance layout, arbitrary geometry, clutter, and the need for lexicon-free recognition.

## 4. Effects on OCR and text spotting models

TextOCR was introduced not only as a benchmark but also as a training set that substantially improves OCR systems. In word-level recognition experiments, the TPS-ResNet-BiLSTM-Attn model trained on COCO-Text achieved 43.20% on the TextOCR test set, whereas the same architecture trained on TextOCR achieved 65.65%; the TextOCR-trained model also improved on COCO-Text and on irregular-text benchmarks such as ICDAR15, SVTP, and CUTE80 [2105.05486]. When models pre-trained on Synth90k and SynthText were fine-tuned on TextOCR, the TextOCR test accuracy rose from 43.07% to 58.61% for CRNN, from 43.16% to 60.85% for Rosetta, from 48.23% to 66.84% for STAR-Net, and from 50.37% to 69.49% for TPS-ResNet-BiLSTM-Attn [2105.05486].

The same effect appears in end-to-end text spotting. Mask TextSpotter v3 trained on COCO-Text achieved 32.5 F-measure on TextOCR, while the same model trained on TextOCR achieved 45.8; fine-tuning the official Mask TextSpotter v3 on TextOCR increased TextOCR F-measure from 34.9 to 50.8 and also improved Total-Text and COCO-Text results [2105.05486]. This supports the claim in the original benchmark that TextOCR helps achieve state-of-the-art performance on multiple OCR datasets as well as exposing weaknesses of prior models on real, text-heavy images.

Later spotters treat TextOCR as a decisive benchmark. GLASS reports end-to-end Hmean of 67.1 on the TextOCR test set, compared with 50.8 for Mask TextSpotter v3, and attributes the improvement to global-to-local attention and an orientation-aware rotated box loss that are especially useful under scale and angle extremities [2208.03364]. DeepSolo, a DETR-like text spotter with explicit point queries and a single decoder, uses TextOCR in extended pretraining and reports substantial gains on Total-Text, ICDAR 2015, and Inverse-Text when TextOCR is added to the training mixture [2211.10772]. This suggests that TextOCR functions simultaneously as a benchmark of scene-text difficulty and as a source of supervision that transfers to other datasets.

## 5. Role in scene-text reasoning

A distinctive feature of TextOCR is its direct linkage to TextVQA and TextCaps. Because the images are drawn from TextVQA, the benchmark makes it possible to separate OCR errors from reasoning errors and to train end-to-end systems in which OCR is part of the scene-text reasoning pipeline rather than a fixed external component [2105.05486].

The original paper uses a TextOCR-trained OCR model to build PixelM4C, an end-to-end model for scene text based reasoning. Relative to the earlier M4C setup using Rosetta OCR, PixelM4C with TextOCR-trained Mask TextSpotter v3 improves TextVQA validation accuracy, reaching 41.23% with the full TextOCR OCR configuration and 42.12% when ST-VQA is added as additional training data; the ground-truth TextOCR upper-bound setting reaches 48.04% with ST-VQA, while human accuracy is 85.01% [2105.05486]. For TextCaps, PixelM4C-Captioner with TextOCR-trained OCR improves BLEU-4 from 23.30 to 24.10 and CIDEr from 89.60 to 91.99 relative to the Rosetta-based baseline [2105.05486].

These results clarify TextOCR’s dual status. It is an OCR benchmark in the conventional sense, but it is also an enabling layer for multimodal reasoning systems that attend to recognized tokens, OCR features, and object features jointly. The benchmark therefore supports analysis at two levels: recognition quality itself, and the effect of recognition quality on downstream question answering and caption generation [2105.05486].

## 6. TextOCR as substrate for restoration and compression

TextOCR has also been repurposed beyond recognition. The motion-blur restoration work "Hybrid CNN-ViT Framework for Motion-Blurred Scene Text Restoration" constructs a TextOCR-derived benchmark by taking a subset of 5,000 sharp TextOCR images, generating synthetically blurred counterparts with realistic motion-blur kernels from Shen et al., and assembling 6,000 strictly aligned blurred-sharp pairs split into 5,000 training, 500 validation, and 500 test samples [2511.06087]. Images are resized to \(256 \times 256 \times 3\), normalized to \([0,1]\), and paired one-to-one with blurred versions; TextOCR’s annotations are not used directly for restoration training, but restored outputs remain compatible with the original text labels for downstream OCR or detection [2511.06087]. On this TextOCR-derived blur benchmark, the proposed model reports 32.20 dB PSNR, 0.934 SSIM, 2.83 million parameters, and an average inference time of 61 ms [2511.06087].

TextOCR also appears in ultra-low bitrate compression research. TextBoost fine-tunes its text-aware stage on the TextOCR training set and evaluates on the TextOCR validation split, using text detection F1, end-to-end recognition F1, PSNR, MS-SSIM, LPIPS, and bpp as metrics [2603.04115]. It exploits TextOCR annotations in two ways: binary text masks built from word-level boxes for the guidance-consistent loss
\[
\mathcal{L}_{\text{gc}} = \mathrm{MSE}\big(m \odot x,\; m \odot \hat{x}\big),
\]
and size-based filtering of small text through average character area thresholds \(T_{\text{train}} = 150\) on \(256 \times 256\) crops and \(T_{\text{test}} = 800\) on full-resolution images [2603.04115]. On TextOCR at approximately 0.033 bpp, TextBoost reports DET F1 \(= 0.404\), while the best baseline ELIC reports DET F1 \(= 0.2515\), corresponding to the paper’s stated relative improvement of up to 60.6% higher text-recognition F1 at comparable PSNR and bpp [2603.04115].

These later uses show that TextOCR has become a general-purpose substrate for scene-text fidelity research. This suggests that its value lies not only in transcription supervision, but also in the combination of dense real-image text, arbitrary geometry, and word-level spatial annotations.

## 7. Terminological extensions and adjacent systems

Although TextOCR originally denotes the 2021 dataset and benchmark, later literature sometimes uses the term more broadly. DTrOCR explicitly describes “TextOCR” as the general problem of recognizing text from images—scene text on signs and storefronts, printed text in documents and receipts, and handwriting—and proposes a decoder-only Transformer that consumes image patches as tokens and autoregressively generates text [2308.15996]. In that usage, TextOCR names a task family rather than a particular benchmark.

A more explicit terminological shift appears in "Text images processing system using artificial intelligence models", where “TextOCR” is used to denote an end-to-end processing system rather than the benchmark itself [2512.11691]. That system converts images to grayscale, applies `RealESRGAN_x2` with tiled inference and CLAHE, detects text with DBNet++ through differentiable binarization and an FPN over ResNet-50, and classifies extracted content into Invoice, Form, Letter, or Report with `facebook/bart-large-mnli`, reporting about 94.62% text recognition accuracy on Total-Text over approximately 10 hours of testing [2512.11691]. This broader usage reflects a pipeline interpretation of TextOCR.

A separate but similarly named line of work is TexOCR, which concerns page-level reconstruction of scientific PDFs into compilable LaTeX rather than scene text spotting [2604.22880]. TexOCR-Bench contains 2,135 expertly annotated documents, TexOCR-Train contains 404K page-level pairs derived from 57K arXiv papers, and the 2B-parameter TexOCR model is trained with supervised fine-tuning and reinforcement learning with verifiable rewards for compilability and referential integrity [2604.22880]. Despite the lexical resemblance, TexOCR addresses document-level page-to-LaTeX reconstruction, not the scene-text dataset introduced in 2021.

Accordingly, the most precise usage distinguishes three layers: TextOCR as the 2021 arbitrary-shaped scene text benchmark [2105.05486]; TextOCR as a broader shorthand for scene text OCR pipelines in some later papers [2512.11691] [2308.15996]; and TexOCR as a separate document OCR program centered on compilable LaTeX reconstruction [2604.22880].

Source: https://www.emergentmind.com/topics/textocr