Street View Text Dataset

• Street View Text Datasets are extensive collections of urban images annotated for text detection, recognition, segmentation, and geo-localization, featuring diverse scripts and challenging conditions.
• They integrate fully supervised benchmarks (e.g., FSNS, SVHN) with partially supervised corpora (e.g., LSVT, C-SVT) to support varied evaluation metrics and end-to-end text spotting tasks.
• These datasets drive advancements in scene text recognition and vision-language models by addressing real-world challenges such as noise, occlusion, and cross-modal alignment.

Street View Text Datasets are large-scale, systematically annotated image corpora designed to support the detection, segmentation, recognition, and structured analysis of text appearing in urban and suburban outdoor environments, as captured by vehicle- or pedestrian-mounted cameras. Such datasets include both classic benchmarks (e.g., French Street Name Signs, Street View House Numbers) and recent, multi-modal corpora featuring partial or weak supervision, diverse scripts, or explicit geographic metadata. Fundamental for evaluating and improving Scene Text Recognition (STR), text spotting, and geo-localization systems, these datasets underpin much of the advancement in vision-language research for urban perception.

1. Defining Street View Text Datasets: Scope and Key Characteristics

Street View Text Datasets are composed of images captured in situ along public streets, typically from sources such as Google Street View, Tencent, Mapillary, or crowdsourcing platforms like Flickr and mobile uploads. Their primary intent is to enable robust text detection and recognition under challenging real-world conditions: arbitrary orientations, diverse scripts, varied lighting, occlusions, clutter, and wide scene variability.

Crucial characteristics include:

• Annotation granularity: Instance-level polygons/quadrangles (bounding boxes for text lines/words/characters) and full or partial transcriptions.
• Language/script coverage: Latin, Cyrillic, Chinese, and other scripts; multi-language support is increasingly prevalent.
• Diversity: Images span urban, suburban, and rural settings, capturing commercial/residential facades, signage, street furniture, vehicles, billboards, and other sources of visible text.
• Task alignment: Datasets are constructed for detection (localization), recognition, end-to-end text spotting, as well as for downstream structured information extraction and cross-modal geo-localization.

The following table provides exemplars distinguished by region, language, annotation, and main task.

Dataset	Region / Language	Annotation	Main Task
FSNS	France / French	Tile-level text	Sign name recognition, sequence OCR
SVHN	US / numeric (house #)	Box, sequence	Multi-digit number recognition
LSVT	China / Chinese, mixed	Polygon/keyword	Detection, end-to-end text spotting
CTW, ShopSign	China / Chinese	Char./line-box	Single-character recognition/detection
Berlin Flickr	Germany / DE+EN	Auto-poly+transc	Building attribute mapping/GIS fusion
CVG-Text	US/AU/JP / EN	Free-form descr.	Text-guided cross-view geo-localization

2. Major Datasets: Composition and Annotation Protocols

2.1. French Street Name Signs (FSNS)

• Scale: ≈1,044,868 train, 16,150 validation, 20,404 test images. Each sample: four 150×150 px tiles (multiview) (Smith et al., 2017).
• Annotation: Each sample carries a normalized, title-case transcription reflecting the map-usable street name (not verbatim sign content). No bounding boxes; sequence string per image. Four non-overlapping train/val/test splits using geographic partitioning. Disjoint vocabulary enforced among splits.
• Acquisition: Tiles detected via sign detector from Google Street View panoramas, heavy use of OCR+human verification.

2.2. Street View House Numbers (SVHN)

• Scale: ∼200,000 full images, ∼600,000 digit crops (Goodfellow et al., 2013).
• Annotation: For every full image: bounding boxes and sequence string for all digits present (multi-character, variable length). Labels enable both per-digit and sequence-level recognition.
• Preprocessing: Centered cropping, per-image mean subtraction, random crop augmentation.

2.3. Large-scale Street View Text (LSVT, ICDAR 2019)

• Scale: 450,000 total images: 50,000 fully annotated (30K train + 20K test) and 400,000 weakly annotated (Sun et al., 2019).
• Annotation: For “full” images: polygon(s)/quadrangle(s) per text instance with UTF-8 transcript, “Do Not Care” tags for illegible/inapplicable text. For “weak,” only 1–few “keywords of interest” per image.
• Format: TXT per image (coordinates, transcript per instance), or global CSV/JSON for weak key phrases.
• Task definitions: Separate detection and end-to-end text spotting, including recognition.

2.4. Chinese Text in the Wild (CTW), ShopSign, and Chinese Street View Text (C-SVT)

• CTW: 32,285 images, 1,018,402 annotated characters, 3,850 unique Chinese characters (Yuan et al., 2018).
• ShopSign: 25,770 images, 196,010 text lines, 626,280 Hanzi instances, five “hard” categories (mirror, wooden, deformed, exposed, obscured) (Zhang et al., 2019).
• C-SVT: 430,000 images: 29,966 full (quadrangle+transcription) and 400,000 weak (keyword-level) (Sun et al., 2019).
• Annotation: Character- or line-level boxes, explicit transcription; class/attribute flags in CTW, text-line-level polygons in ShopSign and C-SVT, with C-SVT pioneering partially supervised annotation leveraging both full and weak labels.

2.5. Berlin Flickr Street-View Text Dataset

• Scale: 3,431 street-view images, matched to OpenStreetMap footprints and building function classes; 1,558 images yield confident STR outputs (Sun et al., 2023).
• Annotation: Auto-detected (TextSnake, SAR) polygons + text, filtered with confidence and string rules, minor subset (numbers) manually validated—90.6% correct recognition.
• No explicit groundtruth or train/val/test splits; used primarily for GIS/urban analytics.

2.6. CVG-Text (Cross-View Geo-localization with Scene Text)

• Scale: 60,000 geo-aligned reference images, 30,000 queries with >100-token natural language scene descriptions (generated via LMM-augmented pipeline), spanning New York, Brisbane, Tokyo (Ye et al., 2024).
• Annotation: Each sample links a panoramic and single street-view, satellite tile, OSM raster, free-form text, plus GPS and POI tags.

3. Evaluation Metrics and Benchmarking Practices

Street-view text datasets support a spectrum of granular and task-defined evaluation criteria:

• Detection: Precision ($P$), Recall ($R$), F₁/H-mean ($F1$), Intersection-over-Union (IoU) for box-to-groundtruth matches. E.g., LSVT uses polygonal IoU, same for ShopSign and CTW.
• Recognition:
  • Full-sequence accuracy: $$\mathrm{Accuracy} = N_{\mathrm{correct}} / N_{\mathrm{total}}$$—exact string match required (FSNS, SVHN).
  • Character/Word-Level: Word Error Rate (WER), Character Error Rate (CER).
  • Normalized metrics: As in LSVT, normalized Levenshtein distance aggregating matches/nonmatches.
  • Partial label handling: Weak-label enabled evaluation (keyword precision/recall; e.g., C-SVT, LSVT).
• End-to-end tasks: True positive if region IoU ≥ threshold and transcript matches (case-sensitive/case-insensitive, script-harmonized).
• Geo-localization (CVG-Text): Recall@k, localization accuracy at radius thresholds, text/image similarity metrics.

Absence of large-scale, manually labeled ground truth (e.g., Berlin Flickr) precludes classic aggregate metrics; manual sampling or task-specific proxies are then employed (Sun et al., 2023).

4. Methodological Innovations: Annotation, Supervision, and Task Expansion

The evolution of street-view text datasets reflects increasing annotation complexity and methodological breadth:

• Classic fully supervised annotation: Exhaustive, box-level or sequence-level gold standards (FSNS, CTW).
• Partially/weakly supervised protocols: LSVT and C-SVT significantly enlarge scale and accelerate annotation by introducing weak supervision (keyword-only or coarse-mask labels), supporting partially supervised learning frameworks that mix strong and weak labels in end-to-end networks (Sun et al., 2019, Sun et al., 2019).
• Attribute-level/metadata labels: CTW annotates occlusion, complex background, distortion, type (handwritten/printed), and other factors for every character.
• Multi-modal and geo-anchored annotation: CVG-Text directly aligns text descriptions, images (pano/single/sat), OSM, and GPS, enabling cross-modal retrieval and scene description evaluation (Ye et al., 2024).
• Synthetic vs. crowdsourced content: Datasets like Berlin Flickr exploit public image uploads with automated STR and ex post filtering, while others (LSVT, C-SVT, ShopSign) use purpose-collected images and trained annotators.
• Normalization and canonicalization: FSNS uses normalized, map-usable transcriptions and enforces disjoint ground-truths across splits, modeling real address parsing needs (Smith et al., 2017).

5. Representative Benchmark Results and Practical Observations

• Recognition performance on classic datasets: Google Inception (CTW) yields 80.5% top-1 accuracy for single-character recognition; YOLOv2 achieves 71.0% mAP for multi-class detection (Yuan et al., 2018). SVHN full-sequence recognition exceeds 96% on public test splits (Goodfellow et al., 2013).
• End-to-end text spotting: 2019 RRC-LSVT top teams achieved H-mean (IoU ≥ 0.5) of 86.42% (detection) and 60.97% (end-to-end spotting) (Sun et al., 2019).
• Partially supervised learning impact: C-SVT’s partially supervised framework delivers +4.03% F-score over comparably costed full-label annotation, demonstrating the efficacy of leveraging large weakly labeled corpora (Sun et al., 2019).
• Cross-dataset generalization: Models trained on one Chinese text dataset exhibit low recall on others (e.g., EAST on CTW→ShopSign yields only 24% horizontal recall), highlighting domain gaps and the importance of diversity (Zhang et al., 2019).
• Challenging “hard” categories: ShopSign’s mirror or obscured categories retain low recall (<40%) even after fine-tuning, suggesting persistent unsolved sub-problems.

6. Limitations, Open Challenges, and Future Directions

Current datasets reveal several persistent limitations:

• Domain and coverage bias: Hotspot clustering in crowdsourced or panoramic data leaves many urban areas unrepresented (e.g., Berlin Flickr, CVG-Text) (Sun et al., 2023, Ye et al., 2024).
• Ground truth paucity: Many datasets (Berlin Flickr, portions of LSVT, C-SVT) lack pixel/box-level annotation across their full scale, limiting precise benchmarking.
• Annotation cost/scale trade-off: Weak/partial labeling strategies reduce cost but introduce noise and ambiguity, requiring new learning paradigms to leverage them optimally (Sun et al., 2019, Sun et al., 2019).
• Rare class and OOV handling: Pronounced long-tail character distributions in CTW, ShopSign, and C-SVT defy classic balanced-class assumptions and stress the need for synthetic augmentation and multimodal transfer (Yuan et al., 2018, Sun et al., 2019).
• Generalization across scripts, styles, and modalities: Multi-lingual and multi-modal (image, text, GIS) datasets are needed for robust practical deployment, motivating research on sequence-to-sequence alignment, retrieval, and explainable scene parsing (Ye et al., 2024).
• Real-world noise and compositionality: Synthetic or LMM-generated descriptions (CVG-Text) exhibit a domain gap vs. actual user utterances; extension to raw, noisy, or incomplete input remains an open area (Ye et al., 2024).

Proposed strategies include integrating multiple data sources, object-level pre-filtering for building-focused analytics, active sampling for target class balance, and embracing active or crowdsourced partial labeling to further scale (Sun et al., 2023, Sun et al., 2019).

7. Dataset Access, Licensing, and Recommended Practices

Most major street-view text datasets (FSNS, CTW, ShopSign, LSVT, CVG-Text, Berlin Flickr) are publicly accessible for academic research with explicit licensing:

Dataset	Access URL	License / Terms
FSNS	Google Research FSNS website	Research use (see publisher)
CTW	https://ctwdataset.github.io/	Public, non-commercial
ShopSign	https://github.com/chongshengzhang/shopsign	Academic, non-commercial
LSVT	http://rrc.cvc.uab.es	Registration, citation required
Berlin Flickr	https://github.com/ya0-sun/STR-Berlin	MIT code, images under Flickr terms
CVG-Text	https://yejy53.github.io/CVG-Text/	See repository
C-SVT	Contact authors, code/data released	Released with publication

Best practices for dataset use and benchmark reproducibility include:

• Strict adherence to train/val/test splits and normalization pipelines where defined (FSNS, SVHN).
• Consistent augmentation (random crops, color jitter) during both model training and evaluation.
• Adoption of multi-level metrics (string accuracy, word/character error rates, spatial recall) aligned with the task.
• Citing the respective dataset publication in all derived work and checking for updates or augmented annotations in repository releases.
• When building new datasets, balanced investment in full and partial annotation schemes is recommended to maximize scale and cost-effectiveness (Sun et al., 2019).

Street View Text Datasets continue to evolve both in annotation richness and task complexity, enabling advances in robust visual-language modeling for real-world, multilingual, and multi-modal environments.