CHURRO-DS: Historical OCR/HTR Dataset
- CHURRO-DS is an open, page-level historical OCR/HTR dataset that unifies 155 corpora spanning 22 centuries across 46 language clusters and 14 scripts.
- It standardizes full-page diplomatic transcriptions without bounding-box targets, enabling direct image-to-text modeling for both printed and handwritten documents.
- The dataset addresses underrepresented historical documents by including diverse layouts, script types, and degradation challenges, and serves as a training substrate for the CHURRO vision-language model.
CHURRO-DS is an open, page-level historical OCR/HTR dataset introduced with the CHURRO vision-LLM to support direct image-to-text recognition on historical materials that differ sharply from modern OCR benchmarks in language, script, layout, and degradation profile. It unifies 155 historical corpora into 99,491 page-level examples spanning 22 centuries, from the 3rd century BCE to the 20th century, across 46 language clusters, 14 scripts, and three writing directions. Its defining design choice is to pair each page image with a single full-page diplomatic transcription in correct reading order, thereby reducing historical OCR/HTR to page-level generation without requiring bounding-box targets (Semnani et al., 24 Sep 2025).
1. Motivation and task definition
CHURRO-DS was created to address a specific mismatch between prevailing OCR/VLM benchmarks and historical text recognition. Existing benchmarks overwhelmingly emphasize standardized, contemporary documents, often in English or Chinese, with regular layouts and short outputs. Historical materials instead exhibit diverse languages, including historical variants and dead languages, multiple scripts and writing directions, irregular page organization, inconsistent orthography, dense abbreviations, and substantial physical or digitization-related degradation. The dataset therefore targets conditions such as multi-column pages, vertical text, marginalia, rubrication, illumination, Fraktur and Gothic typography, creases, bleed-through, curvature, and scanning artifacts (Semnani et al., 24 Sep 2025).
A further motivation is methodological. Many historical datasets do not provide bounding-box annotations, which limits their utility for pipeline OCR but makes them suitable for page-level image-to-text modeling. CHURRO-DS standardizes such humanities-sourced corpora into a common format with curated, human-annotated, page-level transcriptions in correct reading order. In this formulation, each example consists of a page image, page-level metadata indicating language and script, and a single text string representing the complete page content. This makes the benchmark natively compatible with VLMs that can model long outputs and variable-resolution inputs without line or word segmentation.
The dataset directly addresses three gaps. First, it expands language and script diversity through 46 language clusters and 14 scripts across five script families, including historical variants such as Old or Middle High German and dead languages such as Classical Latin. Second, it covers left-to-right, right-to-left, and vertical top-to-bottom, right-to-left East Asian layouts. Third, it incorporates document-level visual phenomena that are common in historical archives but underrepresented in modern OCR evaluation, including ligatures, abbreviations, marginalia, rubrication, and illumination. The inclusion of the first publicly available historical OCR datasets for Persian and Turkish, and the first publicly available HTR dataset for Ottoman Turkish, reflects this emphasis on previously underserved material.
2. Corpus composition and representational coverage
In scale, CHURRO-DS is described as the largest historical text recognition dataset to date. The corpus contains 99,491 page-level image–transcription pairs drawn from 155 sources. Coverage extends over 22 centuries of textual heritage and includes 46 language clusters overall; 29 of these appear in validation and test. The validation/test language set comprises Arabic, Bangla, Bulgarian, Catalan, Chinese, Czech, Dutch, English, Finnish, French, German, Greek, Hebrew, Hindi, Italian, Japanese, Khmer, Latin, Norwegian, Persian, Polish, Portuguese, Romanian, Sanskrit, Slovenian, Spanish, Swedish, Turkish, and Vietnamese (Semnani et al., 24 Sep 2025).
Scriptal coverage is organized using ISO 15924 and Glottolog families. The dataset includes European alphabets, Middle Eastern abjads, Indic abugidas, a Southeast Asian abugida, and East Asian logo-syllabaries. Counts reported for CHURRO-DS are: Latin 71,749; Latin Fraktur variant 12,987; Latin Gaelic variant 117; Cyrillic 952; Greek 434; Hebrew 159; Arabic 3,021; Devanagari 698; Bengali 64; Malayalam 50; Newa 195; Khmer 624; Japanese 2,779; and Han 5,662. This distribution shows both breadth and clear concentration in Latin-script material.
The dataset also separates printed and handwritten materials. Among printed clusters, German has 21,024 total examples, French 14,648, Spanish 8,426, Latin 5,158, Czech 3,145, Dutch 3,016, Polish 2,781, Slovenian 2,723, English 2,504, and Japanese 1,954; lower-resource printed clusters include Chinese with 66 and Bangla with 62. Among handwritten clusters, Chinese has 5,113 examples, Spanish 4,512, French 3,627, English 3,439, Dutch 3,262, Arabic 2,367, German 2,358, Latin 1,596, Japanese 892, Khmer 622, Swedish 552, and Norwegian 441, with smaller clusters such as Portuguese at 72. Several clusters appear only in training.
Document types include newspapers, books, diaries, letters, government records, notarial deeds, censuses, catalogues, Bibles, and manuscripts. Layouts include single-column and double-column pages, tabular structures, complex mixed-size glyphs, and vertical East Asian formats. Pages may also contain marginalia, rubrication, illumination, and non-text decorative elements. Images are correctly oriented but can be slightly misaligned, include extra margins, and display common degradation. This combination of linguistic, scriptal, and material heterogeneity makes CHURRO-DS a benchmark not merely for character recognition but for long-context page understanding under historical constraints.
3. Annotation scheme, transcription standard, and quality control
CHURRO-DS uses page-level diplomatic transcription as its principal annotation standard. Models are evaluated on what they see, so spelling, punctuation, capitalization, historical glyphs such as the long s, abbreviations, and hyphenation are preserved. Modernized editions were excluded if faithful transcriptions could not be recovered. Only minimal normalization is applied for typography, including normalization of Unicode fractions such as “1/4” and ligatures such as U+FB06 “st” to “st”; for Arabic-script evaluation, diacritics and hamza are normalized using PyArabic for fairness (Semnani et al., 24 Sep 2025).
Reading order is a central component of the curation protocol. When ALTO or PAGE XML specified explicit reading order, that information was used directly. Otherwise, dataset-specific heuristics were applied for regular layouts and VLM-guided ordering for complex layouts. Manual spot checks indicate more than 98% reading-order accuracy. Although no bounding boxes are required as training or evaluation outputs, bounding boxes from source formats or VLM-assisted analysis were used during curation when necessary to repair spacing, line breaks, and diacritics.
The curation process also included targeted repair of source-level transcription errors. Sixteen source datasets had notable issues, including marginal omissions and frequent transcription mistakes. Gemini 2.5 Pro was prompted with images and bounding boxes to propose minimal corrections, and the authors manually validated all fixes. Character-level edit distance between original and edited transcriptions remained below 5%. This indicates an effort to preserve source fidelity while repairing clearly identifiable errors.
Filtering and metadata standardization are equally important. Pages with fewer than 30 tokens under Qwen 2.5 VL tokenization were removed, which excludes title pages and chapter dividers. Approximately 15,000 near-duplicate pages were eliminated using MinHash. Images were resized to fit within 2500×2500 pixels while preserving aspect ratio. Language detection uses ISO 639-3 and macrolanguage groupings, with VLM prompting constrained by dataset documentation and supplemented by manual review. Script classification follows ISO 15924 and Glottolog families.
Residual quality issues remain. The dataset notes rare omissions in gold text and common model confusions between visually similar historical glyphs, such as versus . East Asian vertical layouts and very small characters are identified as particularly challenging. These observations do not negate the utility of the benchmark, but they delimit the interpretive confidence appropriate for fine-grained error analysis.
4. Processing pipeline, data splits, access, and licensing
The construction pipeline began with an extensive survey that identified 153 historical corpora; the final unified dataset comprises 155 sources, including humanities editions, archival ground truths, and competition datasets. Source materials were standardized from ALTO/PAGE XML, JSON, TEI/XML, and plain text into a single string per page in proper reading order. Cleaning included minimal typographic normalization, Arabic diacritics normalization for evaluation, LLM-assisted restoration of spacing and line breaks guided by bounding boxes, deduplication, token-length filtering, and image resizing (Semnani et al., 24 Sep 2025).
The split design is balanced at the language-cluster and document-type level. Validation and test sets are formed by sampling 60 pages per language cluster per document type, printed and handwritten, and splitting those evenly into validation and test. Only 29 of the 46 language clusters have enough examples to support this protocol; the remainder are training-only. The resulting counts are 97,151 training pages, 1,170 validation pages, and 1,170 test pages. Because training size varies widely across clusters, with examples such as printed Chinese having only 6 training pages and handwritten Portuguese 12, the split structure also exposes low-resource failure modes rather than obscuring them in aggregate scores.
Release conditions are explicitly research-oriented. CHURRO-DS is distributed under Creative Commons Attribution Share Alike 4.0 (CC BY-SA 4.0), selected to comply with the licenses of included sources, and all included datasets permit research use. No crowdsourcing was conducted. The repository provides the dataset, model, and code, together with source lists, licenses, evaluation scripts and prompts, and the reported benchmarks.
Each data point contains an image file, a UTF-8 text file with the complete page transcription, and metadata fields for language cluster and script. The text is diplomatic and in proper reading order, the images are resized to at most 2500×2500, and the evaluation pipeline applies the same normalization to predictions and gold texts. This explicit formatting makes the benchmark comparatively easy to reproduce and extend, provided its transcription and normalization standards are preserved.
5. Evaluation protocol and benchmark function
CHURRO-DS defines two primary tasks: page-level transcription for printed materials and page-level transcription for handwritten materials. Zero-shot VLM evaluation uses a uniform diplomatic transcription prompt, and the fine-tuned CHURRO model is evaluated identically. The benchmark’s primary metric is normalized Levenshtein similarity, computed at character level. If is the system output, the gold string, the Levenshtein distance, and , then similarity is defined as . Scores are averaged per example and reported separately for printed and handwritten subsets (Semnani et al., 24 Sep 2025).
The benchmark also documents Character Error Rate and Word Error Rate for completeness, although they are not used for primary reporting. The same normalization procedures applied to gold texts are also applied to predictions, including typographic normalization and Arabic diacritics or hamza normalization via PyArabic. This matters because CHURRO-DS evaluates diplomatic fidelity rather than normalized or modernized transcription; a mismatch in normalization policy can produce misleading comparisons.
The recommended reproduction protocol is explicit. Benchmarking should use the provided validation and test splits, the shared diplomatic transcription prompt, and normalized Levenshtein similarity computed per page. Predictions and gold text should undergo the same normalization. The paper further recommends reporting language-cluster breakdowns, which is methodologically important because global averages can conceal strong performance on dominant clusters and weak performance on low-resource ones.
Generation settings also form part of the benchmark definition. Evaluation uses temperature $0$ and generous generation limits to accommodate long pages: 20,000 tokens for non-reasoning models and 40,000 for reasoning models. Since reading order is already curated in the gold annotations, the benchmark is explicitly page-level and does not assume or reward reliance on bounding boxes. A plausible implication is that CHURRO-DS evaluates end-to-end historical page transcription rather than modular OCR subcomponents, which distinguishes it from many prior OCR testbeds.
6. Empirical role in CHURRO and benchmark results
CHURRO-DS is not only an evaluation benchmark but also the training substrate for CHURRO, a 3B-parameter open-weight VLM obtained by fine-tuning Qwen 2.5 VL (3B). The base model was selected for strong zero-shot OCR, compact size, and support for high-resolution, variable-sized inputs. Fine-tuning was performed for 5 epochs on 32 NVIDIA H100 GPUs with gradient accumulation, effective batch size 128, a learning rate of with cosine schedule, and a training time of approximately 25 hours. Images were resized to a maximum of 5,120 vision patches of pixels, and total image-plus-text tokens per example were kept below approximately 25,000. The training objective remained page-level image-to-text generation in correct reading order, with no bounding-box targets (Semnani et al., 24 Sep 2025).
On the CHURRO-DS printed test subset, CHURRO achieves 82.3% normalized Levenshtein similarity, exceeding Gemini 2.5 Pro at 80.9% by 1.4 percentage points. It also outperforms the strongest open-weight zero-shot VLMs specialized for page OCR, including NuMarkdown (8B) at 72.7%, olmOCR (8B) at 69.8%, and Nanonets OCR (3B) at 69.7%, as well as OCR systems such as Azure OCR at 71.9% and Mistral OCR at 64.0%. The Azure OCR plus Gemini 2.5 Pro hybrid underperforms both of its components at 52.6%, while the oracle ensemble upper bound is 88.9%.
On the handwritten subset, CHURRO reaches 70.1%, surpassing Gemini 2.5 Pro at 63.6% by 6.5 points. The best open-weight zero-shot baseline reported is Qwen 2.5 VL (72B) at 54.5%, followed by NuMarkdown (8B) at 51.2%, RolmOCR (8B) at 49.0%, and Nanonets OCR (3B) at 43.2%. OCR systems perform less well, with Azure OCR at 47.7%, Mistral OCR at 29.4%, and Azure OCR plus Gemini at 40.4%; the oracle ensemble upper bound is 76.3%.
The reported gains clarify the interaction between model specialization and dataset design. Relative to base Qwen 2.5 VL (3B), CHURRO improves by an average of 14.5 percentage points across printed languages and 27.2 points across handwritten languages. Printed fine-tuning gains are largest for Japanese at +37.0, Finnish at +32.9, and Swedish at +22.9. Handwritten gains are largest for Greek at +62.6, Japanese at +54.7, Turkish at +42.3, Hebrew at +42.1, and Persian at +36.8. Conversely, printed Chinese performance is reported as 6.2%, which the paper attributes to the cluster having only 6 training examples. This supports the broader claim that CHURRO-DS functions not merely as a large benchmark, but as domain-specific supervision whose coverage and imbalance directly shape downstream behavior. The paper also reports that CHURRO is 15.5 times more cost-effective than Gemini 2.5 Pro under the stated cost assumptions.
7. Limitations, biases, and extension criteria
CHURRO-DS has several strengths: it is open, large, page-level, human-annotated, explicitly ordered by reading sequence, broad in language and script coverage, inclusive of degraded scans and historical typography, and released with a unified evaluation pipeline. At the same time, the dataset has clearly stated limitations. Language coverage is imbalanced, with strong representation of European languages and Latin scripts, and more limited coverage for lower-resource languages. No African indigenous languages are currently included. Some clusters have extremely sparse training data, which depresses performance and complicates comparison across languages (Semnani et al., 24 Sep 2025).
Annotation noise also persists despite manual verification and VLM-assisted correction. Rare omissions or transcription errors remain, and minor typographic normalization can affect edge cases. Layout diversity is broad but not exhaustive. Particularly difficult page types include complex East Asian newspapers with non-rectangular region boundaries and interspersed Latin text. These limitations mean that aggregate scores should not be treated as uniformly representative across all historical document regimes.
The paper therefore recommends a conservative reporting practice. Results should be broken down by language cluster rather than conflated across imbalanced categories. Comparisons should respect the diplomatic transcription standard, especially when other resources use modernized editions. Normalization procedures, including typographic normalization and Arabic diacritics handling, should be disclosed explicitly for reproducibility. Strong conclusions should be avoided for clusters with very small training sets unless those clusters are supplemented with additional data.
Criteria for extension are also specified. New corpora should satisfy license compatibility, preferably through Creative Commons or explicit research-use permission; maintain diplomatic, page-level transcription in correct reading order; provide language and script metadata using ISO 639-3 and ISO 15924; apply only minimal typography normalization and consistent script-specific diacritic handling where needed; and undergo deduplication, token-length filtering above 30 tokens, and manual spot checks. For complex layouts, VLM-guided ordering with human verification is recommended. Contributions from underrepresented regions, including African languages and scripts, and from additional dead or historical variants are explicitly encouraged. This suggests that CHURRO-DS is intended less as a closed benchmark than as an extensible infrastructure for historically grounded OCR/HTR research.
