---
title: 'GlotOCR Bench: OCR Evaluation Across 158 Scripts'
url: https://www.emergentmind.com/topics/glotocr-bench
type: topic
---

# GlotOCR Bench: OCR Evaluation Across 158 Scripts

GlotOCR Bench, also called GlotOCR, is a multilingual optical character recognition benchmark designed to evaluate OCR generalization across Unicode scripts rather than across a small set of high- and mid-resource languages. It covers **158 Unicode scripts**, provides synthetic **images of text** at sentence- or word-level in **clean** and **degraded** variants, and supplies **exact ground-truth transcriptions** and metadata for evaluation. Its stated aim is to determine whether modern OCR and vision–language models can recognize text in *almost all writing systems encoded in Unicode*, rather than only in widely used scripts such as Latin, Chinese, Japanese, Korean, or Arabic [2604.12978].

## 1. Definition and script-centric scope

GlotOCR Bench is organized around **script coverage** as the primary evaluation axis. For each of **158 Unicode scripts**, it provides rendered text images together with exact transcriptions. The benchmark is explicitly intended to test whether OCR systems can generalize beyond the narrow subset of scripts that dominate existing OCR and multimodal evaluation [2604.12978].

The benchmark distinguishes three resource tiers derived from script prevalence in web content. The **High** tier contains **Latin only**. The **Mid** tier contains **9 scripts**: Arabic (**Arab**), Cyrillic (**Cyrl**), Devanagari (**Deva**), Han (**Hani**), Japanese (**Jpan**, mixed Kana+Kanji), Hangul (**Hang**), Greek (**Grek**), Hebrew (**Hebr**), and Thai (**Thai**). The **Low** tier contains the remaining **148 scripts**. This tiering is central to the benchmark’s analysis because the reported results are stratified by these categories [2604.12978].

The underlying text sources are language–script pairs from **GlotLID v3** and subsequent web-derived corpora such as **GlotCC** and **FineWeb2**, with supplemental material from **Wiktionary**, **Wikisource**, **Omniglot** sample texts, **Google Fonts language metadata**, and script conversion tools such as **Aksharamukha**. Text is filtered to sentences in **30–100 characters**, verified with **GlotScript** script identification, and constrained so that samples are not purely random character strings but instead aim to be *linguistically valid* [2604.12978].

Sampling is uneven by design. The benchmark includes up to **100 sentences per script**, except for several special cases: **Latin** has **4,000 sentences**, some mid-tier scripts have **400 sentences**, and the total corpus contains **16,375 sentences** across scripts. For **68 scripts**, fewer than **100 samples** are available, often at word level rather than sentence level. The paper notes that for these low-sample scripts, models often fail even at **ScriptAcc**, but that this failure pattern is consistent across models [2604.12978].

## 2. Construction pipeline and rendering methodology

The rendering pipeline is a defining component of GlotOCR Bench because many scripts require correct shaping, combining marks, ligatures, and directional handling. Fonts are drawn from the **Google Fonts repository**, restricted to **SIL Open Font License** fonts. The benchmark reports **3101 font families** in total. Font availability is highly imbalanced: Latin has **1907 fonts**; the **Mid** tier has a median of approximately **59 fonts per script** with a range of **29–323**; the **Low** tier has a median of **1 font per script**, with minimum **1** and maximum **29** [2604.12978].

Font selection follows a four-stage per-sentence filtering procedure: **script match**, **codepoint coverage**, **glyph rendering test**, and **random selection** from the remaining candidates. This design addresses a specific failure mode documented in the benchmark: some fonts declare support for a script but do not actually render all required glyphs [2604.12978].

Text shaping is performed with **HarfBuzz**, which handles contextual forms, ligatures, combining marks, and shaping rules for scripts such as Arabic and Indic scripts. Rasterization is performed with **FreeType**. The pipeline supports both **LTR** and **RTL** scripts, but **sentences with mixed bidi content are excluded**, so each sample is purely left-to-right or purely right-to-left. **Vertically oriented scripts** such as Mongolian are forced to **horizontal** rendering because vertical layout is not supported in the pipeline [2604.12978].

The benchmark provides two image variants per text sample. The **clean variant** uses a **white background**, **48 px** font size, **1000 px** canvas width with **40 px** padding, and slight random rotation up to $\pm 1^\circ$. The **degraded (“old document”) variant** applies a sequence of perturbations: blending onto a scanned paper texture patch with rotation up to $\pm 2^\circ$; elastic deformation with Gaussian kernel $17 \times 17$ and amplitude up to $\pm 8$ px; Gaussian noise with $\sigma = 8$; **10–30 white rectangular patches** up to $40 \times 15$ px to simulate ink dropout; global intensity scaling to **50–85%** of original with textured noise of $\sigma = 10$; downsampling to **40–70%** of original dimensions and upsampling back; **JPEG compression** with quality between **30 and 80**; and perspective distortion with corner warps up to **10%** of image width or height. Additional glyph-level perturbations modify character spacing by **−2 to +4 px**, apply **dilation** with probability **0.4**, **erosion** with probability **0.25** using a **2×2 kernel**, add line vertical jitter up to **±3 px**, and displace glyphs vertically by a **parabolic curvature** [2604.12978].

Manual review is integrated into the benchmark construction. For each script, **10 rendered images** were manually inspected across diverse sizes. For common scripts, outputs were compared with standard text editors; for rare scripts, reviewers compared glyphs **character-by-character against Unicode charts**. The audit was used to verify that all characters were rendered, directionality was correct, and ligatures, diacritics, and shaping behaved correctly as far as supported. Degradation parameters were also validated by human inspection in **Latin, Greek, Cyrillic, Arabic** so that degraded images remained legible to humans while still challenging models [2604.12978].

## 3. Data format and evaluation protocol

The released dataset includes **clean and degraded image variants**, **ground-truth transcriptions**, and metadata. The metadata includes **script code** in **ISO 15924** form, **language ID**, and **text source**. The benchmark is explicitly a **textline/sentence-level OCR** resource and does **not** include bounding boxes or segmentation annotations [2604.12978].

Evaluation is based on three primary metrics: **Character Error Rate (CER)**, **Acc@k**, and **ScriptAcc**. **CER** is defined as normalized character-level edit distance with whitespace ignored:
$$
\text{CER} = \min\!\left(1, \frac{S + D + I}{N}\right),
$$
where $S$ is the number of substitutions, $D$ deletions, $I$ insertions, and $N$ the reference length. The score is bounded by **1** [2604.12978].

A notable aspect of the protocol is that CER is computed under four normalizations and the **minimum** is taken: original output, reversed output string, lowercased output, and output with Unicode combining marks removed. This compensates for expected variants such as **RTL/LTR reversal**, **case variation**, and **diacritic variation**. A sentence counts as correct for acceptance-based metrics if **any** normalized output yields sufficiently low CER [2604.12978].

**Acc@k** is defined as the percentage of sentences with $\text{CER} \le k/100$:
$$
\text{Acc@k} = \frac{1}{M} \sum_{m=1}^M \mathbf{1}\{\text{CER}_m \le k/100\},
$$
where $M$ is the number of sentences. GlotOCR reports **Acc@0** for exact match and **Acc@5** for near-perfect match with $\text{CER} \le 0.05$. The paper identifies **Acc@5** as its primary accuracy metric because it acts as a coarse but interpretable measure of whether a model can function in a given script at all [2604.12978].

**ScriptAcc** measures whether the model output is in the **correct script** independent of transcription correctness. The dominant script of the prediction is inferred with **GlotScript**, and **ScriptAcc** is the fraction of sentences for which predicted script matches expected script. This metric is used to diagnose whether errors arise from fine-grained character confusion within the correct script or from **cross-script hallucination** into another script [2604.12978].

No **WER** is reported. The benchmark justifies this omission by noting that text spans many languages and scripts with differing tokenization conventions [2604.12978].

## 4. Evaluated systems and zero-shot setup

GlotOCR Bench evaluates **14 models**, spanning OCR-centric systems, general multimodal VLMs used for OCR, and proprietary APIs. The open-weight OCR or OCR-oriented VLM models listed are **dots.ocr**, **dots.mocr**, **olmOCR-2**, **RolmOCR**, **LightOnOCR-2**, **Nanonets-OCR2**, **PaddleOCR-VL-1.5**, **FireRed-OCR**, **GLM-OCR**, **DeepSeek-OCR-2**, and **HunyuanOCR**. The benchmark also includes **Qwen3-VL-8B** as a general open-weight VLM that supports text recognition. The proprietary systems are **Gemini 3.1 Flash-Lite** and **GPT-4.1** [2604.12978].

The paper characterizes most evaluated systems as VLM-based OCR models composed of a transformer or MoE visual encoder plus a language model trained on document images. This framing is important because later analysis argues that performance strongly tracks script-level pretraining coverage and that current systems rely on language-model pretraining as much as on visual recognition [2604.12978].

All models are evaluated **zero-shot** using the **uv-scripts/ocr inference suite**. The prompt is a simple instruction to transcribe the text in the image, return it wrapped in tags, and provide no explanation. For chat-based APIs and chat-style VLMs, the **chat template** is used; otherwise, a raw prompt is applied. The benchmark passes the **original rendered resolution** directly to the model and does not introduce extra resizing or cropping at evaluation time [2604.12978].

A separate experiment is conducted only for **GPT-4.1** using **script-aware hinting**. In that condition, the prompt provides the **script**, **language**, and the **exact deduplicated set of characters** appearing in the image. Short texts under **10 characters** are excluded, leaving **149 scripts** in the analysis [2604.12978].

## 5. Empirical findings across resource tiers

The main empirical result is a sharp tier-dependent discontinuity. On **Latin**, all models achieve **Acc@5 ≥ 75%**, and the best models exceed **90%**. The strongest reported Latin result is **Gemini 3.1 Flash-Lite** with **0.9% CER** and **Acc@5 95.3%**, with **dots.mocr** and **dots.ocr** close behind [2604.12978].

On the **Mid-tier scripts**—Arabic, Cyrillic, Devanagari, Han, Japanese, Hangul, Greek, Hebrew, and Thai—performance drops substantially but remains usable for the strongest systems. The benchmark reports **Gemini Flash-Lite** at **CER 3.0, Acc@5 82.7%** and **dots.ocr** at **CER 5.0, Acc@5 78.3%**. Averaged over models, the drop from **High** to **Mid** is **27.6 points** in **Acc@5** [2604.12978].

On the **148 Low-tier scripts**, performance collapses. The best three models—**Gemini**, **dots.ocr**, and **dots.mocr**—achieve only **Acc@5 ≈ 7.7%**, and many other models remain below **1%**. The macro-average **Acc@5** drops from about **60.0%** on **Mid** scripts to **2.3%** on **Low** scripts, a **57.7-point collapse**. The paper identifies this as **Finding 1**, describing it as a threshold phenomenon determined by pretraining coverage rather than a smooth degradation [2604.12978].

Across all tiers, **Gemini 3.1 Flash-Lite** has the best macro mean with **Macro CER 27.7**, **Acc@0 52.4%**, and **Acc@5 61.9%**, with **dots.mocr** and **dots.ocr** described as **very close** behind [2604.12978].

Per-language analysis within scripts shows that script identity alone does not determine difficulty. **Latin** typically has median **Acc@5** above **90%** but still exhibits substantial variation across languages. **Devanagari** shows lower medians and moderate spread, with complex conjuncts causing errors. **Arabic** is reported as the most difficult of the mid-tier scripts, with low medians and wide interquartile ranges, while **Cyrillic** is somewhat better. This intra-script variance is summarized as **Finding 2** [2604.12978].

The relation between **ScriptAcc** and **Acc@5** is strongly positive, but not identical. Scripts with high script recognition accuracy tend to have high OCR accuracy, yet the deviations are diagnostically important. **Arabic** has high **ScriptAcc** but lower **Acc@5**, indicating intra-script transcription difficulty. **Hebrew** has lower **ScriptAcc** because outputs are often misclassified as **Thai**, reducing **Acc@5** through cross-script confusion. **Japanese** is reported to achieve surprisingly high **Acc@5**, even above Latin, despite mixing multiple writing systems. This pattern is reported as **Finding 3** [2604.12978].

The **script-aware hinting** experiment with **GPT-4.1** shows limited global improvement: among **149 scripts**, **125** show **no change**, **21** improve, and **3** get worse, for an average gain of **+0.7 percentage points** in **Acc@5**. The largest improvement is on **Han**, with a gain of more than **20 points**, and **Cyrillic** and **Thai** also improve notably. Some low-resource scripts such as **Italic** and **Coptic** move from near zero to roughly **5–10% Acc@5**, but remain very poor overall. This is **Finding 4**, which the paper interprets as evidence that script-aware hinting is not the main bottleneck for low-resource scripts [2604.12978].

Under image degradation, all evaluated top models lose performance across all tiers. The benchmark notes moderate drops on Latin, larger absolute and relative drops on mid-resource scripts, and small absolute but large relative drops on low-resource scripts because baseline performance is already near zero. **GPT-4.1** is described as slightly more robust among the top models, with a **13.8% relative drop** on Latin. This behavior is summarized as **Finding 5**, with clean images functioning as an upper bound [2604.12978].

## 6. Failure modes, interpretation, limitations, and reproducibility

GlotOCR Bench devotes particular attention to **cross-script hallucination**, which it categorizes into four mutually exclusive prediction classes: **Correct script**, **Cross-script hallucination**, **Silent**, and **Artifact**. Averaged across scripts, the benchmark reports **12.5%** correct-script outputs, **68.4%** hallucinations, **6.0%** silent outputs, and **13.1%** artifacts. The dominant failure mode is therefore not abstention but production of fluent-looking text in an incorrect script [2604.12978].

Model-specific differences in hallucination behavior are also reported. **dots.ocr** is relatively conservative, with **Correct 15.8%**, **Hallucination 38.3%**, **Silent 42.1%**, and **Artifacts 3.8%**. **Gemini Flash-Lite** yields **Correct 22.6%**, **Hallucination 70.2%**, **Silent 0.5%**, and **Artifacts 6.7%**. **olmOCR-2** reaches **Correct 14.4%**, **Hallucination 81.7%**, **Silent 0.0%**, and **Artifacts 3.9%**. **DeepSeek-OCR-2** is noted for especially high artifact rates at **26.2%** [2604.12978].

The appendix-level analysis identifies common hallucination attractors. **Latin**, **Arabic**, and **Devanagari** are the most frequent targets. Some substitutions reflect script proximity, such as **Syriac → Arabic**, **Grantha → Tamil**, **Coptic → Greek**, **Newa → Devanagari**, **Tangut → Han**, and **Lisu → Latin**. Others appear primarily distributional, such as **Old Uyghur / Mongolian → Arabic** or **Ogham → Latin**. This is reported as **Finding 6**, and the paper treats it as strong evidence of language-model bias overriding lack of visual familiarity [2604.12978].

The benchmark’s interpretive claim is that OCR performance broadly tracks **script-level pretraining coverage** and that modern OCR VLMs function as **language models with a visual front-end**. The evidence cited includes the tier-wise performance collapse, the strong relation between **ScriptAcc** and **Acc@5**, the limited effect of script-aware hints on low-resource scripts, and the prevalence of fluent hallucinations in high-resource scripts rather than random noise [2604.12978]. This suggests that current systems are not script-agnostic recognizers of graphemic form.

Several limitations are explicit. The images are **synthetic**, not real scanned documents; the benchmark is predominantly **single-line/single-block text**; and real documents include additional complexities such as multi-column layouts, marginal annotations, stamps, and non-uniform scanning artifacts. Typographic variation is highly uneven because many low-resource scripts have only **one usable font** and very few samples. The pipeline does not support vertical layout and therefore renders vertical scripts horizontally. Text sources are biased toward web text and open resources, and some scripts rely on artificially constructed or transliterated text. The paper accordingly presents GlotOCR as a controlled, script-level evaluation environment and states that real-world performance is likely worse [2604.12978].

The benchmark is released with a **Hugging Face dataset** at `https://hf.co/datasets/cis-lmu/glotocr-bench` and a **GitHub rendering and evaluation pipeline** at `https://github.com/cisnlp/glotocr-bench`. The dataset contains images, transcriptions, script and language metadata, and source information, and is released under an **evaluation-only** license. The pipeline is released under **Apache 2.0** and includes rendering, font selection, evaluation code for **CER**, **Acc@k**, **ScriptAcc**, and cross-script hallucination analysis, as well as scripts to regenerate benchmark images from seed text [2604.12978].

A plausible implication is that GlotOCR Bench functions not merely as a scorecard for current OCR systems but as a stress test for claims of multilingual OCR coverage. Within the benchmark’s own framing, OCR is effectively “near solved” only for **Latin**, workable for a narrow set of **mid-resource** scripts, and unusable for most of the remaining Unicode inventory under **Acc@5** evaluation. The benchmark therefore foregrounds the gap between nominal Unicode support and actual zero-shot OCR competence across scripts [2604.12978].

Source: https://www.emergentmind.com/topics/glotocr-bench