GlotOCR Bench: OCR Evaluation Across 158 Scripts

• GlotOCR Bench is a benchmark that evaluates OCR generalization across 158 Unicode scripts using a script-centric tiering system.
• The benchmark employs a sophisticated rendering pipeline with HarfBuzz and FreeType to produce both clean and degraded text images.
• Empirical results show high accuracy for Latin scripts, moderate performance for mid-resource scripts, and a collapse in low-resource scripts highlighting pretraining challenges.

GlotOCR Bench, also called GlotOCR, is a multilingual optical character recognition benchmark designed to evaluate OCR generalization across Unicode scripts rather than across a small set of high- and mid-resource languages. It covers 158 Unicode scripts, provides synthetic images of text at sentence- or word-level in clean and degraded variants, and supplies exact ground-truth transcriptions and metadata for evaluation. Its stated aim is to determine whether modern OCR and vision–LLMs can recognize text in almost all writing systems encoded in Unicode, rather than only in widely used scripts such as Latin, Chinese, Japanese, Korean, or Arabic (Kargaran et al., 14 Apr 2026).

1. Definition and script-centric scope

GlotOCR Bench is organized around script coverage as the primary evaluation axis. For each of 158 Unicode scripts, it provides rendered text images together with exact transcriptions. The benchmark is explicitly intended to test whether OCR systems can generalize beyond the narrow subset of scripts that dominate existing OCR and multimodal evaluation (Kargaran et al., 14 Apr 2026).

The benchmark distinguishes three resource tiers derived from script prevalence in web content. The High tier contains Latin only. The Mid tier contains 9 scripts: Arabic (Arab), Cyrillic (Cyrl), Devanagari (Deva), Han (Hani), Japanese (Jpan, mixed Kana+Kanji), Hangul (Hang), Greek (Grek), Hebrew (Hebr), and Thai (Thai). The Low tier contains the remaining 148 scripts. This tiering is central to the benchmark’s analysis because the reported results are stratified by these categories (Kargaran et al., 14 Apr 2026).

The underlying text sources are language–script pairs from GlotLID v3 and subsequent web-derived corpora such as GlotCC and FineWeb2, with supplemental material from Wiktionary, Wikisource, Omniglot sample texts, Google Fonts language metadata, and script conversion tools such as Aksharamukha. Text is filtered to sentences in 30–100 characters, verified with GlotScript script identification, and constrained so that samples are not purely random character strings but instead aim to be linguistically valid (Kargaran et al., 14 Apr 2026).

Sampling is uneven by design. The benchmark includes up to 100 sentences per script, except for several special cases: Latin has 4,000 sentences, some mid-tier scripts have 400 sentences, and the total corpus contains 16,375 sentences across scripts. For 68 scripts, fewer than 100 samples are available, often at word level rather than sentence level. The paper notes that for these low-sample scripts, models often fail even at ScriptAcc, but that this failure pattern is consistent across models (Kargaran et al., 14 Apr 2026).

2. Construction pipeline and rendering methodology

The rendering pipeline is a defining component of GlotOCR Bench because many scripts require correct shaping, combining marks, ligatures, and directional handling. Fonts are drawn from the Google Fonts repository, restricted to SIL Open Font License fonts. The benchmark reports 3101 font families in total. Font availability is highly imbalanced: Latin has 1907 fonts; the Mid tier has a median of approximately 59 fonts per script with a range of 29–323; the Low tier has a median of 1 font per script, with minimum 1 and maximum 29 (Kargaran et al., 14 Apr 2026).

Font selection follows a four-stage per-sentence filtering procedure: script match, codepoint coverage, glyph rendering test, and random selection from the remaining candidates. This design addresses a specific failure mode documented in the benchmark: some fonts declare support for a script but do not actually render all required glyphs (Kargaran et al., 14 Apr 2026).

Text shaping is performed with HarfBuzz, which handles contextual forms, ligatures, combining marks, and shaping rules for scripts such as Arabic and Indic scripts. Rasterization is performed with FreeType. The pipeline supports both LTR and RTL scripts, but sentences with mixed bidi content are excluded, so each sample is purely left-to-right or purely right-to-left. Vertically oriented scripts such as Mongolian are forced to horizontal rendering because vertical layout is not supported in the pipeline (Kargaran et al., 14 Apr 2026).

The benchmark provides two image variants per text sample. The clean variant uses a white background, 48 px font size, 1000 px canvas width with 40 px padding, and slight random rotation up to $\pm 1^\circ$. The degraded (“old document”) variant applies a sequence of perturbations: blending onto a scanned paper texture patch with rotation up to $\pm 2^\circ$; elastic deformation with Gaussian kernel $17 \times 17$ and amplitude up to $\pm 8$ px; Gaussian noise with $\sigma = 8$; 10–30 white rectangular patches up to $40 \times 15$ px to simulate ink dropout; global intensity scaling to 50–85% of original with textured noise of $\sigma = 10$; downsampling to 40–70% of original dimensions and upsampling back; JPEG compression with quality between 30 and 80; and perspective distortion with corner warps up to 10% of image width or height. Additional glyph-level perturbations modify character spacing by −2 to +4 px, apply dilation with probability 0.4, erosion with probability 0.25 using a 2×2 kernel, add line vertical jitter up to ±3 px, and displace glyphs vertically by a parabolic curvature (Kargaran et al., 14 Apr 2026).

Manual review is integrated into the benchmark construction. For each script, 10 rendered images were manually inspected across diverse sizes. For common scripts, outputs were compared with standard text editors; for rare scripts, reviewers compared glyphs character-by-character against Unicode charts. The audit was used to verify that all characters were rendered, directionality was correct, and ligatures, diacritics, and shaping behaved correctly as far as supported. Degradation parameters were also validated by human inspection in Latin, Greek, Cyrillic, Arabic so that degraded images remained legible to humans while still challenging models (Kargaran et al., 14 Apr 2026).

3. Data format and evaluation protocol

The released dataset includes clean and degraded image variants, ground-truth transcriptions, and metadata. The metadata includes script code in ISO 15924 form, language ID, and text source. The benchmark is explicitly a textline/sentence-level OCR resource and does not include bounding boxes or segmentation annotations (Kargaran et al., 14 Apr 2026).

Evaluation is based on three primary metrics: Character Error Rate (CER), Acc@k, and ScriptAcc. CER is defined as normalized character-level edit distance with whitespace ignored:

$$\text{CER} = \min\!\left(1, \frac{S + D + I}{N}\right),$$

where $S$ is the number of substitutions, $D$ deletions, $\pm 2^\circ$0 insertions, and $\pm 2^\circ$1 the reference length. The score is bounded by 1 (Kargaran et al., 14 Apr 2026).

A notable aspect of the protocol is that CER is computed under four normalizations and the minimum is taken: original output, reversed output string, lowercased output, and output with Unicode combining marks removed. This compensates for expected variants such as RTL/LTR reversal, case variation, and diacritic variation. A sentence counts as correct for acceptance-based metrics if any normalized output yields sufficiently low CER (Kargaran et al., 14 Apr 2026).

Acc@k is defined as the percentage of sentences with $\pm 2^\circ$2:

$\pm 2^\circ$3

where $\pm 2^\circ$4 is the number of sentences. GlotOCR reports Acc@0 for exact match and Acc@5 for near-perfect match with $\pm 2^\circ$5. The paper identifies Acc@5 as its primary accuracy metric because it acts as a coarse but interpretable measure of whether a model can function in a given script at all (Kargaran et al., 14 Apr 2026).

ScriptAcc measures whether the model output is in the correct script independent of transcription correctness. The dominant script of the prediction is inferred with GlotScript, and ScriptAcc is the fraction of sentences for which predicted script matches expected script. This metric is used to diagnose whether errors arise from fine-grained character confusion within the correct script or from cross-script hallucination into another script (Kargaran et al., 14 Apr 2026).

No WER is reported. The benchmark justifies this omission by noting that text spans many languages and scripts with differing tokenization conventions (Kargaran et al., 14 Apr 2026).

4. Evaluated systems and zero-shot setup

GlotOCR Bench evaluates 14 models, spanning OCR-centric systems, general multimodal VLMs used for OCR, and proprietary APIs. The open-weight OCR or OCR-oriented VLM models listed are dots.ocr, dots.mocr, olmOCR-2, RolmOCR, LightOnOCR-2, Nanonets-OCR2, PaddleOCR-VL-1.5, FireRed-OCR, GLM-OCR, DeepSeek-OCR-2, and HunyuanOCR. The benchmark also includes Qwen3-VL-8B as a general open-weight VLM that supports text recognition. The proprietary systems are Gemini 3.1 Flash-Lite and GPT-4.1 (Kargaran et al., 14 Apr 2026).

The paper characterizes most evaluated systems as VLM-based OCR models composed of a transformer or MoE visual encoder plus a LLM trained on document images. This framing is important because later analysis argues that performance strongly tracks script-level pretraining coverage and that current systems rely on language-model pretraining as much as on visual recognition (Kargaran et al., 14 Apr 2026).

All models are evaluated zero-shot using the uv-scripts/ocr inference suite. The prompt is a simple instruction to transcribe the text in the image, return it wrapped in tags, and provide no explanation. For chat-based APIs and chat-style VLMs, the chat template is used; otherwise, a raw prompt is applied. The benchmark passes the original rendered resolution directly to the model and does not introduce extra resizing or cropping at evaluation time (Kargaran et al., 14 Apr 2026).

A separate experiment is conducted only for GPT-4.1 using script-aware hinting. In that condition, the prompt provides the script, language, and the exact deduplicated set of characters appearing in the image. Short texts under 10 characters are excluded, leaving 149 scripts in the analysis (Kargaran et al., 14 Apr 2026).

5. Empirical findings across resource tiers

The main empirical result is a sharp tier-dependent discontinuity. On Latin, all models achieve Acc@5 ≥ 75%, and the best models exceed 90%. The strongest reported Latin result is Gemini 3.1 Flash-Lite with 0.9% CER and Acc@5 95.3%, with dots.mocr and dots.ocr close behind (Kargaran et al., 14 Apr 2026).

On the Mid-tier scripts—Arabic, Cyrillic, Devanagari, Han, Japanese, Hangul, Greek, Hebrew, and Thai—performance drops substantially but remains usable for the strongest systems. The benchmark reports Gemini Flash-Lite at CER 3.0, Acc@5 82.7% and dots.ocr at CER 5.0, Acc@5 78.3%. Averaged over models, the drop from High to Mid is 27.6 points in Acc@5 (Kargaran et al., 14 Apr 2026).

On the 148 Low-tier scripts, performance collapses. The best three models—Gemini, dots.ocr, and dots.mocr—achieve only Acc@5 ≈ 7.7%, and many other models remain below 1%. The macro-average Acc@5 drops from about 60.0% on Mid scripts to 2.3% on Low scripts, a 57.7-point collapse. The paper identifies this as Finding 1, describing it as a threshold phenomenon determined by pretraining coverage rather than a smooth degradation (Kargaran et al., 14 Apr 2026).

Across all tiers, Gemini 3.1 Flash-Lite has the best macro mean with Macro CER 27.7, Acc@0 52.4%, and Acc@5 61.9%, with dots.mocr and dots.ocr described as very close behind (Kargaran et al., 14 Apr 2026).

Per-language analysis within scripts shows that script identity alone does not determine difficulty. Latin typically has median Acc@5 above 90% but still exhibits substantial variation across languages. Devanagari shows lower medians and moderate spread, with complex conjuncts causing errors. Arabic is reported as the most difficult of the mid-tier scripts, with low medians and wide interquartile ranges, while Cyrillic is somewhat better. This intra-script variance is summarized as Finding 2 (Kargaran et al., 14 Apr 2026).

The relation between ScriptAcc and Acc@5 is strongly positive, but not identical. Scripts with high script recognition accuracy tend to have high OCR accuracy, yet the deviations are diagnostically important. Arabic has high ScriptAcc but lower Acc@5, indicating intra-script transcription difficulty. Hebrew has lower ScriptAcc because outputs are often misclassified as Thai, reducing Acc@5 through cross-script confusion. Japanese is reported to achieve surprisingly high Acc@5, even above Latin, despite mixing multiple writing systems. This pattern is reported as Finding 3 (Kargaran et al., 14 Apr 2026).

The script-aware hinting experiment with GPT-4.1 shows limited global improvement: among 149 scripts, 125 show no change, 21 improve, and 3 get worse, for an average gain of +0.7 percentage points in Acc@5. The largest improvement is on Han, with a gain of more than 20 points, and Cyrillic and Thai also improve notably. Some low-resource scripts such as Italic and Coptic move from near zero to roughly 5–10% Acc@5, but remain very poor overall. This is Finding 4, which the paper interprets as evidence that script-aware hinting is not the main bottleneck for low-resource scripts (Kargaran et al., 14 Apr 2026).

Under image degradation, all evaluated top models lose performance across all tiers. The benchmark notes moderate drops on Latin, larger absolute and relative drops on mid-resource scripts, and small absolute but large relative drops on low-resource scripts because baseline performance is already near zero. GPT-4.1 is described as slightly more robust among the top models, with a 13.8% relative drop on Latin. This behavior is summarized as Finding 5, with clean images functioning as an upper bound (Kargaran et al., 14 Apr 2026).

6. Failure modes, interpretation, limitations, and reproducibility

GlotOCR Bench devotes particular attention to cross-script hallucination, which it categorizes into four mutually exclusive prediction classes: Correct script, Cross-script hallucination, Silent, and Artifact. Averaged across scripts, the benchmark reports 12.5% correct-script outputs, 68.4% hallucinations, 6.0% silent outputs, and 13.1% artifacts. The dominant failure mode is therefore not abstention but production of fluent-looking text in an incorrect script (Kargaran et al., 14 Apr 2026).

Model-specific differences in hallucination behavior are also reported. dots.ocr is relatively conservative, with Correct 15.8%, Hallucination 38.3%, Silent 42.1%, and Artifacts 3.8%. Gemini Flash-Lite yields Correct 22.6%, Hallucination 70.2%, Silent 0.5%, and Artifacts 6.7%. olmOCR-2 reaches Correct 14.4%, Hallucination 81.7%, Silent 0.0%, and Artifacts 3.9%. DeepSeek-OCR-2 is noted for especially high artifact rates at 26.2% (Kargaran et al., 14 Apr 2026).

The appendix-level analysis identifies common hallucination attractors. Latin, Arabic, and Devanagari are the most frequent targets. Some substitutions reflect script proximity, such as Syriac → Arabic, Grantha → Tamil, Coptic → Greek, Newa → Devanagari, Tangut → Han, and Lisu → Latin. Others appear primarily distributional, such as Old Uyghur / Mongolian → Arabic or Ogham → Latin. This is reported as Finding 6, and the paper treats it as strong evidence of language-model bias overriding lack of visual familiarity (Kargaran et al., 14 Apr 2026).

The benchmark’s interpretive claim is that OCR performance broadly tracks script-level pretraining coverage and that modern OCR VLMs function as LLMs with a visual front-end. The evidence cited includes the tier-wise performance collapse, the strong relation between ScriptAcc and Acc@5, the limited effect of script-aware hints on low-resource scripts, and the prevalence of fluent hallucinations in high-resource scripts rather than random noise (Kargaran et al., 14 Apr 2026). This suggests that current systems are not script-agnostic recognizers of graphemic form.

Several limitations are explicit. The images are synthetic, not real scanned documents; the benchmark is predominantly single-line/single-block text; and real documents include additional complexities such as multi-column layouts, marginal annotations, stamps, and non-uniform scanning artifacts. Typographic variation is highly uneven because many low-resource scripts have only one usable font and very few samples. The pipeline does not support vertical layout and therefore renders vertical scripts horizontally. Text sources are biased toward web text and open resources, and some scripts rely on artificially constructed or transliterated text. The paper accordingly presents GlotOCR as a controlled, script-level evaluation environment and states that real-world performance is likely worse (Kargaran et al., 14 Apr 2026).

The benchmark is released with a Hugging Face dataset at https://hf.co/datasets/cis-lmu/glotocr-bench and a GitHub rendering and evaluation pipeline at https://github.com/cisnlp/glotocr-bench. The dataset contains images, transcriptions, script and language metadata, and source information, and is released under an evaluation-only license. The pipeline is released under Apache 2.0 and includes rendering, font selection, evaluation code for CER, Acc@k, ScriptAcc, and cross-script hallucination analysis, as well as scripts to regenerate benchmark images from seed text (Kargaran et al., 14 Apr 2026).

A plausible implication is that GlotOCR Bench functions not merely as a scorecard for current OCR systems but as a stress test for claims of multilingual OCR coverage. Within the benchmark’s own framing, OCR is effectively “near solved” only for Latin, workable for a narrow set of mid-resource scripts, and unusable for most of the remaining Unicode inventory under Acc@5 evaluation. The benchmark therefore foregrounds the gap between nominal Unicode support and actual zero-shot OCR competence across scripts (Kargaran et al., 14 Apr 2026).