Composite Grounded-OCR Score

• Composite Grounded-OCR Score is a unified metric for evaluating vision-language models on both reading and detection tasks in document images.
• The method decomposes performance into six equally weighted subtasks, covering plain, layout-sensitive, structured, localized, and conditional detection.
• Empirical results demonstrate that models like GutenOCR-7B significantly improve reading accuracy and detection F1 scores compared to previous benchmarks.

The composite grounded-OCR score is a unified evaluation metric for vision-LLMs performing grounded optical character recognition (OCR) across reading and detection tasks on documents. It quantifies system-level performance using an unweighted average over six core task families encompassing structured and layout-sensitive reading along with both unconditional and conditional detection, explicitly grounding all evaluations in the document image and its spatial-structural annotations. This score is integral to the evaluation framework introduced in "GutenOCR: A Grounded Vision–Language Front-End for Documents," and is designed for broad applicability to business documents, scientific literature, and synthetic grounded data (Heidenreich et al., 20 Jan 2026).

1. Formal Definition and Task Decomposition

The composite grounded-OCR score $C \in [0,1]$ is formally defined as the mean of six subtask scores, each corresponding to a core grounded-OCR capability:

• $T_1$: Full-page plain reading (“text”)
• $T_2$: Full-page layout-sensitive reading (“text2d”)
• $T_3$: Full-page structured reading; lines serialized in order (CER$_{\text{e2e}}$)
• $T_4$: Localized reading (specified region/box)
• $T_5$: Full-page detection (e.g., line/paragraph box detection)
• $T_6$: Conditional detection (detection conditioned on a query, e.g., “where is x?”)

For each reading task $T_i$ that produces a string output, a normalized character error rate $\epsilon_i \in [0,1]$ is evaluated:

• For $T_1$0: $T_1$1
• For $T_1$2: $T_1$3 (after line-box alignment and serialization)

Detection subtasks ($T_1$4, $T_1$5) are evaluated using $T_1$6 at Intersection-over-Union (IoU) threshold 0.5:

• $T_1$7
• $T_1$8 (for matches conditioned on the query)

All reading errors are converted to scores $T_1$9 (higher is better), and the composite score is:

$T_2$0

This framework ensures consistent scaling, comparability, and equal weighting across diverse grounded OCR sub-tasks.

2. Evaluation Protocol and Dataset Composition

The evaluation protocol is applied over both in-domain and out-of-domain benchmarks, with the principal in-domain set consisting of 10,500 held-out pages. The ground-truth preparation involves Unicode NFKC normalization and whitespace cleanup for transcripts, with bounding boxes clipped to image bounds and degenerate boxes eliminated. Pages are rasterized at 72 dpi for standardization.

In-domain evaluation sources:

• OCR-IDL (noisy business documents: IDs, invoices, forms)
• TabMe++ (business documents processed by alternative OCR engines)
• PubMed-OCR (multi-column scientific articles, including mathematics and specialized symbols)

Sub-benchmark protocols:

• Full-page reading: CER on plain or layout-annotated text
• Structured reading: CER$T_2$1 after matching ordered line boxes
• Localized reading: CER on crops given GT boxes
• Detection: $T_2$2@IoU~$T_2$3 on lines/paragraphs
• Conditional detection: $T_2$4@IoU~$T_2$5 for query-matched lines

External/transfer benchmarks:

• Fox (English OCR): evaluates token-F1 and CER for page OCR; region/line/color-guided OCR using CER on region crops
• OmniDocBench v1.5: evaluates component-level text recognition by CER, formula recognition by CER and CDM, and text detection ([email protected] for partial ground-truth spans; precision is undefined)

This protocol ensures fine-grained, multi-task, and cross-domain assessment of both content and spatial layout recognition ability.

3. Aggregation and Weighting Strategy

All six task families are assigned equal weight in the computation of the composite score. This design choice avoids biasing the metric toward any single grounded-OCR capability, reflecting a holistic system requirement.

• Reading tasks ($T_2$6–$T_2$7): Assessed by (1–CER), directly penalizing transcription errors at the character level or, where specified, after end-to-end box-to-string serialization.
• Detection tasks ($T_2$8, $T_2$9): Assessed by $T_3$0, rewarding balanced recall and precision at reasonable spatial alignment thresholds.

Each subscore is normalized and combined without further task-specific weighting. This uniformity facilitates interpretability and system-level comparison across markedly different VLM front-ends or OCR paradigms.

4. Comparative Performance: Qwen2.5-VL-7B vs. GutenOCR-7B

Empirical results on the in-domain evaluation suite (10.5K held-out pages) reveal substantial improvements when upgrading from the Qwen2.5-VL-7B backbone to GutenOCR-7B:

Subtask	Qwen2.5-VL-7B	GutenOCR-7B	Relative Improvement
Composite grounded-OCR (all 6 tasks)	0.396	0.819	+0.423 (>100%)
Text CER ($T_3$1)	0.333	0.202	–39% rel.
Text2d CER ($T_3$2)	0.522	0.280	–46% rel.
Lines CER$T_3$3 ($T_3$4)	0.633	0.147	–77% rel.
Local CER ($T_3$5)	0.530	0.129	–76% rel.
Detection F1 ($T_3$6)	0.111	0.787	+609% rel.
Cond-Detection F1 ($T_3$7)	0.285	0.882	+210% rel.

Results indicate that model specialization via grounded fine-tuning yields the largest relative gains in structured reading (T₃, T₄) and detection (T₅, T₆), with moderate but meaningful reductions in error for plain and layout-sensitive full-page reading.

5. Observed Trade-offs and Failure Modes

The multidimensionality of the composite score enables the identification of systematic trade-offs resulting from model specialization:

• Page-level linearization vs. layout sensitivity (Fox): GutenOCR-7B maintains high content token-F1 (≈0.973 vs. 0.984 for Qwen2.5-VL-7B), yet page CER increases (0.025→0.164) as model adherence to 2D document structure supersedes the “canonical” Markdown order criteria of Fox.
• Region and line-pointer OCR (Fox): Region CER improves from 0.163→0.067; line CER from 0.701→0.211, demonstrating more accurate spatially-grounded reading.
• Color-guided OCR (Fox): CER degrades dramatically from 0.109 to 0.963 (catastrophic forgetting), reflecting a lack of color-pointer examples in the GutenOCR-7B fine-tuning mix.
• Out-of-domain detection (OmniDocBench): [email protected] increases from ~0.02 to 0.55, validating cross-domain transfer of line detection skills, but with precision undefined due to incomplete ground truth.
• Formula recognition (OmniDocBench): CDM slightly decreases (0.935→0.927), and formula CER increases marginally (0.216→0.221), indicating mild negative transfer for formula-dense pages.

These results exemplify how the composite grounded-OCR score enables rigorous auditing of model strengths and weaknesses across a spectrum of real-world document analyses.

6. Relationship to Other Composite OCR Evaluation Metrics

While the composite grounded-OCR score targets multidimensional grounded OCR in documents, alternative metrics such as DISGO WER (Hwang et al., 2023) have been introduced for scene-text OCR. DISGO WER combines errors from deletions, insertions, substitutions, and grouping/ordering at the word level, using IoU-based alignment and detailed error codes. Its design emphasizes word-level evaluation, layout (block/order) errors, and diagnostic decomposability. A plausible implication is that the composite grounded-OCR score’s task-diverse, end-to-end emphasis fills a complementary role for structured document analysis, whereas DISGO WER provides more granular diagnostics at the word/block level in scene OCR.

7. Significance and Implications for Vision-LLM Evaluation

The composite grounded-OCR score establishes a rigorous, interpretable standard for evaluating unified document understanding in vision-LLMs. Its equal-weighted, task-spanning design avoids overfitting to a single dimension of performance and reflects the end-user requirement of simultaneous competence in reading, spatial understanding, structure extraction, and conditional search. As vision-language architectures continue to evolve, this composite metric provides a foundation for quantifying real-world OCR capability, benchmarking progress, and exposing the nuanced trade-offs introduced by model specialization and dataset curation (Heidenreich et al., 20 Jan 2026).