---
title: Composite Grounded-OCR Score
url: https://www.emergentmind.com/topics/composite-grounded-ocr-score
type: topic
---

# Composite Grounded-OCR Score

The composite grounded-OCR score is a unified evaluation metric for vision-language models performing grounded optical character recognition (OCR) across reading and detection tasks on documents. It quantifies system-level performance using an unweighted average over six core task families encompassing structured and layout-sensitive reading along with both unconditional and conditional detection, explicitly grounding all evaluations in the document image and its spatial-structural annotations. This score is integral to the evaluation framework introduced in "GutenOCR: A Grounded Vision–Language Front-End for Documents," and is designed for broad applicability to business documents, scientific literature, and synthetic grounded data [2601.14490].

## 1. Formal Definition and Task Decomposition

The composite grounded-OCR score $C \in [0,1]$ is formally defined as the mean of six subtask scores, each corresponding to a core grounded-OCR capability:

- $T_1$: Full-page plain reading (“text”)
- $T_2$: Full-page layout-sensitive reading (“text2d”)
- $T_3$: Full-page structured reading; lines serialized in order (CER$_{\text{e2e}}$)
- $T_4$: Localized reading (specified region/box)
- $T_5$: Full-page detection (e.g., line/paragraph box detection)
- $T_6$: Conditional detection (detection conditioned on a query, e.g., “where is x?”)

For each reading task $T_i$ that produces a string output, a normalized character error rate $\epsilon_i \in [0,1]$ is evaluated:
- For $T_1, T_2, T_4$: $\epsilon_i = \text{CER}$
- For $T_3$: $\epsilon_3 = \text{CER}_{\text{e2e}}$ (after line-box alignment and serialization)

Detection subtasks ($T_5$, $T_6$) are evaluated using $F_1$ at Intersection-over-Union (IoU) threshold 0.5:
- $F_5 = \text{F}_1@0.5$
- $F_6 = \text{F}_1@0.5$ (for matches conditioned on the query)

All reading errors are converted to scores $s_i = 1 - \epsilon_i$ (higher is better), and the composite score is:

$$
C = \frac{1}{6}\left[(1-\epsilon_1) + (1-\epsilon_2) + (1-\epsilon_3) + (1-\epsilon_4) + F_5 + F_6\right]
$$

This framework ensures consistent scaling, comparability, and equal weighting across diverse grounded OCR sub-tasks.

## 2. Evaluation Protocol and Dataset Composition

The evaluation protocol is applied over both in-domain and out-of-domain benchmarks, with the principal in-domain set consisting of 10,500 held-out pages. The ground-truth preparation involves Unicode NFKC normalization and whitespace cleanup for transcripts, with bounding boxes clipped to image bounds and degenerate boxes eliminated. Pages are rasterized at 72 dpi for standardization.

**In-domain evaluation sources:**
- OCR-IDL (noisy business documents: IDs, invoices, forms)
- TabMe++ (business documents processed by alternative OCR engines)
- PubMed-OCR (multi-column scientific articles, including mathematics and specialized symbols)

**Sub-benchmark protocols:**
- *Full-page reading*: CER on plain or layout-annotated text
- *Structured reading*: CER$_{\text{e2e}}$ after matching ordered line boxes
- *Localized reading*: CER on crops given GT boxes
- *Detection*: $F_1$@IoU~$0.5$ on lines/paragraphs
- *Conditional detection*: $F_1$@IoU~$0.5$ for query-matched lines

**External/transfer benchmarks:**
- *Fox* (English OCR): evaluates token-F1 and CER for page OCR; region/line/color-guided OCR using CER on region crops
- *OmniDocBench v1.5*: evaluates component-level text recognition by CER, formula recognition by CER and CDM, and text detection (recall@0.5 for partial ground-truth spans; precision is undefined)

*This protocol ensures fine-grained, multi-task, and cross-domain assessment of both content and spatial layout recognition ability.*

## 3. Aggregation and Weighting Strategy

All six task families are assigned equal weight in the computation of the composite score. This design choice avoids biasing the metric toward any single grounded-OCR capability, reflecting a holistic system requirement.

- **Reading tasks ($T_1$–$T_4$):** Assessed by (1–CER), directly penalizing transcription errors at the character level or, where specified, after end-to-end box-to-string serialization.
- **Detection tasks ($T_5$, $T_6$):** Assessed by $F_1@0.5$, rewarding balanced recall and precision at reasonable spatial alignment thresholds.

Each subscore is normalized and combined without further task-specific weighting. This uniformity facilitates interpretability and system-level comparison across markedly different VLM front-ends or OCR paradigms.

## 4. Comparative Performance: Qwen2.5-VL-7B vs. GutenOCR-7B

Empirical results on the in-domain evaluation suite (10.5K held-out pages) reveal substantial improvements when upgrading from the Qwen2.5-VL-7B backbone to GutenOCR-7B:

| Subtask                              | Qwen2.5-VL-7B    | GutenOCR-7B   | Relative Improvement |
|---------------------------------------|------------------|---------------|---------------------|
| Composite grounded-OCR (all 6 tasks)  | 0.396            | 0.819         | +0.423 (>100%)      |
| Text CER ($T_1$)                      | 0.333            | 0.202         | –39% rel.           |
| Text2d CER ($T_2$)                    | 0.522            | 0.280         | –46% rel.           |
| Lines CER$_{\text{e2e}}$ ($T_3$)      | 0.633            | 0.147         | –77% rel.           |
| Local CER ($T_4$)                     | 0.530            | 0.129         | –76% rel.           |
| Detection F1 ($T_5$)                  | 0.111            | 0.787         | +609% rel.          |
| Cond-Detection F1 ($T_6$)             | 0.285            | 0.882         | +210% rel.          |

Results indicate that model specialization via grounded fine-tuning yields the largest relative gains in structured reading (T₃, T₄) and detection (T₅, T₆), with moderate but meaningful reductions in error for plain and layout-sensitive full-page reading.

## 5. Observed Trade-offs and Failure Modes

The multidimensionality of the composite score enables the identification of systematic trade-offs resulting from model specialization:

- **Page-level linearization vs. layout sensitivity (Fox):** GutenOCR-7B maintains high content token-F1 (≈0.973 vs. 0.984 for Qwen2.5-VL-7B), yet page CER increases (0.025→0.164) as model adherence to 2D document structure supersedes the “canonical” Markdown order criteria of Fox.
- **Region and line-pointer OCR (Fox):** Region CER improves from 0.163→0.067; line CER from 0.701→0.211, demonstrating more accurate spatially-grounded reading.
- **Color-guided OCR (Fox):** CER degrades dramatically from 0.109 to 0.963 (catastrophic forgetting), reflecting a lack of color-pointer examples in the GutenOCR-7B fine-tuning mix.
- **Out-of-domain detection (OmniDocBench):** Recall@0.5 increases from ~0.02 to 0.55, validating cross-domain transfer of line detection skills, but with precision undefined due to incomplete ground truth.
- **Formula recognition (OmniDocBench):** CDM slightly decreases (0.935→0.927), and formula CER increases marginally (0.216→0.221), indicating mild negative transfer for formula-dense pages.

These results exemplify how the composite grounded-OCR score enables rigorous auditing of model strengths and weaknesses across a spectrum of real-world document analyses.

## 6. Relationship to Other Composite OCR Evaluation Metrics

While the composite grounded-OCR score targets multidimensional grounded OCR in documents, alternative metrics such as DISGO WER [2308.13173] have been introduced for scene-text OCR. DISGO WER combines errors from deletions, insertions, substitutions, and grouping/ordering at the word level, using IoU-based alignment and detailed error codes. Its design emphasizes word-level evaluation, layout (block/order) errors, and diagnostic decomposability. A plausible implication is that the composite grounded-OCR score’s task-diverse, end-to-end emphasis fills a complementary role for structured document analysis, whereas DISGO WER provides more granular diagnostics at the word/block level in scene OCR.

## 7. Significance and Implications for Vision-Language Model Evaluation

The composite grounded-OCR score establishes a rigorous, interpretable standard for evaluating unified document understanding in vision-language models. Its equal-weighted, task-spanning design avoids overfitting to a single dimension of performance and reflects the end-user requirement of simultaneous competence in reading, spatial understanding, structure extraction, and conditional search. As vision-language architectures continue to evolve, this composite metric provides a foundation for quantifying real-world OCR capability, benchmarking progress, and exposing the nuanced trade-offs introduced by model specialization and dataset curation [2601.14490].

Source: https://www.emergentmind.com/topics/composite-grounded-ocr-score