OmniDocBench Evaluation Overview

• OmniDocBench is a comprehensive document parsing benchmark that standardizes evaluation across diverse document types with detailed annotations and multi-level evaluation protocols.
• It employs rigorous methodologies including OCR, table parsing, formula recovery, and adaptive matching with metrics like NED, TEDS, and CDM to quantify performance.
• The benchmark drives advances in state-of-the-art vision-language and multimodal models while addressing real-world challenges through dedicated physical evaluation tracks.

OmniDocBench Evaluation

OmniDocBench is a comprehensive document parsing benchmark designed to advance and rigorously standardize evaluation in structured content extraction across diverse document domains. The evaluation paradigm for OmniDocBench and its successors has shaped state-of-the-art systems, catalyzed developments in robust vision-LLMs (VLMs) and multimodal LLMs (MLLMs), and surfaced key limitations intrinsic to both digital and "in-the-wild" document parsing.

1. Definition, Coverage, and Evolution of OmniDocBench

OmniDocBench originated as a benchmark comprising high-quality annotations across nine document types, including academic papers, textbooks, slides, handwritten notes, and densely typeset newspapers, with a multi-level evaluation strategy encompassing 19 layout categories and 15 attribute labels (Ouyang et al., 2024). The initial v1.0 version consisted of 981 PDF pages, with later iterations (v1.5) expanding to 1,355 pages, providing balanced representation of English and Chinese, as well as increased complexity in layout and annotation precision (Cui et al., 29 Jan 2026, Zhou et al., 4 Mar 2026).

OmniDocBench’s core ambition is to drive fair, diverse, and granular assessment of document parsing systems—from modular pipelines to monolithic VLMs—and to provide a substrate supporting both end-to-end and module-specific evaluation. The task scope includes text recognition, table parsing, LaTeX formula recovery, structural layout modeling, and reading order.

Rapid saturation of OmniDocBench leaderboards by SOTA models achieving >90% overall scores motivated development of successor protocols and datasets—particularly OmniDocBench v1.6 (introducing multi-granularity adaptive matching and a "Hard" subset), Real5-OmniDocBench (physical scenario reconstruction), and Wild-OmniDocBench (robustness via real-world captures) (Wang et al., 6 Apr 2026, Zhou et al., 4 Mar 2026, Li et al., 25 Mar 2026).

2. Evaluation Protocols, Matching Algorithms, and Metric Formulation

OmniDocBench employs a rigorous multi-metric, multi-level evaluation protocol, designed to capture both block- and sequence-level correspondence. The standard pipeline involves:

• End-to-End Evaluation: Models convert PDF or captured page images into Markdown (or equivalent) structured output. Core subtasks are:
  • Text Extraction: Full transcription at block and page-level.
  • Formula Recovery: Extraction as LaTeX or MathML.
  • Table Parsing: Structural recovery as HTML/XML trees.
  • Reading Order: Ordered permutation of recognized blocks.
• Module-Specific Evaluation: Component-wise assessment (layout mAP, OCR nED, table TEDS, formula CDM).
• Attribute-Level Analysis: Stratified by document type and page attributes (e.g., column layout, scan artifacts, color background).

Key formal metrics:

• Normalized Edit Distance (NED):

$$\mathrm{NED} = \frac{\mathrm{ED}(p, g)}{\max(|p|, |g|)}$$

where $\mathrm{ED}$ denotes Levenshtein distance between prediction $p$ and ground-truth $g$.

• Tree Edit Distance Similarity (TEDS):

$$\mathrm{TEDS}(G_a, G_p) = 1 - \frac{\text{EditDist}(G_a, G_p)}{\max(|G_a|, |G_p|)}$$

for table structure alignment.

• Formula Character Detection Matching (CDM): F₁ score based on pixel-level, character localization in rendered LaTeX.
• Overall Score: Typically a (possibly weighted) mean over core metrics, e.g.,

$$\mathrm{Overall} = \frac{ [(1 - \mathrm{TextNED}) \times 100 + \mathrm{TEDS} + \mathrm{CDM}] }{3 }$$

OmniDocBench v1.6 introduced Multi-Granularity Adaptive Matching (MGAM), mitigating penalization due to block segmentation style by adaptively merging/splitting predictions for optimal matching to GT (Wang et al., 6 Apr 2026). This substantially increases discriminative power, especially for robust systems whose segmentation conventions differ from those of the annotators.

3. Physical-World Robustness: Real5-OmniDocBench and Wild-OmniDocBench

While digital benchmarks enabled rapid SOTA convergence in clean settings, the lack of controlled, ground-truth-aligned physical-world evaluation motivated creation of Real5-OmniDocBench (Zhou et al., 4 Mar 2026, Cui et al., 29 Jan 2026) and Wild-OmniDocBench (Li et al., 25 Mar 2026).

Real5-OmniDocBench is a full one-to-one physical reconstruction of the OmniDocBench v1.5 test set (1,355 digital pages × 5 scenarios = 6,775 images) across five canonical real-world perturbation axes:

Scenario Type	Key Subconditions
Scanning	Flatbed, multi-gen, slant, book-curve, clip/shadow
Warping	Fold, curve, crumple, dog-ear, spine
Screen-Photography	Moiré, glare, LCD/OLED device variety
Illumination	Low light, shadow, color-cast, reflection
Skew	Pitch, roll, yaw, compound, extreme tilt

Images inherit ground truth for layout, text, formulas, tables, and reading order, realizing a fully causal framework for factor-wise attribution of model failure—enabling diagnosis of geometric, photometric, and model-limited degradation.

Wild-OmniDocBench emphasizes "in-the-wild" physical capture: printed OmniDocBench pages are physically warped, crumpled, and photographed under variable lighting, or displayed on screens and re-photographed, to induce real-world distortion spanning folds, wrinkles, moiré, blur, and reflection (Li et al., 25 Mar 2026).

In both, evaluation strictly follows OmniDocBench protocol, permitting rigorous attribution from digital to real-world robustness loss.

4. Quantitative Results and Comparative System Behavior

Table: Representative Overall Scores Across Benchmarks

Model Type	Method	Digital (OmniDoc)	Real5 (Avg 5)	Δ (Wild) [O→W]
Pipeline Tools	Marker-1.8.2	94.8	60.1	-
	PP-StructureV3	96.4	64.5	-
General VLMs	Qwen3-VL-235B	98.2	88.9	-9.46
	Gemini-3 Pro	98.5	89.2	-
Specialized VLMs	PaddleOCR-VL-1.5	97.8	92.1	-19.74
End-to-End MLLM	DocHumming (Wild)	93.75	87.03	-6.72

For SOTA generalist VLMs, the mean performance drop from digital to physical is ~8–9 points. Pipeline-based solutions exhibit catastrophic degradation, typically 30+ points. Specialized compact VLMs, e.g., PaddleOCR-VL-1.5, lose ~5–6 points and achieve the highest real-world robustness. End-to-end trained MLLMs such as DocHumming, with structure-aware curriculum and synthetic data augmentation, show the least degradation (-6.72).

Component analysis reveals:

• Geometric distortions (warping, skew) primarily degrade table structural accuracy (TEDS, -10 to -40 points) and multiply reading order (RO) error.
• Optical artifacts (glare, moiré) induce localized OCR failures (Text NED -0.05 to -0.10).
• Generalist VLMs maintain semantic accuracy but lack built-in geometric correction; classic pipelines fail drastically under compounded artifacts.
• Domain-tuned VLMs with multi-task and synthetic training demonstrate stronger invariance to physical noise (Zhou et al., 4 Mar 2026, Li et al., 25 Mar 2026).

5. Auditing, Contamination, and Benchmark Limitations

Extensive audit pipelines have revealed inherent limitations in both the annotation process and evaluation reliability on OmniDocBench (Li et al., 8 May 2026):

• A three-stage audit (automatic dual-OCR, human+vision adjudication, relabeling) of the v1.5 test set demonstrated a 12.08% annotation-error rate (2,580/21,353 evaluated blocks). Errors include character substitutions (73%), structural annotation drift (25%), and miscategory (2%).
• 31.8% of test pages contain at least one score-influencing error.
• Public release of full annotation and document assets increases contamination risk. Over 20 open-source models released tend to overfit or memorize the benchmark, with leaderboard compression surpassing the annotation-error floor.
• Saturation above 90% overall score on clean data compresses the inter-model gap to less than the true annotation error rate, undermining leaderboard discrimination.

These limitations—together with the risk of benchmark-specific overfitting—motivate data-centric redesigns (source-traceable sets such as PureDocBench and adaptive matching protocols) and inclusion of real-capture evaluation tracks.

6. Key Insights for Practical Benchmarking and Model Development

• No single distortion axis dominates: Geometric factors inflict the greatest performance drop, but real-world failures compound multiple artifacts.
• Scaling model size is not sufficient for physical-world robustness. Inductive biases for geometry (e.g., polygonal layouts, geometric unwarping) and photometric invariance are critical, as evidenced by compact VLMs matching or surpassing larger competitors.
• Synthetic data augmentation and tailored training (e.g., realism-aware augmentation, progressive structure-token curriculum) substantially enhance robustness and mitigate repetitive or hallucinated output (Li et al., 25 Mar 2026).
• End-to-end architectures aligned with document structure display reduced vulnerability to segmentation failures, outperforming cascaded pipelines under capture-induced distortion.
• Benchmark design influences SOTA progress: The introduction of adaptive matching and hard/easy splits in evaluation protocols has restored discriminative power to leaderboard rankings, reflecting true progress on both standard and rare (‘long-tail’) document structures (Wang et al., 6 Apr 2026).

7. Open Challenges and Prospective Directions

Current limitations point to necessary directions for both evaluation and modeling:

• Incomplete coverage of real-world noise: Motion blur, heavy occlusion, and severe environmental interference are not yet systematically modeled in current tracks (Zhou et al., 4 Mar 2026, Li et al., 25 Mar 2026).
• Annotation and contamination: Source-traceable (programmatic) ground truth and robust, multi-stage QA pipelines are required to ensure long-term reliability and fair comparison (Li et al., 8 May 2026).
• Modeling advancements needed: Research is underway towards end-to-end geometric unwarping, photometric-invariant encoders, and 3D scene-aware reasoners for document parsing.
• Evaluation must encompass multi-track (clean, degraded, real-capture) settings, with metrics moving beyond edit distances toward layout fidelity, semantic completeness, and block-level equivalence.

These advances in benchmark design, evaluation rigor, and model architecture collectively define the state and trajectory of OmniDocBench and its evaluation protocols within document intelligence research.