Real5-OmniDocBench: Robust Parsing Benchmark

Updated 27 June 2026

Real5-OmniDocBench is a physically reconstructed benchmark that evaluates document parsing models across five distortion channels with one-to-one digital/physical mapping.
The benchmark employs controlled capture methods for scanning, warping, screen-photo, illumination, and skew to enable precise, factor-wise performance analysis.
Key results show that compact, distortion-aware models maintain >90% accuracy under real-world distortions, while general-purpose pipelines suffer significant degradation.

Real5-OmniDocBench is a full-scale, physically reconstructed benchmark designed to systematically evaluate document parsing models under realistic, in-the-wild distortions, explicitly exposing the resilience (or brittleness) of state-of-the-art systems outside laboratory digital test conditions. Originating from the OmniDocBench v1.5 test set (1,355 pages), Real5-OmniDocBench introduces five physically instantiated distortion channels—Scanning, Warping, Screen-Photography, Illumination, and Skew—yielding 6,775 images with one-to-one digital/physical ground-truth mapping. The benchmark supports rigorous factor-wise analysis of model performance degradation, providing critical insights for the development of robust document parsing pipelines and vision-LLMs (VLMs) suited for real deployment scenarios (Zhou et al., 4 Mar 2026, Cui et al., 29 Jan 2026).

1. Physical Benchmark Construction and Scenarios

Real5-OmniDocBench reconstructs every page from OmniDocBench v1.5 as five distinct physical variants, using tightly controlled procedures and a standardized apparatus:

Reconstruction Pipeline: Each of the 1,355 source pages is printed at 1200 dpi on a Canon C5840, using an A3/A4-aware workflow to preserve original layout spatiality.
Distortion Scenarios:
- Scanning: Pages scanned under five setups (standard, low-quality, slanted, stapled, bound) introduce blur, misalignment, and edge artifacts.
- Warping: Non-rigid deformations including folding, cylindrical curvature, crumpling, dog-ear, and book-spine arc. Realizations involve capturing the physical effects with smartphones of varying ISP/sensor designs.
- Screen-Photography: Pages displayed on diverse screens (office monitor, professional monitor, laptop, tablet, mobile) and photographed hand-held. Modulates moiré, glare, pixelation, and color fidelity as a function of $f_\text{moiré}=|f_\text{display}-f_\text{sensor}|$ .
- Illumination: Captured under lighting diversity (low-light, strong shadow, color-cast, directed flashlight, glass refraction); brightness modeled as $B(x,y)=B_0(1+\alpha \exp(-((x-x_0)^2+(y-y_0)^2)/\sigma^2))$ .
- Skew: Captures at varied poses—pitch, roll, yaw up to $\sim\!60^\circ$ —with geometric transformation modeled by homography $H=K\cdot R(\theta_p,\theta_r,\theta_y)\cdot K^{-1}$ .
Annotation Protocol: Ground-truth transcribed directly from the original digital pages, including region-level class, bounding box, text, table structure, formula (LaTeX), and reading order. After physical capture, correspondence validation is conducted with both automated anomaly detection (VLM committee) and multiple manual audits.

This systematic protocol preserves semantic and geometric ground-truth correspondence across all instances, uniquely enabling factor-wise attribution of model failures.

2. Dataset Composition and Evaluation Protocol

Real5-OmniDocBench encompasses 5 distinct distortion subsets, each mapped one-to-one to the source corpus, and maintains exact split parity for meaningful model comparison:

Distortion	Capture Modality	Sample Count	Key Artifacts / Effects
Scanning	Flatbed/A3-A4 scanner	1,355	Blur, grain, misalign
Warping	Physical deformation + photo	1,355	Curvature, folds
Screen-photo	Hand-held camera of screen	1,355	Moiré, glare, tilt
Illumination	Variable lighting setups	1,355	Shadows, color shift
Skew	Hand-held, extreme tilt	1,355	3D perspective, trapezoid
Total		6,775

Test-Only Suite: There are no validation/train splits; Real5-OmniDocBench is strictly for test-time robustness evaluation.
Ground-Truth Inheritance: Layout, table, formula, reading-order, and region-level annotations are inherited unaltered from OmniDocBench v1.5 (Ouyang et al., 2024, Cui et al., 29 Jan 2026).

The evaluation protocol exactly matches that of OmniDocBench: character-level Normalized Edit Distance (NED) for text, Character Detection Matching (CDM) for formulas, Tree-Edit-Distance-based Similarity (TEDS) for tables, and sequence NED for reading order. An overall score aggregates the three sub-task metrics as a weighted sum (Cui et al., 29 Jan 2026).

3. Factor-wise Performance Attribution

Real5-OmniDocBench's unprecedented physical one-to-one mapping enables precise, scenario-level performance drop analysis. For any scenario $s$ :

$\Delta_s = \text{Overall}_{\text{digital}} - \text{Overall}_s,\quad \text{RelDrop}_s = \frac{\Delta_s}{\text{Overall}_{\text{digital}}}\times 100\%$

This enables partitioning failures by geometric/optical root cause, an analysis impossible with synthetically degraded or partially sampled benchmarks.

Geometric Distortions (Warping, Skew): Induce the steepest accuracy degradation, with TEDS dropping sharply in pipeline models and only modestly in VLMs tuned for geometric invariance.
Optical Artifacts (Screen-photo, Illumination): Cause 5–7% relative drops, primarily due to moiré, glare, and shadows fragmenting layout or occluding content.
Empirical Failures: Pipelines show high variance and severe RoA drops on non-rigid and perspective-affected inputs; state-of-the-art compact VLMs (e.g., PaddleOCR-VL-1.5) retain $>$ 90% accuracy overall (Cui et al., 29 Jan 2026).

4. Results: Robustness Benchmarking Across Architectures

The following table summarizes scenario-wise and overall accuracy for representative models, as reported in (Cui et al., 29 Jan 2026, Zhou et al., 4 Mar 2026):

Model	Scanning	Warping	Screen-photo	Illumination	Skew	Overall
PaddleOCR-VL-1.5 (0.9B)	93.43	91.25	91.76	92.16	91.66	92.05
Qwen3-VL-235B	89.43	89.99	89.27	89.27	86.56	88.90
Gemini-3 Pro	89.47	88.90	88.86	89.53	89.45	89.24
PP-StructureV3	84.68	59.34	66.89	73.38	37.98	64.45

Compact Specialist VLMs (PaddleOCR-VL-1.5): Achieve SOTA robustness across all scenarios and outperform larger general-purpose VLMs (Qwen3-VL-235B, Gemini-3) by 2–4 pp overall, with relative drops below 5% even on Skew and Warping.
Generalist VLMs: Perform well on semantic tasks but do not automatically guarantee geometric invariance. Their performance flattens as scenario complexity increases.
Pipeline Methods: Exhibit large variance and catastrophic failure, particularly under severe warping/skew (e.g., PP-StructureV3 drops from 84.68% in Scanning to 37.98% in Skew).

5. Domain-Specific Robustness Drivers

Key architectural and training features that underpin Real5-OmniDocBench robustness:

Polygonal/Quadrilateral Layout Modeling: PP-DocLayoutV3 eschews axis-aligned boxes for multi-point segmentation, preserving element boundaries on curved/skewed pages.
Distortion-Aware Pretraining/Augmentation: Layout and recognition models undergo augmentation with Moiré, geometric skew, and low-light during training; Uncertainty-Aware Cluster Sampling accentuates hard negative cases (extreme warping, illumination).
Task-Specific Fine-Tuning: Modules for seal recognition and 4-point curved text-spotting boost rare and outlier scenario performance.
Joint Reading-Order Prediction: Pairwise anti-symmetric scoring ( $S_{i,j}$ ) and voting mitigate non-linear traversal errors.

These features collectively contribute to a 14.19 pp improvement on Skewed pages and consistent gains ( $\sim$ 3–5 pp) on Screen-photo and Illumination scenarios.

6. Implications, Research Directions, and Recommendations

Persistent Reality Gap: Even SOTA VLMs experience a 4–7% absolute drop moving from digital to physically distorted pages, with geometric warping and 3D skew responsible for the largest errors (Zhou et al., 4 Mar 2026).

Architectural Strategy:

Integrate explicit geometric unwarping, moiré/glare removal, and multi-task layout restoration as front-end modules.
Employ physically plausible augmentations and scenario-level validation for new architectures.
Focus on compact, highly specialized models for resource-efficient, real-world deployments, as parameter scaling alone does not confer geometric invariance.

Diagnostic Value: Real5-OmniDocBench provides the first rigorous, multi-factor, full-coverage diagnostic for document parsing robustness. Researchers are encouraged to report per-scenario and factor-wise RelDrop as a standard for parser evaluation.

Future Work:

Development of joint restoration-parsing models able to "see through" severe physical artifacts.
Extension to further real-world phenomena, such as historical scans and non-Latin scripts.
Richer structural metrics for block completeness, layout fidelity, and multi-page coherence.

7. Conclusion

Real5-OmniDocBench offers the document intelligence community a comprehensive, physically grounded, and annotation-aligned testbed for in-the-wild document parsing. Its design sets a new standard for robust system evaluation, closing the gap between digital-domain assurance and real-world reliability. The empirical findings—particularly the superiority of distortion-aware, compact VLMs over even much larger general-purpose systems—will inform both theoretical and applied advances in the field (Zhou et al., 4 Mar 2026, Cui et al., 29 Jan 2026).