---
title: 'OmniParsingBench: Unified Multimodal Parsing'
url: https://www.emergentmind.com/topics/omniparsingbench
type: topic
---

# OmniParsingBench: Unified Multimodal Parsing

OmniParsingBench is a multimodal parsing benchmark introduced in the "Logics-Parsing-Omni Technical Report" to unify evaluation across documents, static images, and audio-visual streams, while operationalizing a three-level parsing paradigm that bridges low-level perception and high-level cognition [2603.09677]. It is designed around the premise that multimodal parsing has been fragmented into separate pipelines for OCR/layout, chart parsing, video captioning, and related tasks, and that high-level semantic descriptions often lack rigorous grounding in detected facts and tend to hallucinate. The benchmark therefore couples hierarchical task decomposition with an evidence anchoring requirement: semantic outputs must cite the factual sub-outputs—such as boxes, timestamps, and symbols—from which they derive, so that unstructured signals can be transformed into standardized knowledge that is locatable, enumerable, and traceable [2603.09677].

## 1. Motivation and benchmark objective

OmniParsingBench was created to address three stated goals: to unify evaluation across documents, static images, and audio-visual streams; to evaluate a three-level parsing paradigm that bridges low-level perception and high-level cognition; and to enforce evidence anchoring by requiring all semantic answers to cite the factual sub-outputs from which they derive [2603.09677]. In the associated framework, multimodal parsing is not treated as a single monolithic prediction problem. Instead, it is decomposed into hierarchical levels that progress from spatial-temporal grounding, to symbolization and attribute extraction, to semantic and logical interpretation.

The benchmark is therefore aligned with the broader Omni Parsing framework rather than with a single modality-specific task family. In the source report, the framework integrates three hierarchical levels: **Holistic Detection**, which achieves precise spatial-temporal grounding of objects or events to establish a geometric baseline for perception; **Fine-grained Recognition**, which performs symbolization and attribute extraction on localized objects to complete structured entity parsing; and **Multi-level Interpreting**, which constructs a reasoning chain from local semantics to global logic [2603.09677]. A pivotal advantage claimed for this organization is the evidence anchoring mechanism, which enforces strict alignment between high-level semantic descriptions and low-level facts.

A plausible implication is that OmniParsingBench is intended not merely to compare end-task outputs, but to diagnose whether high-level reasoning quality is supported by verifiable low-level perception. That emphasis distinguishes it from benchmarks in which captioning or question answering can be scored without checking whether the underlying claims are grounded in identifiable evidence.

## 2. Domain coverage and held-out test composition

OmniParsingBench consists of six domain-specific modules, each with a held-out test split, and no training or fine-tuning is performed on these test sets [2603.09677]. The benchmark spans page images, static natural scenes, charts and geometry diagrams, audio clips, general video, and text-rich long-form video.

| Module | Test composition | Modality emphasis |
|---|---|---|
| Document Parsing (LogicsDocBench) | 900 page-level PDF images covering OCR/layout, tables, formulas | Documents |
| Natural Image Parsing | 1,000 general scenes + 1,000 knowledge-aware scenes with unambiguous real-world entities | Images |
| Graphics Parsing | 500 charts + 500 single-image geometry problems | Charts and geometry |
| Audio Parsing | 2,434 audio clips, sampled from AudioSet-strong and Librispeech | Speech and acoustic events |
| Natural Video Parsing | 1,100 general video segments + 725 camera-motion clips | General video |
| Text-Rich Video Parsing | 452 long-form instructional videos with frame-level OCR, segment-level ASR, and global captions | Long-form instructional video |

The dataset composition is itself part of the benchmark’s design argument. Document Parsing covers OCR/layout, tables, and formulas. Natural Image Parsing includes both “general” scenes and “knowledge-aware” scenes with unambiguous real-world entities such as landmarks, brands, and species. Graphics Parsing is divided into a chart submodule with line/bar/pie charts and flowcharts, and a geometry submodule with single-image geometry problems. The audio and video modules extend the benchmark beyond conventional document-vision evaluation by incorporating speech, acoustic events, camera motion, frame-level OCR, segment-level ASR, and global narrative outputs [2603.09677].

This breadth matters because the benchmark is intended to test a unified parsing stack rather than isolated subsystems. The held-out, read-only test design also makes the reported setting explicitly evaluative rather than adaptive.

## 3. Hierarchical taxonomy and benchmark tasks

OmniParsingBench evaluates each domain along three hierarchical parsing levels, denoted **L1**, **L2**, and **L3** [2603.09677]. These levels are both a taxonomy and a task decomposition.

At **L1 (Holistic Detection)**, the benchmark scores spatial or temporal grounding of bounding boxes or segments for objects, text blocks, chart regions, video events, camera shots, and audio events. The task is defined as follows: given raw input, localize all relevant instances with bounding-box or timestamp outputs and assign a coarse category label [2603.09677]. In effect, L1 establishes the geometric or temporal substrate on which later parsing stages depend.

At **L2 (Fine-grained Recognition)**, the benchmark scores symbol extraction and attribute extraction. The reported examples include OCR text, table cells, code blocks, object labels, chart data points, geometric primitives, acoustic labels, and speaker IDs. The task is defined as extracting structured symbols, normalized coordinates for geometry, or speech transcripts and acoustic tags on each L1 region [2603.09677]. Here the benchmark moves from mere localization to structured entity parsing.

At **L3 (Multi-level Interpreting)**, the benchmark scores semantic captioning, question answering, chart interpretations such as HTML or mermaid-code, geometric reasoning, video narratives, course summaries, and logical induction, including chart trends, geometry proofs, and audio-visual causal chains [2603.09677]. The task is defined as using L1 and L2 results to produce a structured, evidence-anchored interpretation in JSON, such as a global caption, QA-style reasoning output, or code/table reverse rendering.

The benchmark’s internal categories make clear that “parsing” is being used in a broad sense. It includes not only region detection and token recognition, but also semantic normalization and logically structured explanation. This suggests a deliberate attempt to bridge perception-oriented evaluation and cognition-oriented evaluation within one benchmark schema.

## 4. Evidence anchoring and the JSON evaluation workflow

The defining procedural feature of OmniParsingBench is its **evidence anchoring mechanism** [2603.09677]. Every L3 output must reference the exact L1/L2 facts that justify each semantic statement; the report gives the example “Box#3: ‘Bar in April = 42’.” Bench scripts then verify anchoring by matching cited identifiers—box IDs, timestamps, and symbol IDs—against the ground-truth L1/L2 annotations. Failure to anchor a semantic claim is counted as an L3 error [2603.09677].

This requirement changes the meaning of semantic correctness. A semantically plausible answer is insufficient if it does not cite the factual sub-outputs from which it derives. In that sense, the benchmark treats hallucination as a grounding failure rather than merely as an incorrect final answer.

The evaluation procedure is correspondingly standardized. For each sample, the model produces a **single JSON** containing L1, L2, and L3 outputs. The benchmark scripts then extract detections for IoU and mAP, text for CER and WER, and semantic outputs for EM, KR, and LC. Domain-level **Perception**, **Cognition**, and **Overall** scores are computed, and a unified leaderboard ranks models by the sum of their Overall scores across domains [2603.09677].

Because the JSON contains all three levels, the benchmark supports cross-level consistency checks rather than isolated module scoring. A plausible implication is that the benchmark encourages architectures that can retain traceability from raw evidence to final semantic claims, instead of post hoc explanation layers detached from the perception stack.

## 5. Metrics and score aggregation

OmniParsingBench groups metrics by level and domain, then aggregates them into two composite scores, **Perception** for L1+L2 and **Cognition** for L3. The reported **Overall** score per domain is the simple average of Perception and Cognition [2603.09677].

For **L1 detection**, the benchmark uses Intersection-over-Union for each predicted box $b$ and ground-truth $g$:
$$
\mathrm{IoU}(b,g)=\frac{\lvert b\cap g\rvert}{\lvert b\cup g\rvert}.
$$
It also reports Recall@$ \tau_{\mathrm{IoU}} $ and Precision@$ \tau_{\mathrm{IoU}} $:
$$
R(\tau)=\frac{\#\{g:\exists\,b,\mathrm{IoU}(b,g)\ge \tau\}}{\#\{g\}},
\qquad
P(\tau)=\frac{\#\{b:\exists\,g,\mathrm{IoU}(b,g)\ge \tau\}}{\#\{b\}}.
$$
Mean Average Precision is defined through
$$
\mathrm{AP}=\int_0^1 P(r)\,dr,\qquad
\mathrm{mAP}=\frac{1}{Q}\sum_{q=1}^Q \mathrm{AP}_q.
$$

For **L2 recognition**, OCR text accuracy uses Character Error Rate:
$$
\mathrm{CER}=\frac{S+D+I}{N},
$$
where $S$, $D$, and $I$ are substitutions, deletions, and insertions, and $N$ is ground-truth length. ASR accuracy uses Word Error Rate:
$$
\mathrm{WER}=\frac{S+D+I}{N_{\mathrm{words}}}.
$$
Acoustic event detection uses segment-level F1:
$$
\mathrm{F1}=\frac{2\,\mathrm{Precision}\,\mathrm{Recall}}{\mathrm{Precision}+\mathrm{Recall}}.
$$

For **L3 interpreting**, the benchmark uses Exact-Match for QA,
$$
\mathrm{EM}=\frac{1}{N}\sum_{i=1}^N \mathbb{1}\{\mathrm{pred}_i=\mathrm{gold}_i\},
$$
Knowledge-Reference Rate for images,
$$
\mathrm{KR}=\frac{\#\{\text{captions correctly mention all verified entities}\}}{\#\{\text{knowledge images}\}},
$$
and a Logical Consistency Score via LLM-based QA,
$$
\mathrm{LC}=\frac{1}{M}\sum_{j=1}^M \mathbb{1}\{\mathrm{LLM\ judge\ says\ answer}_j\text{ is correct}\}.
$$

The report defines the aggregations as
$$
\mathrm{Perception}=\frac{\text{Detection}_{\mathrm{score}}+\text{Recognition}_{\mathrm{score}}}{2},
\qquad
\mathrm{Cognition}=\frac{\text{Interpreting}_{\mathrm{score}}}{1},
\qquad
\mathrm{Overall}=\frac{\mathrm{Perception}+\mathrm{Cognition}}{2}.
$$

These choices are consequential. They make L3 explicitly measurable as a cognition-oriented layer rather than as an unstructured free-text side task, while still keeping the underlying perception metrics visible.

## 6. Representative results, ablations, and relation to adjacent benchmarks

The report provides representative domain-level results for Logics-Parsing-Omni on OmniParsingBench [2603.09677]. For **Natural Image**, the scores are Overall **62.46**, Perception **50.53**, and Cognition **74.38**. For **Graphics**, they are **87.43**, **82.67**, and **92.19**. For **Document**, the benchmark reports Overall **84.90** and Perception **84.90**, with no L3. For **Audio**, the scores are **53.75**, **70.03**, and **37.48**. For **Natural Video**, they are **43.78**, **51.76**, and **35.79**. For **Text-Rich Video**, they are **66.78**, **54.52**, and **79.03**.

The same source states that Logics-Parsing-Omni outperforms open-weight baselines—Qwen3-Omni, Qwen3-VL, and GPT-5.2—in all six domains, and matches or exceeds proprietary Gemini-3-Pro in most L3 cognitive measures [2603.09677]. The reported Graphics ablation is especially central to the benchmark’s rationale: removing structured L1/L2 supervision severely degrades L3 logical reasoning, down from **92.19** to approximately **80**. In the benchmark’s framing, this underlines the necessity of evidence anchoring.

A common source of confusion is nomenclature. In surrounding benchmark literature, **OmniDocBench** has also been referred to as **OmniParsingBench**, but it denotes a document-centered benchmark for PDF parsing with 981 PDF pages from nine distinct sources, 19 layout categories, and 15 attribute labels [2412.07626]. By contrast, the benchmark in the Logics-Parsing-Omni report is explicitly multimodal, spanning documents, images, graphics, audio, and video [2603.09677]. The overlap in naming suggests a shared interest in unified parsing evaluation, but the scopes are not identical.

There is also a useful historical parallel with **BenchCLAMP**, a benchmark for constrained language model parsing across seven semantic parsing datasets and two syntactic parsing datasets [2206.10668]. BenchCLAMP formalizes output validity with context-free grammars and constrained decoding, whereas OmniParsingBench formalizes multimodal grounding with evidence-anchored JSON outputs. BenchCLAMP’s own discussion of an ideal unified parsing benchmark called “OmniParsingBench” proposed multi-regime splits, formal grammars, standardized metrics, and shared constrained-decoding tooling [2206.10668]. This suggests a convergence between text-centric parsing research and multimodal parsing research around a common objective: standardized, cross-task evaluation with explicit validity constraints, even though the technical mechanisms differ.

Taken together, these results and comparisons position OmniParsingBench as a benchmark centered on traceable multimodal knowledge extraction rather than on isolated captioning or recognition accuracy alone. Its distinctive contribution is the requirement that semantic interpretation remain explicitly tethered to verifiable low-level evidence.

Source: https://www.emergentmind.com/topics/omniparsingbench