DocScope: Benchmark for Verifiable Document Reasoning

• DocScope is a benchmark that reformulates long-document QA as a structured prediction problem, requiring evidence page identification, precise region grounding, fact extraction, and a final concise answer.
• It employs a four-stage audit protocol that independently evaluates page localization, region grounding, fact consistency, and answer verification to ensure trustworthy reasoning.
• Empirical findings show that even correct answers often lack complete evidential chains, highlighting the need for improved visual grounding and faithful fact synthesis.

DocScope is a benchmark for evaluating whether Multimodal LLMs can perform trustworthy, verifiable reasoning over long, visually rich PDF documents by producing not only a final answer but also an auditable reasoning trajectory (Feng et al., 9 May 2026). It reformulates long-document question answering as a structured prediction problem in which a model must identify evidence pages, ground evidence regions, extract supporting factual statements, and provide a concise answer. The benchmark is designed around the premise that end-to-end answer accuracy is insufficient for assessing trustworthiness when users require a verifiable chain of reasoning, particularly in settings such as legal dossiers, scientific studies, and financial reports.

1. Structured reasoning trajectory formulation

DocScope casts long-document QA as prediction of a structured output over a complete PDF document $\mathcal{D}=\{p_1,\dots,p_N\}$ and a question $q$ (Feng et al., 9 May 2026). The required model output is

$$y=\bigl(\mathcal{P},\,\mathcal{R},\,\mathcal{F},\,a\bigr),$$

where $\mathcal{P}\subseteq\{1,\dots,N\}$ is the set of evidence pages, $$\mathcal{R}=\{(i,[x_1,y_1,x_2,y_2])\mid i\in\mathcal{P}\}$$ is a collection of normalized-coordinate bounding boxes on those pages, $\mathcal{F}$ is a list of factual statements distilled from each region, and $a$ is the final concise answer.

This formulation makes explicit that document understanding is not treated as a single scalar prediction problem. A reasoning trajectory is considered unverifiable if any level fails: omission of a necessary page, imprecise or missing regions, or hallucinated or missing facts invalidates the trace even when the final answer is correct. In that sense, DocScope operationalizes verifiability as hierarchical evidence completeness rather than answer agreement alone.

2. Four-stage audit protocol

DocScope evaluates each component of the trajectory independently through inter-stage decoupling (Feng et al., 9 May 2026). Downstream stages are scored only on the subset of pages correctly retrieved, $\hat{\mathcal{P}=\mathcal{P}\cap\mathcal{P}^*}$, so that upstream page-localization errors do not automatically contaminate region and fact evaluation.

The first stage, Page Localization, compares predicted pages $\mathcal{P}$ against gold pages $\mathcal{P}^*$ using micro-averaged Precision, Recall, and F1. Precision and Recall are defined as

$q$0

with F1 given by the harmonic mean.

The second stage, Region Grounding, evaluates whether predicted boxes spatially cover the gold evidence regions on correctly retrieved pages. In the final protocol, an LLM-based judge, GPT-5.5, is shown both gold boxes and predicted boxes overlaid on the page and labels each gold region as “covered,” “imprecise,” or “not_covered.” Two F1 variants are reported. For question $q$1 and page $q$2, let $q$3 denote the number of “covered” gold regions, $q$4 the number of “imprecise” gold regions, $q$5 the total number of gold regions, and $q$6 the number of predicted boxes. Strict scoring uses $q$7, whereas lenient scoring uses $q$8. The corresponding Recall and Precision are

$q$9

again summarized by harmonic-mean F1.

The third stage, Fact Extraction, uses a text-only judge, Qwen3.6-Plus, to label each model-generated factual statement as “consistent” or “not_consistent” relative to the gold facts on the same pages. The metric is micro-averaged consistency rate:

$$y=\bigl(\mathcal{P},\,\mathcal{R},\,\mathcal{F},\,a\bigr),$$0

The fourth stage, Answer Verification, also uses Qwen3.6-Plus as a text-only judge, comparing the predicted answer $$y=\bigl(\mathcal{P},\,\mathcal{R},\,\mathcal{F},\,a\bigr),$$1 with the gold answer $$y=\bigl(\mathcal{P},\,\mathcal{R},\,\mathcal{F},\,a\bigr),$$2 for semantic equivalence on all questions, including unanswerable ones. Accuracy is reported separately for answerable questions, unanswerable questions, and overall.

The benchmark further reports that all judges were selected via human-alignment studies to ensure high agreement, with Krippendorff’s $$y=\bigl(\mathcal{P},\,\mathcal{R},\,\mathcal{F},\,a\bigr),$$3 in most cases. This suggests that the evaluation design attempts to stabilize automated judgment rather than treating LLM judges as uncalibrated black boxes.

3. Dataset scope and annotation pipeline

DocScope comprises 1,124 questions derived from 273 publicly licensed, visually rich English PDFs (Feng et al., 9 May 2026). The documents are 35–100 pages long, with an average of 51.3 pages and an average of 24.6 K tokens. Of the questions, 1,046 are answerable and span seven categories—visual-count, structure, numeric/statistical, procedural, entity-comparison, semantic, and temporal—while 78 are unanswerable. The benchmark is partitioned into 730 test instances (65%) and 394 validation instances (35%).

The evidence structure is explicitly hierarchical. There are 397 single-page questions and 649 multi-page questions. On average, each question has 3.99 bounding boxes, covering 9.8% page area, and 4.99 facts. These statistics indicate that evidence is commonly distributed across multiple pages and regions rather than concentrated in a single local span.

Component	Value
Documents	273 publicly licensed English PDFs
Questions	1,124 total
Answerable / unanswerable	1,046 / 78
Splits	730 test, 394 validation
Evidence pattern	397 single-page, 649 multi-page
Avg. boxes / question	3.99
Avg. facts / question	4.99

The annotation pipeline involved 13 trained annotators performing page selection, region bounding-box marking, fact-bearing statement extraction, and final answer entry. A model-assisted verification pass using Gemini-3.1-Flash-Lite outputs was followed by adjudication by two senior reviewers. Under this process, only 3.7% of answers were revised, and 176 of 1,300 initially synthesized questions were filtered out for specialty, PII, or low quality.

These design choices matter because the benchmark does not stop at answer annotation; it encodes a dense supervision structure for page-level, region-level, and fact-level evidence. A plausible implication is that DocScope is intended not merely as a leaderboard dataset but as an audit framework for studying failure modes in document reasoning systems.

4. Model coverage and experimental setting

The benchmark evaluates a broad set of systems across proprietary, open-weight, retrieval-augmented, and end-to-end document-reading paradigms (Feng et al., 9 May 2026). The six proprietary MLLMs are Gemini 3.1 Pro, Gemini 3.1 Flash-Lite, Claude Opus 4.7, Claude Sonnet 4.6, GPT-5.4, and Qwen3.6 Plus. The twelve open-weight models are Intern-S1-Pro; Qwen3.5 {397 B/122 B/27 B}; Qwen3-VL {235 B/32 B/30 B/8 B}; Gemma 4 {31 B/26 B-MoE}; and Ministral 3 {14 B/8 B}. The benchmark also includes two agentic RAG frameworks, SimpleDoc and VidoRAG, and two end-to-end document readers, URaG and Docopilot 8 B/2 B.

Inference was run via official APIs or open-weight code on four 80 GB L20Z GPUs, with a 16 384-token output limit. This establishes that all systems were evaluated under a common long-output regime appropriate for trajectory prediction rather than short-answer generation.

The breadth of model classes is central to the benchmark’s comparative value. It permits direct comparison between frontier proprietary MLLMs, open-weight alternatives, agentic retrieval pipelines, and specialized document readers under a unified trajectory-level evaluation protocol.

5. Empirical findings and failure structure

The reported results show that answer accuracy does not substitute for trajectory-level evaluation (Feng et al., 9 May 2026). The highest overall answer accuracy is 78.9% for Gemini 3.1 Pro, with 79.1% on answerable questions and 75.5% on unanswerable questions. Only three proprietary models exceed 70% overall accuracy. For top models, page-localization F1 is around 86–88%.

The central failure point in the trajectory is Region Grounding. The best strict region-F1 is 57.4% for GPT-5.4, and most models incur a drop of more than 40 percentage points from page F1 to strict region F1. Fact consistency peaks at 77.4% for Claude Opus 4.7 and often remains in the 60–70% range. These numbers indicate that identifying relevant pages is materially easier than localizing precise evidence and translating that evidence into faithful statements.

A particularly important benchmark finding concerns correct answers. Among correct answers, the fraction with complete chains—defined by page, region, and fact success—is at most 29% for GPT-5.4. The same analysis reports that 19% are completely unreliable and 52% are only partially reliable. This directly contradicts the common assumption that a correct final answer is a sufficient proxy for trustworthy reasoning.

The benchmark also identifies the primary source of difficulty as aggregation of evidence dispersed across long distances and multiple document clusters. That observation is consistent with the dataset’s high proportion of multi-page questions and the hierarchical evidence structure. It suggests that long-context document QA remains constrained not only by local perception but by cross-page evidence integration.

6. Oracle analysis and architectural implications

To isolate capability bottlenecks, DocScope evaluates four representative models—Claude Sonnet 4.6, GPT-5.4, Qwen3-VL 235 B, and Ministral 3 8 B—under cumulative oracle settings: Oracle pages, Oracle regions, and Oracle facts (Feng et al., 9 May 2026). Oracle pages supplies only the gold evidence pages; Oracle regions adds gold bounding-box descriptions; Oracle facts also injects gold atomic facts.

The largest accuracy gains, on the order of +10–15 percentage points, come from Oracle facts, while Oracle regions adds only marginal improvement beyond Oracle pages. The authors therefore identify fact extraction, defined as perception plus faithful statement generation, as the primary bottleneck rather than pure region grounding. This may appear to differ from the earlier observation that region grounding is the weakest trajectory stage, but the two findings address different questions: region grounding is the weakest measured stage in raw performance terms, whereas oracle substitution indicates that faithful fact synthesis is the dominant capability bottleneck for improving answer accuracy.

The oracle experiments also show that smaller or open-weight models benefit more from Oracle pages, indicating that long-context retrieval is more binding for them. Even with full Oracle fact access, noticeable performance gaps remain, specifically 92% versus 79%, which the study interprets as evidence of intrinsic reasoning capacity limits.

Another notable result is that providing Oracle pages sometimes lowers strict region-F1 in stronger models. The reported explanation is a conservative-to-aggressive shift in bounding-box strategy: when freed from large-area conservative boxes, such models attempt finer localization but often miss the exact gold row. This reveals a tension between broad spatial coverage and exact region fidelity.

The benchmark further reports that, within each of the Qwen3.5, Qwen3-VL, and Gemma 4 families, dense variants outperform MoE variants despite the latter’s larger total parameters. The stated interpretation is that activated parameter count matters more than raw scale for constructing high-fidelity reasoning trajectories. Agentic RAG systems outperform end-to-end readers, but still remain far behind frontier proprietary models in end-to-end verifiable performance.

Taken together, these findings position DocScope as a benchmark for separating answer correctness from evidential trustworthiness. Its results indicate that progress in long-document MLLMs requires improvements not only in long-context retrieval, but also in high-precision visual grounding and faithful fact synthesis if document-QA systems are to become verifiable in the benchmark’s strict sense.