EviBench: Multi-Page Document VQA Dataset

• EviBench is a reasoning-annotated training dataset designed for document visual question answering, incorporating chain-of-thought reasoning and evidence-page labels.
• It aggregates 4.8k examples from diverse single-page and multi-page datasets, ensuring structured supervision for accurate evidence retrieval.
• A two-stage, model-assisted annotation pipeline validates answer consistency and formatting, enabling effective reward-driven learning in DocR1.

EviBench is a reasoning-annotated training dataset for document visual question answering introduced in “DocR1: Evidence Page-Guided GRPO for Multi-Page Document Understanding” (Xiong et al., 10 Aug 2025). In that work, it is described as “a high-quality training set with 4.8k examples” and serves as the dataset that makes the Evidence Page-Guided GRPO (EviGRPO) framework feasible on multi-page documents. EviBench is not a held-out benchmark; it is “the training dataset used in our study,” designed so that format, evidence-page, and answer-accuracy rewards are all verifiable under the DocR1 RL pipeline. Its annotations follow the schema >, <evidence_page>, and <answer>, enabling DocR1 to learn a structured reasoning format, evidence-page localization, and answer generation over both single-page and multi-page documents (Xiong et al., 10 Aug 2025).

1. Definition and function within DocR1

EviBench was introduced to “bridge this gap” between existing open-source document QA datasets and the requirements of EviGRPO. The stated problem is that “most existing open-source datasets are not directly compatible with our EviGRPO framework due to mismatches in input-output formats and reward structures.” In DocR1, EviBench therefore functions primarily as a training dataset whose labels are constructed specifically so that the three reward components in EviGRPO—format, evidence-page, and answer accuracy—are directly computable (Xiong et al., 10 Aug 2025).

The dataset’s role is unusually central because its structure is not incidental to the model; it is co-designed with the system prompt, the reward computation, and the curriculum schedule. The paper presents EviBench as the core supervised signal, via verifiable rewards, that lets DocR1 learn to internalize a structured reasoning style, retrieve relevant pages before answering, and operate across both single-page and multi-page document settings. This coupling distinguishes EviBench from conventional VQA corpora that provide only image-question-answer triples.

2. Corpus scope, composition, and source datasets

EviBench is defined as a reasoning-annotated training dataset for document visual question answering with explicit evidence-page labels and chain-of-thought-style reasoning. It is constructed by re-annotating samples from existing document and chart QA datasets so that they conform to the DocR1 output schema think → evidence_page → answer. The dataset supports single-page document QA, multi-page document QA, evidence-page retrieval, and multi-hop reasoning (Xiong et al., 10 Aug 2025).

The paper gives two equivalent views of its scale: “4.8k examples” in the abstract, and “1.3k single-page and 3.5k multi-page document samples” in the data-construction section.

Subset	Size	Source structure
Single-page	1.3k	13 datasets, 100 samples each, Images = 1
Multi-page	3.5k	DUDE 1000; MP-DocVQA 500; TATDoc 500; SlideVQA 500; Multihiertt 500; ArxivFullQA$_\text{train}$ 500

The single-page subset draws 100 samples each from DocVQA, InfographicVQA, ChartQA, DeepForm, DVQA, FigureQA, KleisterCharity, OCRVQA, TabFact, TextCaps, TextVQA, VisualMRC, and WikiTableQuestions. The multi-page subset draws from DUDE, MP-DocVQA, TATDoc, SlideVQA, Multihiertt, and ArxivFullQA$_\text{train}$. The page ranges are dataset-dependent: DUDE uses 1–21 pages per sample, MP-DocVQA 1–36, TATDoc 1–3, SlideVQA exactly 20, Multihiertt 3–7, and ArxivFullQA$_\text{train}$ 1–29 (Xiong et al., 10 Aug 2025).

EviBench is therefore not a monolithic corpus collected from scratch. It is a uniform reasoning-annotated overlay over a heterogeneous mixture of existing benchmarks. The paper does not specify language distribution explicitly; all listed source datasets are predominantly English. Domains span forms, invoices, business documents, tables, charts, infographics, text-rich images, and scientific articles.

3. Two-stage annotation pipeline

EviBench is built with a two-stage, model-assisted annotation pipeline. The annotator and verifier are both Gemini 2.5 Flash. In Stage 1, the model receives a sample from a base dataset together with a task-specific system prompt. For EviBench, the main prompt is the “page selection format” system prompt, which instructs the model to produce intermediate reasoning in <think>...</think>, page-level evidence predictions in <evidence_page>...</evidence_page>, and the final answer in <answer>...</answer> (Xiong et al., 10 Aug 2025).

The Stage 1 output is filtered by answer correctness. The paper states that “The sample advances to the next stage only if its predicted answer is consistent with the ground truth.” In Stage 2, the same MLLM is given the annotated content again in a controlled verification setting. The annotation is kept only if “the model's output once again aligns with the ground truth under this controlled setting.” Human annotators are not mentioned in this process; the pipeline is fully automated, with quality controlled through answer-consistency and self-consistency filtering.

This design makes EviBench a dataset of structured annotations rather than merely relabeled answers. Each retained sample contains a reasoning trace, an evidence-page decision sequence, and an answer aligned to the original dataset label. The same general pipeline is reused to create ArxivFullQA, although ArxivFullQA differs in that the initial input for Step 1 is LaTeX text rather than images, and the prompt is adapted to scientific papers and multiple question types.

4. Evidence-page labels, output schema, and reward coupling

The defining annotation in EviBench is the evidence-page label. In DocR1, “evidence pages” are page-level indicators specifying which images among a multi-page document contain relevant evidence for answering the question. The system prompt instructs the annotator to identify how many pages are present and then emit a comma-separated sequence of T and F, one per page, in order. A T marks a page containing relevant evidence; an F marks a page judged irrelevant (Xiong et al., 10 Aug 2025).

Each EviBench sample has the following effective structure. The input is one or more document images $X = \{x_1, \ldots, x_N\}$ together with a question $q$. The labels consist of a reasoning trace inside <think>..., an evidence-page sequence inside <evidence_page>...</evidence_page>, and a free-form answer inside <answer>...</answer>. There is no explicit mention of bounding boxes or region coordinates; the evidence supervision is page-level rather than region-level.

The reward design in EviGRPO is explicitly tied to these labels. For a candidate response $o_i$, the total reward is

$$r_i = r_i^{\text{format}} + r_i^{\text{acc}} + r_i^{\text{evi}}.$$

Here, $r_i^{\text{format}} = 1$ if the output strictly follows the required tagged format and $0$ otherwise; $r_i^{\text{acc}}$ is the ANLS score between the generated answer and the ground-truth answer; and $_\text{train}$0 is an F1-style score over predicted and ground-truth evidence-page sets. With $_\text{train}$1 the set of predicted evidence pages, $_\text{train}$2 the ground-truth evidence-page set, $_\text{train}$3 the number of predicted judgements, and $_\text{train}$4 the number of input pages, the evidence reward is

$_\text{train}$5

This formulation imposes a strict length constraint: if the model does not provide a judgement for exactly all pages, the evidence reward is zero. The paper motivates the F1 choice by noting that plain page-wise accuracy would permit trivial high scores in sparse-evidence settings by predicting all pages as irrelevant. The reward is then normalized into an advantage,

$_\text{train}$6

and optimized with a GRPO-style clipped objective with KL regularization. EviBench is what makes this reward decomposition operational, because it provides ground-truth evidence-page sets $_\text{train}$7 for every training example.

5. Curriculum use and evaluation status

EviBench is inseparable from the DocR1 curriculum. Training proceeds in two RL stages. First, the model is trained on the single-page subset of EviBench “for one epoch” so that it can “internalize the desired output format and reasoning style” and “activate [the] latent reasoning capability” of the Qwen2.5-VL-Instruct base model. Second, the model is trained on the multi-page subset “for another epoch” to develop multi-page reasoning and evidence retrieval across longer contexts (Xiong et al., 10 Aug 2025).

The paper reports ablations comparing EviGRPO with single-page data, multi-page data, mixed data, and a non-curriculum mixed strategy. The curriculum strategy—single-page warm-up followed by multi-page training—yields the best and most balanced gains. This suggests that EviBench’s explicit partition into simpler single-page reasoning examples and harder multi-page evidence-retrieval examples is functionally important rather than merely organizational.

EviBench itself is not used as a benchmark in the result tables. Evaluation is conducted on external benchmarks and on ArxivFullQA. During training, however, the operative metrics are precisely those defined by EviGRPO: the binary format reward, ANLS answer reward, and evidence-page F1 reward. During evaluation on external multi-page tasks, ANLS is the main metric, while evidence-page behavior is analyzed separately through recall on test benchmarks such as MP-DocVQA, DUDE, and MultiHiertt.

6. Naming, adjacent resources, and disambiguation

In the DocR1 literature, “EviBench” specifically denotes the training dataset described above. The string is not, however, a stable community-wide proper name. In biomedical NLP, “EvidenceBench” is a distinct benchmark “often shortened informally to ‘EviBench’ in discussion”; it targets sentence-level evidence retrieval for biomedical hypotheses and includes both a 426-instance benchmark and the 107,461-instance EvidenceBench-100k training resource (Wang et al., 25 Apr 2025).

A second distinct usage appears in financial NLP. “EvasionBench” is “sometimes abbreviated informally as ‘EviBench’ in discussion,” although that paper consistently uses the full name EvasionBench. EvasionBench is a benchmark for detecting evasive answers in earnings-call Q&A, with 30,000 training samples and 1,000 human-annotated test samples across three evasion levels (Ma et al., 14 Jan 2026).

EventBench is another neighboring but separate resource. It is an event-based MLLM benchmark and the paper explicitly notes that “EviBench” is not mentioned by name there; rather, EventBench is described as the kind of unified event-stream benchmark one might have in mind under an “EviBench-style” evaluation (Liu et al., 23 Nov 2025). These naming collisions matter because the DocR1 EviBench is a training corpus for evidence-page-guided RL on documents, whereas EvidenceBench, EvasionBench, and EventBench are public-facing evaluation resources in biomedical retrieval, financial discourse analysis, and event-based multimodal reasoning, respectively.

7. Reproducibility, design choices, and limitations

The DocR1 paper does not explicitly provide a URL or license for EviBench. It refers to EviBench as “the training dataset used in our study” rather than as a public benchmark release. The paper does, however, disclose the main structural components needed to reconstruct a similar resource: the PSF-3 system prompt defining the <think>, <evidence_page>, and <answer> format; alternative prompt variants used in ablations; the dataset selection and sample counts; and pseudocode for the EviGRPO training loop (Xiong et al., 10 Aug 2025).

Several design choices are explicit or strongly implied. EviBench favors quality over scale: only 4.8k samples are annotated, but each sample carries rich supervision. It favors multi-domain coverage by sampling from many benchmarks rather than constructing a single-domain corpus. It also chooses coarse page-level evidence rather than finer region-level grounding, which matches the coarse-to-fine EviGRPO strategy of retrieving relevant pages before generating the answer.

The paper does not present a dedicated limitations subsection for EviBench, but several limitations are implicit. All annotations depend on Gemini 2.5 Flash, so annotator biases and errors may propagate into the dataset. Evidence-page labels are not manually verified; they are validated indirectly through answer correctness and self-consistency. The supervision is page-level only, and the absolute scale—4.8k training examples—is modest. The paper also states that “large-scale chain-of-thought training data for multi-page document understanding is extremely scarce, and collecting such annotations is highly resource-intensive,” which situates EviBench as a compromise between annotation richness and annotation cost.

Future extensions are not explicitly described, but natural directions noted in the discussion include extending EviBench with more samples per base dataset, adding region-level evidence annotations, and incorporating more domains or languages. A plausible implication is that EviBench’s lasting significance lies less in its raw size than in its role as a reward-compatible supervision layer for multi-page document RL: its schema, prompts, and verifiable evidence labels define the training signal that enables DocR1’s coarse-to-fine evidence retrieval and answer generation.