DocHop-QA Benchmark

Updated 9 July 2026

DocHop-QA is a comprehensive benchmark that advances multi-hop reasoning by integrating evidence from multiple documents and modalities such as text, tables, and layouts.
It employs a structured question-generation pipeline with techniques like TF-IDF, BERTScore, and layout-aware methods to ensure robust, cross-document evidence synthesis.
The benchmark supports both discriminative index extraction and generative answer formulation, challenging models with tasks like bounding-box entity prediction and free-form text extraction.

to=functions.arxiv_search_arxiv 北京赛车有 format=json-stringify content={"all_fields":"DocHop-QA multimodal document collections multi-hop reasoning", "start":0, "max_results":5} to=functions.arxivsearch_arxiv 天天中彩票官方 format=json-stringify content={"all_fields":"PluriHop exhaustive recall-sensitive QA distractor-rich corpora", "start":0, "max_results":5} to=functions.arxivsearch_arxiv 久赢 format=json-stringify content={"all_fields":"DocHopper Iterative Hierarchical Attention complex questions long documents", "start":0, "max_results":5} to=functions.arxivsearch_arxiv ￣影音先锋 format=json-stringify content={"all_fields":"SimpleDoc Multi-Modal Document Understanding Dual-Cue Page Retrieval Iterative Refinement", "start":0, "max_results":5} to=functions.arxiv_get_arxiv_paper_details 体育彩票天天 format=json-stringify content={"arxiv_id":"(Park et al., 20 Aug 2025)"} DocHop-QA is a large-scale benchmark for multimodal, multi-document, multi-hop question answering, comprising 11,379 QA instances constructed from publicly available scientific documents sourced from PubMed (Park et al., 20 Aug 2025). It was introduced to address a gap between practical information-seeking tasks and earlier QA benchmarks that are confined to single-paragraph or single-document settings, rely heavily on Wikipedia-based content and unimodal plain text, and often involve shallow reasoning paths with brief phrase-level or single-sentence answers. In DocHop-QA, evidence may be distributed across multiple documents, modalities, and structural formats, including textual passages, tables, and layout cues, and the benchmark does not rely on explicitly hyperlinked documents; instead, it supports open-ended reasoning through semantic similarity and layout-aware evidence synthesis (Park et al., 20 Aug 2025).

1. Problem formulation and scope

DocHop-QA treats question answering as a cross-document reasoning problem in which the answer may depend on evidence drawn from one or more scientific articles, possibly spanning paragraphs, tables, and page-layout structure (Park et al., 20 Aug 2025). The benchmark is described as domain-agnostic despite being sourced from PubMed, because its construction is organized around recurrent scientific question concepts rather than narrow task-specific templates.

Each QA instance $i$ is associated with a question text $q_i$ , a context set of entities $E_i=\{e_1,\dots,e_n\}$ drawn from one or more documents, and a gold answer index set $G_i \subseteq \{1,\dots,n\}$ (Park et al., 20 Aug 2025). For layout-based settings, each entity may also carry a bounding box $b_j$ . This formulation supports both discriminative and generative evaluation, with tasks ranging from multi-label index prediction to free-form answer generation.

A central distinction in the benchmark is between paragraph-oriented and table-oriented reasoning. Paragraph-oriented instances are described as “fan-hop,” while table-oriented instances are described as “chain-hop” (Park et al., 20 Aug 2025). In the former, the system must gather semantically aligned snippets; in the latter, it must follow a structured evidence path such as problem paragraph $\rightarrow$ solution paragraph $\rightarrow$ table caption $\rightarrow$ referencing paragraph. This design shifts the benchmark away from hyperlink-following or single-document reading and toward evidence synthesis across heterogeneous document structures.

2. Corpus assembly and question generation

The source corpus is drawn from PubMed Central open-access articles, approximately 2 million full-text papers in XML and PDF form (Park et al., 20 Aug 2025). After filtering by basic heuristics, including removal of very short or non-biomedical papers, and computing pairwise similarities on titles and abstracts via TF-IDF with cosine threshold $\tau = 0.3$ , the final pool comprises approximately 20,000 document pairs spanning 9,250 unique documents (Park et al., 20 Aug 2025).

The question-generation pipeline has six stages (Park et al., 20 Aug 2025):

Seed collection: manually author 90 cross-document QA exemplars.
Keyword scaffolds: extract approximately 15 high-frequency semantic placeholders such as 〈Problem〉, 〈Solution〉, and 〈Effect〉.
Question concepts: define 11 templated multi-hop structures.
LLM fine-tuning: train three separate Qwen2-Instruct LoRA models (4-bit) for non-comparison, comparison, and problem-table types.
Generation and post-filter: prompt each LLM with two abstracts and a concept template, then filter malformed or concept-mismatched outputs.
Deterministic table generation: generate the two table-oriented subtypes by inserting table captions into placeholders.

The 11 question concepts define the semantic backbone of the benchmark.

QType	Template
PS	〈Problem〉⟨sep⟩〈Solution〉
PSM	〈Problem〉⟨sep⟩〈Solution〉⟨sep⟩〈Mechanism〉
PSR	〈Problem〉⟨sep⟩〈Solution〉⟨sep⟩〈Result〉
PSL	〈Problem〉⟨sep⟩〈Solution〉⟨sep⟩〈Limitation〉
ME	〈Mechanism〉⟨sep⟩〈Effect〉
SAL	〈StudyTopic〉⟨sep⟩〈Advantage〉⟨sep⟩〈Limitation〉
FF	〈Feature〉⟨sep⟩〈Feature〉
SS	〈Solution〉⟨sep⟩〈Solution〉
PST	〈Problem〉⟨sep⟩〈Solution〉⟨sep⟩〈Table〉
TS	〈Table〉⟨sep⟩〈StudyTopic〉
TSL	〈Table〉⟨sep⟩〈StudyTopic〉⟨sep⟩〈Limitation〉

Evidence retrieval during dataset construction also differs by subtype. For paragraph-oriented instances, the benchmark retrieves the top 5 snippets per question using a hybrid score $0.2 \cdot \mathrm{TF\mbox{-}IDF} + 0.8 \cdot \mathrm{BERTScore}$ and keeps only instances where all snippets are at least $q_i$ 0 (Park et al., 20 Aug 2025). For table-oriented instances, it assembles 3–4 snippets via a fixed logic chain using keyword filters and semantic thresholds. This curation strategy is designed to preserve explicit multi-hop structure rather than merely producing plausible answers.

3. Formal tasks and representation

DocHop-QA defines four tasks spanning structured extraction, entity localization, and free-form generation (Park et al., 20 Aug 2025).

Task 1: BBox Entity Index Extraction

The input consists of page images, OCR text, and bounding boxes $q_i$ 1. The output is a predicted index set $q_i$ 2. This is a multi-label classification problem in which each bounding box receives a score $q_i$ 3, and thresholding determines the predicted set.

The task also introduces three spatial metrics (Park et al., 20 Aug 2025):

Belong: fraction of predicted $q_i$ 4 whose boxes lie entirely inside some gold $q_i$ 5.
Contain: fraction of gold $q_i$ 6 boxes entirely inside some predicted $q_i$ 7.
Overlap: fraction of predicted $q_i$ 8 with non-zero intersection with some gold $q_i$ 9.

Task 2: XML Entity Index Extraction

The input is a linearized XML sequence of entities such as paragraphs and tables, optionally augmented with CLIP image embeddings for each entity (Park et al., 20 Aug 2025). The goal remains multi-label classification over the entity set $E_i=\{e_1,\dots,e_n\}$ 0. Two setups are defined: Concat, which feeds the entire sequence into one model such as BigBird or Longformer, and PerEnt, which encodes each entity separately and then classifies it.

Task 3: Structured Generative Answering

A vision-language LLM such as Qwen2.5-VL or InternVL2 is prompted to extract the entity IDs (Park et al., 20 Aug 2025). The output is a structured list $E_i=\{e_1,\dots,e_n\}$ 1, which is evaluated against the gold index set under multi-label metrics.

Task 4: Generative Text Extraction

An LLM such as Qwen2.5, InternVL2, or Gemini-2.5 is prompted in question-only, zero-shot with instruction, or one-shot settings and may receive text-only, image-only, or text+image inputs (Park et al., 20 Aug 2025). The output is a free-form answer $E_i=\{e_1,\dots,e_n\}$ 2, evaluated with BLEU-4 and ROUGE-N, ROUGE-L, and ROUGE-Lsum.

The benchmark also makes the hop-linking mechanism explicit through cosine similarity over embeddings:

$E_i=\{e_1,\dots,e_n\}$ 3

Here, $E_i=\{e_1,\dots,e_n\}$ 4 and $E_i=\{e_1,\dots,e_n\}$ 5 are BERT-based embeddings, or TF-IDF vectors in the first-stage filtering setup (Park et al., 20 Aug 2025).

Layout-aware evidence synthesis is represented through normalized bounding-box coordinates $E_i=\{e_1,\dots,e_n\}$ 6 and models such as LayoutLMv3 and LayoutXLM, which ingest both text tokens and positional embeddings (Park et al., 20 Aug 2025). In some XML-based settings, CLIP image embeddings are concatenated with text representations in PerEnt + CLIP configurations.

4. Benchmark composition and reasoning structure

DocHop-QA contains 11,379 QA pairs, of which 75.3% (approximately 8,547) are paragraph-oriented and 24.7% (approximately 2,812) are table-oriented (Park et al., 20 Aug 2025). Instances may be single-document or multi-document, but the majority are multi-document. The modalities include unstructured text, structured tables, and layout cues such as OCR bounding boxes and page images. Document complexity is substantial: the average document has approximately 10 pages, 20 subsections, 60 paragraphs, and 1,500–2,500 tokens (Park et al., 20 Aug 2025).

The benchmark’s examples illustrate several distinct reasoning motifs. In a paragraph-oriented PS instance, one abstract identifies the problem and another abstract provides the solution, yielding an answer that integrates both (Park et al., 20 Aug 2025). In a table-oriented PST instance, the evidence path combines a problem paragraph, a solution paragraph, a table caption, and a discussion paragraph referencing the table. In a TSL instance, the system must connect table metadata, an explanatory paragraph, the abstracts of cited trial articles, and a limitations paragraph from a referenced paper. These examples show that the benchmark does not merely require retrieving related snippets; it requires tracking the role each snippet plays within a typed reasoning schema.

A plausible implication is that DocHop-QA operationalizes multi-hop reasoning at several levels simultaneously: semantic linking across documents, structural linking between narrative text and tables, and spatial linking between textual entities and page regions. That combination is unusual among existing QA benchmarks and helps explain why the benchmark includes both index-oriented and text-generation tasks.

5. Evaluation protocol and empirical difficulty

For Tasks 1–3, DocHop-QA uses sample-level multi-label precision, recall, and F1, defined over predicted index set $E_i=\{e_1,\dots,e_n\}$ 7 and gold set $E_i=\{e_1,\dots,e_n\}$ 8 as follows (Park et al., 20 Aug 2025):

$E_i=\{e_1,\dots,e_n\}$ 9

$G_i \subseteq \{1,\dots,n\}$ 0

For Task 4, the benchmark reports BLEU-4 and ROUGE-1, ROUGE-2, ROUGE-L, and ROUGE-Lsum (Park et al., 20 Aug 2025).

Representative baseline results indicate that DocHop-QA remains difficult across both structured and generative settings.

Task	Model/setup	Reported result
Task 1 (BBox)	LayoutLMv3	F1 = 12.35, Recall = 34.97, Precision = 8.47, Overlap = 95.5%
Task 1 (BBox)	LayoutXLM	F1 = 7.13, Recall = 4.49, Precision = 24.06, Overlap = 33.96%
Task 2 (XML)	BigBird, PerEnt + CLIP	F1 = 23.24, Recall = 21.29, Precision = 25.77, Overlap = 64.98%
Task 3 (Structured gen.)	Qwen2.5-VL zero-shot text-only	F1 = 14.06, Recall = 51.62%
Task 4 (Free-form gen.)	Qwen2.5 zero-shot image-only	BLEU-4 = 21.56, ROUGE-1 = 38.45, ROUGE-2 = 16.92, ROUGE-L = 23.90

Several benchmark findings directly challenge common assumptions about multimodal QA (Park et al., 20 Aug 2025). Very low containment accuracy in Task 1 highlights the difficulty of exact spatial grounding. Structured index generation in Task 3 remains brittle in formatting. Adding both text and image often degrades performance relative to a single modality in Task 4, indicating that naive fusion is suboptimal rather than automatically beneficial. These results position DocHop-QA less as a saturated leaderboard benchmark than as a diagnostic testbed for evidence alignment, multimodal fusion, and answer-set precision.

6. Relation to adjacent document-QA paradigms

DocHop-QA occupies a distinct position within document QA because it combines multi-document scope with multimodal evidence and multiple output paradigms (Park et al., 20 Aug 2025). Related systems illuminate different facets of this design space.

DocHopper, introduced for complex questions over long documents, iteratively attends to hierarchically structured documents and can retrieve either short passages or long sections, achieving state-of-the-art results on three of four evaluated tasks while being 3–10 times faster than the baselines (Sun et al., 2021). Its core setting is long-document navigation within a document hierarchy. By contrast, DocHop-QA’s majority multi-document instances and explicit table/layout tasks move beyond within-document iterative attention toward cross-document evidence composition.

SimpleDoc addresses DocVQA through a retrieval-augmented framework that first retrieves candidate pages through embedding similarity, then filters and reranks them based on page summaries, with a VLM reasoner that iteratively retrieves fresh pages until the question is answered or deemed unanswerable (Jain et al., 16 Jun 2025). It reports an average accuracy of 70.1% on four DocVQA datasets, a +3.2 absolute improvement over the best prior RAG while retrieving fewer than 4 pages versus 12–20 (Jain et al., 16 Jun 2025). This suggests a methodological affinity with DocHop-QA’s multimodal emphasis, but SimpleDoc is framed as a system architecture, whereas DocHop-QA is a benchmark spanning structured index prediction and generative answering.

PluriHop formalizes a different but related difficulty regime: pluri-hop questions are defined by recall sensitivity, exhaustiveness, and exactness, and the accompanying PluriHopWIND dataset is built from 191 real-world wind-industry reports in German and English (Sveistrys et al., 16 Oct 2025). None of the tested approaches in that setting exceed 40% statement-wise F1, and the proposed PluriHopRAG improves relative F1 by 18–52% depending on the base LLM (Sveistrys et al., 16 Oct 2025). A plausible implication is that DocHop-QA and PluriHop stress complementary failure modes: DocHop-QA emphasizes multimodal cross-document reasoning and layout grounding, whereas PluriHop emphasizes exhaustive retrieval in distractor-rich recurring-report corpora.

In the biomedical domain, the UETQuintet MedHopQA system distinguishes direct from sequential questions, decomposes sequential questions into sub-questions, retrieves mixed context from web search and Wikipedia, and achieves an Exact Match score of 0.84 on the BioCreative IX - MedHopQA Shared Task (Nguyen et al., 11 Jan 2026). That work demonstrates the continued relevance of selective multi-hop reasoning and contextual retrieval in biomedical QA, but its task formulation differs from DocHop-QA’s entity-index extraction, XML-based classification, and multimodal layout-aware evidence synthesis.

Taken together, these comparisons suggest that DocHop-QA sits at an intersection of several previously separate research threads: long-document navigation, multimodal page retrieval, biomedical multi-hop decomposition, and recall-sensitive cross-document aggregation. Its distinctive contribution is to unify these pressures within one benchmark built around typed multi-hop concepts, multi-document evidence, and both structured and generative evaluation (Park et al., 20 Aug 2025).