---
title: 'MuDABench: Multi-Document Analytical QA Benchmark'
url: https://www.emergentmind.com/topics/mudabench
type: topic
---

# MuDABench: Multi-Document Analytical QA Benchmark

Searching arXiv for the benchmark and closely related work to ground the article.
arXiv search: MuDABench, MDBench, and related multi-document QA benchmarks.
MuDABench is a benchmark for multi-document analytical question answering over large, semi-structured financial document collections. It was introduced for settings in which answering a question requires extracting and synthesizing information across numerous documents to perform quantitative analysis, rather than locating a single supporting passage or reasoning over a small fixed set of pages [2604.22239]. The benchmark treats the repository as a semi-structured database linked by document-level metadata such as company, fiscal year, and document type, and it is explicitly designed for cases where the collection exceeds the context window of current long-context LLMs. In this formulation, success depends on metadata-aware filtering, robust single-document extraction, and cross-document aggregation, with the further constraint that missing one relevant document or misreading one table can invalidate the result [2604.22239].

## 1. Task setting and problem formulation

MuDABench targets analytical QA over large-scale document collections in the financial domain. The intended task is not conventional retrieval-augmented fact lookup, but a pipeline in which a system must identify the right subset of documents, extract per-document facts, and then compute or compare over the extracted values. The paper’s canonical example is the question “Which companies changed their accounting firms in 2024?”, which requires finding relevant annual reports, extracting the accounting firm for each company-year, aligning across years, and aggregating the results [2604.22239].

This design distinguishes MuDABench from benchmarks in which multi-document reasoning is effectively a small-scale multi-hop problem. The benchmark emphasizes four interacting properties: many documents per question, document metadata, cross-document aggregation, and quantitative analytical questions rather than simple fact lookup. A plausible implication is that MuDABench operationalizes a stricter notion of document-grounded reasoning than benchmarks where the main challenge is passage retrieval, because the answer is often a computed conclusion over a metadata-filtered subset of filings rather than a sentence recoverable from any single document [2604.22239].

The benchmark also assumes substantial heterogeneity in document format and content. Annual reports are often long and table-heavy, announcements can be scanned or irregular, and ESG reports have complex visual layouts. This heterogeneity is part of the benchmark’s difficulty, because the system must preserve fidelity across both textual and semi-structured evidence sources [2604.22239].

## 2. Corpus construction, metadata, and formal dataset structure

MuDABench was built with distant supervision plus expert curation. The authors crawled financial documents from **cninfo** for Chinese listed companies and the **SEC** for U.S. listed companies, and they also included announcements and ESG reports. Additional structured data were drawn from commercial financial databases such as **CSMAR** and **Wind**. Each document is annotated with three metadata fields: **ticker symbol**, **fiscal year**, and **document type** [2604.22239].

Question construction proceeds from structured financial data. The authors created a master spreadsheet in which each row corresponds to a document and is indexed by metadata. Domain experts then wrote question templates for analytical tasks such as trend analysis and peer comparison, and these templates were instantiated using the structured data to yield natural-language questions. Experts also manually transcribed the required structured indicators into natural-language fact statements, forming the intermediate information set $\mathcal{S}_j$ for each question. This intermediate set supports process evaluation in addition to final-answer scoring [2604.22239].

Formally, the dataset is defined as

$$
\mathcal{X} = \left\{ \left( Q_j, \mathcal{D}_j, \mathcal{M}_j, \mathcal{S}_j \right) \mid j = 1, 2, \ldots, n \right\},
$$

where $Q_j$ is the question, $\mathcal{D}_j$ the relevant document set, $\mathcal{M}_j$ the metadata, and $\mathcal{S}_j$ the set of intermediate fact descriptions [2604.22239].

In scale, the benchmark contains **over 80,000 pages**, **589 documents** in the collection described in the benchmark section, and **332 QA instances**. It uses about **15 documents per question on average**, and the paper also describes question sets with **roughly 5–38 PDF documents** per question, with the average reported as 15 documents. For cross-year questions, the intermediate fact set can be about half the size of the document collection for those tasks. For announcement documents, the authors use a merge strategy to account for disclosure lag across years [2604.22239].

## 3. Question classes and analytical workload

MuDABench includes both **simple** and **complex** analytical questions. The questions usually ask for ranking companies by a metric, computing ranges or variances, comparing values across years, identifying changes over time, calculating growth rates, finding outlier firms based on mean $\pm 2$ standard deviations, analyzing time intervals between corporate events, or comparing firms that satisfy a condition such as “audited by Big 4” [2604.22239].

What makes these questions multi-document analytical tasks is that the answer is not contained in one document. Instead, the system must identify all documents matching the metadata constraint, extract specific numeric or categorical facts from each document, merge facts across documents or across years, and perform a final computation or ranking. For complex questions, this may also involve conditional filtering before aggregation, such as restricting analysis to companies with certain year-over-year changes or only firms meeting a threshold over a period [2604.22239].

This structure shifts the locus of difficulty away from isolated reasoning steps toward end-to-end analytical integrity. The paper emphasizes that the collection behaves like a semi-structured database, not a single long input. A plausible implication is that MuDABench is especially sensitive to compounding error: even moderate failures in per-document extraction can collapse downstream aggregation, because the benchmark’s questions often require a near-complete set of relevant facts rather than a sparse evidential chain [2604.22239].

## 4. Evaluation protocol and diagnostic metrics

MuDABench uses three metrics: **process accuracy**, **final-answer accuracy**, and **full accuracy**. Final-answer accuracy is the main metric. For each question $Q_i$, let $A_i$ be the gold final answer and $\hat{A}_i$ the prediction; a judge model determines whether $\hat{A}_i$ is semantically equivalent to $A_i$, yielding a binary correctness indicator $T_i \in \{0,1\}$ [2604.22239].

Process accuracy measures whether the system extracted the needed supporting facts. For standard RAG, the benchmark defines gold supporting facts $\mathcal{S}_i$ and extracted facts $\mathcal{I}_i$, with coverage

$$
C_i = \frac{|\mathcal{I}_i \cap \mathcal{S}_i|}{|\mathcal{S}_i|},
$$

where the intersection is based on semantic matching judged by an LLM rather than exact string match. Because a single judge may overestimate coverage, the paper also introduces a double-check error estimate

$$
E_i=\frac{\#\text{ incorrect or missing facts in } \mathcal{S}_i}{|\mathcal{S}_i|},
$$

and defines conservative coverage as

$$
\tilde{C}_i=\min(C_i,\;1-E_i).
$$

For the document-grounded workflow, the benchmark uses a cell-wise evaluation on aligned rows. If $\mathcal{C}_i$ is the set of required gold metric cells and $\hat{\mathcal{C}}_i$ the correctly extracted cells, then the corresponding coverage is the ratio of correctly extracted cells to required cells [2604.22239].

**Full accuracy** is a strict joint metric: a sample is correct only if both the process is fully correct and the final answer is correct. The paper treats process coverage mainly as a diagnostic signal rather than a perfect standalone metric, because equivalent evidence may be expressed in non-atomic forms. This makes the evaluation protocol explicitly process-aware: MuDABench does not only ask whether a system arrived at the right output, but also whether it assembled the necessary document-grounded evidence on the way [2604.22239].

## 5. Reported system behavior and the multi-agent workflow

The experiments show that **standard RAG performs poorly**, even with large retrieval budgets and even when metadata is injected. Increasing the chunk budget generally improves process coverage somewhat, but final-answer accuracy stays low, especially on complex questions. The paper therefore argues that the main limitation is not only retrieval recall; the system also struggles to synthesize fragmented evidence into a correct aggregated answer. In the reported results, standard RAG final-answer accuracy is often in the **single digits to low teens**. A noise experiment in which irrelevant documents are injected shows further performance drops, especially on complex questions [2604.22239].

To address these limitations, the authors propose a **metadata-aware multi-agent workflow**. Its main components are a **planning agent**, a **document-level information extractor**, a **normalization agent**, and a **code-generation / code agent**. The planning agent decomposes the global question into sub-query templates rather than directly answering it, using the metadata schema to create per-document or per-document-type query templates. For each document $D_i$, the system instantiates the templates using metadata $M_i$ and runs a standard RAG system on the single document, producing localized textual evidence. The normalization agent converts extraction outputs into flat JSON records, and the code agent receives the task, the schema, a few JSON examples, and a path to the structured data, then writes executable Python code to analyze the JSON and compute the final answer [2604.22239].

The workflow substantially improves both process and final-answer performance, but it remains well below human performance. In one reported setting, the workflow with **GPT-4.1 mini** reaches roughly **24.3% process / 24.3% final** on simple questions and **16.2% process / 16.2% final** on complex questions, with better results at larger chunk budgets. Human performance in the main table is much higher, around **0.833 final** on simple and **0.733 final** on complex questions. The authors identify two primary bottlenecks: **single-document information extraction accuracy** and **insufficient domain-specific knowledge**. They also note secondary code-level issues such as JSON/schema mismatches, encoding or path-reading issues, and runtime errors caused by inconsistent field names [2604.22239].

The reported implementation details reinforce the benchmark’s emphasis on workflow design. PDF parsing uses **ChatDOC / commercial PDF processing**; standard RAG uses **OpenAI File Search** and **GPT-4o-2024-11-20**; the agentic workflow uses **DeepSeek-R1-0528** for planning and code generation, **DeepSeek-Chat-V3-0324** for normalization, and OpenAI file search for single-document QA; some experiments use **GPT-4.1 mini** for cost control; all LLMs are run at **temperature 0**; and judge models include **DeepSeek-V3.2** for cell-wise rejudging and **Kimi K2** for other correctness judgments [2604.22239].

## 6. Benchmark position, limitations, and name disambiguation

MuDABench is situated against several neighboring benchmark families. The paper contrasts it with **HotpotQA, 2WikiMultiHopQA, MuSiQue, FanOutQA,** and **MoreDocsSameLen**, which emphasize multi-hop reasoning over small sets of web pages; with **LongBench, RULER,** and **LongDocURL**, which typically assume the relevant content fits within one context window; with **FinanceBench**, which is mostly single-document QA; with **FinAgentBench**, which focuses on retrieval precision and locating the right passages or documents rather than downstream aggregation; and with **M3DocVQA**, which studies multi-document multimodal understanding on a much smaller scale. The paper also notes that **Aryn** and **DocETL** propose workflows for multi-step document processing but do not provide a large public benchmark [2604.22239].

A recurrent source of confusion is nomenclature. MuDABench should not be conflated with two distinct benchmarks named **MDBench**. One is a synthetic benchmark for LLM evaluation on multi-document reasoning generated from structured seed knowledge and rendered into natural-language document sets [2506.14927]. The other is an open-source benchmarking framework for model discovery on dynamical systems, evaluating data-driven recovery of governing differential equations on ODEs and PDEs [2509.20529].

| Benchmark | Domain | Core task |
|---|---|---|
| MuDABench | Financial document collections | Multi-document analytical QA with metadata-aware filtering and cross-document aggregation |
| MDBench | Multi-document reasoning | Synthetic QA over generated document sets from structured seed knowledge |
| MDBench | Dynamical systems | Benchmarking data-driven methods for model discovery |

MuDABench’s limitations are stated explicitly. It is restricted to the financial domain because dense semi-structured data are easier to find there than in many other domains. The dataset size is intentionally limited because evaluation is costly. The evaluation is sensitive to issues of fact granularity and semantic equivalence. The documents come from publicly available disclosures, so no private information is involved, and the authors note that the dataset is being corrected over time, so reported results may not be fully up to date [2604.22239].

These limitations clarify the benchmark’s intended role. It is not presented as a general benchmark for arbitrary long-context QA, nor as a pure retrieval benchmark, nor as a substitute for naturally occurring benchmarks in other domains. Rather, it functions as a controlled testbed for large-scale analytical reasoning over document collections in which correct document selection, robust per-document extraction, metadata-aware planning, and accurate aggregation are all first-class requirements [2604.22239].

Source: https://www.emergentmind.com/topics/mudabench