MuDABench: Multi-Document Analytical QA Benchmark

• MuDABench is a benchmark for multi-document analytical QA in finance that leverages metadata filtering and cross-document aggregation for quantitative analysis.
• It simulates tasks where systems must extract per-document facts from over 80,000 pages and synthesize evidence across diverse, semi-structured financial reports.
• The evaluation emphasizes process and final-answer accuracy, highlighting challenges in metadata-aware filtering, robust extraction, and aggregation under complex conditions.

Searching arXiv for the benchmark and closely related work to ground the article. arXiv search: MuDABench, MDBench, and related multi-document QA benchmarks. MuDABench is a benchmark for multi-document analytical question answering over large, semi-structured financial document collections. It was introduced for settings in which answering a question requires extracting and synthesizing information across numerous documents to perform quantitative analysis, rather than locating a single supporting passage or reasoning over a small fixed set of pages (Li et al., 24 Apr 2026). The benchmark treats the repository as a semi-structured database linked by document-level metadata such as company, fiscal year, and document type, and it is explicitly designed for cases where the collection exceeds the context window of current long-context LLMs. In this formulation, success depends on metadata-aware filtering, robust single-document extraction, and cross-document aggregation, with the further constraint that missing one relevant document or misreading one table can invalidate the result (Li et al., 24 Apr 2026).

1. Task setting and problem formulation

MuDABench targets analytical QA over large-scale document collections in the financial domain. The intended task is not conventional retrieval-augmented fact lookup, but a pipeline in which a system must identify the right subset of documents, extract per-document facts, and then compute or compare over the extracted values. The paper’s canonical example is the question “Which companies changed their accounting firms in 2024?”, which requires finding relevant annual reports, extracting the accounting firm for each company-year, aligning across years, and aggregating the results (Li et al., 24 Apr 2026).

This design distinguishes MuDABench from benchmarks in which multi-document reasoning is effectively a small-scale multi-hop problem. The benchmark emphasizes four interacting properties: many documents per question, document metadata, cross-document aggregation, and quantitative analytical questions rather than simple fact lookup. A plausible implication is that MuDABench operationalizes a stricter notion of document-grounded reasoning than benchmarks where the main challenge is passage retrieval, because the answer is often a computed conclusion over a metadata-filtered subset of filings rather than a sentence recoverable from any single document (Li et al., 24 Apr 2026).

The benchmark also assumes substantial heterogeneity in document format and content. Annual reports are often long and table-heavy, announcements can be scanned or irregular, and ESG reports have complex visual layouts. This heterogeneity is part of the benchmark’s difficulty, because the system must preserve fidelity across both textual and semi-structured evidence sources (Li et al., 24 Apr 2026).

2. Corpus construction, metadata, and formal dataset structure

MuDABench was built with distant supervision plus expert curation. The authors crawled financial documents from cninfo for Chinese listed companies and the SEC for U.S. listed companies, and they also included announcements and ESG reports. Additional structured data were drawn from commercial financial databases such as CSMAR and Wind. Each document is annotated with three metadata fields: ticker symbol, fiscal year, and document type (Li et al., 24 Apr 2026).

Question construction proceeds from structured financial data. The authors created a master spreadsheet in which each row corresponds to a document and is indexed by metadata. Domain experts then wrote question templates for analytical tasks such as trend analysis and peer comparison, and these templates were instantiated using the structured data to yield natural-language questions. Experts also manually transcribed the required structured indicators into natural-language fact statements, forming the intermediate information set $\mathcal{S}_j$ for each question. This intermediate set supports process evaluation in addition to final-answer scoring (Li et al., 24 Apr 2026).

Formally, the dataset is defined as

$$\mathcal{X} = \left\{ \left( Q_j, \mathcal{D}_j, \mathcal{M}_j, \mathcal{S}_j \right) \mid j = 1, 2, \ldots, n \right\},$$

where $Q_j$ is the question, $\mathcal{D}_j$ the relevant document set, $\mathcal{M}_j$ the metadata, and $\mathcal{S}_j$ the set of intermediate fact descriptions (Li et al., 24 Apr 2026).

In scale, the benchmark contains over 80,000 pages, 589 documents in the collection described in the benchmark section, and 332 QA instances. It uses about 15 documents per question on average, and the paper also describes question sets with roughly 5–38 PDF documents per question, with the average reported as 15 documents. For cross-year questions, the intermediate fact set can be about half the size of the document collection for those tasks. For announcement documents, the authors use a merge strategy to account for disclosure lag across years (Li et al., 24 Apr 2026).

3. Question classes and analytical workload

MuDABench includes both simple and complex analytical questions. The questions usually ask for ranking companies by a metric, computing ranges or variances, comparing values across years, identifying changes over time, calculating growth rates, finding outlier firms based on mean $\pm 2$ standard deviations, analyzing time intervals between corporate events, or comparing firms that satisfy a condition such as “audited by Big 4” (Li et al., 24 Apr 2026).

What makes these questions multi-document analytical tasks is that the answer is not contained in one document. Instead, the system must identify all documents matching the metadata constraint, extract specific numeric or categorical facts from each document, merge facts across documents or across years, and perform a final computation or ranking. For complex questions, this may also involve conditional filtering before aggregation, such as restricting analysis to companies with certain year-over-year changes or only firms meeting a threshold over a period (Li et al., 24 Apr 2026).

This structure shifts the locus of difficulty away from isolated reasoning steps toward end-to-end analytical integrity. The paper emphasizes that the collection behaves like a semi-structured database, not a single long input. A plausible implication is that MuDABench is especially sensitive to compounding error: even moderate failures in per-document extraction can collapse downstream aggregation, because the benchmark’s questions often require a near-complete set of relevant facts rather than a sparse evidential chain (Li et al., 24 Apr 2026).

4. Evaluation protocol and diagnostic metrics

MuDABench uses three metrics: process accuracy, final-answer accuracy, and full accuracy. Final-answer accuracy is the main metric. For each question $Q_i$, let $A_i$ be the gold final answer and $\hat{A}_i$ the prediction; a judge model determines whether $$\mathcal{X} = \left\{ \left( Q_j, \mathcal{D}_j, \mathcal{M}_j, \mathcal{S}_j \right) \mid j = 1, 2, \ldots, n \right\},$$0 is semantically equivalent to $$\mathcal{X} = \left\{ \left( Q_j, \mathcal{D}_j, \mathcal{M}_j, \mathcal{S}_j \right) \mid j = 1, 2, \ldots, n \right\},$$1, yielding a binary correctness indicator $$\mathcal{X} = \left\{ \left( Q_j, \mathcal{D}_j, \mathcal{M}_j, \mathcal{S}_j \right) \mid j = 1, 2, \ldots, n \right\},$$2 (Li et al., 24 Apr 2026).

Process accuracy measures whether the system extracted the needed supporting facts. For standard RAG, the benchmark defines gold supporting facts $$\mathcal{X} = \left\{ \left( Q_j, \mathcal{D}_j, \mathcal{M}_j, \mathcal{S}_j \right) \mid j = 1, 2, \ldots, n \right\},$$3 and extracted facts $$\mathcal{X} = \left\{ \left( Q_j, \mathcal{D}_j, \mathcal{M}_j, \mathcal{S}_j \right) \mid j = 1, 2, \ldots, n \right\},$$4, with coverage

$$\mathcal{X} = \left\{ \left( Q_j, \mathcal{D}_j, \mathcal{M}_j, \mathcal{S}_j \right) \mid j = 1, 2, \ldots, n \right\},$$5

where the intersection is based on semantic matching judged by an LLM rather than exact string match. Because a single judge may overestimate coverage, the paper also introduces a double-check error estimate

$$\mathcal{X} = \left\{ \left( Q_j, \mathcal{D}_j, \mathcal{M}_j, \mathcal{S}_j \right) \mid j = 1, 2, \ldots, n \right\},$$6

and defines conservative coverage as

$$\mathcal{X} = \left\{ \left( Q_j, \mathcal{D}_j, \mathcal{M}_j, \mathcal{S}_j \right) \mid j = 1, 2, \ldots, n \right\},$$7

For the document-grounded workflow, the benchmark uses a cell-wise evaluation on aligned rows. If $$\mathcal{X} = \left\{ \left( Q_j, \mathcal{D}_j, \mathcal{M}_j, \mathcal{S}_j \right) \mid j = 1, 2, \ldots, n \right\},$$8 is the set of required gold metric cells and $$\mathcal{X} = \left\{ \left( Q_j, \mathcal{D}_j, \mathcal{M}_j, \mathcal{S}_j \right) \mid j = 1, 2, \ldots, n \right\},$$9 the correctly extracted cells, then the corresponding coverage is the ratio of correctly extracted cells to required cells (Li et al., 24 Apr 2026).

Full accuracy is a strict joint metric: a sample is correct only if both the process is fully correct and the final answer is correct. The paper treats process coverage mainly as a diagnostic signal rather than a perfect standalone metric, because equivalent evidence may be expressed in non-atomic forms. This makes the evaluation protocol explicitly process-aware: MuDABench does not only ask whether a system arrived at the right output, but also whether it assembled the necessary document-grounded evidence on the way (Li et al., 24 Apr 2026).

5. Reported system behavior and the multi-agent workflow

The experiments show that standard RAG performs poorly, even with large retrieval budgets and even when metadata is injected. Increasing the chunk budget generally improves process coverage somewhat, but final-answer accuracy stays low, especially on complex questions. The paper therefore argues that the main limitation is not only retrieval recall; the system also struggles to synthesize fragmented evidence into a correct aggregated answer. In the reported results, standard RAG final-answer accuracy is often in the single digits to low teens. A noise experiment in which irrelevant documents are injected shows further performance drops, especially on complex questions (Li et al., 24 Apr 2026).

To address these limitations, the authors propose a metadata-aware multi-agent workflow. Its main components are a planning agent, a document-level information extractor, a normalization agent, and a code-generation / code agent. The planning agent decomposes the global question into sub-query templates rather than directly answering it, using the metadata schema to create per-document or per-document-type query templates. For each document $Q_j$0, the system instantiates the templates using metadata $Q_j$1 and runs a standard RAG system on the single document, producing localized textual evidence. The normalization agent converts extraction outputs into flat JSON records, and the code agent receives the task, the schema, a few JSON examples, and a path to the structured data, then writes executable Python code to analyze the JSON and compute the final answer (Li et al., 24 Apr 2026).

The workflow substantially improves both process and final-answer performance, but it remains well below human performance. In one reported setting, the workflow with GPT-4.1 mini reaches roughly 24.3% process / 24.3% final on simple questions and 16.2% process / 16.2% final on complex questions, with better results at larger chunk budgets. Human performance in the main table is much higher, around 0.833 final on simple and 0.733 final on complex questions. The authors identify two primary bottlenecks: single-document information extraction accuracy and insufficient domain-specific knowledge. They also note secondary code-level issues such as JSON/schema mismatches, encoding or path-reading issues, and runtime errors caused by inconsistent field names (Li et al., 24 Apr 2026).

The reported implementation details reinforce the benchmark’s emphasis on workflow design. PDF parsing uses ChatDOC / commercial PDF processing; standard RAG uses OpenAI File Search and GPT-4o-2024-11-20; the agentic workflow uses DeepSeek-R1-0528 for planning and code generation, DeepSeek-Chat-V3-0324 for normalization, and OpenAI file search for single-document QA; some experiments use GPT-4.1 mini for cost control; all LLMs are run at temperature 0; and judge models include DeepSeek-V3.2 for cell-wise rejudging and Kimi K2 for other correctness judgments (Li et al., 24 Apr 2026).

6. Benchmark position, limitations, and name disambiguation

MuDABench is situated against several neighboring benchmark families. The paper contrasts it with HotpotQA, 2WikiMultiHopQA, MuSiQue, FanOutQA, and MoreDocsSameLen, which emphasize multi-hop reasoning over small sets of web pages; with LongBench, RULER, and LongDocURL, which typically assume the relevant content fits within one context window; with FinanceBench, which is mostly single-document QA; with FinAgentBench, which focuses on retrieval precision and locating the right passages or documents rather than downstream aggregation; and with M3DocVQA, which studies multi-document multimodal understanding on a much smaller scale. The paper also notes that Aryn and DocETL propose workflows for multi-step document processing but do not provide a large public benchmark (Li et al., 24 Apr 2026).

A recurrent source of confusion is nomenclature. MuDABench should not be conflated with two distinct benchmarks named MDBench. One is a synthetic benchmark for LLM evaluation on multi-document reasoning generated from structured seed knowledge and rendered into natural-language document sets (Peper et al., 17 Jun 2025). The other is an open-source benchmarking framework for model discovery on dynamical systems, evaluating data-driven recovery of governing differential equations on ODEs and PDEs (Bideh et al., 24 Sep 2025).

Benchmark	Domain	Core task
MuDABench	Financial document collections	Multi-document analytical QA with metadata-aware filtering and cross-document aggregation
MDBench	Multi-document reasoning	Synthetic QA over generated document sets from structured seed knowledge
MDBench	Dynamical systems	Benchmarking data-driven methods for model discovery

MuDABench’s limitations are stated explicitly. It is restricted to the financial domain because dense semi-structured data are easier to find there than in many other domains. The dataset size is intentionally limited because evaluation is costly. The evaluation is sensitive to issues of fact granularity and semantic equivalence. The documents come from publicly available disclosures, so no private information is involved, and the authors note that the dataset is being corrected over time, so reported results may not be fully up to date (Li et al., 24 Apr 2026).

These limitations clarify the benchmark’s intended role. It is not presented as a general benchmark for arbitrary long-context QA, nor as a pure retrieval benchmark, nor as a substitute for naturally occurring benchmarks in other domains. Rather, it functions as a controlled testbed for large-scale analytical reasoning over document collections in which correct document selection, robust per-document extraction, metadata-aware planning, and accurate aggregation are all first-class requirements (Li et al., 24 Apr 2026).