FDABench: Multi-Source Data Agent Benchmark

• FDABench is a comprehensive benchmark designed to evaluate LLM-based data agents that integrate structured databases with unstructured multimodal sources.
• It comprises 2,007 tasks across diverse domains and difficulty levels, emphasizing trade-offs in quality, response latency, and token cost.
• The benchmark supports detailed performance analysis on retrieval, tool invocation, and report generation to compare various analytical agent architectures.

Searching arXiv for FDABench and the benchmark papers it is positioned against. FDABench is a benchmark for evaluating data agents on analytical queries over heterogeneous data. It targets a setting in which LLM-based agents must integrate structured databases with unstructured and multimodal sources such as documents, web content, audio, and video, and is presented as the first data agent benchmark specifically designed for multi-source data analytical scenarios. The benchmark contains 2,007 tasks spanning different data sources, domains, difficulty levels, and task types, and is designed to expose architectural trade-offs in response quality, accuracy, latency, and token cost across agent systems (Wang et al., 2 Sep 2025).

1. Problem setting and formalization

FDABench is motivated by the observation that modern analytics increasingly requires combining relational databases with extra-database content. In this setting, a data agent is defined as an agent system that uses an LLM to reason, select tools, and orchestrate interactions over heterogeneous data to answer analytical queries. Its defining properties are multi-source reasoning over structured and unstructured data, tool selection and invocation based on the task, context, and observations, and adaptive decision-making and refinement via feedback (Wang et al., 2 Sep 2025).

The benchmark formalizes a workload as $W = (q, D, K)$, where $q$ is a natural-language query, $D$ denotes structured sources such as relational databases, and $K$ denotes unstructured sources such as documents or audio/video. Agent processing is written as

$$\mathcal{A}(q, \mathcal{D}, \mathcal{K}, \mathcal{T}) = \mathcal{L}(\pi(q, \mathcal{D}, \mathcal{K}, \mathcal{T}, \mathcal{O})),$$

where $\mathcal{L}$ is the foundation LLM, $\mathcal{T}$ the available tools, $\pi$ the agent policy, and $\mathcal{O}$ the observations generated during tool interactions. This formalization emphasizes that analytical competence is not reducible to a single retrieval or SQL-generation step; it includes iterative orchestration across data modalities and tools.

FDABench defines heterogeneous, multi-source analytical tasks as queries that require retrieval, transformation, aggregation, and synthesis across both databases and extra-database sources such as PDFs, web pages, and vector indexes of text, audio, or video. The expected outputs may be precise numbers, explanatory reports, or combinations of quantitative results with qualitative context. This positioning distinguishes the benchmark from text-to-SQL benchmarks that evaluate structured querying only, RAG benchmarks that emphasize text retrieval and generation quality, and general agent benchmarks that test tool use without centering multi-source data analytics or report-generation quality (Wang et al., 2 Sep 2025).

2. Benchmark composition and analytical coverage

FDABench is constructed at a scale intended to support broad comparative evaluation. It contains 2,007 tasks across 50+ domains and 130+ databases. The benchmark is organized into three task types—Single-choice (SC), Multiple-choice (MC), and Report—and into three difficulty levels with the following distribution: Easy 415 (20.68%), Medium 659 (32.84%), and Hard 933 (46.49%) (Wang et al., 2 Sep 2025).

Its structured component comprises 110+ relational databases collected from Spider, Spider 2.0, BIRD, and DABStep, with coverage of joins, aggregations, filters, nested queries, and dirty data handling. Its unstructured component includes more than 1,600 files across 50+ domains, including PDF documents, audio, and video embedded into vector databases, together with web content gathered with Perplexity search and file-system corpora. The tool environment includes SQL/query tools, vector search and retrievers, web search, document operators such as extract and summarize, and semantic operators including sem_filter, sem_agg, and sem_sim_join.

The three task categories are differentiated by output form and analytical demand. Single-choice tasks target precise numerical or statistical queries requiring exact results, including aggregation across tables, conditional filters, and single-source or cross-source joins. Multiple-choice tasks require multi-answer analytical inference, such as identifying all statements consistent with database results and document evidence. Report tasks require comprehensive, structured reports that synthesize quantitative findings derived from databases with qualitative insights from unstructured sources, including trends, context, and comparisons (Wang et al., 2 Sep 2025).

Difficulty labeling is based on reasoning token usage and tool-call counts, both of which increase with complexity, together with query composition. Easy tasks correspond to patterns such as single-table aggregation plus single unstructured snippet lookup, with few tool calls and a short reasoning trace. Medium tasks involve multi-table joins together with multi-document retrieval and synthesis. Hard tasks include multi-step nested SQL, cross-source joins or alignments, multi-hop retrieval, and synthesis under noisy or long contexts, with higher tool-call volume and extended reasoning chains.

The illustrative tasks clarify the intended workload. A report-mode finance task combines a relational database of quarterly revenue and segment tables with annual-report PDFs and web articles, then asks for a summary of FY2023 revenue trend by segment together with an explanation of training cost drivers mentioned by management. A healthcare single-choice task combines a patient-admissions database with a public guideline PDF and asks for the average length of stay for cardiology admissions in Q2 2024. A retail multiple-choice task joins store-level sales data with web articles on promotions to determine which statements are consistent with both sales changes and promotion evidence. These examples show that FDABench evaluates not only retrieval or SQL correctness, but also cross-source alignment and synthesis.

3. Agent–expert construction workflow and reliability controls

FDABench is built through an agent–expert collaboration framework designed to avoid the weaknesses of both LLM-only generation and purely manual benchmark construction. The benchmark authors state that LLM-only generation of multi-source tasks is unreliable, whereas human-only construction is costly. The construction process is therefore organized into a three-phase workflow referred to as Algorithm 1 (Wang et al., 2 Sep 2025).

Phase 1 – Initial context gathering: database schema is extracted and gold SQL from validated datasets is executed to obtain gold structured results. Unstructured context is then gathered through web search, vector search, and file search tailored to the initial query $q_0$ and database context. These materials are collated into a context bundle

$q$0

Phase 2 – Expert verification and refinement: an agent drafts benchmark instances, including query variants, options, gold subtasks, and report references. Human experts then assign one of three statuses, $q$1. If a draft is marked REVISE, the agent regenerates it using the revision history $q$2, and the cycle continues until ACCEPT or a maximum number of iterations is reached.

Phase 3 – Finalization: accepted instances are stored together with gold subtasks and references, whereas disposed or max-iteration drafts are discarded. Difficulty labels are assigned on the basis of reasoning tokens and tool-call counts.

The division of labor is explicit. The agent drafts tasks with structured–unstructured linkages, proposes gold subtasks, and standardizes outputs by category. Experts validate database–document relationships, answer correctness, option quality, and report coverage, and they enforce acceptance thresholds. The paper does not report inter-annotator agreement or $q$3; quality assurance is instead implemented through expert accept/revise/dispose gating. This suggests that the benchmark’s reliability claims rest on controlled review and iterative rejection rather than on annotator-consensus statistics.

4. Portability, evaluation pipeline, and metrics

A central design objective of FDABench is generalization across target systems and frameworks. The benchmark supports end-to-end LLM agents using planning, reflection, tool-use, or multi-agent patterns, as well as SQL-capable agents, semantic-operator systems such as LOTUS, Palimpsest, and DocETL, and RAG-based systems such as NaiveRAG, GraphRAG, HippoRAG2, and CORAG. Its unified evaluation pipeline normalizes inputs and outputs across SC, MC, and report tasks, and records intermediate traces including plans, tool invocations, and observations for capability analysis (Wang et al., 2 Sep 2025).

The evaluation setup reported in the paper uses an Ubuntu server with dual AMD EPYC 9555 processors (64-core $q$4), 2.2 TB RAM, and $q$5 NVIDIA H200 NVL GPUs. DeepSeek-V3 via OpenRouter is the default foundation model for many experiments. Input and output are standardized by task type: SC and MC instances provide a question and options, and the agent returns choice or choices with optional justification; report instances provide a question and sources, and the agent returns a structured report. In all cases, intermediate steps are captured.

Ground truth is constructed separately for structured and unstructured components. Structured answers are obtained by executing gold SQL from validated datasets such as Spider and BIRD to derive precise numeric answers. Unstructured content is selected through search and retrieval and validated by experts, together with gold references. Judging is rule-based rather than LLM-mediated: exact match is used for SC and MC, and ROUGE-1 and ROUGE-L are used against gold report or reference texts for report tasks. Tool capability is measured through Tool Recall and Success Rate, and the paper reports no LLM-as-judge setup, no composite weighted score, and no statistical tests (Wang et al., 2 Sep 2025).

The reported metric definitions include

$q$6

Latency is summarized by

$q$7

with recorded breakdown categories Decision, Execute, Retry, and Generate. Token cost is defined as

$q$8

where $q$9 and $D$0 are model-dependent per-token unit costs. Monetary cost is reported, although currency conversion is not specified.

5. Empirical findings and characteristic failure modes

FDABench is used to compare multiple classes of data-agent systems and to surface trade-offs among quality, accuracy, efficiency, and cost. Among general analytical agents evaluated with DeepSeek-V3 across easy, medium, and hard difficulties, DAgent exhibits the lowest latency at 178/192/211 seconds and fewer external model calls at 4.1/4.7/5.1, but only moderate quality, with ROUGE-1 of 0.39/0.38/0.34 and EX$D$1 of 0.48/0.44/0.29. AOP, which combines planning and reflection, shows the best overall effectiveness among general agents, with ROUGE-1 of 0.51/0.44/0.41, ROUGE-L of 0.24/0.21/0.19, EX$D$2 of 0.62/0.48/0.35, and EX$D$3 of 0.41/0.38/0.31, with latency 165/189/225 seconds and 15.2/17.6/19.8 external calls. Taiji incurs higher external-call counts, 26.4–31.1, and latency, 202–255 seconds, with moderate quality, while AgenticData achieves strong EX$D$4 on easy tasks at 0.66 but with higher cost and latency at 226–298 seconds (Wang et al., 2 Sep 2025).

Semantic-operator systems occupy a different region of the performance frontier. LOTUS and DocETL attain high EX$D$5 on easy and hard tasks, approximately 0.79–0.81 and 0.76–0.78, together with strong ROUGE-1 around 0.42–0.48, but their monetary cost is reported as 6–20$D$6 higher than that of general agents, with roughly 98–102 external calls and latency around 642–677 seconds. Palimpzest shows still higher overhead, with external calls around 152–159 and latency around 1,015–1,039 seconds, together with mixed exact-match performance but competitive ROUGE.

Among RAG systems with a planning adapter, CORAG is reported as offering the best balance of quality and efficiency, with ROUGE-1 of 0.41/0.39/0.36, EX$D$7 of 0.52/0.45/0.32, external calls of 4.0–5.1, latency around 197–202 seconds, and the lowest costs among RAG methods. GraphRAG and HippoRAG2 require more external calls and higher latency, while NaiveRAG has the lowest calls and cost but also the weakest quality, including ROUGE-1 of 0.33/0.31/0.29 and EX$D$8 of 0.41/0.35/0.24.

The benchmark also supports comparisons across workflow patterns and foundation models. Reflection patterns yield top ROUGE and robust exact-match performance for some models, whereas planning patterns minimize cost while maintaining high Tool Recall. Multi-agent patterns often achieve the highest SC/MC exact match for models such as Kimi-k2, with EX approximately 0.584, but at higher latency and cost. Tool-use patterns show model-dependent variance in Success Rate; one example reported is DeepSeek-R1 with $D$9 under tool-use. Efficiency breakdowns make these differences operationally explicit: planning agents show stable time allocation and minimal retry overhead below 3%, with total latency around 202–225 seconds; reflection agents incur retry overhead of 26–29% and total latency around 456–657 seconds; multi-agent systems show increasing coordination overhead, with Decision time accounting for approximately 28–41% as difficulty rises.

The principal failure modes identified in the benchmark are schema linkage and cross-table join errors under complex database schemas, cross-source alignment mistakes linking database entities to document mentions, retrieval failures or irrelevant chunk selection, hallucinations in report narratives without sufficient grounding, and excessive tool invocations in multi-agent systems without proportional quality gains. These error categories are significant because they correspond closely to the benchmark’s core competencies: structured reasoning, cross-source entity alignment, retrieval precision, grounded synthesis, and efficient orchestration.

6. Relation to adjacent benchmarks, reproducibility, and limitations

FDABench is positioned as a complement to text-to-SQL benchmarks such as Spider, Spider 2.0, and BIRD, because it adds unstructured or multimodal sources and evaluates report quality in addition to database querying. It extends beyond RAG benchmarks such as CRAG, HotpotQA, and MS MARCO by requiring database reasoning and tool orchestration, and differs from general agent benchmarks such as AgentBoard, GAIA, WebArena, and MINT by focusing specifically on multi-source analytics, tool invocation correctness, and report generation quality. Relative to DABStep, it broadens modality coverage to DB + PDFs/audio/video/web, increases scale to 2,007 tasks, expands evaluation dimensions, and standardizes SC, MC, and Report categories (Wang et al., 2 Sep 2025).

The name is potentially confusing because adjacent literatures use similar acronyms for unrelated benchmarks. A paper queried as FinDABench or FDABench corresponds to BIBench rather than a separate FDABench release (Liu et al., 2024). Other similarly named systems include FedTADBench for federated time-series anomaly detection (Liu et al., 2022), FedAD-Bench for federated unsupervised anomaly detection in tabular data (Anwar et al., 2024), and FD-Bench for data-driven fluid simulation (Wang et al., 25 May 2025). In the present usage, FDABench denotes the benchmark for data agents operating over heterogeneous analytical sources.

For reproducibility, the benchmark’s source code and data artifacts are available at https://github.com/fdabench/FDAbench. The repository includes the benchmark and adapters; licensing and versioning are not specified in the paper. The implementation is in Python, Ubuntu/Linux is recommended, access to foundation models is provided through OpenRouter, and deployments require a vector database for unstructured embeddings together with standard SQL engines. New systems are incorporated by implementing the adapter interface for a workflow pattern, mapping native input and output to FDABench schemas, validating tool availability, and enabling logging so that tool metrics can be computed.

The benchmark’s limitations are also explicit. Although its domain and modality coverage is broad, it may not capture all real-world data-lake heterogeneity, with logs and images beyond videos not emphasized. Report evaluation relies on ROUGE and SC/MC evaluation relies on rule-based exact match; no inter-annotator agreement, human rubric scoring, or statistical significance testing is reported. Ground truth depends on gold SQL for structured answers and curated unstructured content for qualitative evidence, which implies continuing expert curation as the benchmark expands. The paper identifies future directions including broader modality coverage, more domains and tools, rubric-based human or LLM judging for nuanced report quality, robustness tests with noisy or unreliable sources and adversarial retrieval, and standardized cost-aware scoring. Taken together, these limitations indicate that FDABench is a rigorous and portable benchmark for multi-source analytical agents, but not a final specification of the space.