DABench-LLM: Data Analysis Agent Benchmark

• The paper introduces InfiAgent-DABench, the first closed-form benchmark designed to evaluate LLM agents on end-to-end CSV data analysis tasks.
• It employs a multi-stage dataset creation process, combining GPT-generated CSV descriptions, expert-guided question formulation, and strict output formatting for automatic grading.
• Baseline results reveal that even leading LLMs struggle with structured data analysis, highlighting the need for benchmark-specific agent tuning.

DABench-LLM commonly refers to InfiAgent-DABench, a benchmark introduced to evaluate LLM-based agents on end-to-end data analysis tasks grounded in real CSV files. It was presented as “the first benchmark specifically designed to evaluate LLM-based agents on data analysis tasks,” and it targets agents that must plan, write code, invoke a Python execution environment, self-debug if needed, and produce a final answer in a strict machine-parseable format (Hu et al., 2024). In its original formulation, the benchmark combines DAEval, a closed-form evaluation dataset, with an agent framework that supports ReAct-style reasoning, file interaction, and Python sandbox execution (Hu et al., 2024). Subsequent work used InfiAgent-DABench as a central evaluation target for notebook-centric and search-based agents, including Jupiter, which reports results under the same benchmark family (Li et al., 11 Sep 2025). The name is not fully stable across the literature: later papers use “DABench” for unrelated affective-memory evaluation (Lu et al., 31 Oct 2025), and a distinct 2025 hardware paper uses “DABench-LLM” for benchmarking dataflow accelerators for LLM training rather than agentic data analysis (Hu et al., 4 Dec 2025).

1. Definition, scope, and motivation

InfiAgent-DABench was built because existing code-completion benchmarks such as HumanEval, MBPP, and DS-1000 were characterized as “not suitable for evaluating LLM-based agents which require complex abilities, such as planning, self-debugging” (Hu et al., 2024). The benchmark therefore focuses on realistic data analysis tasks that are “challenging but practically useful” and require “the harmonization ability of LLMs in natural language and code” (Hu et al., 2024).

Its scope is explicitly end-to-end. Rather than measuring isolated function synthesis or snippet completion, it evaluates agents that receive a grounded question over a CSV file, reason in a ReAct loop, generate and execute Python code, inspect outputs, revise when necessary, and then conclude with a structured answer (Hu et al., 2024). This framing distinguishes it from code benchmarks centered on single-function synthesis and from agent benchmarks such as WebShop, WebArena, ToolBench, and Gentopia, which emphasize web interaction or general tool use rather than CSV-centric data analysis with closed-form evaluation (Hu et al., 2024).

The benchmark’s main contributions were stated as threefold: proposing InfiAgent-DABench itself; benchmarking 34 LLMs and finding that even state-of-the-art models still face substantial difficulty on data analysis tasks; and introducing DAInstruct together with DAAgent, an open-source data analysis agent that outperforms GPT-3.5 on the benchmark by 3.9% (Hu et al., 2024). This suggests that the benchmark was intended not only as an evaluation suite but also as a vehicle for studying specialization in data-analysis agents.

2. Dataset construction and task distribution

The benchmark’s public evaluation component, DAEval, contains 257 data analysis questions derived from 52 CSV files collected from GitHub under criteria intended to ensure meaningfulness, English headers or content, and adequate scale, specifically columns greater than 5 and rows at least 20 (Hu et al., 2024). On the validation set, the reported statistics are 52 CSV files, 257 questions, 461 subquestions, a maximum of 8 subquestions per question, an average of 9639.7 rows per file, and an average of 10.6 columns per file (Hu et al., 2024).

The construction workflow is multi-stage. First, GPT-3.5 generates a description of each CSV, including schema, data types, and missing values. Second, expert interviews identify seven concept categories: Summary Statistics, Distribution Analysis, Correlation Analysis, Outlier Detection, Comprehensive Data Preprocessing, Feature Engineering, and Machine Learning. Third, GPT-4 generates open-ended questions grounded in file descriptions and concepts, with prompts designed to increase difficulty. Fourth, a “format-prompting technique” converts those questions into closed-form form by adding constraints and explicit output requirements as a sequence of @answer_name[answer] subquestions. Fifth, labels are produced with OpenAI ADA, and a label is kept only when ADA generates the same answer in all three trials. Sixth, experts assess Suitableness, Reasonableness, Value, Restrictiveness, Alignment, and Correctness, after which “85% samples are qualified and kept in final” (Hu et al., 2024).

The benchmark is diversified along several axes. Files are clustered into 9 domains: Finance and Economics; Health and Medical; Demographics and Social Science; Marketing and Consumer Behavior; Energy and Environmental Monitoring; Transportation, Logistics, and Tourism; Culture, Entertainment, and Media; Scientific Research and Technology; and Other Categories (Hu et al., 2024). Question complexity is annotated both by concepts per question and by overall difficulty. The concept-count distribution is 150 questions with 1 concept, 97 with 2 concepts, 9 with 3 concepts, and 1 with 4 concepts. The difficulty distribution is 82 easy, 87 medium, and 88 hard questions (Hu et al., 2024).

Expert comparison between human-authored and GPT-4-generated questions yielded slightly higher scores for GPT-4 on Reasonableness, Value, Restrictiveness, and Alignment, and the win rate for difficulty favored GPT-4 questions at 60.66% versus 39.34% (Hu et al., 2024). A plausible implication is that the benchmark treats synthetic question generation not merely as scalable augmentation, but as a quality-controlled source of task instances.

3. Closed-form formulation and automatic evaluation

A central design decision in DABench-LLM is the conversion of open-ended data-analysis questions into closed-form tasks that can be automatically graded without human supervision. The paper motivates this by noting that data analysis questions are often open-ended and difficult to evaluate reliably, while GPT-4 as an auto-evaluator of open-ended answers achieved only “about 67% consistency with human experts” (Hu et al., 2024).

The format-prompting technique adds detailed constraints on methods, workflows, and statistical parameters, and imposes an explicit output schema consisting of ordered tag-value pairs such as @answer_name[answer] (Hu et al., 2024). These tags may specify type, rounding, and valid range. In a correlation example, the constraints require the Pearson correlation coefficient, a significance level of 0.05, p-value reporting, and a rule for determining whether the relationship is linear, nonlinear, or none; the corresponding format block requires fields such as @correlation_coefficient[r_value], @p_value[p_value], and @relationship_type[relationship_type], with explicit numeric formatting (Hu et al., 2024).

Formally, if a question $q$ is posed over a table $T$, the gold answer is defined as

$$a^* = f(q, T) = \big(@n_1[v_1^*], \dots, @n_m[v_m^*]\big),$$

where the deterministic procedure $f$ is fixed by the question’s constraints, including fixed random_state where applicable (Hu et al., 2024). A model response $R$ is parsed by a reformatter and regex extractor into

$$\hat{a} = g(R) = \big(@n_1[\hat{v}_1], \dots, @n_m[\hat{v}_m]\big).$$

Question-level correctness is then

$$\text{Correct}(q) = \prod_{j=1}^{m} \mathbf{1}[\, \hat{v}_j = v_j^* \,],$$

so a question is correct if and only if every subanswer matches exactly after the specified rounding and formatting constraints are applied (Hu et al., 2024).

At the benchmark level, if $N$ is the number of questions, accuracy is

$$\mathrm{Acc} = \frac{1}{N} \sum_{i=1}^{N} \mathbf{1}[\, \hat{a}_i = a_i^* \,].$$

Equivalently, the reported score is the proportion of questions for which all subquestions are answered correctly (Hu et al., 2024). There is no separate $\varepsilon$-tolerance for numeric comparison; numeric tolerance is encoded by the question itself through rounding and range requirements, and the final check is exact string matching (Hu et al., 2024).

4. Agent framework and execution environment

DABench-LLM includes not only a dataset but also an agent framework. The framework allows LLMs to “reason in a ReAct way, interact with files and invoke tools such as a Python code sandbox,” and it supports both proprietary models via API and local models via vLLM (Hu et al., 2024). Python execution is isolated in a Docker-based local sandbox intended to avoid safety issues (Hu et al., 2024).

The standard interaction pattern is a ReAct loop:

• Thought
• Action: python_code_sandbox
• Action Input: code
• Observation: sandbox output
• repetition as needed
• Final Answer using the required tags (Hu et al., 2024)

The prompt template imposes operational discipline. For example, “For all outputs in code, THE print() function MUST be called,” and agents are explicitly instructed not to fabricate sandbox outputs such as system notifications (Hu et al., 2024). This matters because the benchmark is designed to assess actual tool-mediated execution rather than purely latent reasoning.

The same framework supports evaluation, model serving, and the generation of DAInstruct instruction-tuning data (Hu et al., 2024). Experimental settings reported in the paper include temperature 0.2, top_p 1.0, and frequency penalty 0.0, together with standardized model versions such as gpt-3.5-turbo-0613 and gpt-4-0613 (Hu et al., 2024). Timeouts, memory limits, and token budgets are not specified in that paper (Hu et al., 2024).

A notable implementation component is a reformatting step performed with GPT-3.5 to align raw model outputs to the exact @answer_name[answer] schema. Ablations show that this step is necessary for fair automatic evaluation and “doesn’t effect to the precision of evaluation” (Hu et al., 2024). This indicates that output-format brittleness is a major confound in agent benchmarking unless normalization is handled explicitly.

5. Baseline results, DAAgent, and benchmark difficulty

The original benchmark study evaluated 34 LLMs and concluded that “Data analysis tasks are challenging for current LLMs” (Hu et al., 2024). GPT-4-0613 achieved 78.99% accuracy, while the best open-source LLM reported there, Qwen-72B-Chat, reached 59.92%, leaving a substantial gap (Hu et al., 2024). At the same time, the paper observed that “most powerful open-source LLMs achieve comparable performance with GPT-3.5,” with Qwen-72B-Chat at 59.92% and GPT-3.5-turbo-0613 at 60.70% (Hu et al., 2024).

The reported results include:

Model or framework	Accuracy (%)
GPT-4-0613	78.99
AutoGen (gpt-4-0613)	71.48
DAAgent-34B	64.59
GPT-3.5-turbo-0613	60.70
Qwen-72B-Chat	59.92
Gemini-Pro	56.42
Qwen-Agent (Qwen-72B-Chat)	55.25
DAAgent-13B	55.86
DAAgent-7B	51.36
XAgent (gpt-4-0613)	47.52
Claude-2.1	47.47

DAAgent is the benchmark’s specialized open-source agent trained with DAInstruct. Its 34B variant achieved 64.59%, surpassing GPT-3.5-turbo-0613 by 3.89 percentage points (Hu et al., 2024). The gains from DAInstruct instruction tuning are large relative to CodeLlama-Instruct baselines: 24.61 to 51.36 for 7B, 26.67 to 55.86 for 13B, and 31.13 to 64.59 for 34B (Hu et al., 2024). This suggests that benchmark-specific specialization can substantially improve performance even without matching the strongest proprietary frontier model.

The reformatting ablation further quantifies format brittleness. Accuracy improved from 72.76 to 78.99 for GPT-4, from 44.75 to 59.92 for Qwen-72B-Chat, from 18.29 to 37.50 for Qwen-14B-Chat, and from 6.23 to 38.67 for Mistral-7B-Instruct-v0.2 when reformatting was applied (Hu et al., 2024). The paper interprets this as evidence that part of the challenge lies not only in solving the data task but in adhering exactly to the output contract.

6. Subsequent use and methodological extensions

Later work treated InfiAgent-DABench as a canonical benchmark for notebook-centric data-analysis agents. Jupiter, which formulates data analysis as a search problem over notebook states and applies MCTS together with a value model trained on NbQA, evaluates directly on the InfiAgent-DABench variant and reports “Accuracy by Questions (%)” as the main metric (Li et al., 11 Sep 2025). Under the reported setup—maximum tree depth 10, up to 40 iterations, per-iteration expansion $T$0, temperature 0.7, and candidate aggregation by mode among collected terminal answers—the method achieved 77.82% ABQ with Qwen2.5-7B-Instruct and 86.38% ABQ with Qwen2.5-14B-Instruct (Li et al., 11 Sep 2025).

Jupiter’s ablations are notable because they isolate the effect of value-guided search on the same benchmark. For Qwen2.5-14B-Instruct, ReAct achieved 77.04, Majority Voting 83.66, and Jupiter with the value model and no exploration term achieved 86.38; with exploration enabled, the same Jupiter configuration dropped to 74.71 (Li et al., 11 Sep 2025). For Qwen2.5-7B-Instruct, the best Jupiter setting reached 77.82 compared with 68.09 for ReAct and 75.10 for Majority Voting (Li et al., 11 Sep 2025). The paper argues that sparse valid program paths make exploration costly, so disabling the exploration term at inference and relying on the value model yields better use of the search budget (Li et al., 11 Sep 2025).

Jupiter also broadens the ecological picture around DABench-LLM by introducing NbQA, a large dataset extracted from real Jupyter notebooks. The pipeline crawls 1.6M notebooks and 3.2M associated data files from about 47k GitHub repositories, producing 38,635 task-solution pairs, of which 6,845 have complete interactive dependencies suitable for trajectory collection and value-model training (Li et al., 11 Sep 2025). Visualization tasks are present in NbQA but excluded from the main LLM experiments and separated for later release (Li et al., 11 Sep 2025). This indicates a trajectory in the literature from manually curated closed-form CSV tasks toward large-scale notebook-derived executable supervision.

A related but distinct line is Data Interpreter, an LLM agent for data science that reports strong performance on its own ML-Benchmark and open-ended task benchmark rather than directly adopting InfiAgent-DABench terminology in the paper text (Hong et al., 2024). The reported benchmarks are “very close in spirit” to DABench-LLM-like evaluation because they emphasize data cleaning, analysis, visualization, end-to-end ML pipelines, and open-ended data-science workflows executed through code with dynamic dependencies (Hong et al., 2024). A plausible implication is that the broader research area converges on a common desideratum—end-to-end agentic data science with execution and verification—even when benchmark nomenclature differs.

7. Naming ambiguities, limitations, and significance

The term “DABench-LLM” requires disambiguation. In the context of data-analysis agents, it refers to InfiAgent-DABench and its evaluation protocol (Hu et al., 2024, Li et al., 11 Sep 2025). However, a 2025 paper on personalized affective agents introduces a different “DABench” centered on emotional expression and emotional change toward objects, not data analysis over CSV files (Lu et al., 31 Oct 2025). Separately, a 2025 hardware systems paper introduces another “DABench-LLM,” this time as a standardized benchmark for post-Moore dataflow AI accelerators running LLM workloads, with tiers for intra-chip profiling and inter-chip scalability analysis (Hu et al., 4 Dec 2025). These are unrelated benchmarks sharing overlapping names.

Within the original InfiAgent-DABench line, an explicit limitation is the exclusion of visualization questions. The 2024 paper describes this as a “significant limitation,” arguing that closed-form scoring for visual outputs is difficult, code-only evaluation may miss visual correctness, and multimodal evaluators such as GPT-4(V) remain “suboptimal” and “costly” (Hu et al., 2024). The authors identify “the precise evaluation of visualization in data analysis” as future work (Hu et al., 2024). Jupiter reiterates that visualization tasks are not used in its LLM evaluations, even though they are part of its broader notebook-derived dataset construction (Li et al., 11 Sep 2025).

Another limitation concerns evaluator design. The closed-form protocol was introduced partly because GPT-4 as an auto-evaluator of open-ended answers achieved only about 67% consistency with human experts (Hu et al., 2024). This strongly shaped the benchmark’s emphasis on deterministic procedures, output tags, and exact string matching. A plausible implication is that DABench-LLM operationalizes a specific philosophy of benchmark design: restrict the answer space enough to make evaluation model-free, even at the cost of excluding some open-ended or visually grounded forms of analysis.

In that form, DABench-LLM occupies a distinct place in the evaluation landscape. It measures agents on grounded CSV inputs, code execution in a sandbox, structured closed-form outputs, and exact-match scoring (Hu et al., 2024). Compared with code benchmarks such as HumanEval, MBPP, and DS-1000, it emphasizes multi-step reasoning, tool use, and self-debugging rather than code completion (Hu et al., 2024). Compared with broader agent benchmarks, it prioritizes structured data-analysis workflows and precise correctness (Hu et al., 2024). That positioning has made it a recurring reference point for later work on notebook-based supervision, value-guided search, and data-analysis agent specialization (Li et al., 11 Sep 2025).