DARE-bench: Data Science LLM Benchmark

• DARE-bench is a benchmark that assesses LLM agents’ ability to follow instructions and build accurate ML models under controlled, reproducible conditions.
• It employs 6,300 Kaggle-derived tasks with defined train/test splits and reference code to verify both process fidelity and outcome performance.
• Its sandboxed execution environment enforces deterministic outcomes through fixed seeds and real-world constraints for tool-augmented data science workflows.

DARE-bench is a benchmark designed for machine learning modeling and data science instruction following. It evaluates and trains LLM agents that execute multi-step, tool-augmented data science workflows under realistic constraints, and it is built around verifiable ground truth rather than human- or model-based judges. The benchmark consists of 6,300 Kaggle-derived tasks and serves both as an evaluation benchmark and a large, labeled training corpus across instruction following and modeling, including time series (Shu et al., 27 Feb 2026).

1. Scope, objectives, and measured capabilities

DARE-bench targets two fundamental DS-agent capabilities. The first is instruction following (process fidelity), defined as faithfully executing a specified workflow with deterministic outcomes under fixed randomness and explicit procedural constraints. The second is ML modeling (outcome performance), defined as building predictive models that maximize performance under standard, reproducible metrics. This emphasis on process-aware, verifiable evaluation addresses two gaps identified by the benchmark: the lack of standardized, process-aware evaluation of instruction adherence and process fidelity, and the scarcity of accurately labeled, scalable training data (Shu et al., 27 Feb 2026).

The benchmark is organized into three task families and six variants:

Variant	Capability emphasis	Scoring basis
Classification-IF	Instruction following	Acc
Classification-MM	ML modeling	macro-F1
Regression-IF	Instruction following	Acc
Regression-MM	ML modeling	clipped $R^2$
Time-series-XF	Time series with exogenous features available for train/test	clipped $R^2$
Time-series-CF	Canonical forecasting; exogenous features available for training but test contains only timestamp/entity columns	clipped $R^2$

The task design is explicitly agentic. Inputs comprise a natural-language question $Q$, dataset metadata $M$, training data $D_{\text{train}} = \{(x_i, y_i)\}$, a test split $D_{\text{test}}$ with masked targets, and a code execution tool $T$ with runtime limit $T_{\max}$ and an interaction budget $K$ turns. The agent produces executable code $R^2$0; the tool executes $R^2$1 to fit on $R^2$2 and predict on $R^2$3: $R^2$4. Outputs are standardized as a prediction.csv aligned by row_id with the required target(s) (Shu et al., 27 Feb 2026).

2. Corpus construction and task packaging

DARE-bench contains 6,300 tasks total, curated automatically from Kaggle datasets, not just competitions, with augmented metadata from web crawl. The split is approximately 95/5 train/test, and the most recently updated tasks are designated as test. Domain coverage is broad; examples reported in the paper include Finance at 16.9% of train and 17.1% of test, Health at 10.2% and 8.4%, and Others at 47.5% and 51.9% (Shu et al., 27 Feb 2026).

The curation pipeline has four stages. Dataset sourcing uses the Kaggle API plus web crawling, with filtering by tabular format, license, size, and metadata. LLM-assisted task design and feasibility analysis identifies targets, features, and data types; for time series it detects timestamp, frequency, and exogenous features, and checks feasibility. Post-processing applies a random train/test split for classification and regression, and a chronological split for time series; instruction-following tasks inject controlled noise into approximately 20% of training data, while test remains clean reference data; time series tasks detect entity identifiers to prevent leakage across groups and resample irregular series to uniform intervals using LLM-suggested aggregation. Finalization verifies solvability for IF tasks by executing reference code in a sandbox to produce deterministic $R^2$5 under fixed seeds, while MM tasks use dataset labels directly and do not require sandbox execution for reference production (Shu et al., 27 Feb 2026).

This packaging is central to DARE-bench’s verifiability. Each task folder contains train/val files, metadata.txt, and a natural-language question. For IF tasks, a reference code path is provided to generate $R^2$6; for MM tasks, test labels are masked. The benchmark also enforces safeguards against leakage through entity ID detection for grouped series, chronological splits, masked ground truth in test files, and a standardized I/O contract based on row_id alignment (Shu et al., 27 Feb 2026).

3. Execution environment and sandboxed agent interaction

DARE-bench is designed for tool-augmented agents that write and run code. Its execution environment is a stateless Python code executor that runs user-provided scripts in an isolated sandbox. The authors use SandboxFusion (ByteDance) for execution and enforce deterministic settings through fixed seeds and reproducible environments (Shu et al., 27 Feb 2026).

The tool schema, denoted python_executor, is single-shot. It uses a 200-second per-run time limit, allows up to 3 tool invocations per conversation, and imposes 200 MB file upload/download limits. Required arguments are code as a string, files_to_load as a list of paths, and files_to_save as a list of paths. The tool returns full stdout/stderr, so explicit prints are required for logs. Although the paper describes a 10-minute execution limit as a design feature, the main evaluations adopt 5 turns and 200 s per tool run as a balanced configuration (Shu et al., 27 Feb 2026).

The benchmark’s examples illustrate why these constraints matter. One IF example requires applying specified range filters while retaining rows with missing values but dropping rows violating value ranges, selecting features in a specified order, imputing numerics by mean and categoricals by most frequent, applying StandardScaler for numerics and OneHotEncoder(handle_unknown=ignore, sparse_output=False) for categoricals, training LogisticRegression with random_state=86, and emitting prediction.csv aligned by row_id. A failure example shows an agent ignoring an explicit seed requirement for MLPClassifier(random_state=51), causing non-deterministic predictions and an evaluation failure. A plausible implication is that DARE-bench treats procedural compliance not as stylistic preference, but as a reproducibility condition (Shu et al., 27 Feb 2026).

4. Evaluation semantics, metrics, and reward definitions

The benchmark operationalizes instruction adherence and process fidelity through outcome equivalence under controlled randomness. For instruction-following tasks, the reference solution $R^2$7 is produced by executing a reference program $R^2$8 that strictly follows the instructions. Under controlled randomness, following the same steps yields deterministic outputs. Evaluation is exact-match: $R^2$9 if $R^2$0, else $R^2$1. The paper does not introduce a step-level rubric or a weighted composite across procedural steps; instead, IF tasks use outcome equivalence as the verifiable indicator of process fidelity (Shu et al., 27 Feb 2026).

For classification IF tasks, the benchmark reports standard accuracy,

$R^2$2

For Classification-MM, it uses macro-F1: $R^2$3

$R^2$4

$R^2$5

For Regression-MM and the two time-series tracks, it uses clipped coefficient of determination: $R^2$6 For multi-target tasks, the metric is averaged across targets (Shu et al., 27 Feb 2026).

DARE-bench also defines verifiable rewards for RL. For IF tasks with reference solution $R^2$7 and model predictions $R^2$8, the reward is $R^2$9 if $Q$0, $Q$1 if $Q$2 exists, and $Q$3 otherwise. For predictive ML tasks with ground-truth $Q$4 and model predictions $Q$5, the reward is $Q$6 if $Q$7 exists, and $Q$8 otherwise, where $Q$9 is macro-F1 for classification or clipped $M$0 for regression and time series; for multiple targets, the reward averages across targets. Each task is run three times and the average score is reported, with greedy decoding whenever applicable (Shu et al., 27 Feb 2026).

5. Empirical findings, failure modes, and training utility

DARE-bench reports substantial sensitivity to interaction budgets and runtime limits. In the hyperparameter study with gpt-o4-mini, Classification-IF increases from 37.16 at 3 turns and 300 s to 67.56 at 5 turns and 200 s, and the best observed Classification-IF is 76.80 at 15 turns and 100 s. For balanced cost and latency, the benchmark adopts 5 turns and 200 s for main comparisons; under that setting, gpt-o4-mini reaches 67.56 on class-IF, 57.89 on class-MM, 53.62 on reg-IF, 57.60 on reg-MM, 42.29 on time-XF, and 9.67 on time-CF (Shu et al., 27 Feb 2026).

On the test tasks under the 5-turn, 200-second configuration, Claude-Sonnet-3.7 leads most modeling columns, with 61.03 on classification-MM, 63.20 on regression-MM, 49.88 on time-XF, and 13.70 on time-CF. GPT-5 leads the IF columns with 69.81 on class-IF and 57.24 on reg-IF. The same table reports the following score vectors by family order class-IF, class-MM, reg-IF, reg-MM, time-XF, time-CF: gpt-4o at 32.88, 40.45, 20.28, 40.60, 35.54, 4.77; gpt-4.1 at 55.82, 57.83, 52.17, 58.62, 40.78, 6.60; Claude-Sonnet-4 at 16.21, 18.27, 15.21, 11.33, 4.80, 0.01; Qwen3-32B at 17.11, 30.71, 15.21, 35.86, 26.96, 0.00; and Qwen3-4B at 3.60, 5.23, 0.72, 3.29, 6.97, 0.00. Even highly capable models struggle, especially in Time-series-CF, where near-zero scores are common among open-source models (Shu et al., 27 Feb 2026).

The benchmark’s failure analysis is process-oriented. Reported failure modes include incorrect tool argument passing, wrong step order, missing required arguments such as random_state, skipping mandatory preprocessing, fragile preprocessing, unreliable type inference, hard-coding metadata, and especially poor formatting and forecasting behavior on canonical forecasting tasks. In Time-series-CF, near-zero performance is associated with format errors and trivial heuristics such as last value or mean. This suggests that DARE-bench is not merely measuring whether a model can emit plausible code, but whether it can maintain procedural and statistical discipline inside a constrained execution loop (Shu et al., 27 Feb 2026).

A major feature of DARE-bench is its use as training data. Supervised fine-tuning boosts Qwen3-32B’s accuracy by 1.83x, increasing the baseline total score from 23.25 to roughly 42–43 depending on trace-selection strategy. Reported SFT totals are 42.42 for SFT-FV, 42.91 for SFT-AV, 42.83 for SFT-BV, and 41.12 for SFT-DV. Reinforcement learning boosts Qwen3-4B’s accuracy by more than 8x, increasing the total from 4.39 to 37.40, while Model-Perf rises from 54.18 to 62.55. The paper further reports that RL on Qwen3-4B reduces code errors by 48% and halves execution-limit errors, and that SFT encourages checking metadata.txt first. On DSBench transfer, SFT-DV improves Qwen3-32B competition-level accuracy from 32.38 to 42.41, and against DataWiseAgent the same model reaches 42.41 versus 29.17, although time-series-CF remains 0.0 for both (Shu et al., 27 Feb 2026).

6. Naming, adjacent uses, and limitations

The name “DARE-bench” is not unique across recent arXiv literature. In the diffusion-LLM framework “DARE: Diffusion LLMs Alignment and Reinforcement Executor,” the benchmarking component is an OpenCompass-based evaluation platform integrated into DARE; when referred to as a standalone component, the summary names it DARE-bench, although the paper itself uses “evaluation platform” or “DARE evaluation” rather than the explicit moniker “DARE-bench” (Yang et al., 5 Apr 2026). That usage is separate from the data-science benchmark described here. It is also distinct from “DARE: Diverse Visual Question Answering with Robustness Evaluation,” a multiple-choice VQA benchmark focused on robustness across prompt variations, subsets of answer options, output format constraints, and multiple correct answers (Sterz et al., 2024).

Within the data-science benchmark itself, the stated limitations are narrow but consequential. Current tasks are primarily tabular, so multimodal inputs such as text plus image are not covered. Generating large numbers of executable traces is costly, and rejection sampling may bias toward shorter trajectories. The paper does not list per-variant task counts or difficulty tiers. Planned extensions include figures, speeches, clustering, stronger procedural constraints and verifier-based objectives, and anomaly detection tracks for tabular and time series with event- or segment-level metrics and weak or unsupervised scoring protocols (Shu et al., 27 Feb 2026).

In this sense, DARE-bench occupies a specific niche: an executable, process-aware benchmark for LLM data-science agents, built around deterministic verification, realistic sandbox constraints, and training-time utility. Its central methodological claim is that process fidelity in data science can be evaluated objectively through reproducible outcomes rather than through subjective judges, provided that the environment, seeds, tools, and I/O contracts are controlled tightly enough (Shu et al., 27 Feb 2026).