---
title: 'DARE-bench: Data Science LLM Benchmark'
url: https://www.emergentmind.com/topics/dare-bench
type: topic
---

# DARE-bench: Data Science LLM Benchmark

DARE-bench is a benchmark designed for machine learning modeling and data science instruction following. It evaluates and trains LLM agents that execute multi-step, tool-augmented data science workflows under realistic constraints, and it is built around verifiable ground truth rather than human- or model-based judges. The benchmark consists of 6,300 Kaggle-derived tasks and serves both as an evaluation benchmark and a large, labeled training corpus across instruction following and modeling, including time series [2602.24288].

## 1. Scope, objectives, and measured capabilities

DARE-bench targets two fundamental DS-agent capabilities. The first is **instruction following (process fidelity)**, defined as faithfully executing a specified workflow with deterministic outcomes under fixed randomness and explicit procedural constraints. The second is **ML modeling (outcome performance)**, defined as building predictive models that maximize performance under standard, reproducible metrics. This emphasis on process-aware, verifiable evaluation addresses two gaps identified by the benchmark: the lack of standardized, process-aware evaluation of instruction adherence and process fidelity, and the scarcity of accurately labeled, scalable training data [2602.24288].

The benchmark is organized into three task families and six variants:

| Variant | Capability emphasis | Scoring basis |
|---|---|---|
| Classification-IF | Instruction following | Acc |
| Classification-MM | ML modeling | macro-F1 |
| Regression-IF | Instruction following | Acc |
| Regression-MM | ML modeling | clipped \(R^2\) |
| Time-series-XF | Time series with exogenous features available for train/test | clipped \(R^2\) |
| Time-series-CF | Canonical forecasting; exogenous features available for training but test contains only timestamp/entity columns | clipped \(R^2\) |

The task design is explicitly agentic. Inputs comprise a natural-language question \(Q\), dataset metadata \(M\), training data \(D_{\text{train}} = \{(x_i, y_i)\}\), a test split \(D_{\text{test}}\) with masked targets, and a code execution tool \(T\) with runtime limit \(T_{\max}\) and an interaction budget \(K\) turns. The agent produces executable code \(C\); the tool executes \(C\) to fit on \(D_{\text{train}}\) and predict on \(D_{\text{test}}\): \(y = G(Q, D_{\text{train}}, D_{\text{test}}, M, T(T_{\max}, K))\). Outputs are standardized as a `prediction.csv` aligned by `row_id` with the required target(s) [2602.24288].

## 2. Corpus construction and task packaging

DARE-bench contains 6,300 tasks total, curated automatically from Kaggle datasets, not just competitions, with augmented metadata from web crawl. The split is approximately 95/5 train/test, and the most recently updated tasks are designated as test. Domain coverage is broad; examples reported in the paper include Finance at 16.9% of train and 17.1% of test, Health at 10.2% and 8.4%, and Others at 47.5% and 51.9% [2602.24288].

The curation pipeline has four stages. **Dataset sourcing** uses the Kaggle API plus web crawling, with filtering by tabular format, license, size, and metadata. **LLM-assisted task design and feasibility analysis** identifies targets, features, and data types; for time series it detects timestamp, frequency, and exogenous features, and checks feasibility. **Post-processing** applies a random train/test split for classification and regression, and a chronological split for time series; instruction-following tasks inject controlled noise into approximately 20% of training data, while test remains clean reference data; time series tasks detect entity identifiers to prevent leakage across groups and resample irregular series to uniform intervals using LLM-suggested aggregation. **Finalization** verifies solvability for IF tasks by executing reference code in a sandbox to produce deterministic \(y_{\text{ref}}\) under fixed seeds, while MM tasks use dataset labels directly and do not require sandbox execution for reference production [2602.24288].

This packaging is central to DARE-bench’s verifiability. Each task folder contains train/val files, `metadata.txt`, and a natural-language question. For IF tasks, a reference code path is provided to generate \(y_{\text{ref}}\); for MM tasks, test labels are masked. The benchmark also enforces safeguards against leakage through entity ID detection for grouped series, chronological splits, masked ground truth in test files, and a standardized I/O contract based on `row_id` alignment [2602.24288].

## 3. Execution environment and sandboxed agent interaction

DARE-bench is designed for tool-augmented agents that write and run code. Its execution environment is a stateless Python code executor that runs user-provided scripts in an isolated sandbox. The authors use SandboxFusion (ByteDance) for execution and enforce deterministic settings through fixed seeds and reproducible environments [2602.24288].

The tool schema, denoted `python_executor`, is single-shot. It uses a 200-second per-run time limit, allows up to 3 tool invocations per conversation, and imposes 200 MB file upload/download limits. Required arguments are `code` as a string, `files_to_load` as a list of paths, and `files_to_save` as a list of paths. The tool returns full stdout/stderr, so explicit prints are required for logs. Although the paper describes a 10-minute execution limit as a design feature, the main evaluations adopt 5 turns and 200 s per tool run as a balanced configuration [2602.24288].

The benchmark’s examples illustrate why these constraints matter. One IF example requires applying specified range filters while retaining rows with missing values but dropping rows violating value ranges, selecting features in a specified order, imputing numerics by mean and categoricals by most frequent, applying `StandardScaler` for numerics and `OneHotEncoder(handle_unknown=ignore, sparse_output=False)` for categoricals, training `LogisticRegression` with `random_state=86`, and emitting `prediction.csv` aligned by `row_id`. A failure example shows an agent ignoring an explicit seed requirement for `MLPClassifier(random_state=51)`, causing non-deterministic predictions and an evaluation failure. A plausible implication is that DARE-bench treats procedural compliance not as stylistic preference, but as a reproducibility condition [2602.24288].

## 4. Evaluation semantics, metrics, and reward definitions

The benchmark operationalizes **instruction adherence** and **process fidelity** through outcome equivalence under controlled randomness. For instruction-following tasks, the reference solution \(y_{\text{ref}}\) is produced by executing a reference program \(C_{\text{ref}}\) that strictly follows the instructions. Under controlled randomness, following the same steps yields deterministic outputs. Evaluation is exact-match: \(\mathrm{Acc}_{\mathrm{IF}} = 1\) if \(y = y_{\text{ref}}\), else \(0\). The paper does not introduce a step-level rubric or a weighted composite across procedural steps; instead, IF tasks use outcome equivalence as the verifiable indicator of process fidelity [2602.24288].

For classification IF tasks, the benchmark reports standard accuracy,
\[
\mathrm{Acc} = \frac{1}{n}\sum_{i=1}^{n} \mathbf{1}(y_i = \hat{y}_i).
\]
For Classification-MM, it uses macro-F1:
\[
\mathrm{Precision}_c = \frac{TP_c}{TP_c + FP_c}, \quad
\mathrm{Recall}_c = \frac{TP_c}{TP_c + FN_c},
\]
\[
F1_c = \frac{2 \cdot \mathrm{Precision}_c \cdot \mathrm{Recall}_c}{\mathrm{Precision}_c + \mathrm{Recall}_c},
\]
\[
\mathrm{Macro\text{-}F1} = \frac{1}{C} \sum_{c=1}^{C} F1_c.
\]
For Regression-MM and the two time-series tracks, it uses clipped coefficient of determination:
\[
R^2 = 1 - \frac{\sum_{i=1}^{n} (y_i - \hat{y}_i)^2}{\sum_{i=1}^{n} (y_i - \bar{y})^2},
\qquad
\mathrm{clip}(R^2) = \max(R^2, 0).
\]
For multi-target tasks, the metric is averaged across targets [2602.24288].

DARE-bench also defines verifiable rewards for RL. For IF tasks with reference solution \(y_{\mathrm{ref}}\) and model predictions \(\hat{y}\), the reward is \(1.1\) if \(\hat{y} = y_{\mathrm{ref}}\), \(0.1\) if \(\hat{y}\) exists, and \(0\) otherwise. For predictive ML tasks with ground-truth \(y_{gt}\) and model predictions \(\hat{y}\), the reward is \(0.1 + d(\hat{y}, y_{gt})\) if \(\hat{y}\) exists, and \(0\) otherwise, where \(d(\cdot)\) is macro-F1 for classification or clipped \(R^2\) for regression and time series; for multiple targets, the reward averages across targets. Each task is run three times and the average score is reported, with greedy decoding whenever applicable [2602.24288].

## 5. Empirical findings, failure modes, and training utility

DARE-bench reports substantial sensitivity to interaction budgets and runtime limits. In the hyperparameter study with gpt-o4-mini, Classification-IF increases from 37.16 at 3 turns and 300 s to 67.56 at 5 turns and 200 s, and the best observed Classification-IF is 76.80 at 15 turns and 100 s. For balanced cost and latency, the benchmark adopts 5 turns and 200 s for main comparisons; under that setting, gpt-o4-mini reaches 67.56 on class-IF, 57.89 on class-MM, 53.62 on reg-IF, 57.60 on reg-MM, 42.29 on time-XF, and 9.67 on time-CF [2602.24288].

On the test tasks under the 5-turn, 200-second configuration, Claude-Sonnet-3.7 leads most modeling columns, with 61.03 on classification-MM, 63.20 on regression-MM, 49.88 on time-XF, and 13.70 on time-CF. GPT-5 leads the IF columns with 69.81 on class-IF and 57.24 on reg-IF. The same table reports the following score vectors by family order class-IF, class-MM, reg-IF, reg-MM, time-XF, time-CF: gpt-4o at 32.88, 40.45, 20.28, 40.60, 35.54, 4.77; gpt-4.1 at 55.82, 57.83, 52.17, 58.62, 40.78, 6.60; Claude-Sonnet-4 at 16.21, 18.27, 15.21, 11.33, 4.80, 0.01; Qwen3-32B at 17.11, 30.71, 15.21, 35.86, 26.96, 0.00; and Qwen3-4B at 3.60, 5.23, 0.72, 3.29, 6.97, 0.00. Even highly capable models struggle, especially in Time-series-CF, where near-zero scores are common among open-source models [2602.24288].

The benchmark’s failure analysis is process-oriented. Reported failure modes include incorrect tool argument passing, wrong step order, missing required arguments such as `random_state`, skipping mandatory preprocessing, fragile preprocessing, unreliable type inference, hard-coding metadata, and especially poor formatting and forecasting behavior on canonical forecasting tasks. In Time-series-CF, near-zero performance is associated with format errors and trivial heuristics such as last value or mean. This suggests that DARE-bench is not merely measuring whether a model can emit plausible code, but whether it can maintain procedural and statistical discipline inside a constrained execution loop [2602.24288].

A major feature of DARE-bench is its use as training data. Supervised fine-tuning boosts Qwen3-32B’s accuracy by 1.83x, increasing the baseline total score from 23.25 to roughly 42–43 depending on trace-selection strategy. Reported SFT totals are 42.42 for SFT-FV, 42.91 for SFT-AV, 42.83 for SFT-BV, and 41.12 for SFT-DV. Reinforcement learning boosts Qwen3-4B’s accuracy by more than 8x, increasing the total from 4.39 to 37.40, while Model-Perf rises from 54.18 to 62.55. The paper further reports that RL on Qwen3-4B reduces code errors by 48% and halves execution-limit errors, and that SFT encourages checking `metadata.txt` first. On DSBench transfer, SFT-DV improves Qwen3-32B competition-level accuracy from 32.38 to 42.41, and against DataWiseAgent the same model reaches 42.41 versus 29.17, although time-series-CF remains 0.0 for both [2602.24288].

## 6. Naming, adjacent uses, and limitations

The name “DARE-bench” is not unique across recent arXiv literature. In the diffusion-LLM framework “DARE: Diffusion Large Language Models Alignment and Reinforcement Executor,” the benchmarking component is an OpenCompass-based evaluation platform integrated into DARE; when referred to as a standalone component, the summary names it DARE-bench, although the paper itself uses “evaluation platform” or “DARE evaluation” rather than the explicit moniker “DARE-bench” [2604.04215]. That usage is separate from the data-science benchmark described here. It is also distinct from “DARE: Diverse Visual Question Answering with Robustness Evaluation,” a multiple-choice VQA benchmark focused on robustness across prompt variations, subsets of answer options, output format constraints, and multiple correct answers [2409.18023].

Within the data-science benchmark itself, the stated limitations are narrow but consequential. Current tasks are primarily tabular, so multimodal inputs such as text plus image are not covered. Generating large numbers of executable traces is costly, and rejection sampling may bias toward shorter trajectories. The paper does not list per-variant task counts or difficulty tiers. Planned extensions include figures, speeches, clustering, stronger procedural constraints and verifier-based objectives, and anomaly detection tracks for tabular and time series with event- or segment-level metrics and weak or unsupervised scoring protocols [2602.24288].

In this sense, DARE-bench occupies a specific niche: an executable, process-aware benchmark for LLM data-science agents, built around deterministic verification, realistic sandbox constraints, and training-time utility. Its central methodological claim is that process fidelity in data science can be evaluated objectively through reproducible outcomes rather than through subjective judges, provided that the environment, seeds, tools, and I/O contracts are controlled tightly enough [2602.24288].

Source: https://www.emergentmind.com/topics/dare-bench