DSPredict: End-to-End Predictive Benchmark

• DSPredict is a benchmark within DSGym that organizes end-to-end Kaggle-style predictive modeling challenges across domains such as computer vision and molecular prediction.
• It requires agents to execute complete workflows including data exploration, preprocessing, feature engineering, and model training under realistic constraints.
• Benchmark results show high valid submission rates but low competitive performance, highlighting challenges in robust feature engineering and pipeline optimization.

Searching arXiv for the relevant paper and topic name to ground the article in the published record. DSPredict is a component of DSGym that organizes predictive-modeling challenges as end-to-end, execution-grounded benchmark tasks for data science agents. It is introduced as part of the broader DSGym framework in “DSGym: A Holistic Framework for Evaluating and Training Data Science Agents” (Nie et al., 22 Jan 2026). Within DSGym-Tasks, DSPredict comprises Kaggle-style competitions in which an agent receives a training set $D_{\mathrm{train}}$, a test set $D_{\mathrm{test}}$, and an official evaluation metric $m$, and must generate executable code that explores and preprocesses data, engineers features, trains predictive models, produces a valid submission.csv, and maximizes the target metric. The benchmark spans domains including computer vision, molecular property prediction, and single-cell perturbation, and is intended to measure whether agents can carry out realistic predictive workflows under execution constraints rather than solve static benchmark items by pattern matching alone (Nie et al., 22 Jan 2026).

1. Position within DSGym

DSPredict appears in DSGym as one of the benchmark expansions designed to broaden task coverage beyond conventional analysis benchmarks. DSGym describes it as comprising “challenging prediction tasks spanning domains such as computer vision, molecular prediction, and single-cell perturbation” (Nie et al., 22 Jan 2026). The surrounding DSGym framework emphasizes standardized evaluation interfaces, self-contained execution environments, and modularity for tasks, agent scaffolds, and tools, and DSPredict inherits those design principles.

Within DSGym-Tasks, predictive modeling is partitioned into two splits, DSPredict-Easy and DSPredict-Hard (Nie et al., 22 Jan 2026). This split structure is central to the benchmark’s role: it distinguishes between tasks on which agents can more readily generate workable solutions and tasks on which end-to-end modeling remains difficult. A plausible implication is that DSPredict is intended not merely as a scorecard for final predictive accuracy, but as a stress test of the entire modeling pipeline under heterogeneous data modalities and computational friction.

2. Task formulation and benchmark scope

Each DSPredict task is defined as an end-to-end Kaggle-style competition. The agent is given:

• a training set $D_{\mathrm{train}}$
• a test set $D_{\mathrm{test}}$
• an official evaluation metric $m$

The required workflow consists of four explicit steps: exploring and preprocessing the data, engineering features and training one or more predictive models, writing out a valid submission.csv, and maximizing the target metric $m$ such as SMAPE, ROC-AUC, or classification accuracy (Nie et al., 22 Jan 2026).

The benchmark covers three stated domain families:

• Computer Vision: image classification, segmentation, and pose estimation
• Molecular Property Prediction: quantum-chemistry scalar couplings and enzyme stability
• Single-Cell Perturbation: open-problems-single-cell-perturbations and vaccine response prediction

The data summary reported for DSPredict indicates 54 “Hard” competitions and 38 “Easy” ones, with dataset sizes ranging from 200 MB to more than 10 GB (Nie et al., 22 Jan 2026). Example tasks include digit-recognizer with 123 MB and 42,000 $28\times 28$ grayscale images across 10 classes; siim-acr-pneumothorax-segmentation with 426 MB and 12,000 labeled fluoroscopy images; champs-scalar-coupling with 1.6 GB and approximately 2,000,000 molecular pairs with quantum coupling constants; novozymes-enzyme-stability-prediction with 16 MB and approximately 130,000 enzyme variants with melting temperatures; open-problems-single-cell-perturbations with 4.3 GB; and stanford-covid-vaccine with 2.6 GB (Nie et al., 22 Jan 2026).

These examples show that DSPredict is not restricted to a single statistical regime. The benchmark combines small and large datasets, classification and regression, dense and sparse modalities, and workloads whose preprocessing and feature-engineering requirements differ materially across domains.

3. Data modalities and preprocessing conventions

DSPredict includes modality-specific preprocessing patterns that reflect common competition practice rather than a single canonical pipeline. For tabular data, typical preprocessing pipelines include imputing missing values, one-hot or ordinal encoding categoricals, and scaling numeric columns to zero mean and unit variance (Nie et al., 22 Jan 2026). For time series, the reported conventions are filling gaps by forward-fill or $0$, engineering lag features, and applying date-based cyclical transforms using sine and cosine of day-of-week.

For image tasks, preprocessing typically includes resizing to a fixed shape, normalizing pixel values to $[0,1]$, and optional augmentation such as random crop or flip (Nie et al., 22 Jan 2026). For molecular tasks, preprocessing includes parsing SMILES via RDKit, computing atomic descriptors such as Morgan fingerprints, and normalizing features. For single-cell counts, the benchmark description lists the $D_{\mathrm{test}}$0 transform, top-variance gene selection, and per-cell library-size scaling.

These pipeline descriptions are benchmark-level regularities rather than fixed requirements. The DSPredict setup explicitly leaves room for agent-driven choices, such as median imputation or another strategy “dictated by agents” for missing values (Nie et al., 22 Jan 2026). This suggests that the benchmark is designed to evaluate procedural competence in pipeline construction, not only knowledge of a predetermined recipe.

4. Metrics and evaluation protocol

DSPredict preserves the original leaderboard metric $D_{\mathrm{test}}$1 for each task (Nie et al., 22 Jan 2026). The paper enumerates a set of common metrics and gives their mathematical forms:

• Mean Squared Error

$D_{\mathrm{test}}$2

• Root MSE

$D_{\mathrm{test}}$3

• Mean Absolute Error

$D_{\mathrm{test}}$4

• Symmetric MAPE

$D_{\mathrm{test}}$5

• Accuracy

$D_{\mathrm{test}}$6

• ROC-AUC

$D_{\mathrm{test}}$7

• Pearson correlation

$D_{\mathrm{test}}$8

In addition to task-specific leaderboard metrics, DSPredict tracks three competition-level aggregates: Valid Submission Rate, Above Median Rate, and Any Medal Rate (Nie et al., 22 Jan 2026). Valid Submission Rate is the fraction of tasks where the agent produced a syntactically correct submission.csv; Above Median Rate is the fraction of tasks where the agent’s score exceeds the human-median leaderboard score; and Any Medal Rate is the fraction of tasks earning at least a Bronze medal per Kaggle rules.

This evaluation design matters because it distinguishes execution reliability from competitiveness. An agent can achieve a high Valid Submission Rate while still performing poorly on Above Median Rate and Any Medal Rate, thereby exposing the gap between producing runnable code and producing strong predictive solutions.

5. Agent scaffolding and training pipeline

Although DSPredict is described primarily as an evaluation suite, DSGym states that its infrastructure also allows agent finetuning via execution-verified data synthesis (Nie et al., 22 Jan 2026). The default agent scaffold named “CodeAct” generates interleaved <reasoning> and <python> blocks, with reasoning used for step-by-step planning and code executed in an isolated Docker worker.

The reported modeling repertoire includes several standard library-based approaches. Tree-based models include lightgbm.LGBMRegressor(loss='mse', n_estimators=100, max_depth=6, learning_rate=0.1, early_stopping_rounds=10). Linear and logistic models include sklearn.linear_model.LinearRegression() and LogisticRegression(). For computer vision tasks, the benchmark description gives a small CNN example in PyTorch or TensorFlow with convolution, ReLU, max pooling, flattening, and a final linear layer (Nie et al., 22 Jan 2026). Optimization examples include regression trees minimizing MSE via gradient boosting and neural networks minimizing cross-entropy

$D_{\mathrm{test}}$9

Agents are also described as often setting hyperparameters directly in code, including learning_rate=0.01, batch_size=64, and epochs=10.

The execution-verified synthesis pipeline, adapted from DSGym’s Section 6, is summarized as self-play to generate new Kaggle-style queries, sampling diverse solution trajectories, and filtering with an LLM-Judge (Nie et al., 22 Jan 2026). The paper notes that although this synthesis pipeline is demonstrated on analysis tasks, the same infrastructure can bootstrap upstream DSPredict training data by soliciting new synthetic Kaggle-like challenges. This suggests that DSPredict has a dual role: benchmark and potential source domain for agent training.

6. Reported benchmark performance

The benchmark results reported for DSPredict show a substantial disparity between valid execution and strong leaderboard performance (Nie et al., 22 Jan 2026). For DSPredict-Hard, with $m$0, private test results are reported for four agent configurations.

Agent	Valid	Above Median	Any Medal
GPT-5.1 (high reasoning)	85.7%	14.3%	4.8%
GPT-5.1 (medium)	81.0%	7.1%	4.8%
Claude Sonnet 4.5	71.4%	4.8%	0%
Qwen3-Coder 480B	66.7%	5.9%	2.4%

For DSPredict-Easy, with $m$1, results are given as percentile rank: GPT-5.1 (high) at the 60.4th percentile, with GD at the 75th percentile; GPT-5.1 (medium) at the 55.7th percentile; Claude Sonnet 4.5 at the 49th percentile; and Qwen3-Coder 480B at the 42.9th percentile (Nie et al., 22 Jan 2026).

The accompanying takeaways are stated directly. Agents “reliably generate valid submissions” at greater than 65% on Hard and approximately 100% on Easy, but “genuine competitive performance is rare,” with median Above Median Rate below 15% and medal rates near 0% (Nie et al., 22 Jan 2026). GPT-5.1 with “hard” reasoning effort is reported as achieving the best leaderboard standing. These findings position DSPredict as a benchmark in which pipeline completion is no longer the only bottleneck; robust feature engineering, model iteration, and task-specific adaptation remain limiting factors.

7. Limitations, failure modes, and open questions

The benchmark description identifies several main bottlenecks in DSPredict (Nie et al., 22 Jan 2026). Environment access restrictions cause failures when agents attempt to import heavy libraries such as cv2 or PyTorch and encounter ModuleNotFound errors or timeouts. API incompatibilities arise when agents hallucinate deprecated arguments, with early_stopping_rounds in LightGBM given as an example. Simplicity bias is described as a tendency to fall back to trivial heuristics such as global medians or zero-fill baselines when agents encounter friction. Time and resource constraints are especially salient for datasets larger than 10 GB, which can trigger memory limits or timeouts and lead agents to downscale or abandon more complex architectures. Low medal rates persist even when submissions are valid, indicating weak feature engineering and ensembling in comparison with human competition baselines.

The paper frames several open questions around these limitations, including how to better guide agents through multi-stage pipeline design from data profiling to hyperparameter tuning, and how to integrate domain-specific tools such as RDKit for chemistry and Scanpy for single-cell analysis under DSGym’s scaffolding (Nie et al., 22 Jan 2026). A plausible implication is that DSPredict functions not only as a benchmark of current capability but also as a testbed for interface design, tool integration, and curriculum construction in autonomous data science.

Taken together, DSPredict defines a realistic benchmark for end-to-end predictive modeling by embedding agents in execution-grounded competition tasks across heterogeneous scientific domains. Its reported results show that syntactic validity and basic workflow completion are increasingly attainable, whereas consistently competitive predictive performance remains difficult (Nie et al., 22 Jan 2026).