---
title: 'DataSciBench: Data Science Evaluation Benchmarks'
url: https://www.emergentmind.com/topics/datascibench
type: topic
---

# DataSciBench: Data Science Evaluation Benchmarks

DataSciBench refers to a suite of methodology-driven, rigorously engineered benchmarks that evaluate the competency of algorithms, large language models, and supercomputing platforms in data science workflows. The term encompasses several separate but related initiatives, each addressing limitations in prior evaluation frameworks and advancing systematic measurement for code generation, data analysis agents, scientific benchmarking, and HPC-oriented analytics.

## 1. Origins and Scope of DataSciBench

Early benchmarking in machine learning and scientific computing focused on isolated, well-defined tasks or synthetic workloads, exemplified by datasets such as ImageNet or microbenchmarks in high-performance computing (HPC). However, the rapid proliferation of LLM-powered data science assistants and new computational paradigms highlighted several deficiencies: restricted task coverage, narrow test cases, unrealistic or non-representative code snippets, and metrics insensitive to multi-step, real-world workflows [2502.13897][2505.15621][1811.02287].

DataSciBench as a term has thus evolved to cover:

- Multi-stage, contextually rich data-science agent evaluation tasks [2502.13897].
- Real-world, moderately sized, and thoroughly validated data-science code generation problems [2505.15621].
- Representative analytics kernels and performance metrics for next-generation supercomputer procurement [1811.02287].

Each DataSciBench instantiation is characterized by its emphasis on realism, breadth across data science subdomains, and reproducible, fine-grained evaluation metrics.

## 2. Benchmark Structure and Construction Methodologies

### Problem and Prompt Design

DataSciBench benchmarks aggregate prompts and tasks that span canonical data-science functions: data cleaning and preprocessing, exploratory statistics, data visualization, predictive modeling, pattern/cluster mining, and interpretability reporting. Examples include:

- A curated corpus of 222 multi-stage prompts combining two or more data-science sub-tasks, sourced from real-world LLM platforms (CodeGeeX), refined open benchmarks (e.g., BigCodeBench), manual expert-crafted scenarios, and high-quality LLM-generated samples [2502.13897].
- 1,000 code-generation tasks extracted from active, high-star GitHub repositories, enforcing ≥3 distinct API calls per problem, with the average solution spanning 22.5 lines of code (Table 1) [2505.15621].

### Ground Truth and Validation

Many data-science tasks lack easily obtainable or oracle-style ground-truth. To address this:

- A semi-automated pipeline leverages strong LLMs (e.g., GPT-4o) for self-consistency: multiple solutions are synthesized and executed, and the modal or majority output is adopted, with borderline or ambiguous outcomes resolved by expert human annotation [2502.13897].
- Rigorous, automated test suites accompany code generation tasks; for instance, each DS-bench item ships with a generative Python test script producing 200 randomized inputs, further reinforced by self-repair loops until mean code coverage exceeds 97.8% [2505.15621].

### Multi-Modality and Data Representations

Benchmarks cover tabular, graph, image, sequential/time series, signal, and textual modalities. Distributional coverage and edge-case challenge are enhanced through approaches such as convex-hull boundary identification (for scientific data) and multi-label task tagging (for agent benchmarks) [2506.23419][2502.13897][2507.23336].

## 3. Evaluation Metrics and Frameworks

### Task–Function–Code (TFC) Framework

The TFC schema formalizes the benchmark into tuples $(T_i, F_i, C_i)$, where each $T_i$ is a data-science subtask, $F_i$ a programmatically coded metric (e.g., data cleaning completeness, silhouette score), and $C_i$ the Python code evaluating $F_i$ against candidate outputs [2502.13897]. This supports stepwise assessment, including both binary and continuous outcomes.

### Aggregate Scores

Key aggregate metrics include:

- Completion Rate (CR): Proportional credit across sub-steps, $$ CR = (\sum_{t=1}^T s_t)/(2T) $$ where $s_t$ is a stepwise graded pass metric.
- Success Rate (SR): Fraction of runs where all steps succeed in a single pass over multiple trials.
- Weighted overall scores: $$ \text{Score} = 0.65 \cdot CR + 0.05 \cdot SR + 0.05 \cdot S_{VLM} + 0.05 \cdot \sum_{i=1}^5 F_i $$ where $S_{VLM}$ is a VLM-judged score for visualization quality [2502.13897].
- For code generation, pass@$k$ metrics quantify the rate at which $k$ independent generations pass all tests:
  $$
  \operatorname{pass}@k = 1 - \prod_{i=1}^k \frac{n-c}{n-i+1}
  $$
  where $n$ is the number of samples, $c$ the number of passes [2505.15621].

### Fine-Grained and Domain-Specific Criteria

- Binary success/failure as judged by VLM-as-judge protocols for real-world queries.
- Distributional divergence (e.g., KS p-value, KL/JS divergence, Wasserstein, MMD) to quantify challenge in train/test splits for novel scientific datasets [2506.23419].
- Latency, throughput, scaling efficiency (strong and weak), and figure-of-merit (FOM, in TB/s) for system-level HPC benchmarks [1811.02287].

## 4. Experimental Insights and Model Performance

In comprehensive experiments across open-source and API-restricted LLMs:

- API-based LLMs, notably GPT-4o, attain the highest aggregate scores (e.g., 66.31% SR, 68.44% CR), with strong code-specialist open-source models (Deepseek-Coder-33B-Instruct: 61.23% CR) narrowing the gap for code-centric tasks [2502.13897].
- Despite high HumanEval benchmarks, models exhibit major failure modes on DataSciBench: incomplete data cleaning, library misuse, I/O noncompliance, or missing explicit output files.
- In DS-bench code generation, leading models achieve pass@1 rates of only 20.2% (GPT-4o), with performance dropping sharply for data visualization and API-heavy tasks (e.g., Matplotlib: 7.3%) [2505.15621].
- Analysis on DSBC shows multi-cell (ReAct-based) prompting improves agent accuracy by ≈10% over SmolAgent baselines, with temperature introducing up to 4% variance in complex task setups [2507.23336].
- For HPC analytics, DataSciBench's minimal three-kernel suite reveals that workload mix and problem size, rather than BLAS flavor or R version, dominate system throughput. Systems scaled to hundreds of nodes show nearly ideal performance for memory-bound workloads (PCA, SVM), while communication-bound tasks (k-means) begin to reveal bottlenecks beyond 16–128 nodes [1811.02287].

## 5. Practical Guidelines and Benchmark Use

Deployment of DataSciBench-enabled evaluation involves:

- Structured context engineering: dataset descriptions (row/col counts, schema, summaries) are encoded as JSON to fit within token and privacy constraints [2507.23336].
- Standardized prompt templates are used for both zero-shot and multi-step code generation, fostering reproducibility and facilitating agent routing between simple and complex queries.
- Code, datasets, and evaluation scripts are systematically released under open-source licenses, notably at https://github.com/THUDM/DataSciBench [2502.13897], https://github.com/ShuyinOuyang/DS_bench [2505.15621], and https://github.com/traversaal/DSBC [2507.23336].
- For scientific data partitioning, deterministic edge-case selection ensures test set challenge and prevents information leakage; users prepare nonnegative data, use provided APIs, and re-run splits as the dataset evolves [2506.23419].
- HPC vendors and researchers employ kernel-centric suites and reference code (MPI + BLAS) to audit and optimize machine-level analytic throughput [1811.02287].

## 6. Limitations and Future Directions

Current DataSciBench benchmarks exhibit several boundaries:

- Predominance of English-language and tabular or code-based tasks; multimodal and multilingual extensions are proposed [2507.23336].
- Challenges remain in chart and visualization assessment, with VLM judges imprecise compared to domain-specific critics [2502.13897].
- Code evaluation focuses on correctness and code coverage; expansion to efficiency, robustness to bad inputs, and code quality metrics is a stated agenda [2505.15621].
- Data stream, time series, and interactive/real-time tasks are only sparsely represented.
- Agent routing schemes, which optimize inference cost by predicting task complexity, are suggested as routes for adaptive, cost-efficient deployment [2507.23336].

*This suggests* DataSciBench will continue to evolve, integrating advances in natural language understanding, code synthesis, evaluation science, and computational agent design, to sustain rigorous, transparent progress in real-world data-science automation.

Source: https://www.emergentmind.com/topics/datascibench