---
title: 'AutoSDT-5K: Dataset for Scientific Coding Tasks'
url: https://www.emergentmind.com/topics/autosdt-5k
type: topic
---

# AutoSDT-5K: Dataset for Scientific Coding Tasks

AutoSDT-5K is a large-scale, openly available dataset of real-world scientific coding tasks, constructed to support research in data-driven scientific discovery and the development of AI co-scientists. Automatically assembled by the AutoSDT pipeline, AutoSDT-5K provides comprehensive disciplinary coverage, high ecological validity, and is explicitly designed for both training and evaluating large language models (LLMs) in diverse scientific programming workflows.

## 1. Dataset Composition and Structure

AutoSDT-5K comprises 5,404 unique coding tasks, each represented as a paired tuple—task instruction and code solution—drawn from authentic scientific research codebases. The dataset is distinguished by:

- **Coverage of Scientific Disciplines**: Four core domains are included: bioinformatics (1,466 tasks), computational chemistry (1,345), geographical information science (1,541), and psychology/cognitive neuroscience (1,052).
- **Workflow Characteristics**: Each task reflects a real scientific workflow, averaging 4.3 subtasks and spanning common stages such as data preprocessing, visualization, statistical analysis, and advanced domain-specific analytics.
- **Python Package Diversity**: 756 unique packages are represented, including both general-purpose libraries like `sklearn` and `scipy`, and specialist toolkits such as `ase` (atomic simulations), `nibabel` (neuroimaging), and `geopandas` (geospatial).
- **Task Difficulty**: Distribution is 22.3% easy (avg. 214.7 code lines), 48.4% medium (263.7 lines), and 29.3% hard (403.2 lines), reflecting the complexity encountered in daily scientific research coding.
- **Source Repositories**: Tasks are drawn from 1,325 distinct research-related repositories (out of 2,993 screened), with explicit checks for research relevance and the presence of scholarly or domain-driven workflows.

AutoSDT-5K is, to date, the largest and most diversified open dataset targeting data-driven coding tasks for scientific discovery, in contrast to preceding datasets which are smaller, evaluation-only, or lack real workflow granularity.

## 2. Pipeline Construction and the Role of Large Language Models

The fully automatic AutoSDT pipeline consists of three principal stages, each leveraging the reasoning and coding abilities of state-of-the-art LLMs:

1. **AutoSDT-Search**: Utilizes discipline-specific keywords, expanded by GPT-4o, to exhaustively search GitHub and PapersWithCode. LLMs inspect repository documentation to select only those with clear research aims and valid scientific workflows.
2. **AutoSDT-Select**: After repository cloning, all Python scripts are filtered, with LLM-powered heuristics confirming workflow relevance, dataset usage, and meaningful scientific outputs. LLMs parse codebases to extract a compact, dependency-complete code workspace, reducing average storage from ~265 MB to ~40 MB per workspace.
3. **AutoSDT-Adapt**: Code is automatically refactored by Claude-3.7-Sonnet for standalone execution, minimally adapting I/O and module paths as needed and ensuring functional equivalence through up to three rounds of LLM-based self-debugging. Task instructions are synthesized to be domain-specific, clear, and mimicking realistic scientific directions, again via LLM prompting and summarization. Each resulting (instruction, code) pair then forms a datum in AutoSDT-5K.

Cost efficiency is a hallmark: the dataset was assembled for ~$0.55 per task, compared to traditional annotation costs exceeding $20 per task for similar scientific coding problems.

## 3. Validation, Expert Feedback, and Dataset Quality

To ensure dataset fidelity and practical usability, AutoSDT-5K underwent expert evaluation on a stratified random sample of 256 tasks by nine subject-matter experts across all four target domains:

- **Ecological Validity**: 93% of instructions are considered “scientifically authentic” for the given domain.
- **Instruction Clarity**: 73.4% are unambiguously stated, supplying clear scientific context, goals, inputs, and outputs; remaining deficiencies are attributable largely to poor documentation in original source code.
- **Functional Correctness**: 92.2% of code solutions can be executed to successfully fulfill the given instruction, with 84.4% demonstrating full functional equivalence to the original code’s outputs.
- **Task Realism**: Complexity and composition mirror actual research workflows, including multi-step analyses and domain-specialized methods.

These results position AutoSDT-5K as a premier resource for both robust model training and rigorous evaluation.

## 4. Benchmarking and Model Performance Improvements

AutoSDT-5K was used to train the AutoSDT-Coder model suite (Qwen2.5-Coder-Instruct LLMs fine-tuned on AutoSDT-5K) and benchmarked on two challenging scientific evaluation sets:

- **ScienceAgentBench**: Assesses correctness and execution of scientific programs.
- **DiscoveryBench**: Evaluates data-driven hypothesis generation and semantic matching of outputs.

Key results include:

| Model                       | ScienceAgentBench SR (%) | DiscoveryBench HMS |
|-----------------------------|-------------------------|--------------------|
| Qwen2.5-Coder-32B (base)    | 3.9                     | 6.9                |
| AutoSDT-Coder-32B (finetuned)| 7.8                    | 8.1                |
| GPT-4o (May 2024)           | 7.5                     | —                  |

This corresponds to a 100% relative improvement in scientific coding success rate, and a 17.4% relative gain in hypothesis matching, matching or exceeding non-reasoning GPT-4o on ScienceAgentBench and closing the performance gap with proprietary models for hypothesis generation. Performance gains scale with model capacity and data volume.

## 5. Comparative Analysis Against Other Models

AutoSDT-Coder-32B, fine-tuned on AutoSDT-5K, achieves higher scientific coding accuracy compared to baseline open-weight models such as Llama-3.1-Instruct-405B and Qwen2.5-Coder-32B:

- Surpasses Qwen2.5-Coder-32B baseline by over 2x on ScienceAgentBench (7.8% vs. 3.9% SR).
- Performance approaches that of GPT-4o (7.5–11.4% SR), though proprietary models with proprietary reasoning data still hold an edge (e.g., OpenAI o1-preview >20% SR).
- AutoSDT-Coder-32B provides best performance among open-weight models on both general program correctness and domain-specific hypothesis discovery, illustrating the impact of large, ecologically valid training sets.

## 6. Implications and Future Directions

AutoSDT-5K represents a substantive advance toward democratizing AI-driven scientific discovery by enabling rigorous training and evaluation of LLM-based “AI co-scientists” without reliance on closed, proprietary datasets. Its distinguishing features—scale, disciplinary diversity, workflow realism, and high validation—establish it as a keystone for:

- **Enabling open research**: Supports the transparent development and reproducible evaluation of scientific LLMs, reducing barriers to entry for under-resourced research communities.
- **Catalyzing further dataset development**: The AutoSDT pipeline generalizes to additional disciplines and languages by modifying seed keywords, suggesting broad extensibility.
- **Driving model advances**: The dataset supports scaling to larger models and the incorporation of more sophisticated reasoning (e.g., chain-of-thought, automatic evaluator generation).
- **Domain impact**: Improved LLM performance on core scientific workflows has implications for automating data analysis, hypothesis generation, and collaborative research across STEM and social sciences.

## 7. Summary Table of Key Statistics

| Property                   | Value / Description                                              |
|----------------------------|-----------------------------------------------------------------|
| Total coding tasks         | 5,404                                                           |
| Disciplines                | 4 (bioinformatics, chemistry, geoscience, neuroscience/psych)   |
| Unique Python packages     | 756                                                             |
| Source repositories        | 1,325 (of 2,993 screened)                                       |
| Average subtasks per task  | 4.3                                                             |
| Task difficulty spread     | 22.3% easy, 48.4% medium, 29.3% hard                            |
| Expert-validated accuracy  | 93% ecological validity; 92.2% code correctness                 |
| Relative improvement (SR)  | 100% (3.9%→7.8%, ScienceAgentBench, Qwen2.5-32B)                |
| Hypothesis matching gain   | +17.4% (6.9→8.1, DiscoveryBench, Qwen2.5-32B)                   |

AutoSDT-5K thus establishes a new standard for large-scale, automatic, open-data resources supporting the next generation of data-driven scientific AI agents.

Source: https://www.emergentmind.com/topics/autosdt-5k