D3-Gym: Constructing Real-World Verifiable Environments for Data-Driven Discovery

Abstract: Despite recent progress in LLMs and agents for scientific data-driven discovery, further advancing their capabilities is held back by the absence of verifiable environments representing real-world scientific tasks.To fill this gap, we introduce D3-Gym, the first automatically constructed dataset with verifiable environments for scientific Data-Driven Discovery. D3-Gym comprises (1) 565 tasks sourced from 239 real scientific repositories across four disciplines where (2) each task is equipped with a natural language instruction, an executable environment with pre-installed dependencies, input dataset and artifact previews, a reference code solution, and an automatically synthesized evaluation script. Rigorous evaluation of the quality of the verification signal in D3-Gym confirms that our evaluation scripts achieve 87.5% agreement with human-annotated gold standards and strong alignment in domain-specific evaluation logic, showing their scientific soundness. Further, training on trajectories sampled from D3-Gym yields consistent and substantial gains across Qwen3 models of varying sizes on ScienceAgentBench, boosting Qwen3-32B by 7.8 absolute points and substantially shrinking the gap with strong proprietary models. All D3-Gym artifacts (environments, creation workflow, trajectories, and models) can be found at https://github.com/OSU-NLP-Group/D3-Gym.

• The paper introduces D3-Gym, a large-scale dataset of 565 real-world scientific tasks with automatic, verifiable evaluation for data-driven discovery.
• It presents a four-stage automated pipeline that extracts, previews, validates, and synthesizes evaluation scripts for authentic scientific coding workflows.
• Empirical results demonstrate that fine-tuning on D3-Gym boosts model performance by up to +7.8 SR@3, outperforming static datasets and approaching proprietary benchmarks.

D3-Gym: An Automated Dataset for Real-World Verifiable Environments in Data-Driven Scientific Discovery

Motivation and Contributions

The development of robust language agents for scientific data-driven discovery is significantly constrained by the scarcity of open, realistic, executable environments with domain-grounded verification signals. While advances in LLMs have enabled flexible agents across various scientific domains, progress is limited by the lack of benchmarks that accurately capture scientific workflows, require handling of specialized data modalities, and provide task-specific correctness checks. The "D3-Gym: Constructing Real-World Verifiable Environments for Data-Driven Discovery" (2604.27977) paper introduces D3-Gym, the first large-scale, automatically constructed dataset of verifiable scientific environments tailored for this setting.

D3-Gym comprises 565 task environments extracted from 239 real-world scientific repositories across four domains: bioinformatics, computational chemistry, geographic information science, and cognitive neuroscience. Each task features a natural language instruction, input data with schema previews, a reference code solution, and an automatically synthesized evaluation script producing a domain-grounded pass/fail signal. Notably, these environments are fully executable, support complex dependencies, and utilize rigorous LLM-backed verification for output validity.

The paper presents:

• A scalable, automated pipeline for extracting, filtering, and validating real scientific coding tasks.
• A planning+coding paradigm for LLM-powered synthesis of high-quality task-specific evaluation scripts, enabling executable correctness signals absent in prior work.
• Thorough empirical validation of verification signals against human-annotated gold standards.
• Demonstration of the efficacy of D3-Gym as a training environment: rejection sampling fine-tuning (RFT) on D3-Gym substantially boosts open-weight model performance on ScienceAgentBench, with gains scaling with model size and closing the gap to proprietary models.

Dataset Construction Pipeline

The creation of D3-Gym is accomplished via a four-stage workflow:

1. Candidate Extraction & Filtering: Tasks are sourced by crawling real scientific repositories (using AutoSDT) to extract relevant data-processing scripts, which are filtered to retain only those using authentic data and dependencies. Data integrity is enforced by verifying file existence and content.
2. Dataset Preview and Execution: Surviving candidates undergo dataset preview generation for schema context, followed by isolated execution to validate code and produce reference outputs.
3. Rigorous Output Validation: Outputs are judged by a multimodal LLM-as-judge to ensure results are substantive and non-degenerate. Only tasks passing these quality gates proceed to evaluation script generation.
4. Evaluation Script Synthesis: Key innovation arises here: evaluation scripts are synthesized in a two-stage LLM-driven fashion—first, planning out evaluation logic (metric selection, artifact targets, acceptance criteria) with access to instruction, outputs, and data schema; then, translating the plan into an executable Python script capable of robustly distinguishing correct from incorrect solutions.

This design addresses the lack of pre-existing evaluation scripts in scientific repositories and the intrinsic domain specificity of outputs, both of which preclude straightforward adaptation of unit-testing-based methodologies prevalent in software engineering environment builders.

Evaluation of Verification Signals

To rigorously assess the quality of the verification signals:

• Agreement with Human Gold Standards: On a set of 50 held-out tasks with human-expert gold evaluation scripts, D3-Gym’s silver (LLM-generated) scripts achieve 87.5% pass/fail agreement, high specificity ($91.0\%$), and close alignment in evaluation logic (Likert-scale scores $>4.0$ on metric choice, threshold, and artifacts).
• Ablations confirm that the planning phase and access to dataset schema/previews significantly enhance reliability; removing these components leads to drastic drops in recall and overall alignment.
• Strictness Analysis: Silver scripts tend to adopt marginally stricter criteria than their gold counterparts, occasionally rejecting borderline-correct outputs. However, accepted outputs are reliably correct (high specificity), reinforcing the scientific utility of the verification.

Dataset Characteristics and Task Diversity

D3-Gym encompasses:

• Task and Modality Diversity: Tasks span statistical analysis, ML, visualization, signal/image processing, and computational simulation, with input/output modalities covering tabular/text data, molecular/structural data (e.g., SMILES, FASTA), geospatial logs, images, and time series.
• Package Coverage: Enforcement of execution in a controlled environment captures complex dependency resolution scenarios; domain-specific packages (e.g., rdkit, ase, pysam, geopandas) are strongly represented, raising the difficulty level for both code synthesis and interpretability.
• Non-Triviality: On a random sample, top-tier LLMs such as GPT-5.4 and Claude Opus 4.6 solve only $\sim$36–39% of tasks, and execution rates are consistently below 85%. This demonstrates substantial headroom for agentic and model improvements.

Training and Empirical Results

Leveraging D3-Gym environments, rejection sampling fine-tuning (RFT) is applied to a suite of open-weight Qwen3 models across four sizes (4B, 8B, 14B, 32B), using both distillation from a stronger teacher and self-improvement (on-policy).

Results on ScienceAgentBench and ScienceAgentBench-Verified (average of three trials):

• Qwen3-32B fine-tuned with RFT-Self on D3-Gym shows a +7.8 absolute point SR@3 increase, surpassing larger mixture-of-experts counterparts (80B, 235B) and approaching proprietary o1-preview and Claude Sonnet 4.5 models.
• Self-generated trajectories for fine-tuning are empirically better aligned with a model’s distribution than teacher-generated data, especially in larger models, supporting their use for scalable agentic improvement.
• The model performance improves across almost all task categories, with error reduction most pronounced in data schema, datatype/shape, and visualization, though logical and algorithmic errors become the dominant remaining failure mode.
• D3-Gym’s efficacy as fine-tuning data is empirically superior to static SFT datasets (e.g., AutoSDT-5K), both in sample efficiency and overall model improvement, primarily due to the executable, verifiable training environments.

Implications and Future Directions

D3-Gym establishes a new paradigm for constructing agentic scientific coding environments: executable, verifiable, sourced directly from authentic scientific scripts in the wild, and accompanied by domain-grounded, automatically synthesized evaluation. Immediate implications include:

• Advancement of Open-Weight Language Agents: D3-Gym narrows the performance gap between open-weight and proprietary LLMs for scientific workflows. Explicit execution and verifiability support scaling beyond vanilla SFT into trajectory-based learning, rejection sampling, and future RLHF-style approaches.
• Benchmarking and Generalizability: By providing tasks across diverse scientific domains and modalities, D3-Gym offers an ecologically valid benchmark for LLM-based scientific agents, with the potential for extension as more repositories and modalities are incorporated.
• Test Case and Evaluation Logic Co-synthesis: The two-stage planning+coding synthesis for evaluation scripts contextually bridges general LLM knowledge with domain-specific correctness criteria—an advance over prior work in both scientific and software engineering environment construction.

Potential future developments include scaling D3-Gym to match the diversity and size of SW engineering environment datasets, enhancing LLM understanding of highly specialized scientific packages, integrating reinforcement learning protocols over executably verifiable environments, and exploring automated curriculum generation leveraging task difficulty metrics.

Conclusion

D3-Gym (2604.27977) introduces a rigorously filtered, automatically constructed suite of real-world scientific coding environments with full executable and verifiable support. Its task-specific, LLM-synthesized evaluation signals are scientifically robust. Fine-tuning open-weight models in this setting yields strong, consistent performance gains, particularly through self-generated verified trajectory distillation. D3-Gym lays a robust foundation for future advances in LLM-driven scientific discovery and will facilitate domain-credible assessment and development of next-generation scientific agents.