AI Scientist via Synthetic Task Scaling

Abstract: With the advent of AI agents, automatic scientific discovery has become a tenable goal. Many recent works scaffold agentic systems that can perform machine learning research, but don't offer a principled way to train such agents -- and current LLMs often generate plausible-looking but ineffective ideas. To make progress on training agents that can learn from doing, we provide a novel synthetic environment generation pipeline targeting machine learning agents. Our pipeline automatically synthesizes machine learning challenges compatible with the SWE-agent framework, covering topic sampling, dataset proposal, and code generation. The resulting synthetic tasks are 1) grounded in real machine learning datasets, because the proposed datasets are verified against the Huggingface API and are 2) verified for higher quality with a self-debugging loop. To validate the effectiveness of our synthetic tasks, we tackle MLGym, a benchmark for machine learning tasks. From the synthetic tasks, we sample trajectories from a teacher model (GPT-5), then use the trajectories to train a student model (Qwen3-4B and Qwen3-8B). The student models trained with our synthetic tasks achieve improved performance on MLGym, raising the AUP metric by 9% for Qwen3-4B and 12% for Qwen3-8B.

• The paper introduces a fully automatic pipeline that generates and verifies synthetic ML research tasks at scale.
• It employs LLMs for task proposal, code synthesis, and self-debugging to produce high-quality agent trajectories.
• Results on MLGym benchmarks show up to 12% gain in aggregate AUP, demonstrating enhanced autonomous research capabilities.

AI Scientist via Synthetic Task Scaling: Toward Autonomous ML Research Agents

Overview

"AI Scientist via Synthetic Task Scaling" (2603.17216) presents a paradigm for training ML research agents using large-scale, fully automatic synthetic ML task generation and experiential learning via task-trajectory sampling. Unlike prior agentic frameworks that focus on real tasks or curated corpora, this work proposes and implements a systematic pipeline for creating diverse, grounded ML challenges at scale, and demonstrates resultant gains in generalization and benchmark performance. Central contributions include an unsupervised yet validated environment synthesis pipeline, agent trajectory collection via state-of-the-art LLMs, and consequential improvements to agent learning, as evidenced on the MLGym benchmark.

Figure 1: Schematic of agentic ML task execution in the SWE-agent framework, supporting multi-turn research iteration and agent-environment interaction.

Synthetic ML Task Generation Pipeline

The core methodology centers on scalable, robust generation of ML research environments compatible with agentic frameworks. This multistage pipeline operates as follows:

1. Topic Sampling: A large LM is prompted to enumerate diverse, granular ML topics across subfields.
2. Task and Dataset Proposal: For each topic, the model produces a novel research task and queries the HuggingFace API for compatible, real datasets, ensuring data-grounding.
3. Configuration and Code Synthesis: From the validated dataset and task description, the LM generates config files and minimal working baseline code, conforming to SWE-agent/MLGym execution interfaces.

To guarantee that generated tasks are executable, the system implements a self-debugging verification loop. Upon encountering code/compilation errors, the pipeline iterates: it returns the stack trace to the LM, which revises the code/config, with up to $k$ retries per task. Failed cases are pruned, resulting in a large batch of verified, executable ML research tasks.

Figure 2: End-to-end automated task and trajectory generation; tasks are sampled, validated (including automated debugging), and executed without human intervention.

Large-Scale Agent Trajectory Collection

For each verified environment, agentic trajectories (sequences of state-action pairs spanning multiple reasoning and code-editing rounds) are synthesized using a powerful teacher LM (GPT-5). Each episode involves iterative calls, emulating research workflows of hypothesis, code modification, execution, and evaluation. Given computational constraints, the pipeline executes at scale on distributed clusters, targeting hundreds of trajectories per task.

The resulting corpus is then post-processed. Only tasks with at least one successful run are retained, and trajectories exceeding practical token limits are truncated or discarded, yielding a high-quality training set consisting of $\sim$500 grounded ML tasks and $\sim$34,000 agent-environment interaction trajectories.

Figure 3: Distribution of successful trajectory counts for a sampled subset of generated tasks, highlighting heterogeneity in automatic environment complexity and agent solution rates.

Additional analysis (token lengths, trajectory properties, success filtering) confirms the diversity and controlled quality of the dataset.

Figure 4: Summary statistics of training trajectories: token-length distribution, prevalence of truncation, and number of agent-environment interaction turns.

Supervised Fine-Tuning and Benchmark Evaluation

The central empirical validation leverages MLGym [nathani2025mlgymnewframeworkbenchmark], a rigorous ML agent benchmark comprised of 13 distinct machine learning challenges. Two open LLMs (Qwen3-4B and Qwen3-8B) are fine-tuned via SFT on the synthetic trajectory corpus. Performance is then compared against strong baseline LMs (GPT-4o, GPT-5, raw Qwen3-4B/8B) using the standard MLGym protocol.

The evaluation is meticulous: each model-agent iteratively improves provided ML baselines across subtasks, with scores aggregated using the AUP metric to align task heterogeneity. Notably, the SFT-finetuned models outperform their unaugmented counterparts on a majority of subtasks (9/13 for Qwen3-4B), with aggregate AUP improvements of 9% (Qwen3-4B) and 12% (Qwen3-8B). Results suggest that large-scale, diverse, synthetic agentic experience directly benefits agent capabilities for automated research, not just narrow code synthesis.

Figure 5: Model performance (64 runs) per MLGym subtask: SFT-trained Qwen models show improved score distributions versus baseline LMs.

Figure 6: Aggregate model performance (AUP); SFT-trained Qwen3-4B/8B improve substantially.

Implications and Future Directions

This work establishes synthetic task scaling as a practical paradigm for bootstrapping agentic scientific discovery in ML. The results demonstrate that:

• Scalable, autonomous environment synthesis can cover orders of magnitude more research variants than human labeling or static dataset curation.
• Agentic, multi-step learning from rich, trajectory-level experience is critical; training on end artifacts alone (final code/paper) fails to capture the necessary iterative reasoning and error-correction behaviors needed for research.
• Transfer to rigorous benchmarks: Gains observed on a challenging ML research agent benchmark indicate substantive skill improvements, though total disentanglement from benchmark-format familiarity remains unresolved.

However, several limitations are acknowledged:

• Generalization beyond MLGym is not fully established; future work should evaluate zero-shot performance on conceptually different benchmarks (e.g., MLE-Bench, MLRC-Bench), and ablate the relative contributions of each pipeline component (dataset grounding, self-debugging, trajectory filtering).
• Teacher bias and coverage: By construction, student agents can only learn what teacher models (GPT-5) can handle. Areas of teacher deficiency or hallucination propagate to student limitations.
• RL and Task Discovery: Current SFT protocols do not incentivize exploration or reward true novelty. The trajectory corpus can be used for RL, but challenges remain due to long credit assignment and heterogeneous reward scales.

The discussion also notes partial ceiling effects for complex tasks with large or intricate starter repositories (e.g., MS-COCO), where synthetic environments may not encapsulate all necessary structure for robust agent improvement.

Conclusion

"AI Scientist via Synthetic Task Scaling" systematizes and validates a pipeline for large-scale, unsupervised ML research agent training using synthetic, grounded tasks and agentic trajectories (2603.17216). The approach robustly improves agent performance on a benchmark suite, substantiating the claim that appropriately-generated, verifiable synthetic experience can augment agentic reasoning and generalization.

The framework is extensible to other formats, compatible with RL, and points toward an era where ML research agents can be trained through massive volumes of simulated yet executable research experience. Open questions remain in transfer, teacher-student gap minimization, and reward-driven discovery, making this a blueprint for further progress toward fully autonomous AI scientists.