Towards Execution-Grounded Automated AI Research

Abstract: Automated AI research holds great potential to accelerate scientific discovery. However, current LLMs often generate plausible-looking but ineffective ideas. Execution grounding may help, but it is unclear whether automated execution is feasible and whether LLMs can learn from the execution feedback. To investigate these, we first build an automated executor to implement ideas and launch large-scale parallel GPU experiments to verify their effectiveness. We then convert two realistic research problems - LLM pre-training and post-training - into execution environments and demonstrate that our automated executor can implement a large fraction of the ideas sampled from frontier LLMs. We analyze two methods to learn from the execution feedback: evolutionary search and reinforcement learning. Execution-guided evolutionary search is sample-efficient: it finds a method that significantly outperforms the GRPO baseline (69.4% vs 48.0%) on post-training, and finds a pre-training recipe that outperforms the nanoGPT baseline (19.7 minutes vs 35.9 minutes) on pre-training, all within just ten search epochs. Frontier LLMs often generate meaningful algorithmic ideas during search, but they tend to saturate early and only occasionally exhibit scaling trends. Reinforcement learning from execution reward, on the other hand, suffers from mode collapse. It successfully improves the average reward of the ideator model but not the upper-bound, due to models converging on simple ideas. We thoroughly analyze the executed ideas and training dynamics to facilitate future efforts towards execution-grounded automated AI research.

• The paper introduces an automated framework that integrates ideation, code generation, and execution to evaluate AI research ideas.
• It employs modular components—Implementer, Scheduler, and Worker—to translate ideas into executable code, achieving significant performance gains over baselines.
• Execution-guided evolutionary search outperforms reinforcement learning by maintaining idea diversity and sample efficiency, though RL faces mode collapse challenges.

Execution-Grounded Automated AI Research: Systematic Study and Benchmarks

System Architecture and Problem Formulation

"Towards Execution-Grounded Automated AI Research" (2601.14525) introduces a large-scale, automated framework to study whether LLMs can autonomously generate, implement, and validate novel research ideas via direct grounding in execution. The system decomposes the automation pipeline into three principal modules: Implementer, Scheduler, and Worker, yielding an integrated executor capable of handling a broad manifold of open-ended AI research ideas articulated in natural language.

Figure 1: Automated idea executor architecture, showing modular Implementer, Scheduler, and Worker roles and their composition within a closed execution-learning loop.

The Implementer ingests batches of LLM-generated ideas, orchestrates their translation to code diffs using code-generating models, and iteratively patches baseline codebases. Ideas are executed by Workers on distributed GPU clusters, while the Scheduler allocates resources adaptively. By structuring the system this way, the authors facilitate large-scale, parallel evaluation of arbitrary research proposals under controlled, reproducible conditions with detailed execution and error logging.

Two representative, challenging research environments ground the framework: (1) LLM pre-training acceleration (nanoGPT baseline), and (2) post-training for mathematical reasoning (GRPO baseline). These domains offer rich search spaces tractable to automated implementation, well-defined performance metrics, and robust, competitive expert baselines.

Benchmarking LLMs in Ideation and Execution

An initial experimental axis benchmarks frontier LLMs (Claude-4.5-Opus, Claude-4.5-Sonnet, GPT-5, Kimi-K2-Thinking, Qwen3-235B-A22B) as both ideators and executors. The system measures completion rate (fraction of samples executed successfully), average validation accuracy/loss, and best-of-N performance.

Figure 2: Performance and execution rates for several LLMs in ideator and executor roles in self-execution and cross-execution scenarios, across both studied environments.

A substantial fraction of LLM-generated ideas are directly executable (often >80% with self-execution), and best-of-N samples already outperform codebase baselines. Notably, when using distinct models as executor (e.g., ideator: Claude, executor: GPT-5), success rates drop but remain significant, preserving strong performance gains above baselines. These quantitative results establish the feasibility of execution-grounded feedback as a reward signal and motivate further iterative learning.

Evolutionary Search with Execution-Guided Feedback

The primary learning paradigm is an evolutionary search framework that interleaves exploration (sampling novel ideas) and exploitation (generating variants of previously successful ideas). Across multiple environmental instantiations and model backbones, evolutionary search demonstrates marked sample efficiency and robust solution scaling.

Figure 3: Scaling and saturation trends in evolutionary search, showing per-epoch best performance by model, for nanoGPT (left) and GRPO (right) environments.

All models rapidly surpass the baselines; for instance, Claude-4.5-Sonnet achieves 69.4% accuracy on GRPO (vs. 48.0% baseline), and Claude-4.5-Opus yields faster pre-training (19.7 minutes to target loss in nanoGPT, vs. a 35.9-minute baseline). Compared to best-of-N sampling under equal budget constraints, execution-guided search exhibits clear superiority in discovering higher-performing ideas within only several epochs.

Figure 4: Comparative analysis between best-of-N and execution-guided evolutionary search on the nanoGPT environment.

Analysis of the sampled idea corpus reveals both hyperparameter and algorithmic innovations, with the most effective ideas generally arising from algorithmic changes (with the exception of hyperparameter-dominated successes for Claude-4.5-Sonnet in GRPO). Quantitative breakdowns highlight differential generative tendencies between LLMs, and qualitative inspection identifies multiple rediscoveries of recent research results, indicating a capacity for frontier-aligned innovation.

Reinforcement Learning from Execution-Based Rewards

Reinforcement learning (RL) using execution-based feedback is studied as an alternative pathway for optimizing ideator performance. The approach uses the GRPO objective and leverages distributed rollouts, measuring both average and maximal rewards per epoch.

Figure 5: RL reward curves for average and max reward per epoch in both environments; RL robustly increases mean reward but leaves max reward unimproved.

While RL succeeds in reliably increasing the average reward—with trends paralleling RL for mathematics or code domains—it does not improve the maximal reward per epoch (a key desideratum for scientific discovery). In-depth trajectory analysis attributes this to mode collapse: the ideator converges on generic, safe, easy-to-implement ideas at the expense of diversity and exploration, as highlighted by collapsed thinking trace lengths and concentration on a narrow set of modifications such as normalization swaps or EMA application.

Figure 6: Average thinking/idea lengths over RL training, and execution rates as a function of trace length. RL induces shorter, safer proposals that are more likely to execute, but less diverse.

Figure 7: Epoch-wise counts of two converged ideas after RL—demonstrates loss of ideation diversity.

Ad-hoc RL modifications (e.g., dynamic prompt augmentation, length/diversity reward shaping) show limited impact on mitigating diversity collapse, pointing toward the need for more fundamental RL objective reformulations or scaffold-based approaches.

Implications, Theoretical Considerations, and Future Directions

The work substantively demonstrates that execution-grounded closed-loop feedback can be instantiated at scale for open-ended LLM research ideation. Execution-guided evolutionary search is empirically validated as a highly sample-efficient strategy capable of producing human-competitive or superior results for realistic LLM research tasks, provided sufficiently performant ideators and reliable execution infrastructure. However, the search regime’s scaling—both in upper-bound discovery and across research environments—is bottlenecked by LLM saturation effects and limited transferability of core search mechanisms. The RL paradigm, when naively applied with simple execution rewards, is prone to degeneracies inherent to reward sparsity and mode collapse, ultimately failing to push the state of the art in upper-bound performance.

Practically, the study underscores several system limitations: (i) lack of generalized transfer/extrapolative capacity in discovered ideas at larger scales, (ii) the need for richer and more nuanced execution-derived learning signals (beyond scalar rewards), and (iii) substantial reliance on the coverage and adaptability of underlying executor modules. Theoretically, the results emphasize the gap between average-case ideation amplification and true frontier advancement, and suggest new lines of research blending search, RL, richer scaffolds, and diverse reward shaping. Future architectures may need to integrate finer-grained feedback, external tool integration for code generation, and joint optimization for both novelty and effectiveness.

Figure 8: RL with dynamic prompt augmentation—little to no early-epoch improvement, pointing to the limits of auxiliary context in overcoming RL collapse.

Figure 9: RL under length-based reward shaping—maintains thinking trace length but fails to raise overall effectiveness reward.

Figure 10: RL with explicit diversity reward—achieves sustained exploration but no significant advancement over effectiveness-only RL.

Conclusion

"Towards Execution-Grounded Automated AI Research" (2601.14525) establishes a comprehensive empirical and methodological foundation for automating the full AI research loop via execution-grounded feedback, encompassing ideation, implementation, evaluation, and iterative learning. Execution-guided evolutionary search with LLMs demonstrates rapid baseline overperformance and the emergence of non-trivial algorithmic hypotheses; in contrast, RL from execution alone induces convergence to generic solutions without boosting the upper bound. The results pinpoint both avenues for further research—particularly around richer learning signals, diversity maintenance, and improved executor/generalization mechanisms—and non-trivial obstacles to fully autonomous open-ended scientific discovery. This work constitutes a rigorous benchmark and calls for subsequent innovation in execution-grounded AI agent design, training, and evaluation.