AIRA_2: Overcoming Bottlenecks in AI Research Agents

Abstract: Existing research has identified three structural performance bottlenecks in AI research agents: (1) synchronous single-GPU execution constrains sample throughput, limiting the benefit of search; (2) a generalization gap where validation-based selection causes performance to degrade over extended search horizons; and (3) the limited capability of fixed, single-turn LLM operators imposes a ceiling on search performance. We introduce AIRA$_2$, which addresses these bottlenecks through three architectural choices: an asynchronous multi-GPU worker pool that increases experiment throughput linearly; a Hidden Consistent Evaluation protocol that delivers a reliable evaluation signal; and ReAct agents that dynamically scope their actions and debug interactively. On MLE-bench-30, AIRA$_2$ achieves a mean Percentile Rank of 71.8% at 24 hours - surpassing the previous best of 69.9% - and steadily improves to 76.0% at 72 hours. Ablation studies reveal that each component is necessary and that the "overfitting" reported in prior work was driven by evaluation noise rather than true data memorization.

• The paper demonstrates significant performance improvements using a modular, asynchronous multi-GPU evolutionary framework with ReAct agents.
• It introduces Hidden Consistent Evaluation to prevent reward hacking and stabilize long-horizon performance.
• Experimental results show AIRA_2 outperforms benchmarks, achieving 71.8% rank at 24 hours and 76.0% at 72 hours.

AIRA_2: Overcoming Bottlenecks in AI Research Agents

Introduction

AIRA_2 directly addresses major architectural challenges in scalable agentic AI research, systematically targeting compute throughput, evaluation instability, and operator capability as formalized bottlenecks in prior literature. Unlike previous open-ended LLM-powered agents, AIRA_2's architectural solutions are separated, controlled, and ablated, reliably driving improvements across the MLE-bench-30 benchmark. The key abstractions are an asynchronous, multi-GPU evolutionary search with consistent externalized evaluation, and highly capable ReAct operators for interactive, stateful reasoning and tool use.

Figure 1: AIRA_2 achieves SOTA on MLE-bench-30, outperforming prior agents across varying compute budgets and displaying efficient scaling with additional resources.

Architectural Innovations

AIRA_2 decomposes into a global orchestrator maintaining candidate populations, with asynchronous GPU workers instantiating containerized ReAct agents. The evolutionary framework (steady-state evolution with temperature-scaled rank selection) enables robust diversity and rapid exploration. The crucial improvement lies in hardware-optimized parallelization: by decoupling reasoning and experimentation with container orchestration, AIRA_2 linearly amplifies sample throughput with available GPUs, compressing days of search into hours without the deadtime of synchronous designs.

Figure 2: The two-tier AIRA_2 architecture, illustrating decentralized parallel mutation, isolated evaluation, decoupling of search and selection, and stateful agent-environment interaction.

The Hidden Consistent Evaluation (HCE) protocol establishes three fixed label splits---train, search, and validation---preventing agents from gaming the evaluation signal or suffering metric instability due to stochastic splits. Crucially, agents never observe metrics or data splits directly; only final scores are externally computed and revealed. This removes "reward hacking" vulnerabilities and shifts selection dynamics from overfitting to true generalization, as demonstrated by the elimination of approximate test/validation drift.

The use of ReAct agents for all mutation and crossover enables dynamic action scoping and multi-step tool use. These agents can execute multi-turn debugging, adapt code, explore data, invoke shell commands, and flexibly allocate compute depending on sub-task. This surpasses fixed single-turn operator paradigms which fail in deeply nonstationary or error-prone research domains.

Experimental Results

AIRA_2 sets a new state-of-the-art on the challenging MLE-bench-30 suite, achieving 71.8% mean Percentile Rank at 24 hours (absolute improvement over 69.9% for MARS+), and further scaling to 76.0% at 72 hours due to sustained exploration and optimization. Notably, performance never degrades over extended horizons, contrasting sharply with previous agents that plateau or regress due to overfitting and escalating evaluation noise.

Ablation studies show all three architectural pillars are critical:

• Removing ReAct agents reduces efficiency by up to 5.5 points at early time budgets.
• Disabling multi-GPU parallelism or evolution causes substantial search inefficiency (Best-of-K baselines saturate at single-GPU asymptotes).
• Reverting to non-consistent evaluation reintroduces long-term performance degradation, highlighting the importance of HCE in stabilizing learning signals.

Figure 3: Increasing parallel GPU workers leads to superior long-run performance, not just faster convergence, as diversity accumulation prevents local optima entrapment.

Figure 4: HCE eliminates overfitting-induced performance collapse, enabling monotonic improvement across extended research horizons, and demonstrates previous "overfitting" was artifactually driven by procedural noise.

Case Studies: Agentic Research Trajectories

Fine-grained analysis confirms complex, dynamic reasoning emerges under the ReAct paradigm. For example, in the "champs-scalar-coupling" task, AIRA_2 identifies underfitting not as a fundamental failure but as an opportunity for further scaling, methodically increasing model capacity and training time after inspecting logs and validation losses. Operator crossovers combine complementary strategies (e.g., data preprocessing with large-scale modeling), yielding solutions unattainable by prior agents.

Figure 5: AIRA_2's multi-turn reasoning and exploration dynamics demonstrate superior sub-task decomposition, error diagnosis, and solution refinement leading to human-competitive medal acquisition.

These behaviours extend to other domains (billion-word imputation, fine-grained visual classification), where AIRA_2 not only surpasses agentic baselines but, in several tasks, is the only method to cross key performance thresholds.

Theoretical and Practical Implications

AIRA_2 demonstrates that agentic research is not rate-limited by LLM inference alone: proper architectural decomposition, throughput-oriented parallelism, external reward control, and operator autonomy are all indispensable for long-horizon scalability. Demonstrated gains are robust across diverse ML challenges and positively correlate with compute allocation, but critically do not lead to reward hacking or evaluation collapse.

The findings challenge previous assumptions regarding long-horizon plateauing in large-scale agentic research, specifically attributing performance deterioration not to true overfitting but to noisy or agent-contaminated evaluation procedures. This underscores the theoretical necessity of externalized, consistent, and hidden evaluation for reliable generalization in autonomous science.

Practically, the results indicate that future AI researchers, for both software and scientific domains, must be architected with high-throughput, asynchronous multi-agent search, evaluation externalization, and dynamic operator design as first-class components. Monolithic, serial, or static-prompt agents are inadequate for open-ended research.

Limitations and Future Directions

Despite its empirical strengths, AIRA_2's results may still confound exploration of genuinely novel research spaces---latent pretraining contamination on public Kaggle solutions cannot be excluded. Evaluation on genuinely "closed" domains will be needed for full separation of research and retrieval abilities. Additionally, AIRA_2 relies on modest manual split curation and may not be optimal for resource-constrained environments where rapid, low-compute search is paramount.

Future developments should include adaptive agent initialization, on-the-fly evaluation split synthesis, broader tool affordance (e.g., internet or API access), and extension to lifelong, open-ended research without externally curated rewards. Quantification of credit assignment among evolutionary, operator, and evaluation modules as compute further scales is a critical next step.

Conclusion

AIRA_2 establishes a high-water mark for ML2-ML research agent performance on the MLE-bench-30 benchmark, grounded in architectural modularity and rigorous ablation. By shifting from brittle, synchronous, and static designs to parallel, decentralized, interactive exploration with externalized evaluation, AIRA_2 demonstrates reliable, scalable, and robust performance in open-ended agent science. These results articulate a new blueprint for autonomous research agents targeting complex, noisy, and computationally intensive domains, and provide foundational evidence for the further decoupling and factorization of agent capabilities in future AI research systems.