AIRS-Bench: a Suite of Tasks for Frontier AI Research Science Agents

Abstract: LLM agents hold significant promise for advancing scientific research. To accelerate this progress, we introduce AIRS-Bench (the AI Research Science Benchmark), a suite of 20 tasks sourced from state-of-the-art machine learning papers. These tasks span diverse domains, including language modeling, mathematics, bioinformatics, and time series forecasting. AIRS-Bench tasks assess agentic capabilities over the full research lifecycle -- including idea generation, experiment analysis and iterative refinement -- without providing baseline code. The AIRS-Bench task format is versatile, enabling easy integration of new tasks and rigorous comparison across different agentic frameworks. We establish baselines using frontier models paired with both sequential and parallel scaffolds. Our results show that agents exceed human SOTA in four tasks but fail to match it in sixteen others. Even when agents surpass human benchmarks, they do not reach the theoretical performance ceiling for the underlying tasks. These findings indicate that AIRS-Bench is far from saturated and offers substantial room for improvement. We open-source the AIRS-Bench task definitions and evaluation code to catalyze further development in autonomous scientific research.

• The paper introduces AIRS-Bench as a contamination-controlled benchmark for assessing end-to-end LLM-driven research agents on practical ML tasks without baseline code.
• It demonstrates that agents achieve a 58.8% valid submission rate and a 23.4% average normalized score, underscoring significant performance gaps versus human SOTA.
• The work highlights scaffold strategies’ impact, with parallel search methods driving notable improvements in agent performance and occasional SOTA exceedances.

AIRS-Bench: Rigorous Evaluation of LLM-Based Autonomous Research Agents

Motivation and Benchmark Positioning

AIRS-Bench ("AI Research Science Benchmark") (2602.06855) is proposed as a standardized, contamination-controlled suite for assessing the full autonomous research workflow of LLM-driven agents. The central aim is to evaluate AI Research Agents—LLM agents coupled with computational scaffolds—on entirely practical ML tasks originating from high-impact state-of-the-art publications, without providing any baseline code. This end-to-end assessment addresses crucial limitations in the evaluation of agentic AI, such as data contamination, environmental standardization discrepancies, and high empirical noise, which have impeded progress in agentic ML research benchmarking.

AIRS-Bench explicitly positions itself vis-à-vis prior benchmarks by insisting on unsaturated, challenging tasks with no baseline solution, and by enforcing a unified task schema compatible with a variety of agentic frameworks. This enables robust model-agnostic comparisons and algorithmic ablations. As detailed in the cross-benchmark comparison—

Figure 1: Comparative overview of leading agentic AI research benchmarks, highlighting AIRS-Bench's unique coverage of the full scientific research pipeline, stringent baseline restrictions, and demanding compute profile.

—AIRS-Bench is distinguished by covering all phases of the scientific method (hypothesis generation, implementation, experimentation, analysis), not providing any starter code, focusing on long-horizon research problems, and requiring significant GPU resources.

Task Suite Design and Coverage

AIRS-Bench comprises 20 distinct tasks sourced from 17 recent ML papers, spanning seven problem categories. These include core ML domains—NLP (question answering, text classification, extraction/matching), code, math, molecular/biochemical modeling, and time-series forecasting.

Figure 2: Distribution of AIRS-Bench tasks across the seven defined categories, with NLP and molecular/protein ML tasks being the most represented.

Each task is meticulously constructed as a $\{$problem, dataset, metric$\}$ triplet, matching the format of the originating ML research. Importantly, agents must programmatically generate and execute code to train and validate models, rather than simply inferring output. Agents are evaluated strictly using the code-generated predictions and well-defined, paper-aligned metrics—

Figure 3: Illustration of the AIRS-Bench task format, specifying the computational challenge, input dataset, and quantitative evaluation metric.

The design ensures agents operate as actual autonomous research scientists, from ideation and implementation to iterative refinement and empirical analysis. Task diversity, schema standardization, and systematic human verification enable extension to arbitrary ML domains while preserving consistent, interpretable evaluation.

Agent, Scaffold, and Harness Architecture

AIRS-Bench's agentic definition follows AI research literature: an "agent" is a composition of an LLM and a scaffold. The LLM provides core reasoning, while the scaffold is an algorithmic orchestrator (e.g., greedy, ReAct, MCTS, evolutionary search) that governs solution space exploration, tool invocation, validation, and code refinement. Harnesses (such as AIRA-dojo and MLGym) instantiate agent+scaffold pairs, expose interfaces for environment interaction, manage solution generation and artifact submission, and regulate computational resource usage.

Figure 4: Schematic of the agent-scaffold-harness-environment architecture adopted in AIRS-Bench, supporting rigorous solution-space search and controlled agentic experimentation.

Parallel scaffolds (tree search, evolutionary) and sequential scaffolds (e.g., ReAct) are both supported and evaluated, enabling fair direct comparison under equivalent compute and task constraints.

Experimental Protocol and Evaluation Metrics

Evaluation encompasses 14 agents (various model+scaffold combinations, both open and closed source), subjected to uniform hardware (1×H200 GPU per run), strict compute-time quotas (24h), and a minimum of 10 seeds per task-agent pair for robust statistics. The benchmark enforces pre-cached pretrained checkpoints, anonymized test splits, and complete isolation from SOTA methodologies and code.

Aggregated evaluation employs three core metrics:

• Mean Valid Submission Rate (VSR): Fraction of runs yielding syntactically valid, scorable submissions.
• Average Normalized Score: Progress towards SOTA using a "march of 9s" transform, capturing the logarithmic reduction of distance to optimal, and normalized so that $0.0$ is worst valid solution observed, $1.0$ is human SOTA, with potential for exceeding $1.0$ if agents surpass human results.
• Elo Rating: Relative skill estimation adapted from the Bradley–Terry model, treating each agent-task outcome as a head-to-head match, including human SOTA as an artificial "opponent."

Empirical Results and Performance Analysis

Overall, AIRS-Bench yields a challenging empirical landscape for research agents:

• The mean valid submission rate across all agents and tasks is $58.8\%$—indicating that even producing a working solution is non-trivial.
• The average normalized score across all runs and agents is just $23.4\%$, underscoring the substantial gap to human SOTA.
• Strong correlation between the ability to submit valid results and achieving higher relative scores is observed.
• Notably, only $1.55\%$ of agent-task combinations exceeded SOTA, almost exclusively driven by parallel (greedy tree search) scaffolds.

  Figure 5: Aggregate comparison of valid submission rate, normalized score, and Elo rating for all 14 agents, showing clear advantages for large LLMs paired with parallel search scaffolds.

  

  Figure 6: Valid submission rate distribution per agent, highlighting agent reliability and the marked difficulty of many AIRS-Bench task-agent pairings.

  

  Figure 7: Distribution of raw performance per agent over all tasks—agents rarely occupy the 'best' or 'above-average' performance bins.

  

  Figure 8: Detailed agent-by-agent VSR, indicating the performance advantage of larger transformer models when equipped with population-based search.

  

  Figure 9: Average normalized scores, reinforcing the high difficulty floor across the suite, and the limited progress toward SOTA.

  

  Figure 10: Task-by-task normalized performance (averaged across seeds and agents), highlighting a small number of cases where SOTA is exceeded and a long tail of difficult, unsolved tasks.

  

  Figure 11: Average normalized score as a function of task difficulty band, delineating the disparities between 'easy', 'medium', 'hard', and 'expert' categories—gap to SOTA remains high for all.

  

  Figure 12: Elo ranking of agents (including human SOTA as a pseudo-agent), with the top Greedy scaffold agents still considerably below human-level performance.

Exceeding SOTA Cases:

Four tasks saw agents outperforming human SOTA, predominantly via model ensembling or novel code search. Notably, for TextualClassificationSickAccuracy, a greedy gpt-oss-120b agent constructed a cross-validated RoBERTa+DeBERTa ensemble and logistic meta-learner, resulting in accuracy improvements (from 90.5% SOTA to 93.1%). Analogous gains are observed for semantic similarity, time series forecasting, and challenging coreference tasks—with agent-devised solutions occasionally diverging significantly from published approaches.

Practical and Theoretical Implications

Practical implications of AIRS-Bench are multi-fold:

• Benchmark Saturation: The marked performance gap implies significant remaining headroom; advances in agent algorithmics and LLM base model quality are both necessary for substantive progress.
• Scaffold Dependence: Solution quality is highly sensitive to the choice and sophistication of the scaffold; test-time exploration and iterative code search (as in greedy/evolutionary scaffolds) are disproportionately effective, suggesting further work on algorithmically guided agentic workflows is critical.
• Generalization beyond Existing Solutions: When agents exceed SOTA, it is often through solutions not explicitly described in the literature, indicating unanticipated compositionality and creativity potential in well-orchestrated agentic LLMs.
• Compute and Infrastructure Limits: The high resource requirements for end-to-end agentic evaluation, combined with the need for rigorous contamination control, point to the methodological necessity for shared, scalable, and standardized evaluation pipelines in the community.

Theoretically, AIRS-Bench lays groundwork for formalizing agentic scientific reasoning and the measurement of research automation. It enables systematic ablations of scaffold strategies, model scale, memory, and compute horizons under unified and contamination-minimal experimental conditions.

Outlook and Future Directions

Several clear research directions emerge:

• Scaling Benchmark Size and Diversity: Although the current 20-task suite covers broad ML subfields, further extension to additional domains (e.g., natural sciences, engineering) and to even more unsolved problems would enhance diagnostic fidelity.
• Automatic Task Generation and Curation: Human bottlenecks in task validation and reproducibility checking are identified as limiting factors; semi-automated, community-driven pipelines are a likely evolution.
• Advanced Scaffolds: The consistent superiority of population-based and tree search methodologies underlines the importance of algorithmic innovation at the scaffold level, especially methods that effectively utilize test-time compute and handle code synthesis failures.
• Meta-Benchmarks and Reproducibility Enforcement: Given ongoing challenges in ML reproducibility, AIRS-Bench’s schema and artifact format could be aligned with publishing standards, enabling back-testing of new agentic discoveries against a continually growing suite of validated scientific problems.

Conclusion

AIRS-Bench establishes a robust, rigorous foundation for measuring autonomous research progress in LLM-based agents. The combination of contamination-resistant, practically-relevant tasks, a unified and extensible agent-task schema, and comprehensive metrics enables disciplined benchmarking of agentic AI in genuine scientific workflows. Despite isolated instances of agents exceeding SOTA, the large overall performance gap documents the unsolved nature of research automation and identifies substantial opportunities for scaffold innovation and LLM advancement. AIRS-Bench provides the methodological scaffolding upon which emergent agentic AI techniques can be systematically compared, driving both theoretical understanding and practical progress in automated scientific discovery.