RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human experts

Abstract: Frontier AI safety policies highlight automation of AI research and development (R&D) by AI agents as an important capability to anticipate. However, there exist few evaluations for AI R&D capabilities, and none that are highly realistic and have a direct comparison to human performance. We introduce RE-Bench (Research Engineering Benchmark, v1), which consists of 7 challenging, open-ended ML research engineering environments and data from 71 8-hour attempts by 61 distinct human experts. We confirm that our experts make progress in the environments given 8 hours, with 82% of expert attempts achieving a non-zero score and 24% matching or exceeding our strong reference solutions. We compare humans to several public frontier models through best-of-k with varying time budgets and agent designs, and find that the best AI agents achieve a score 4x higher than human experts when both are given a total time budget of 2 hours per environment. However, humans currently display better returns to increasing time budgets, narrowly exceeding the top AI agent scores given an 8-hour budget, and achieving 2x the score of the top AI agent when both are given 32 total hours (across different attempts). Qualitatively, we find that modern AI agents possess significant expertise in many ML topics -- e.g. an agent wrote a faster custom Triton kernel than any of our human experts' -- and can generate and test solutions over ten times faster than humans, at much lower cost. We open-source the evaluation environments, human expert data, analysis code and agent trajectories to facilitate future research.

• The paper presents a framework of seven evaluation environments that distinguishes AI agents' initial performance gains from humans' long-term research progression.
• The methodology involves comparative experiments with 35 AI agent trials and 44 human expert runs to assess task-specific optimization challenges.
• The findings highlight the need for improved long-horizon planning and resource management in AI systems to match human R&D capabilities and inform safety policies.

Evaluation of AI R&D Abilities: Comparative Performance and Implications

The paper presents a conceptual framework to assess the AI-driven research and development (R&D) capabilities critical for advancing ML systems. It includes a suite of seven specifically designed evaluation environments, each representing a different ML optimization challenge, such as minimizing loss functions or enhancing runtime performance. Critically, these environments aim to encapsulate complex problem-solving skills requisite for AI R&D, reflecting an intent to differentiate solution quality with fine granularity.

The evaluation environments were tested with 44 human expert runs and 35 trials of Claude-3.5-Sonnet, an AI agent. Results revealed that while AI agents initially surpass human performance in the first hour, they tend to plateau, whereas human researchers demonstrate consistent progression beyond this point. Notably, AI agents occasionally achieved superior solutions compared to weaker human results, although their overall performance fails to sustain superiority across extended task durations.

A key contribution of this work is the identification of desired properties for the evaluation environments. These include a broad scoring range to capture variations in solution quality, minimal ambiguities in task instructions, a significant degree of novelty, and comprehensive coverage of core R&D tasks. Despite advancements, the research identifies considerable room for enhancing AI agents' capabilities in long-horizon planning, robust scaling of problem-solving skills, and effective resource management.

From a practical standpoint, advanced AI R&D abilities pose potential risks, as underscored by their inclusion in frontier safety policies by leading AI developers. Achieving parity with human R&D skills might signal a critical threshold, raising the necessity for enhanced security and evaluation strategies. The proposed evaluation framework suggests a pragmatic method for anticipating when AI systems may approach transformative capabilities—utilizing these environments as early warning indicators.

In addition to scrutinizing AI and human performances, the paper proposes an expansion of the evaluation suite. The authors highlight the possibility of increased environment diversity, extended evaluation durations, and the incorporation of improved tools and scaffolding to boost agent performance. Such developments are anticipated to bridge existing gaps in agent capabilities and human-equivalent performance metrics over realistic time horizons.

In conclusion, this paper contributes an insightful methodology for evaluating AI's autonomous research capabilities. While current results suggest that AI agents have yet to fully rival human expertise across comprehensive research tasks, they also underscore the growing potential and inherent risks associated with AI-driven R&D automation. Future research directions include fine-tuning these environments to better capture the nuances of AI research skills and refining the ability of AI systems to manage and operate over extended periods on complex tasks.