MLRC-BENCH: Autonomous ML Research Benchmark

• The benchmark introduces a novel 'Relative Improvement to Human' metric to quantitatively assess LLM agents' performance in solving open research problems.
• MLRC-BENCH distinguishes itself by emphasizing innovation, iterative code development, and precise empirical evaluation over subjective judgments.
• It features reproducible tasks, clear evaluation protocols, and a dynamic, community-driven framework that adapts to evolving ML research challenges.

MLRC-BENCH is a dynamic and rigorously designed benchmark developed to assess the capability of language agents—primarily LLM agents—to solve challenging, open-ended machine learning research competition tasks. Unlike earlier benchmarks that focus on plug-and-play ML engineering subtasks or subjective LLM-as-a-judge pipelines, MLRC-BENCH prioritizes the objective measurement of innovation and implementation in unconstrained research settings. The benchmark assesses not only the ideation of novel ML methodologies but also repository-level code development and demonstrable empirical improvement, providing a granular and transparent evaluation of AI research capabilities (Zhang et al., 13 Apr 2025).

1. Benchmark Scope and Task Selection

MLRC-BENCH is structured around a curated suite of seven machine learning research competition tasks, each carefully selected to satisfy three primary criteria:

• Novelty: Every task is inherently an open research problem, where trivial solutions or rote application of standard models (such as XGBoost or simple prompt engineering) are insufficient.
• Non-Triviality: The tasks require conceptual advances or algorithmic inventiveness (e.g., machine unlearning requires navigating trade-offs between knowledge retention and erasure, rather than simple parameter optimization).
• Reproducibility and Feasibility: Each task is accompanied by publicly released starter code, pre-defined data splits, and clear evaluation protocols. Computational resource constraints (such as GPU memory and allowable runtime) are explicitly specified, ensuring that submissions are both tractable and easy to verify.

This structure distinguishes MLRC-BENCH from prior efforts like AI Scientist and MLAgentBench, which often conflate engineering execution with true research advancement or rely on subjective LLM-judged innovation.

2. Evaluation Protocol and Objective Metrics

All participating agents interact with the same development pipeline, allowing multiple iterations ("snapshots") to refine implementation. Only edits to the designated methods/ directory are permitted, preventing tampering with evaluation code and hardcoding of solutions.

Performance on each task is quantified using a Relative Improvement to Human metric, computed as: $$S_{\text{normalized}} = \frac{S_{\text{agent}} - S_{\text{baseline}}}{S_{\text{top human}} - S_{\text{baseline}}} \times 100$$ where $S_{\text{agent}}$ is the agent's score, $S_{\text{baseline}}$ is the published baseline, and $S_{\text{top human}}$ is the best human score. The normalized score anchors 0 to the baseline and 100 to the best-known human result, allowing for straightforward cross-task comparison and aggregation.

Additional tracked metrics include:

• Effectiveness: Task-specific quantitative measures (e.g., accuracy, mAP, recall).
• Efficiency: Training and inference time, with strict upper bounds to simulate real-world constraints.
• Simplicity: Logical lines of code (LLOC) in the submission, serving as a proxy for maintainability and engineering parsimony.

This rigorous and transparent evaluation scheme differentiates MLRC-BENCH from metrics based only on pass/fail or LLM-judged innovation scores.

3. Quantitative Findings and Performance Analysis

The inaugural evaluation of MLRC-BENCH across leading LLM agents revealed a substantial gap in performance relative to top human participants. The best-tested agent (gemini-exp-1206 under the MLAB framework) closed only 9.3% of the gap between the baseline and the best human score, i.e.,

$S_{\text{normalized}} \approx 9.3$

This result is notable given the inclusion of extensive code-editing iterations, sophisticated agent scaffolds, and the use of state-of-the-art models. Such a low gap closure signals that current autonomous language agents remain far from replicating the methodological or engineering leap required for impactful ML research solutions.

MLRC-BENCH further demonstrates that solutions judged highly “innovative” or “clear” by LLM-as-a-judge (subjective metrics) often fail to yield meaningful test set gains. For example, candidate policies such as median aggregation in LLM merging challenges performed worse than simple mean aggregation, and "gradient ascent unlearning" did not properly balance forgetting and retention under the precise evaluation protocol.

4. Challenges and Insights from Agentic Evaluation

Key observed challenges for agents in MLRC-BENCH include:

• Insufficient Transfer from Subjective Novelty to Objective Gain: There is a salient misalignment between ideas rated as innovative by LLM-judges and their actual empirical effectiveness.
• Over-Editing and Inefficient Code: Iterative agentic code editing often introduces additional complexity and resource usage (increased LLOC or runtime) without a commensurate performance boost.
• Difficulty in Navigating Multi-objective or Constraint-heavy Problems: Problems such as unlearning require sophisticated optimization over conflicting goals, which most agents struggle to formalize and implement reliably.

These results highlight a pivotal challenge: the limited capacity for LLMs to autonomously synthesize novel, correct, and efficient solutions to open research problems (as opposed to well-defined engineering subtasks).

5. Dynamic Benchmark Evolution and Future Prospects

MLRC-BENCH is explicitly designed as a dynamic benchmark. As new research competitions arise (e.g., from machine learning conferences or challenge workshops), they can be incorporated, while tasks that become saturated (i.e., where agents match or exceed human solutions) are retired. This ensures continuous alignment with the evolving state of the field and ongoing benchmarking relevance.

The leaderboard—hosted at the provided Hugging Face Spaces URL—displays current performance across agent scaffolds, models, and research tasks. The full codebase and task descriptions are released to guarantee transparency and reproducibility, fostering open, community-driven advances in agentic ML research.

6. Community Engagement and Broader Impact

MLRC-BENCH is structured to facilitate broad participation and ongoing methodological refinement. Researchers are encouraged to:

• Submit new research problems or benchmarks for inclusion.
• Propose and test new agent scaffoldings, toolchains, or workflow methodologies.
• Suggest refinements to the evaluation protocol or additional control metrics.

The persistent gap between agentic and human research performance sets a clear, quantifiable target for future advances in both LLM model design and scaffold engineering. A likely implication is that continued progress will require not only more powerful base models but also enhanced reasoning architectures, active exploration strategies, and mechanisms for integrating multi-iteration feedback and debugging.

7. Summary Table: Distinctives of MLRC-BENCH

Aspect	MLRC-BENCH	Prior Benchmarks (e.g., AI Scientist)
Task Type	Open ML research competitions	Engineering/pipeline, closed-form tasks
Primary Evaluation	Objective, empirical improvement	LLM-as-a-judge (subjective)
Metric	Relative Improvement to Human, Efficiency	Often binary or point-based
Dynamic Content	Yes: tasks added/retired as field evolves	Typically fixed set
Community Involvement	Leaderboard, open source, feedback loop	Limited

MLRC-BENCH is thus positioned as a rigorous, reproducible, and continuously evolving standard for benchmarking AI agents’ ability to perform authentic, high-level ML research, offering quantitative transparency and a clear bar for progress toward genuinely autonomous scientific discovery.