Multi-Level Existence Bench (MLE-Bench)

• MLE-Bench is a benchmark that formalizes AI research agents as graph-based search algorithms iteratively refining candidate ML solutions.
• It employs advanced search strategies like MCTS and evolutionary search with adaptive operator sets to explore and optimize complex model transformations.
• The framework evaluates agent performance via medal rates and generalization gaps, driving advancements in automated machine learning workflows.

Multi-Level Existence Bench (MLE-Bench) is a benchmark designed to rigorously evaluate AI research agents as they compete across real-world machine learning tasks, primarily in the context of Kaggle-style competitions. MLE-Bench formalizes the development, search, and validation of automated solutions via agent-driven exploration of candidate models and code artifacts. The environment facilitates reproducible, scalable agent operation and exposes the agent to tasks that encompass diverse machine learning domains. The benchmark is uniquely architected to probe the interplay between search strategies, operator sets, and evaluation methodologies, targeting the automation of machine learning workflows.

1. Formalization of AI Research Agents

MLE-Bench conceptualizes AI research agents as graph-based search algorithms that iteratively refine candidate solutions for machine learning tasks. The agent maintains an expanding search graph $\mathcal{G}_t = (V_t, E_t)$, where each node $v \in V_t$ represents an executable artifact—code or trained model—with associated meta-data.

Agents leverage an internal fitness function, most commonly a 5-fold cross-validation (CV) score, to guide navigation through $\mathcal{G}_t$. At each iteration, the agent proposes a transformation of the current best candidate, resulting in new nodes and edges within the search graph. The system executes and evaluates these candidates on real-world ML tasks using dedicated computational resources, such as an NVIDIA H200 GPU and 24 CPUs within the AIRA-dojo environment. This environment provides enforced compute isolation and reproducibility guarantees, enabling fair comparison and scaling experiments over a diverse set of tasks (Toledo et al., 3 Jul 2025).

2. Search Policies and Exploration Strategies

MLE-Bench supports and investigates multiple agent search strategies for exploring the space of candidate solutions. The baseline is a greedy search, which always selects the current best node based on its fitness value. The benchmark also implements more advanced policies, notably Monte Carlo Tree Search (MCTS) and evolutionary search:

• MCTS uses an Upper Confidence Bound (UCT) algorithm for node selection. The UCT score is:

$$h_{\text{UCT}}(v|u) = Q(v) + c \cdot \sqrt{\frac{\log N(u)}{N(v) + \epsilon}}$$

where $Q(v)$ is the running mean fitness of node $v$, $N(u)$ and $N(v)$ are visit counts, $c$ modulates exploration, and $\epsilon$ stabilizes early iterations.

• Evolutionary Search maintains a population of candidate solutions, with parents selected in proportion to their normalized fitness scores. Offspring are generated using specialized operators such as Improve and Crossover, followed by evaluation and population update.

The effectiveness of each search strategy is intricately linked with the operator set. For example, even sophisticated exploration can be bottlenecked by insufficiently diverse or powerful transformation operators. The agent’s ability to explore and exploit the solution space depends critically on this combined design (Toledo et al., 3 Jul 2025).

3. Operator Sets and Transformation Mechanisms

Operators are transformation functions that allow agents to mutate, refine, or debug candidate solutions. The baseline operator set (denoted $\mathcal{O}_{\text{AIDE}}$) comprises:

• Draft (initialization)
• Debug (error detection and correction)
• Improve (performance targeted refinement)
• Memory (summary and recall of previous designs)

MLE-Bench introduces an augmented operator set ("$\mathcal{O}_{\text{AIRA}}$") with advanced capabilities:

• Prompt-Adaptive Complexity: The level of solution complexity presented in the prompt is mapped to the node’s number of children ($n_c$), formulated as:
  • “minimal” if $n_c < 2$
  • “moderate” if $2 \leq n_c < 4$
  • “advanced” if $n_c \geq 5$
• Scoped Memory: Operators retrieve only sibling memories for Draft and Improve (encouraging diversity) and full ancestor chains for Debug.
• Think Tokens: Operators’ prompts include explicit encouragement of chain-of-thought reasoning, producing more structured internal reasoning outputs, while preventing the leakage of reasoning traces to subsequent generations.

These innovations foster strategic diversity in solutions and enable cleaner context management during agent exploration (Toledo et al., 3 Jul 2025).

4. Evaluation Methodology and Metrics

MLE-Bench evaluates agent performance using Kaggle-style “medal” attainment. Each task is scored by standardized metrics (such as AUC or RMSE), with bronze, silver, and gold medals awarded for achieving predefined percentiles relative to the public leaderboard.

Agents’ progress is primarily monitored through the rate at which they win medals, measured on both 5-fold CV (proxy for true generalization) and held-out test sets. The key metric is the medal success rate—the proportion of tasks on which the agent medals (i.e., achieves performance above the threshold for a bronze/silver/gold medal).

A critical analysis in the benchmark quantifies the generalization gap—the difference in medal rates if the final candidate were selected by test set scores rather than validation scores. This exposes overfitting risk when agents rely solely on surrogate fitness measures (Toledo et al., 3 Jul 2025).

Search Policy	Operator Set	Medal Rate (%)	Generalization Adjustment
Greedy	$\mathcal{O}_{\text{AIDE}}$	39.6	+9–13% possible
MCTS/Evolutionary	$\mathcal{O}_{\text{AIRA}}$	47.7	+9–13% possible

5. State-of-the-Art Results

Joint optimization of search strategy and operator set yields significant performance gains. The best-performing agent (advanced operator set with MCTS or evolutionary search) achieves a state-of-the-art medal success rate of 47.7% on MLE-bench lite, up from 39.6% for the baseline. Value propagation in MCTS uses incremental backup:

$$\forall u \in P: N(u) \leftarrow N(u)+1,\ Q(u) \leftarrow Q(u) + \frac{\mathcal{A}F(v_\ell) - Q(u)}{N(u)}$$

ensuring $Q(u)$ tracks mean observed fitness over all evaluations.

These results demonstrate that advanced search strategies, when combined with context-aware operator sets, allow agents to exploit compute resources effectively (even beyond 24 hours per problem) and enhance real-world AutoML performance (Toledo et al., 3 Jul 2025).

6. Implications for Automated Machine Learning

MLE-Bench elucidates that the search policy and operator set are deeply interdependent components: neither alone constrains ultimate agent performance. Operator effectiveness, such as complexity-adaptive prompts and memory scoping, may represent the primary bottleneck in solution space exploration.

A plausible implication is that future automated machine learning systems should prioritize joint optimization of exploration strategies and transformation operators. Additionally, overreliance on surrogate metrics (e.g., validation CV scores) in agent selection can produce substantial generalization gaps, advocating for more robust model selection protocols in AutoML pipelines, potentially based on ensembles or multiple evaluation splits.

The framework provided by MLE-Bench enables systematic investigation and iterative improvement of agent architectures, laying foundations for scalable automated scientific discovery and machine learning engineering (Toledo et al., 3 Jul 2025).