MLS-Bench: Scalable ML Method Discovery

• MLS-Bench is a benchmark that tests if AI systems can invent generalizable ML methods rather than relying solely on engineering-style optimizations.
• It comprises 140 tasks across 12 domains, each targeting a specific ML component to isolate genuine scientific contributions.
• The evaluation protocol emphasizes controlled experiments and transferable improvements to prevent local tuning or scale-based shortcuts.

MLS-Bench, short for ML Science Bench, is a benchmark for evaluating whether AI systems can invent generalizable and scalable machine-learning methods rather than merely execute engineering-style optimization on a fixed setup. It contains 140 tasks across 12 ML domains, and each task requires an agent to improve one targeted component of an ML system or algorithm under controlled conditions, then demonstrate that the improvement transfers across multiple settings and scales. The benchmark is designed to measure “genuine, transferable ML method improvements,” not task-specific leaderboard gains obtained through harness changes, capacity inflation, or local tuning alone (Lyu et al., 9 May 2026).

1. Conceptual scope and motivation

MLS-Bench was introduced to address a distinction that is central to ML research but often blurred in agent evaluation: the difference between ML engineering and ML science. The benchmark’s motivating claim is that many existing evaluations reward forms of improvement such as data cleaning, hyperparameter tuning, debugging, model selection, or local optimization of an implementation. Those activities can improve a score, but they do not necessarily constitute the discovery of a reusable method. MLS-Bench instead asks whether an AI system can discover changes analogous to the kinds of method-level innovations that have historically driven ML progress: new architectures, objectives, optimizers, regularizers, training rules, or system components (Lyu et al., 9 May 2026).

The benchmark therefore centers on a sharper research question: whether AI systems can discover new ML methods that humans would regard as genuine scientific contributions. Its design assumes that a one-off improvement on a single setup is insufficient evidence. A candidate method must generalize across controlled variations and remain meaningful when evaluated at larger or different regimes. This framing makes MLS-Bench a benchmark of scientific capability rather than of isolated task execution.

A common misconception is that “building better AI” can be reduced to opportunistic tuning. MLS-Bench is explicitly organized against that interpretation. Its structure aims to separate method invention from implementation polish, scaffold exploitation, and scale-based shortcuts. This suggests that the benchmark is not primarily about whether an agent can write competent research code, but whether it can support claims of reusable algorithmic insight.

2. Benchmark design and task composition

MLS-Bench is built around eight design principles: Holistic, Atomic, Challenging, Generalizable, Reproducible, Scientific innovation, Scalable, and Unified scoring (Lyu et al., 9 May 2026). “Holistic” refers to broad coverage across major ML subfields; “Atomic” means that each task targets one coherent research question; “Challenging” requires strong human baselines, including reproduced SOTA; “Generalizable” requires testing across multiple settings; “Reproducible” imposes controlled runtimes, pinned dependencies, fixed seeds, and locked versions; “Scientific innovation” restricts gains to the targeted method rather than harness hacks or protocol changes; “Scalable” requires evaluation settings that preserve evidence about whether a method scales; and “Unified scoring” normalizes task outcomes to a bounded scale so that cross-task comparison is possible.

A defining structural feature is the focus on a targeted component. Each task asks the agent to modify one specific part of an ML system or algorithm, such as a loss function, optimizer, attention mechanism, sampler, normalization rule, policy update rule, model architecture block, quantization method, training regularizer, or search strategy. The rest of the harness is frozen. The editable scope is restricted to the relevant component, and tasks are accepted only if that scope is “exactly wide enough” to implement known strong methods and still reproduce their published or reference performance. The paper terms this baseline-calibrated scaffolding (Lyu et al., 9 May 2026).

The 140 tasks are distributed across 12 domains as follows.

Domain	Tasks
LLMs	18
Classical Adaptive Learning	14
Reinforcement Learning	13
Optimization Theory	13
Robotics	12
Vision Generation	11
Deep Learning	11
ML Systems / Efficient ML	10
AI for Science	10
Time Series Forecasting	10
Structured Causal Reasoning	10
Trustworthy Learning	8

The appendix examples illustrate the intended breadth: pretraining optimizer design, attention mechanism design, quantization-aware training, diffusion-policy sampling for robot actions, 3D Gaussian splatting densification, causal graph discovery, federated aggregation, adversarial training, and time-series forecasting architectures. Despite that breadth, the unit of evaluation is deliberately narrow: one coherent method-level question per task.

3. Controlled evaluation protocol

Each MLS-Bench task includes a research question and background, a codebase with editable scopes, at least 3 strong human baselines, at least 3 evaluation settings, a seed policy for multi-seed evaluation when variance matters, a score normalization rule, and a capacity budget when relevant (Lyu et al., 9 May 2026). The environment supports Apptainer, Docker, and conda, and agents interact through four tools: edit, test, submit, and undo.

The evaluation protocol is designed to block common confounds. Harness hacking is prevented by freezing the evaluation code. Hyperparameter knobs shared across methods are locked into protected ranges. Method-specific hyperparameters remain editable only when they are genuinely part of the method. Capacity cheating is controlled by a parameter budget for tasks that modify model components. The paper also states that contamination is reduced by including strong reproduced baselines and disabling web search in the main experiments (Lyu et al., 9 May 2026).

In the main experiments, frontier models are given high reasoning effort, 10,000 thinking tokens, web search disabled, fixed seeds, and a budget of 20 actions with at most 3 test calls. For MLS-Bench-Lite, a smaller subset of 30 tasks spanning all 12 domains, additional models are evaluated with 8 actions and 1 test call. The Lite subset serves as a faster evaluation track while preserving domain coverage.

Generalization is enforced through the requirement of at least three evaluation settings per task. These settings may differ in dataset, environment, model scale, or another controlled condition. Scalability is handled by choosing reduced settings that remain computationally feasible while preserving the ranking of the baselines. The rationale is that a proxy regime is acceptable only if it preserves the comparative structure of known methods; otherwise it ceases to be evidence about scalable performance. This design choice is central to MLS-Bench’s claim that it measures method quality rather than success on an arbitrary small-scale surrogate.

4. Scoring, normalization, and the role of generalization

MLS-Bench uses a baseline-anchored normalization so that scores are comparable across heterogeneous tasks. For each metric, the worst baseline anchors 0, and the best baseline anchors 0.5 on an internal bounded scale. The per-metric transformation is defined as follows (Lyu et al., 9 May 2026):

$$\operatorname{transform}(x)= \begin{cases} \bigl((x-x_\mathrm{floor})/(x_\mathrm{bound}-x_\mathrm{floor})\bigr)^\gamma, & \gamma=\frac{\log 0.5}{\log\bigl((x_\mathrm{ref}-x_\mathrm{floor})/(x_\mathrm{bound}-x_\mathrm{floor})\bigr)},\ \text{if } x_\mathrm{bound}\text{ exists},\ 2\sigma((x-x_\mathrm{floor})/\lambda)-1, & \lambda=(x_\mathrm{ref}-x_\mathrm{floor})/\log 3,\ \text{else}. \end{cases}$$

Here, $x_\mathrm{floor}$ is the worst baseline and $x_\mathrm{ref}$ is the best baseline. Metrics are oriented appropriately before transformation, since some are higher-is-better and others lower-is-better.

Within a setting, metric scores are aggregated by a weighted arithmetic mean:

$$S_\mathrm{setting} = \sum_i w_i s_i \Big/ \sum_i w_i.$$

Across settings, the task score is the geometric mean:

$$S_\mathrm{task} = \bigl(\prod_k S_{\mathrm{setting},k}\bigr)^{1/K}.$$

The use of the geometric mean is a substantive design choice rather than a cosmetic one. It makes MLS-Bench deliberately unforgiving to methods that succeed on one setting but fail on another. A method cannot compensate for collapse in one regime by overperforming in a different regime. This scoring rule operationalizes the benchmark’s central notion of generalization.

The benchmark’s scoring philosophy thus encodes a specific scientific standard: a method claim is only credible if it survives controlled variation. In MLS-Bench, that standard is not left to qualitative interpretation but is built directly into the task score.

5. Empirical findings and observed failure modes

The principal empirical result is that current frontier agents remain far from reliably surpassing human-designed methods on MLS-Bench (Lyu et al., 9 May 2026). The paper evaluates five frontier models on the full benchmark—Claude Opus 4.6, GPT-5.4, Gemini 3.1 Pro, DeepSeek-V3.2, and Qwen-3.6 Plus—and reports a clear gap between reproduced human baselines and agent performance in both first-try and final submitted results. The benchmark is therefore unsaturated: it still leaves substantial headroom between current AI systems and strong human ML research practice.

One of the benchmark’s most important findings is that engineering-style tuning is easier for agents than genuine method invention. When the authors compare a scientific-innovation prompt with an engineering-optimization prompt, agents—especially weaker ones—perform better at tuning parameters, recombining known techniques, polishing implementations, and making local engineering improvements than at proposing genuinely new scientific methods. The paper interprets this as evidence that current agents are more competent in an engineering regime than in a discovery regime.

The study of test-time scaling reaches a similarly qualified conclusion. The paper evaluates Scaling Sampling, Scaling Exploration, Test-time Evolution, and Test-time Training. More compute helps on easier tasks, but gains saturate quickly; on harder tasks, additional scaling still does not close the gap to human SOTA. Moreover, evolutionary and test-time-training approaches can improve visible settings while hurting hidden settings, indicating optimization to observed feedback rather than robust method discovery. More search, in this sense, can increase overfitting rather than scientific validity.

The benchmark also examines adaptive compute allocation on five pretraining tasks where the agent may choose proxy model sizes from

$$\{51\mathrm{M}, 124\mathrm{M}, 199\mathrm{M}, 345\mathrm{M}\}$$

and set token counts for each test call, with compute measured as

$N \cdot D$

where $N$ is model size and $D$ is training tokens. Performance generally drops in this more flexible setting. Improvement is not monotonic with spend: GPT-5.4 uses relatively little budget and is the only model that improves, while Claude Opus 4.6 spends heavily and still degrades. The broader implication drawn in the paper is that the central bottleneck is not merely proposal generation; it is the ability to choose informative experiments, allocate scarce compute, and construct evidence for a scalable claim (Lyu et al., 9 May 2026).

Additional experiments on context provision—adding web search, baseline context, or theory context—show only modest gains. Stronger models sometimes benefit more, but the improvement is often small and frequently comparable to ordinary iterative refinement. This undermines the hypothesis that lack of accessible information is the main constraint. The paper instead argues that the core difficulty lies in transforming knowledge into testable hypotheses and robust implementations.

The case studies reinforce this interpretation. In a Triton fused-attention-kernel task, one system produces a careful H100-tuned implementation judged strong but not truly Hopper-native, while another offers a more interesting Hopper-specific attempt without a clear breakthrough. In adversarial training and latent normalization tasks, the more distinctive proposals are still assessed as narrow combinations or recombinations of known ideas. In quantization-aware training, a proposed hybrid method is undermined because final evaluation recomputes pure RTN scales, illustrating the importance of matching the evaluation path to the intended contribution. In robot-action diffusion sampling, one submission stays within the narrow edit surface and conducts a principled sweep, while another crosses the spirit of the edit boundary by modifying training behavior in a task meant to be inference-time only. These examples show that success in MLS-Bench depends not only on producing code that differs from baseline code, but on making changes that are methodologically appropriate, scientifically motivated, and evaluation-consistent (Lyu et al., 9 May 2026).

6. Scientific interpretation, limitations, and benchmark significance

The paper’s overarching interpretation is that the discovery bottleneck is scientific judgment. Current agents often recombine baseline ingredients rather than invent new mechanisms; genuinely novel components are rare and often weakly justified; and agents struggle with deciding what to test, what evidence matters, and how to scale a claim. The strongest systems differ in style—GPT-5.4 is described as proposing more structurally different ideas but sometimes overclaiming novelty, while Claude Opus 4.6 is portrayed as more disciplined and often preferring careful tuning—but the general limitation is shared across the frontier models (Lyu et al., 9 May 2026).

Several benchmark design choices reflect this diagnosis. Restricting editable scope, freezing the evaluation harness, preserving baseline ranking across scales, and aggregating settings with a geometric mean all serve the same purpose: preventing local improvement from being mistaken for scientific progress. This suggests that MLS-Bench should be understood not merely as another agent benchmark, but as an attempt to formalize the evidentiary standards of ML method development.

The benchmark also has an infrastructural component. The authors release a community platform with a growing leaderboard and position MLS-Bench as a maintained benchmark for cumulative and comparable iteration. In that respect it is not only a static collection of tasks but a research substrate for longitudinal measurement.

Within the broader benchmark landscape, MLS-Bench occupies a distinct niche. Benchmarks such as DesignBench evaluate MLLMs on front-end engineering workflows including generation, edit, and repair across React, Vue, Angular, and vanilla HTML/CSS (Xiao et al., 6 Jun 2025). MergeBench assesses model merging for domain-specialized LLMs across instruction following, mathematics, multilingual understanding, coding, and safety (2505.10833). BenchML provides an extensible framework for large-scale benchmarking of chemical representations under controlled regression protocols (Poelking et al., 2021), and MLBench evaluates machine learning clouds on binary classification tasks derived from Kaggle competitions (Liu et al., 2017). By contrast, MLS-Bench is explicitly concerned with whether an AI system can improve a targeted component of an ML method in a way that is attributable, generalizable, and scalable. Its object of measurement is therefore not application performance in a fixed domain, but the capacity for ML method discovery itself.

A plausible implication is that MLS-Bench functions as both a diagnostic and a boundary marker. It diagnoses where current agents fail—especially in planning, validation, and scaling of scientific claims—and it delineates a stricter notion of AI research capability than benchmarks that reward competent engineering. In that sense, MLS-Bench operationalizes a demanding criterion for “AI systems on building better AI”: the system must not only modify code successfully, but must produce improvements that survive the standards by which human ML research ordinarily distinguishes method invention from tuning.