MLAgentBench: Benchmark for ML Experimentation Agents
- MLAgentBench is a 13-task benchmark designed to test if language agents can perform full ML experimentation, including reading, coding, and iterative improvements.
- It utilizes executable workspaces with starter files and evaluators to measure performance improvements, such as a minimum 10% gain over a baseline.
- The benchmark clearly differentiates itself from similar evaluations like ML-Bench and AgentBench, emphasizing precise task scopes and evaluation metrics.
Searching arXiv for the benchmark and closely related follow-up work to ground the article in current literature. MLAgentBench is a benchmark for evaluating whether language-model agents can conduct machine learning experimentation end to end: reading a problem, inspecting starter code and data, designing and running experiments, interpreting results, iterating on code, and producing a valid submission or checkpoint. It was introduced as a suite of 13 tasks spanning canonical ML problems, Kaggle-style tasks, recent research problems, and code-improvement tasks, with executable workspaces and automatic evaluation rather than subjective judging (Huang et al., 2023). In later literature, the name has remained associated with agentic ML experimentation, but it must be distinguished from the separate repository-level benchmark ML-Bench and its autonomous baseline ML-Agent, which are sometimes informally conflated under the label “ML-Agent-Bench” even though they are not the same benchmark (Tang et al., 2023).
1. Definition and nomenclature
MLAgentBench was introduced to measure a specific capability: open-ended machine learning experimentation by language agents. Its motivating premise is that ML research is not a single inference step but an iterative process involving prior knowledge, code writing, debugging, result interpretation, and revision. The benchmark therefore asks whether an LM-driven agent can operate in a realistic coding environment and improve a baseline system rather than merely answer a question about ML (Huang et al., 2023).
The name is close to several adjacent benchmarks, and the distinctions are technically important. MLAgentBench is the 13-task experimentation benchmark introduced in "MLAgentBench: Evaluating Language Agents on Machine Learning Experimentation" (Huang et al., 2023). By contrast, ML-Bench is a benchmark for repository-level machine learning code generation that evaluates models on using existing open-source repositories and documentation, and ML-Agent is its autonomous baseline for retrieving, reading, planning, and generating code across repositories (Tang et al., 2023). AgentBench is a broader benchmark for LLMs as agents across 8 code-, game-, and web-grounded environments, rather than an ML experimentation benchmark specifically (Liu et al., 2023).
This terminological separation matters because the benchmarks operationalize different problem classes. MLAgentBench centers on experimentation loops inside a task workspace with starter files and evaluators. ML-Bench centers on repository usage and executable code generation from documentation. AgentBench centers on general multi-turn action selection in interactive environments. A plausible implication is that results are not directly interchangeable across these settings, even when all three are described as evaluating “agents.”
2. Benchmark composition and task suite
MLAgentBench contains 13 tasks organized into five categories: Canonical Tasks, Classic Kaggle, Kaggle Challenges, Recent Research, and Code Improvement (Huang et al., 2023). The suite deliberately mixes old, well-understood datasets with newer or more open-ended challenges in order to test generalization and reduce contamination. The tasks span image classification, sentiment classification, node classification, regression, recent Kaggle challenges, research problems such as BabyLM and CLRS, and performance-oriented code-improvement tasks.
| Category | Tasks | Metric or artifact |
|---|---|---|
| Canonical Tasks | CIFAR-10, IMDB, ogbn-arxiv | Classification accuracy |
| Classic Kaggle | house-price, spaceship-titanic | Mean absolute error; classification accuracy |
| Kaggle Challenges | fathomnet, feedback, identify-contrails | SMAPE; MAP@20; Dice coefficient |
| Recent Research | CLRS, BabyLM, Parkinson’s disease progression prediction | Mean square error; perplexity; MCRMSE |
| Code Improvement | llama-inference, vectorization | Wall-clock time |
Each benchmark instance is defined by three elements: a textual task description, starter files, and an evaluator (Huang et al., 2023). The task description specifies the goal and submission format, sometimes with constraints such as training within a limited epoch budget or with hints such as fine-tuning BERT. The starter files include data, data descriptions, metric descriptions, and starter code. The evaluator maps the final submission or checkpoint to a scalar performance score. For many tasks the final artifact is a submission.csv; for CLRS and BabyLM the evaluator instead checks saved checkpoints directly.
The benchmark’s design choice is executable containment. Tasks are meant to be automatically evaluable, and most are designed to run in minutes so that multiple agent trajectories can be compared economically. This gives MLAgentBench a position between toy code-generation benchmarks and fully open-ended research settings: it is broad enough to require planning and iteration, but constrained enough to support repeated, standardized evaluation.
3. Agent interface, action space, and formal interaction model
MLAgentBench places the agent in a general-purpose workspace with allowed file-system and execution actions. The primitive actions are List Files, Read File, Write File, Append File, Copy File, Inspect Script Lines, Undo Edit Script, Execute Script, and Final Answer. The compound actions are Understand File, Edit Script, and Edit Script Segment (Huang et al., 2023). Understand File reads a file and uses an LM to summarize it with respect to a query, returning relevant content and line references. Edit Script and Edit Script Segment support coherent code modification, with the line-bounded variant intended for large codebases such as CLRS and BabyLM.
The benchmark’s reference agent is constructed in the ReAct style and is explicitly organized around “thinking before acting.” At time step , the agent receives a prompt summarizing the task, available tools, and recent history, and it produces both a rationale and an action. The interaction is formalized as
Here is the workspace state, is memory, is the observation returned by the environment, and the memory update incorporates the previous memory, the chosen action, the rationale, and the resulting observation (Huang et al., 2023).
The prompt includes the last three steps of rationale, action, and observation, together with explicit output fields: Reflection, Research Plan and Status, Fact Check, Thought, Action, and Action Input (Huang et al., 2023). Reflection interprets the previous step. Research Plan and Status serves as a persistent, human-readable record of what has been done and what remains. Fact Check is intended to reduce hallucination by forcing the model to verify whether statements are supported by observed outputs. Thought corresponds to pre-action reasoning in the ReAct style. This structure is not merely cosmetic: it provides interpretable traces and a mechanism for human intervention or correction.
4. Evaluation protocol, metrics, and empirical behavior
MLAgentBench evaluates both competence and efficiency. Competence is determined by the final scalar score returned by the task evaluator after the agent finishes, and the benchmark’s main success criterion is whether the agent improves the performance metric by at least 10% over the baseline in the starter code. Success rate is the fraction of runs satisfying that threshold. The benchmark also reports average improvement among runs that end with a valid submission. Efficiency is measured by total wall-clock time and total prompt/completion token consumption (Huang et al., 2023).
The main experiments run 8 trials per agent per task, with a maximum of 50 actions and 5 hours per run for most models; GPT-4 runs are limited to 30 actions because of API cost (Huang et al., 2023). The evaluated agent backends are GPT-4 (0613), GPT-4-turbo (0125), Claude v1.0, Claude v2.1, Claude v3 Opus, Gemini Pro, and Mixtral. The best average success rate is achieved by the Claude v3 Opus agent at 37.5%. GPT-4-turbo and Claude v2.1 both obtain 26.0%, followed by GPT-4 at 19.2%, Gemini Pro at 18.3%, Claude v1.0 at 16.3%, and Mixtral at 3.8% (Huang et al., 2023).
The benchmark distinguishes reliability from magnitude of improvement. In the reported average percentage improvement among successful submissions, GPT-4 has the highest overall value at 41.3%, followed by GPT-4-turbo at 32.8%, Claude v3 Opus at 26.1%, Claude v2.1 at 15.0%, Claude v1.0 at 8.9%, Mixtral at 8.0%, and Gemini Pro at % (Huang et al., 2023). This indicates that GPT-4 can make large gains when it succeeds, whereas Claude v3 Opus is more reliable at crossing the benchmark’s 10% threshold.
Per-task behavior varies sharply. Claude v3 Opus reaches 100% success on house-price and spaceship-titanic, and 87.5% on ogbn-arxiv and feedback, but 0% on Parkinson’s disease, fathomnet, vectorization, and BabyLM (Huang et al., 2023). The paper attributes part of this spread to task recency and dataset familiarity: older established datasets are often easier, whereas recent Kaggle or research tasks can remain unsolved.
A reproduced CIFAR-10 trajectory illustrates the intended benchmark behavior. The agent begins from the baseline train.py, establishes a 52.53% test accuracy after 5 epochs, tries a learning rate of 0.3 that collapses performance to about 10% accuracy, then tries dropout and obtains 49.34%, and finally increases the convolutional width from 6/16 channels to 32/64 channels, achieving 64.31% after 5 epochs and thereby surpassing the 10% relative improvement threshold (Huang et al., 2023). The example is significant because it shows the benchmark testing experiment design and recovery from failed trials, not just one-shot code synthesis.
5. Failure modes and methodological challenges
MLAgentBench exposes several recurrent failure modes. Hallucination is a major one: agents may claim to have improved performance without having run the edited code, or may assert gains contradicted by visible trace information. Response-format errors also occur, including invalid JSON that cannot be parsed. Submission-format errors are another source of failure when the agent alters the required submission.csv structure. Additional failure classes include bad plans, which commit early to unproductive modifications, and small-improvement cases, where the agent makes progress but not enough to satisfy the 10% success threshold (Huang et al., 2023).
Long-horizon control is a persistent challenge. Good trajectories tend to alternate between editing and executing, whereas poor ones drift into irrelevant actions or become trapped in debugging loops. The paper notes that as step counts increase, performance can regress; among the models studied, only Claude v3 Opus consistently maintains or improves performance over longer episodes in the plotted examples (Huang et al., 2023). This suggests that the principal bottleneck is not only code generation quality but also stable experiment management over many turns.
Task-specification sensitivity is another methodological issue. In a Parkinson’s disease example, the agent misunderstood the metric direction and tried to increase SMAPE even though lower is better (Huang et al., 2023). This is a narrow but important failure: it shows that success depends on metric semantics being interpreted correctly, which is separate from the ability to edit or execute code.
Later work using MLAgentBench as a case study sharpened this diagnosis by treating the benchmark as a learning loop for iterative generative optimization. "Understanding the Challenges in Iterative Generative Optimization with LLMs" studies two Kaggle tasks from MLAgentBench—Housing Price and Spaceship Titanic—and shows that changing only the initial program decomposition can change which solutions are reachable (Nie et al., 25 Mar 2026). The paper compares a one-function workflow, in which a single monolithic train_model function is editable, with a many-function workflow split into preprocess, select_features, train_model, and predict. Using OptoPrime with Claude Sonnet-3.5-v2, an 80/20 train-validation split, 20 optimization steps, memory size 5, and 5 runs per configuration, it finds that the better initialization depends on the task: the one-function version is better on Housing Price, whereas the many-function version is better in the best run on Spaceship Titanic (Nie et al., 25 Mar 2026). The same study identifies meta-overfitting, where code revisions improve internal validation performance yet generalize poorly to Kaggle test data. A plausible implication is that MLAgentBench is not only a benchmark of agent competence but also a sensitive probe of how the optimization loop itself is engineered.
6. Subsequent systems, related benchmarks, and broader significance
MLAgentBench has been used as an evaluation environment for later agent architectures. "BudgetMLAgent: A Cost-Effective LLM Multi-Agent system for Automating Machine Learning Tasks" treats MLAgentBench as a benchmark of practical ML engineering work in code and compares single-agent systems against a cost-aware multi-agent system with role specialization, retrieval of past observations, LLM cascades, and an “ask-the-expert” mechanism (Gandhi et al., 2024). In that system, a planner agent reasons over the current task, recent history, and retrieved long-term observations, while worker agents execute actions such as understanding a file, editing a script, reflecting, listing files, or running code. The base LLM is Gemini Pro, with GPT-3.5 or GPT-4 used in cascade or expert modes under a bounded GPT-4 call budget 0. On a selected subset of MLAgentBench tasks, the configuration with profiling plus cascade plus expert reaches a 32.95% average success rate at an average cost of \$r_t, a_t = \text{Agent}(S_{t-1}, m_{t-1}),$10.931 for the GPT-4 single-agent baseline, corresponding to a reported 94.2% reduction in average cost (Gandhi et al., 2024). This suggests that MLAgentBench is also useful for studying capability-cost tradeoffs, not only raw task completion.
Subsequent benchmark work has positioned MLAgentBench as a reference point for more specialized or more difficult settings. The ML Research Benchmark argues that MLAgentBench and ML-Bench are useful for general ML experimentation but are centered on more standard tasks such as classification, segmentation, or classic Kaggle-style problems; it introduces a 7-task competition-level research benchmark intended to probe stronger forms of AI research capability (Kenney, 2024). BioXArena, in turn, adapts the same broad philosophy to biomedical machine learning by requiring agents to write executable code, train predictive models, and submit held-out predictions across 76 tasks and 9 biomedical domains in a 2-hour single-GPU environment (Li et al., 15 May 2026). These developments indicate that MLAgentBench has become a methodological template for end-to-end ML agent evaluation.
The benchmark’s broader significance lies in how it operationalizes ML work. It does not ask whether a model can write an isolated function, nor whether it can navigate a generic interactive environment, nor whether it can simply retrieve repository code. Instead, it measures whether an agent can manage an iterative experimental process in which planning, debugging, metric interpretation, file handling, execution, and submission correctness all matter simultaneously (Huang et al., 2023). That framing helps explain both its utility and its difficulty: the benchmark captures a realistic subset of ML practice, but it also exposes how brittle current agents remain with respect to long-term planning, hallucination, format adherence, metric semantics, and task recency.