---
title: 'MLE-Bench-Lite: A Low-Complexity ML Benchmark'
url: https://www.emergentmind.com/topics/mle-bench-lite
type: topic
---

# MLE-Bench-Lite: A Low-Complexity ML Benchmark

MLE-Bench-Lite is a reduced evaluation suite for machine learning engineering agents built from MLE-bench’s Kaggle-based task distribution. In later papers, it is described as the **“low complexity” split of MLE-bench**, consisting of **22 competitions** drawn from the full **75** offline Kaggle competitions, and it is used because evaluating the full benchmark is expensive and because a smaller subset permits more stable comparison through more seeds per task [2506.15692, 2507.02554, 2603.01692]. The original MLE-bench paper introduced only the 75-competition benchmark and did **not** define a Lite variant, so MLE-Bench-Lite is best understood as a later benchmark subset that emerged in follow-on work rather than as an independently specified artifact in the initial release [2410.07095].

## 1. Definition, provenance, and scope

MLE-bench was introduced as an offline benchmark of **75 curated Kaggle competitions** for evaluating end-to-end machine learning engineering, including training models, preparing datasets, running experiments, and producing valid submission files [2410.07095]. The original benchmark also annotated task complexity as **22 Low**, **38 Medium**, and **15 High** competitions. Later work identifies MLE-Bench-Lite as a **22-task** subset and, more specifically, as the **22 low-complexity tasks** used for efficient evaluation [2603.01692]. This aligns numerically with the original benchmark’s 22 low-complexity competitions [2410.07095].

The subset is not merely a convenience sample. Subsequent work presents it as a curated benchmark intended to preserve task diversity while making repeated evaluation feasible. One paper describes it as a **curated subset of 22 tasks** used because MLE-bench is **high variance**, making it advantageous to spend compute on **more seeds per task** rather than on all 75 tasks [2507.02554]. Another describes it as the **“low complexity” split** chosen because evaluating all 75 competitions is expensive [2506.15692].

| Property | Value |
|---|---|
| Full benchmark size | 75 Kaggle competitions |
| Lite benchmark size | 22 competitions |
| Lite characterization | Low-complexity split / curated subset |
| Original paper status | MLE-bench defined; Lite not separately specified |

## 2. Task inventory and benchmark interface

MLE-Bench-Lite contains the following **22 competitions**: **aerial-cactus-identification**, **aptos2019-blindness-detection**, **denoising-dirty-documents**, **detecting-insults-in-social-commentary**, **dog-breed-identification**, **dogs-vs-cats-redux-kernels-edition**, **histopathologic-cancer-detection**, **jigsaw-toxic-comment-classification-challenge**, **leaf-classification**, **mlsp-2013-birds**, **new-york-city-taxi-fare-prediction**, **nomad2018-predict-transparent-conductors**, **plant-pathology-2020-fgvc7**, **random-acts-of-pizza**, **ranzcr-clip-catheter-line-classification**, **siim-isic-melanoma-classification**, **spooky-author-identification**, **tabular-playground-series-dec-2021**, **tabular-playground-series-may-2022**, **text-normalization-challenge-english-language**, **text-normalization-challenge-russian-language**, and **the-icml-2013-whale-challenge-right-whale-redux** [2506.15692].

The 22 competitions span **image classification**, **image-to-image**, **text classification**, **audio classification**, **tabular prediction**, and **sequence-to-sequence normalization**. Reported dataset sizes range from tiny text tasks at about **0.002 GB** to large image tasks such as **SIIM-ISIC melanoma classification (116.16 GB)** [2506.15692]. This breadth matters because MLE-Bench-Lite is often used as a stand-in for the full benchmark; its value depends on preserving multimodal and multi-objective heterogeneity rather than only reducing runtime.

The interface presented across papers is competition-like rather than question-answering-like. Agents are given the **competition description**, access to the **prepared competition data**, and a **sandboxed execution environment**; they must generate code that trains and evaluates a model and writes a Kaggle-style **`submission.csv`** [2507.02554]. In MLE-STAR, the problem is formalized as a search over executable Python scripts,
$$
s^{*}=\arg\max_{s\in\mathcal{S}} h(s),
$$
where \(h\) is a task-specific validation metric computed by the generated script itself [2506.15692].

## 3. Evaluation protocol and reported metrics

During search, MLE-Bench-Lite does not expose the hidden Kaggle test score. Instead, agents optimize a proxy objective. One paper states that the generated code must print a **5-fold cross-validation** score and use that as fitness during search, formalized as a fitness function \(\mathcal{F}:\mathcal{S}\to[0,1]\), although the paper notes that practical scores are not always literally in \([0,1]\) because metrics such as RMSE may be unbounded [2507.02554]. Another paper evaluates candidate solutions with a task-specific validation metric \(h\) produced after the script splits training data into train/validation, then uses a separate test agent \(\mathcal{A}_{\mathtt{test}}\) to generate final test-time submission code [2506.15692]. Together, these descriptions show that MLE-Bench-Lite is fundamentally a hidden-test benchmark driven by internal validation proxies.

The standard success categories reported on MLE-Bench-Lite are **made submission**, **valid submission**, **above median**, **bronze**, **silver**, **gold**, and **any medal** [2506.15692]. Another paper emphasizes **Medal Success Rate** as the main benchmark metric and also reports **any medal**, **silver-or-above**, and **gold only** [2507.02554]. The use of medal thresholds anchors benchmark performance to Kaggle’s competition structure rather than to a single normalized scalar across tasks.

A shared benchmark constraint is a **24-hour maximum time limit per task** [2506.15692]. Implementations differ in infrastructure and execution policy. One study used **1 NVIDIA H200 GPU**, **24 logical CPU cores**, **100 GB RAM**, and **1 TB scratch storage**, with each code execution capped at **4 hours** [2507.02554]. Another used **96 vCPUs**, **360 GB RAM**, and **8 NVIDIA V100 16GB GPUs** for MLE-STAR experiments [2506.15692]. These are study-level experimental settings rather than a single canonical hardware specification, which is important when comparing results across papers.

## 4. MLE-Bench-Lite as a testbed for MLE agents

MLE-Bench-Lite rapidly became a primary testbed for autonomous MLE agents. In MLE-STAR, the benchmark is used to evaluate a pipeline that combines **web search-based initialization**, **ablation-guided targeted refinement**, and **ensemble planning**. On the Lite benchmark, MLE-STAR with **Gemini-2.5-Pro** achieved **100.0% made submission**, **100.0% valid submission**, **83.3% above median**, **6.1% bronze**, **21.2% silver**, **36.4% gold**, and **63.6% any medal**; with **Gemini-2.0-Flash**, it achieved **43.9% any medal** [2506.15692]. The same paper also reports that against **AIDE with Gemini-2.0-Flash**, MLE-STAR improved any-medal rate from **25.8%** to **43.9%** and above-median rate from **39.4%** to **63.6%**.

A different line of work formalizes MLE agents as search policies over artifacts with a fitness function, selection policy, operator set, operator policy, and termination rule, and studies **Greedy**, **MCTS**, and **Evolutionary** search on MLE-Bench-Lite [2507.02554]. Its best pairing of search strategy and operator set raises medal success from **39.6%** to **47.7%**, and it reports that **operator design** and **search policy** must be considered jointly rather than in isolation. This paper also argues that MLE-Bench-Lite is sufficiently high variance that **at least 10**, and preferably **20**, seeds per task are advisable for reliable comparison.

A later paper, "Reasoning as Gradient: Scaling MLE Agents Beyond Tree Search" [2603.01692], uses the Lite benchmark to argue for a different optimization paradigm. Under a **closed-world protocol** that excludes external retrieval and uses only task-provided materials and execution feedback, **Gome** achieved **68.2% any-medal** on MLE-Bench-Lite with a **12-hour** budget on a **single NVIDIA V100 GPU**. The same paper reports a crossover phenomenon: **tree search** is stronger for weaker reasoning models, while **gradient-style reasoning** becomes stronger as model capability increases [2603.01692].

These results should not be collapsed into a single leaderboard without qualification. Some studies are **open-world**, explicitly using web search and external code examples [2506.15692]; others are **closed-world**, disallowing external retrieval to isolate architectural effects [2603.01692]. This suggests that MLE-Bench-Lite functions less as a single monolithic scoreboard than as a common workload on which differing assumptions about search, retrieval, and reasoning can be compared.

## 5. Methodological significance of the lite format

MLE-Bench-Lite is not only an evaluation subset; it has become a case study in how to design efficient long-horizon benchmarks. One motivation for the Lite split is economic and statistical rather than merely pedagogical: because the full 75-task benchmark is expensive and high variance, the 22-task subset allows more repeated trials per task and therefore more stable model comparison [2507.02554]. Another paper frames the same subset as an efficient evaluation regime centered on the low-complexity stratum of MLE-bench [2603.01692].

A broader efficient-benchmarking literature suggests a second interpretation of benchmark-lite design. "Efficient Benchmarking Is Just Feature Selection and Multiple Regression" argues that efficient benchmarking can be formalized as selecting a subset of benchmark items and learning a predictor of the full benchmark score, with **mRMR** for subset selection and **kernel ridge regression** as a strong default on binary-like benchmarks [2605.25773]. This does **not** describe how MLE-Bench-Lite was originally defined, but it suggests an alternative methodology for future lite variants: a lite benchmark can be treated as a predictive coreset rather than only as a manually chosen easier subset.

"MINCE: Shrinking LLM Evaluation Datasets via Few-Model Monte Carlo Calibration" pushes the same design question in a different direction [2606.22826]. MINCE chooses subset size by Monte Carlo simulation over calibration-model item logs, then freezes a static subset at the smallest size where further growth yields less than **1 percentage point** marginal improvement in the **95th-percentile** drift statistic. This again is not a definition of MLE-Bench-Lite itself, but it suggests that future lite variants could be constructed by explicit score-drift calibration rather than only by manual curation.

Related benchmark work reinforces why MLE-Bench-Lite matters. ALE-Bench positions itself alongside MLE-Bench as a **score-based** long-horizon benchmark that captures iterative improvement rather than one-shot correctness, while noting that ALE-Bench is **CPU-only** and comparatively resource-friendly whereas MLE-Bench uses Kaggle-style ML tasks and requires a **GPU environment** [2506.09050]. The shared design axis is the evaluation of agents under realistic refinement loops, hidden objectives, and nontrivial engineering budgets.

## 6. Reliability, validity, and unresolved issues

A central reliability issue for MLE-Bench-Lite is the **generalization gap** between proxy validation and hidden test performance. The search-policy paper reports that validation-guided search systematically overfits: replacing validation with oracle test information improves medal rate by **9.4%** or **12.4%** for two non-greedy agents and by about **15%** or **16.6%** for two greedy agents, while oracle final-node selection from the already generated graph yields gains of **9 to 11.6 absolute points** [2507.02554]. This is not a defect specific to the Lite split, but the smaller benchmark makes such variance and proxy mismatch easier to expose.

Contamination is another recurring concern because the tasks are drawn from **public Kaggle competitions**. The original MLE-bench paper explicitly treats contamination as a structural risk, yet reports **no evidence** that contamination systematically inflated **GPT-4o** scores: it found **no correlation** between GPT-4o familiarity with competition materials and benchmark performance, and an experiment with **obfuscated competition descriptions** produced **8.5 ± 0.6%** any-medal for the original descriptions versus **8.4 ± 1.0%** for the obfuscated versions [2410.07095]. Later papers remain more cautious. One notes that Kaggle competitions are public and that contamination is therefore plausible, while another reports only a suggestive novelty analysis in which an LLM judge deemed evaluated MLE-STAR solutions “sufficiently novel” relative to top Kaggle discussions from seven competitions [2507.02554, 2506.15692]. The controversy is therefore unresolved rather than settled.

A third issue is benchmark auditing. Although UTBoost studies **SWE-Bench** rather than MLE-Bench-Lite, it argues that lite subsets can amplify the effect of under-specified tests or evaluator defects if they are not independently stress-tested [2506.09289]. This suggests that MLE-Bench-Lite, despite being a competition benchmark rather than a unit-test benchmark, should likewise be treated as an instrument that requires ongoing audit of hidden labels, evaluation scripts, and leaderboard stability rather than as a permanently fixed gold standard.

Finally, Lite is not a synonym for representative completeness. Because later work defines it as the **22 low-complexity tasks** [2603.01692], it emphasizes the part of the MLE-bench distribution where short-horizon reasoning and efficient experimentation are comparatively advantaged. A plausible implication is that MLE-Bench-Lite is especially useful for rapid iteration, ablation, and architecture comparison, but it cannot on its own characterize performance on the medium- and high-complexity portions of the full 75-task benchmark.

Source: https://www.emergentmind.com/topics/mle-bench-lite