MLE-Bench-Lite: A Low-Complexity ML Benchmark

• MLE-Bench-Lite is a curated, low-complexity subset of 22 Kaggle competitions designed to reduce evaluation cost and variance for ML engineering agents.
• It covers diverse tasks including image classification, text normalization, and tabular prediction, ensuring multimodal benchmarking across applications.
• The benchmark employs proxy objectives and cross-validation metrics to facilitate repeated trials and stable comparison, emphasizing efficient evaluation protocols.

MLE-Bench-Lite is a reduced evaluation suite for machine learning engineering agents built from MLE-bench’s Kaggle-based task distribution. In later papers, it is described as the “low complexity” split of MLE-bench, consisting of 22 competitions drawn from the full 75 offline Kaggle competitions, and it is used because evaluating the full benchmark is expensive and because a smaller subset permits more stable comparison through more seeds per task (Nam et al., 27 May 2025, Toledo et al., 3 Jul 2025, Zhang et al., 2 Mar 2026). The original MLE-bench paper introduced only the 75-competition benchmark and did not define a Lite variant, so MLE-Bench-Lite is best understood as a later benchmark subset that emerged in follow-on work rather than as an independently specified artifact in the initial release (Chan et al., 2024).

1. Definition, provenance, and scope

MLE-bench was introduced as an offline benchmark of 75 curated Kaggle competitions for evaluating end-to-end machine learning engineering, including training models, preparing datasets, running experiments, and producing valid submission files (Chan et al., 2024). The original benchmark also annotated task complexity as 22 Low, 38 Medium, and 15 High competitions. Later work identifies MLE-Bench-Lite as a 22-task subset and, more specifically, as the 22 low-complexity tasks used for efficient evaluation (Zhang et al., 2 Mar 2026). This aligns numerically with the original benchmark’s 22 low-complexity competitions (Chan et al., 2024).

The subset is not merely a convenience sample. Subsequent work presents it as a curated benchmark intended to preserve task diversity while making repeated evaluation feasible. One paper describes it as a curated subset of 22 tasks used because MLE-bench is high variance, making it advantageous to spend compute on more seeds per task rather than on all 75 tasks (Toledo et al., 3 Jul 2025). Another describes it as the “low complexity” split chosen because evaluating all 75 competitions is expensive (Nam et al., 27 May 2025).

Property	Value
Full benchmark size	75 Kaggle competitions
Lite benchmark size	22 competitions
Lite characterization	Low-complexity split / curated subset
Original paper status	MLE-bench defined; Lite not separately specified

2. Task inventory and benchmark interface

MLE-Bench-Lite contains the following 22 competitions: aerial-cactus-identification, aptos2019-blindness-detection, denoising-dirty-documents, detecting-insults-in-social-commentary, dog-breed-identification, dogs-vs-cats-redux-kernels-edition, histopathologic-cancer-detection, jigsaw-toxic-comment-classification-challenge, leaf-classification, mlsp-2013-birds, new-york-city-taxi-fare-prediction, nomad2018-predict-transparent-conductors, plant-pathology-2020-fgvc7, random-acts-of-pizza, ranzcr-clip-catheter-line-classification, siim-isic-melanoma-classification, spooky-author-identification, tabular-playground-series-dec-2021, tabular-playground-series-may-2022, text-normalization-challenge-english-language, text-normalization-challenge-russian-language, and the-icml-2013-whale-challenge-right-whale-redux (Nam et al., 27 May 2025).

The 22 competitions span image classification, image-to-image, text classification, audio classification, tabular prediction, and sequence-to-sequence normalization. Reported dataset sizes range from tiny text tasks at about 0.002 GB to large image tasks such as SIIM-ISIC melanoma classification (116.16 GB) (Nam et al., 27 May 2025). This breadth matters because MLE-Bench-Lite is often used as a stand-in for the full benchmark; its value depends on preserving multimodal and multi-objective heterogeneity rather than only reducing runtime.

The interface presented across papers is competition-like rather than question-answering-like. Agents are given the competition description, access to the prepared competition data, and a sandboxed execution environment; they must generate code that trains and evaluates a model and writes a Kaggle-style submission.csv (Toledo et al., 3 Jul 2025). In MLE-STAR, the problem is formalized as a search over executable Python scripts,

$s^{*}=\arg\max_{s\in\mathcal{S}} h(s),$

where $h$ is a task-specific validation metric computed by the generated script itself (Nam et al., 27 May 2025).

3. Evaluation protocol and reported metrics

During search, MLE-Bench-Lite does not expose the hidden Kaggle test score. Instead, agents optimize a proxy objective. One paper states that the generated code must print a 5-fold cross-validation score and use that as fitness during search, formalized as a fitness function $\mathcal{F}:\mathcal{S}\to[0,1]$, although the paper notes that practical scores are not always literally in $[0,1]$ because metrics such as RMSE may be unbounded (Toledo et al., 3 Jul 2025). Another paper evaluates candidate solutions with a task-specific validation metric $h$ produced after the script splits training data into train/validation, then uses a separate test agent $\mathcal{A}_{\mathtt{test}}$ to generate final test-time submission code (Nam et al., 27 May 2025). Together, these descriptions show that MLE-Bench-Lite is fundamentally a hidden-test benchmark driven by internal validation proxies.

The standard success categories reported on MLE-Bench-Lite are made submission, valid submission, above median, bronze, silver, gold, and any medal (Nam et al., 27 May 2025). Another paper emphasizes Medal Success Rate as the main benchmark metric and also reports any medal, silver-or-above, and gold only (Toledo et al., 3 Jul 2025). The use of medal thresholds anchors benchmark performance to Kaggle’s competition structure rather than to a single normalized scalar across tasks.

A shared benchmark constraint is a 24-hour maximum time limit per task (Nam et al., 27 May 2025). Implementations differ in infrastructure and execution policy. One study used 1 NVIDIA H200 GPU, 24 logical CPU cores, 100 GB RAM, and 1 TB scratch storage, with each code execution capped at 4 hours (Toledo et al., 3 Jul 2025). Another used 96 vCPUs, 360 GB RAM, and 8 NVIDIA V100 16GB GPUs for MLE-STAR experiments (Nam et al., 27 May 2025). These are study-level experimental settings rather than a single canonical hardware specification, which is important when comparing results across papers.

4. MLE-Bench-Lite as a testbed for MLE agents

MLE-Bench-Lite rapidly became a primary testbed for autonomous MLE agents. In MLE-STAR, the benchmark is used to evaluate a pipeline that combines web search-based initialization, ablation-guided targeted refinement, and ensemble planning. On the Lite benchmark, MLE-STAR with Gemini-2.5-Pro achieved 100.0% made submission, 100.0% valid submission, 83.3% above median, 6.1% bronze, 21.2% silver, 36.4% gold, and 63.6% any medal; with Gemini-2.0-Flash, it achieved 43.9% any medal (Nam et al., 27 May 2025). The same paper also reports that against AIDE with Gemini-2.0-Flash, MLE-STAR improved any-medal rate from 25.8% to 43.9% and above-median rate from 39.4% to 63.6%.

A different line of work formalizes MLE agents as search policies over artifacts with a fitness function, selection policy, operator set, operator policy, and termination rule, and studies Greedy, MCTS, and Evolutionary search on MLE-Bench-Lite (Toledo et al., 3 Jul 2025). Its best pairing of search strategy and operator set raises medal success from 39.6% to 47.7%, and it reports that operator design and search policy must be considered jointly rather than in isolation. This paper also argues that MLE-Bench-Lite is sufficiently high variance that at least 10, and preferably 20, seeds per task are advisable for reliable comparison.

A later paper, "Reasoning as Gradient: Scaling MLE Agents Beyond Tree Search" (Zhang et al., 2 Mar 2026), uses the Lite benchmark to argue for a different optimization paradigm. Under a closed-world protocol that excludes external retrieval and uses only task-provided materials and execution feedback, Gome achieved 68.2% any-medal on MLE-Bench-Lite with a 12-hour budget on a single NVIDIA V100 GPU. The same paper reports a crossover phenomenon: tree search is stronger for weaker reasoning models, while gradient-style reasoning becomes stronger as model capability increases (Zhang et al., 2 Mar 2026).

These results should not be collapsed into a single leaderboard without qualification. Some studies are open-world, explicitly using web search and external code examples (Nam et al., 27 May 2025); others are closed-world, disallowing external retrieval to isolate architectural effects (Zhang et al., 2 Mar 2026). This suggests that MLE-Bench-Lite functions less as a single monolithic scoreboard than as a common workload on which differing assumptions about search, retrieval, and reasoning can be compared.

5. Methodological significance of the lite format

MLE-Bench-Lite is not only an evaluation subset; it has become a case study in how to design efficient long-horizon benchmarks. One motivation for the Lite split is economic and statistical rather than merely pedagogical: because the full 75-task benchmark is expensive and high variance, the 22-task subset allows more repeated trials per task and therefore more stable model comparison (Toledo et al., 3 Jul 2025). Another paper frames the same subset as an efficient evaluation regime centered on the low-complexity stratum of MLE-bench (Zhang et al., 2 Mar 2026).

A broader efficient-benchmarking literature suggests a second interpretation of benchmark-lite design. "Efficient Benchmarking Is Just Feature Selection and Multiple Regression" argues that efficient benchmarking can be formalized as selecting a subset of benchmark items and learning a predictor of the full benchmark score, with mRMR for subset selection and kernel ridge regression as a strong default on binary-like benchmarks (Bowyer et al., 25 May 2026). This does not describe how MLE-Bench-Lite was originally defined, but it suggests an alternative methodology for future lite variants: a lite benchmark can be treated as a predictive coreset rather than only as a manually chosen easier subset.

"MINCE: Shrinking LLM Evaluation Datasets via Few-Model Monte Carlo Calibration" pushes the same design question in a different direction (Das et al., 22 Jun 2026). MINCE chooses subset size by Monte Carlo simulation over calibration-model item logs, then freezes a static subset at the smallest size where further growth yields less than 1 percentage point marginal improvement in the 95th-percentile drift statistic. This again is not a definition of MLE-Bench-Lite itself, but it suggests that future lite variants could be constructed by explicit score-drift calibration rather than only by manual curation.

Related benchmark work reinforces why MLE-Bench-Lite matters. ALE-Bench positions itself alongside MLE-Bench as a score-based long-horizon benchmark that captures iterative improvement rather than one-shot correctness, while noting that ALE-Bench is CPU-only and comparatively resource-friendly whereas MLE-Bench uses Kaggle-style ML tasks and requires a GPU environment (Imajuku et al., 10 Jun 2025). The shared design axis is the evaluation of agents under realistic refinement loops, hidden objectives, and nontrivial engineering budgets.

6. Reliability, validity, and unresolved issues

A central reliability issue for MLE-Bench-Lite is the generalization gap between proxy validation and hidden test performance. The search-policy paper reports that validation-guided search systematically overfits: replacing validation with oracle test information improves medal rate by 9.4% or 12.4% for two non-greedy agents and by about 15% or 16.6% for two greedy agents, while oracle final-node selection from the already generated graph yields gains of 9 to 11.6 absolute points (Toledo et al., 3 Jul 2025). This is not a defect specific to the Lite split, but the smaller benchmark makes such variance and proxy mismatch easier to expose.

Contamination is another recurring concern because the tasks are drawn from public Kaggle competitions. The original MLE-bench paper explicitly treats contamination as a structural risk, yet reports no evidence that contamination systematically inflated GPT-4o scores: it found no correlation between GPT-4o familiarity with competition materials and benchmark performance, and an experiment with obfuscated competition descriptions produced 8.5 ± 0.6% any-medal for the original descriptions versus 8.4 ± 1.0% for the obfuscated versions (Chan et al., 2024). Later papers remain more cautious. One notes that Kaggle competitions are public and that contamination is therefore plausible, while another reports only a suggestive novelty analysis in which an LLM judge deemed evaluated MLE-STAR solutions “sufficiently novel” relative to top Kaggle discussions from seven competitions (Toledo et al., 3 Jul 2025, Nam et al., 27 May 2025). The controversy is therefore unresolved rather than settled.

A third issue is benchmark auditing. Although UTBoost studies SWE-Bench rather than MLE-Bench-Lite, it argues that lite subsets can amplify the effect of under-specified tests or evaluator defects if they are not independently stress-tested (Yu et al., 10 Jun 2025). This suggests that MLE-Bench-Lite, despite being a competition benchmark rather than a unit-test benchmark, should likewise be treated as an instrument that requires ongoing audit of hidden labels, evaluation scripts, and leaderboard stability rather than as a permanently fixed gold standard.

Finally, Lite is not a synonym for representative completeness. Because later work defines it as the 22 low-complexity tasks (Zhang et al., 2 Mar 2026), it emphasizes the part of the MLE-bench distribution where short-horizon reasoning and efficient experimentation are comparatively advantaged. A plausible implication is that MLE-Bench-Lite is especially useful for rapid iteration, ablation, and architecture comparison, but it cannot on its own characterize performance on the medium- and high-complexity portions of the full 75-task benchmark.