---
title: 'TAM Bench: ML Agent Benchmark'
url: https://www.emergentmind.com/topics/tam-bench
type: topic
---

# TAM Bench: ML Agent Benchmark

TAM Bench is a benchmark for evaluating **general-purpose ML agents**, especially **LLM-based agents** that attempt to automate the full machine learning workflow: understanding a task, reading data descriptions, writing code, training models, generating predictions, and producing a valid competition-style submission [2509.09321]. It was introduced to address limitations in existing benchmarks for end-to-end AutoML and data-science systems, particularly limited task coverage, weak difficulty modeling, and evaluation schemes centered too narrowly on raw leaderboard performance. The benchmark contains **150 curated AutoML tasks** and is organized around three design elements: **automated task acquisition and standardization**, **leaderboard-driven difficulty modeling**, and **multi-dimensional evaluation** [2509.09321].

## 1. Benchmark objective and research setting

TAM Bench targets **realistic, complete ML challenges** rather than isolated subtasks such as code generation, feature engineering in isolation, or narrow tabular prediction [2509.09321]. The evaluated agent is expected to read documentation, understand datasets, perform data analysis, tune or train models, generate code, generate a valid `submission.csv`, and provide a `best_solution.py`. This makes the benchmark explicitly end-to-end in scope.

The benchmark is framed as a response to four limitations in earlier evaluation practice. First, benchmark construction has often required substantial manual labor. Second, prior benchmarks have had narrow or imbalanced task distributions. Third, difficulty labels have often been subjective or absent. Fourth, single-dimensional evaluation based only on task score or medal status can obscure whether an agent actually satisfies task-specific requirements or even produces a valid deliverable [2509.09321]. In this setting, TAM Bench is intended to evaluate whether an ML agent can solve competition-style tasks across multiple domains, modalities, and difficulty levels while obeying output and instruction constraints.

The benchmark is also positioned against specific predecessors. It is described as broader than **MLEBench** in task count and source diversity, more difficulty-aware than **MLAgentBench**, and less domain-restricted than **CALM**, which focuses on tabular finance tasks [2509.09321]. This suggests a benchmark philosophy centered on **generalist ML agency** rather than performance on a single modality or vertical.

## 2. Automated construction pipeline

A central contribution of TAM Bench is its **automated benchmark construction pipeline**, which combines **browser automation** and **LLM-based structuring** to acquire tasks from competition platforms such as **Kaggle**, **AIcrowd**, and **Biendata** [2509.09321]. The paper describes the overall process as: acquire task pages, extract clean task content and leaderboard data, filter tasks, normalize them into a structured schema, reconstruct evaluation datasets, assign difficulty levels, and assemble benchmark subsets.

The acquisition system is inspired by **Model Context Protocol (MCP)** and uses **Browser-Use** for browser control. Its architecture has four layers: an **Agent Layer** based on **LangChain’s ReAct architecture**, a **Controller Layer** that converts agent decisions into browser actions, a **DOM Layer** that parses webpage structure into AI-friendly text, and a **Browser Layer** using **Playwright** for browser control and multi-context management [2509.09321]. The controller is enhanced with an **`extract_markdown`** tool to remove irrelevant page elements such as ads and navigation bars while preserving task-relevant text.

After acquisition, TAM Bench applies a filtering pipeline. The listed rules exclude tasks before **2023**, remove tasks without publicly available datasets, discard tasks lacking ground-truth test labels or reproducible splits, and exclude tasks whose official scoring procedure cannot be replicated [2509.09321]. The 2023 cutoff is explicitly motivated as a way to reduce contamination from model pretraining corpora.

Normalization into a unified schema is performed with **GPT-4o**. The schema includes fields such as `task_type`, `goal_description`, `metric`, `target_col`, `data_information`, `output_format`, and `special_instructions` [2509.09321]. The inclusion of `special_instructions` is significant because those instructions are later used in constraint compliance evaluation, making the schema part of the benchmark’s scoring logic rather than only a documentation layer.

## 3. Coverage, subsets, and task organization

TAM Bench covers six modalities: **Tabular**, **Text**, **Image**, **Audio**, **Graph**, and **Multimodal** [2509.09321]. It also aims to broaden application coverage beyond typical benchmark staples, explicitly mentioning areas such as **financial risk control**, **medical diagnosis**, **e-commerce recommendation**, **bioinformatics**, and graph-based tasks. The paper emphasizes both modality coverage and domain coverage, though the main text does not provide a single consolidated count table by domain.

The benchmark is released in three scales:

| Subset | Tasks | Intended use |
|---|---:|---|
| Lite | 18 | Routine benchmarking and comparative studies |
| Medium | 54 | Broader evaluation at moderate cost |
| Full | 150 | Comprehensive evaluation |

The **Lite** subset is deliberately balanced: it contains **6 modalities**, with **3 tasks per modality**, specifically **one easy**, **one medium**, and **one hard** task per modality [2509.09321]. This construction yields exactly \(6 \times 3 = 18\) tasks and is meant to provide a compact but representative testbed for everyday benchmarking. The paper gives the full Lite task list, including, for example, `new-york-city-taxi-fare-prediction` in tabular data, `lmsys-chatbot-arena` in text, `statoil-iceberg-classifier-challenge` in image, `tensorflow-speech-recognition-challenge` in audio, `WhoIsWho-IND` in graph data, and `planttraits2024` in multimodal evaluation [2509.09321].

A practical issue in competition-derived benchmarking is that hidden test labels are often unavailable. TAM Bench addresses this by reconstructing evaluation datasets from the original training data, partitioning them into new train/test subsets, and then scoring model outputs against the locally reconstructed held-out labels [2509.09321]. The paper states that scores produced by this local evaluation pipeline are **nearly identical** to scores from the competition’s official evaluation system when checked using the provided `sample_submission.csv`. This suggests that the reconstructed evaluation is intended as a faithful offline proxy, though the paper does not report exact discrepancy values.

## 4. Difficulty modeling

A distinctive feature of TAM Bench is its automated **leaderboard-driven difficulty model** [2509.09321]. Rather than relying on manual human estimates, the benchmark derives task difficulty from **Mean score across participants**, **Best score**, and **Participant count**. This is presented as an objective and scalable way to compare heterogeneous tasks.

Because competitions use different metrics, the benchmark first normalizes scores. For the mean and best leaderboard values, the paper gives the following formulas:
\[
\text{NormMean} = \frac{\text{Mean Score} - \text{Min Score}}{\text{Max Score} - \text{Min Score} + \epsilon}
\]
\[
\text{NormBest} = \frac{\text{Best Score} - \text{Min Score}}{\text{Max Score} - \text{Min Score} + \epsilon}
\]
For **unbounded metrics**, \(\text{Max Score}\) and \(\text{Min Score}\) are set to the maximum and minimum observed on the private leaderboard [2509.09321].

For **higher-is-better** metrics such as accuracy or AUC, the difficulty score is:
\[
\text{Difficulty Score} =
w_1 \cdot (1 - \text{NormMean})
+ w_2 \cdot \log_{10}(\text{Participants} + 1)
+ w_3 \cdot (1 - \text{NormBest})
\]

For **lower-is-better** metrics such as loss or RMSE, the score is:
\[
\text{Difficulty Score} =
w_1 \cdot \text{NormMean}
+ w_2 \cdot \log_{10}(\text{Participants} + 1)
+ w_3 \cdot \text{NormBest}
\]

The weights are fixed to:
\[
w_1 = 0.4,\quad w_2 = 0.1,\quad w_3 = 0.5
\]
and tasks are binned as:
- **Easy**: \(\text{Difficulty Score} \le 0.6\)
- **Medium**: \(0.6 < \text{Difficulty Score} \le 0.85\)
- **Hard**: \(\text{Difficulty Score} > 0.85\) [2509.09321]

The benchmark validates this automatic labeling against MLEBench’s manual difficulty assignments. The reported outcome is that tasks labeled **Hard** in MLEBench consistently map to **Medium or Hard** in TAM Bench, and that **no manually difficult task** is misclassified as **Easy** [2509.09321]. The appendix also reports that **24 of 75** shared MLEBench competitions change difficulty level under the TAM Bench procedure, while **68%** remain unchanged. This suggests alignment with prior expert judgment while allowing automated scaling to larger task sets.

## 5. Evaluation methodology

TAM Bench uses a **multi-dimensional evaluation framework** combining performance, constraint compliance, and output validity [2509.09321]. Performance is based on competition-relative ranking rather than raw metric magnitude alone. For task \(i\), the agent’s result is mapped to a rank percentile:
\[
\text{RankPct}_i = \frac{\text{rank}_i}{\text{total participants}_i}
\]
Lower values are better because smaller rank percentiles correspond to better leaderboard placement.

To reduce modality imbalance, the benchmark aggregates ranks with a modality-aware weighting scheme:
\[
\text{WeightedRank} =
\frac{\sum_{i=1}^{N} \frac{1}{f(m_i)} \cdot \text{RankPct}_i}
{\sum_{i=1}^{N} \frac{1}{f(m_i)}}
\]
where \(m_i\) is the modality of task \(i\) and \(f(m_i)\) is its frequency in the evaluation set [2509.09321]. The benchmark also reports grouped average ranks by difficulty and modality, which is intended to expose generalization failures across task types rather than only overall mean performance.

Constraint compliance is derived from the schema’s `special_instructions`. The paper evaluates those instructions with **LLM-as-a-Judge**, prompting a judge model with the special instructions and the final submitted code in `best_solution.py`, and asking whether each constraint is satisfied [2509.09321]. The intended metric is:
\[
\text{ConstraintPass} = \frac{\text{Number of Passed Constraints}}{\text{Total Constraints}}
\]
The benchmark evaluates constraints **only on the final submitted code**, with three stated reasons: only the final code is the actual deliverable, intermediate artifacts may be incomplete or ambiguous, and the evaluation is intended to be outcome-oriented.

Format compliance is split into two practical measures: **Made Submission**, the percentage of tasks on which the agent successfully generates a `submission.csv`, and **Valid Submission**, the percentage on which that submission matches the required structure of `test_labels.csv` in terms of columns and alignment [2509.09321]. This reflects a benchmark view in which failure to produce a usable artifact is a substantive failure, even if the model could in principle train a competitive predictor.

The experimental environment in the paper uses **Docker containers** with **Ubuntu 20.04**, a maximum runtime of **8 hours** per task, **16 vCPUs**, **60 GiB RAM**, and **1 NVIDIA A10 GPU** [2509.09321]. The reported experiments use only **TAM Bench Lite**, and compare two agent frameworks—**AIDE** and **OpenHands**—combined with **GPT-4.1** and **DeepSeek-V3**. The paper notes that Qwen-family models were initially tested but excluded because of **JSON parsing errors and poor stability**.

## 6. Empirical findings, interpretation, and limitations

On TAM Bench Lite, the four evaluated combinations show a clear tension between competitiveness and reliability [2509.09321]. The reported aggregate results are:
- **AIDE + GPT-4.1**: Made Submission **72%**, Valid Submission **56%**, Average Constraint Pass **88.2%**, Average Rank **87**
- **AIDE + DeepSeek-V3**: **39%**, **28%**, **90.4%**, **86**
- **OpenHands + GPT-4.1**: **78%**, **56%**, **84.3%**, **88**
- **OpenHands + DeepSeek-V3**: **56%**, **28%**, **86.9%**, **91**

Since lower average rank is better, **AIDE + DeepSeek-V3** achieves the best reported average rank at **86%**, but the paper immediately cautions that this may be misleading because DeepSeek-based systems have much lower submission and validity rates [2509.09321]. The stated interpretation is that **GPT-4.1-based agents are more reliable**, while **DeepSeek-V3-based agents can be competitive when they work, but are less stable**.

Difficulty-stratified results show degradation as tasks become harder. Averaged across systems, the reported average rank is **80.5** on easy tasks, **91.5** on medium tasks, and **92.75** on hard tasks [2509.09321]. Modality-stratified results further indicate that **Tabular** and **Text** are relatively easier, while **Image** and **Audio** remain difficult, and **Graph** and **Multimodal** are especially challenging, with graph tasks reaching an average rank of **100** across all evaluated systems [2509.09321]. The paper interprets this as evidence that current LLM-centered agents still lack the inductive biases and specialized feature extraction needed for perceptual and structured modalities.

The paper also highlights a notable success case: **OpenHands + DeepSeek-V3** achieved **first place** on `stanford-covid-vaccine` with **MCRMSE = 0.30**, surpassing the best human team’s **0.34198** [2509.09321]. This is presented as a proof-of-possibility result showing that, under some conditions, an LLM-driven agent can exceed human competition performance on a hard benchmark task.

Several limitations are explicitly acknowledged. First, the reported experiments cover **Lite** only, not **Medium** or **Full** [2509.09321]. Second, evaluation is **pass@1**, meaning single-run outcomes only; the paper states that future work should include repeated runs and confidence intervals. Third, the benchmark continues to expand, and the Full version is still being enlarged while maintaining coverage balance. Additional caveats raised in the paper include possible bias from selected competition platforms, dependence on reconstructed rather than original hidden test sets, and potential noise introduced by LLM-based schema annotation and LLM-as-a-Judge evaluation [2509.09321]. This suggests that TAM Bench should be understood not only as a benchmark artifact but also as a proposal for a benchmark construction methodology whose empirical coverage and validation are still developing.

In methodological terms, TAM Bench’s main significance lies in shifting evaluation away from narrow score-centric benchmarking toward a broader assessment of whether an ML agent can complete an entire competition workflow under realistic constraints [2509.09321]. A plausible implication is that its most durable contribution may be less the specific 18-task Lite leaderboard than the combination of automated task acquisition, leaderboard-based difficulty calibration, and benchmark metrics that jointly score predictive performance, instruction following, and artifact validity.

Source: https://www.emergentmind.com/topics/tam-bench