TAM Bench: ML Agent Benchmark

• TAM Bench is a benchmark that assesses general-purpose ML agents on complete competition workflows, from data analysis to valid submission generation.
• It automates task acquisition using browser and LLM-based methods to extract, filter, and normalize tasks from platforms like Kaggle and AIcrowd.
• The benchmark employs leaderboard-driven difficulty modeling and multi-dimensional evaluation to balance performance, constraint compliance, and output validity.

TAM Bench is a benchmark for evaluating general-purpose ML agents, especially LLM-based agents that attempt to automate the full machine learning workflow: understanding a task, reading data descriptions, writing code, training models, generating predictions, and producing a valid competition-style submission (Jia et al., 11 Sep 2025). It was introduced to address limitations in existing benchmarks for end-to-end AutoML and data-science systems, particularly limited task coverage, weak difficulty modeling, and evaluation schemes centered too narrowly on raw leaderboard performance. The benchmark contains 150 curated AutoML tasks and is organized around three design elements: automated task acquisition and standardization, leaderboard-driven difficulty modeling, and multi-dimensional evaluation (Jia et al., 11 Sep 2025).

1. Benchmark objective and research setting

TAM Bench targets realistic, complete ML challenges rather than isolated subtasks such as code generation, feature engineering in isolation, or narrow tabular prediction (Jia et al., 11 Sep 2025). The evaluated agent is expected to read documentation, understand datasets, perform data analysis, tune or train models, generate code, generate a valid submission.csv, and provide a best_solution.py. This makes the benchmark explicitly end-to-end in scope.

The benchmark is framed as a response to four limitations in earlier evaluation practice. First, benchmark construction has often required substantial manual labor. Second, prior benchmarks have had narrow or imbalanced task distributions. Third, difficulty labels have often been subjective or absent. Fourth, single-dimensional evaluation based only on task score or medal status can obscure whether an agent actually satisfies task-specific requirements or even produces a valid deliverable (Jia et al., 11 Sep 2025). In this setting, TAM Bench is intended to evaluate whether an ML agent can solve competition-style tasks across multiple domains, modalities, and difficulty levels while obeying output and instruction constraints.

The benchmark is also positioned against specific predecessors. It is described as broader than MLEBench in task count and source diversity, more difficulty-aware than MLAgentBench, and less domain-restricted than CALM, which focuses on tabular finance tasks (Jia et al., 11 Sep 2025). This suggests a benchmark philosophy centered on generalist ML agency rather than performance on a single modality or vertical.

2. Automated construction pipeline

A central contribution of TAM Bench is its automated benchmark construction pipeline, which combines browser automation and LLM-based structuring to acquire tasks from competition platforms such as Kaggle, AIcrowd, and Biendata (Jia et al., 11 Sep 2025). The paper describes the overall process as: acquire task pages, extract clean task content and leaderboard data, filter tasks, normalize them into a structured schema, reconstruct evaluation datasets, assign difficulty levels, and assemble benchmark subsets.

The acquisition system is inspired by Model Context Protocol (MCP) and uses Browser-Use for browser control. Its architecture has four layers: an Agent Layer based on LangChain’s ReAct architecture, a Controller Layer that converts agent decisions into browser actions, a DOM Layer that parses webpage structure into AI-friendly text, and a Browser Layer using Playwright for browser control and multi-context management (Jia et al., 11 Sep 2025). The controller is enhanced with an extract_markdown tool to remove irrelevant page elements such as ads and navigation bars while preserving task-relevant text.

After acquisition, TAM Bench applies a filtering pipeline. The listed rules exclude tasks before 2023, remove tasks without publicly available datasets, discard tasks lacking ground-truth test labels or reproducible splits, and exclude tasks whose official scoring procedure cannot be replicated (Jia et al., 11 Sep 2025). The 2023 cutoff is explicitly motivated as a way to reduce contamination from model pretraining corpora.

Normalization into a unified schema is performed with GPT-4o. The schema includes fields such as task_type, goal_description, metric, target_col, data_information, output_format, and special_instructions (Jia et al., 11 Sep 2025). The inclusion of special_instructions is significant because those instructions are later used in constraint compliance evaluation, making the schema part of the benchmark’s scoring logic rather than only a documentation layer.

3. Coverage, subsets, and task organization

TAM Bench covers six modalities: Tabular, Text, Image, Audio, Graph, and Multimodal (Jia et al., 11 Sep 2025). It also aims to broaden application coverage beyond typical benchmark staples, explicitly mentioning areas such as financial risk control, medical diagnosis, e-commerce recommendation, bioinformatics, and graph-based tasks. The paper emphasizes both modality coverage and domain coverage, though the main text does not provide a single consolidated count table by domain.

The benchmark is released in three scales:

Subset	Tasks	Intended use
Lite	18	Routine benchmarking and comparative studies
Medium	54	Broader evaluation at moderate cost
Full	150	Comprehensive evaluation

The Lite subset is deliberately balanced: it contains 6 modalities, with 3 tasks per modality, specifically one easy, one medium, and one hard task per modality (Jia et al., 11 Sep 2025). This construction yields exactly $6 \times 3 = 18$ tasks and is meant to provide a compact but representative testbed for everyday benchmarking. The paper gives the full Lite task list, including, for example, new-york-city-taxi-fare-prediction in tabular data, lmsys-chatbot-arena in text, statoil-iceberg-classifier-challenge in image, tensorflow-speech-recognition-challenge in audio, WhoIsWho-IND in graph data, and planttraits2024 in multimodal evaluation (Jia et al., 11 Sep 2025).

A practical issue in competition-derived benchmarking is that hidden test labels are often unavailable. TAM Bench addresses this by reconstructing evaluation datasets from the original training data, partitioning them into new train/test subsets, and then scoring model outputs against the locally reconstructed held-out labels (Jia et al., 11 Sep 2025). The paper states that scores produced by this local evaluation pipeline are nearly identical to scores from the competition’s official evaluation system when checked using the provided sample_submission.csv. This suggests that the reconstructed evaluation is intended as a faithful offline proxy, though the paper does not report exact discrepancy values.

4. Difficulty modeling

A distinctive feature of TAM Bench is its automated leaderboard-driven difficulty model (Jia et al., 11 Sep 2025). Rather than relying on manual human estimates, the benchmark derives task difficulty from Mean score across participants, Best score, and Participant count. This is presented as an objective and scalable way to compare heterogeneous tasks.

Because competitions use different metrics, the benchmark first normalizes scores. For the mean and best leaderboard values, the paper gives the following formulas: $$\text{NormMean} = \frac{\text{Mean Score} - \text{Min Score}}{\text{Max Score} - \text{Min Score} + \epsilon}$$

$$\text{NormBest} = \frac{\text{Best Score} - \text{Min Score}}{\text{Max Score} - \text{Min Score} + \epsilon}$$

For unbounded metrics, $\text{Max Score}$ and $\text{Min Score}$ are set to the maximum and minimum observed on the private leaderboard (Jia et al., 11 Sep 2025).

For higher-is-better metrics such as accuracy or AUC, the difficulty score is: $$\text{Difficulty Score} = w_1 \cdot (1 - \text{NormMean}) + w_2 \cdot \log_{10}(\text{Participants} + 1) + w_3 \cdot (1 - \text{NormBest})$$

For lower-is-better metrics such as loss or RMSE, the score is: $$\text{Difficulty Score} = w_1 \cdot \text{NormMean} + w_2 \cdot \log_{10}(\text{Participants} + 1) + w_3 \cdot \text{NormBest}$$

The weights are fixed to: $w_1 = 0.4,\quad w_2 = 0.1,\quad w_3 = 0.5$ and tasks are binned as:

• Easy: $\text{Difficulty Score} \le 0.6$
• Medium: $0.6 < \text{Difficulty Score} \le 0.85$
• Hard: $$\text{NormMean} = \frac{\text{Mean Score} - \text{Min Score}}{\text{Max Score} - \text{Min Score} + \epsilon}$$0 (Jia et al., 11 Sep 2025)

The benchmark validates this automatic labeling against MLEBench’s manual difficulty assignments. The reported outcome is that tasks labeled Hard in MLEBench consistently map to Medium or Hard in TAM Bench, and that no manually difficult task is misclassified as Easy (Jia et al., 11 Sep 2025). The appendix also reports that 24 of 75 shared MLEBench competitions change difficulty level under the TAM Bench procedure, while 68% remain unchanged. This suggests alignment with prior expert judgment while allowing automated scaling to larger task sets.

5. Evaluation methodology

TAM Bench uses a multi-dimensional evaluation framework combining performance, constraint compliance, and output validity (Jia et al., 11 Sep 2025). Performance is based on competition-relative ranking rather than raw metric magnitude alone. For task $$\text{NormMean} = \frac{\text{Mean Score} - \text{Min Score}}{\text{Max Score} - \text{Min Score} + \epsilon}$$1, the agent’s result is mapped to a rank percentile: $$\text{NormMean} = \frac{\text{Mean Score} - \text{Min Score}}{\text{Max Score} - \text{Min Score} + \epsilon}$$2 Lower values are better because smaller rank percentiles correspond to better leaderboard placement.

To reduce modality imbalance, the benchmark aggregates ranks with a modality-aware weighting scheme: $$\text{NormMean} = \frac{\text{Mean Score} - \text{Min Score}}{\text{Max Score} - \text{Min Score} + \epsilon}$$3 where $$\text{NormMean} = \frac{\text{Mean Score} - \text{Min Score}}{\text{Max Score} - \text{Min Score} + \epsilon}$$4 is the modality of task $$\text{NormMean} = \frac{\text{Mean Score} - \text{Min Score}}{\text{Max Score} - \text{Min Score} + \epsilon}$$5 and $$\text{NormMean} = \frac{\text{Mean Score} - \text{Min Score}}{\text{Max Score} - \text{Min Score} + \epsilon}$$6 is its frequency in the evaluation set (Jia et al., 11 Sep 2025). The benchmark also reports grouped average ranks by difficulty and modality, which is intended to expose generalization failures across task types rather than only overall mean performance.

Constraint compliance is derived from the schema’s special_instructions. The paper evaluates those instructions with LLM-as-a-Judge, prompting a judge model with the special instructions and the final submitted code in best_solution.py, and asking whether each constraint is satisfied (Jia et al., 11 Sep 2025). The intended metric is: $$\text{NormMean} = \frac{\text{Mean Score} - \text{Min Score}}{\text{Max Score} - \text{Min Score} + \epsilon}$$7 The benchmark evaluates constraints only on the final submitted code, with three stated reasons: only the final code is the actual deliverable, intermediate artifacts may be incomplete or ambiguous, and the evaluation is intended to be outcome-oriented.

Format compliance is split into two practical measures: Made Submission, the percentage of tasks on which the agent successfully generates a submission.csv, and Valid Submission, the percentage on which that submission matches the required structure of test_labels.csv in terms of columns and alignment (Jia et al., 11 Sep 2025). This reflects a benchmark view in which failure to produce a usable artifact is a substantive failure, even if the model could in principle train a competitive predictor.

The experimental environment in the paper uses Docker containers with Ubuntu 20.04, a maximum runtime of 8 hours per task, 16 vCPUs, 60 GiB RAM, and 1 NVIDIA A10 GPU (Jia et al., 11 Sep 2025). The reported experiments use only TAM Bench Lite, and compare two agent frameworks—AIDE and OpenHands—combined with GPT-4.1 and DeepSeek-V3. The paper notes that Qwen-family models were initially tested but excluded because of JSON parsing errors and poor stability.

6. Empirical findings, interpretation, and limitations

On TAM Bench Lite, the four evaluated combinations show a clear tension between competitiveness and reliability (Jia et al., 11 Sep 2025). The reported aggregate results are:

• AIDE + GPT-4.1: Made Submission 72%, Valid Submission 56%, Average Constraint Pass 88.2%, Average Rank 87
• AIDE + DeepSeek-V3: 39%, 28%, 90.4%, 86
• OpenHands + GPT-4.1: 78%, 56%, 84.3%, 88
• OpenHands + DeepSeek-V3: 56%, 28%, 86.9%, 91

Since lower average rank is better, AIDE + DeepSeek-V3 achieves the best reported average rank at 86%, but the paper immediately cautions that this may be misleading because DeepSeek-based systems have much lower submission and validity rates (Jia et al., 11 Sep 2025). The stated interpretation is that GPT-4.1-based agents are more reliable, while DeepSeek-V3-based agents can be competitive when they work, but are less stable.

Difficulty-stratified results show degradation as tasks become harder. Averaged across systems, the reported average rank is 80.5 on easy tasks, 91.5 on medium tasks, and 92.75 on hard tasks (Jia et al., 11 Sep 2025). Modality-stratified results further indicate that Tabular and Text are relatively easier, while Image and Audio remain difficult, and Graph and Multimodal are especially challenging, with graph tasks reaching an average rank of 100 across all evaluated systems (Jia et al., 11 Sep 2025). The paper interprets this as evidence that current LLM-centered agents still lack the inductive biases and specialized feature extraction needed for perceptual and structured modalities.

The paper also highlights a notable success case: OpenHands + DeepSeek-V3 achieved first place on stanford-covid-vaccine with MCRMSE = 0.30, surpassing the best human team’s 0.34198 (Jia et al., 11 Sep 2025). This is presented as a proof-of-possibility result showing that, under some conditions, an LLM-driven agent can exceed human competition performance on a hard benchmark task.

Several limitations are explicitly acknowledged. First, the reported experiments cover Lite only, not Medium or Full (Jia et al., 11 Sep 2025). Second, evaluation is pass@1, meaning single-run outcomes only; the paper states that future work should include repeated runs and confidence intervals. Third, the benchmark continues to expand, and the Full version is still being enlarged while maintaining coverage balance. Additional caveats raised in the paper include possible bias from selected competition platforms, dependence on reconstructed rather than original hidden test sets, and potential noise introduced by LLM-based schema annotation and LLM-as-a-Judge evaluation (Jia et al., 11 Sep 2025). This suggests that TAM Bench should be understood not only as a benchmark artifact but also as a proposal for a benchmark construction methodology whose empirical coverage and validation are still developing.

In methodological terms, TAM Bench’s main significance lies in shifting evaluation away from narrow score-centric benchmarking toward a broader assessment of whether an ML agent can complete an entire competition workflow under realistic constraints (Jia et al., 11 Sep 2025). A plausible implication is that its most durable contribution may be less the specific 18-task Lite leaderboard than the combination of automated task acquisition, leaderboard-based difficulty calibration, and benchmark metrics that jointly score predictive performance, instruction following, and artifact validity.