FML-Bench: Autonomous Research Evaluation

• FML-Bench is a comprehensive framework that evaluates automatic ML research agents by testing their ability to propose ideas, modify code, run experiments, and refine hypotheses.
• It spans 8 to 18 tasks across various ML domains, including generalization, fairness, and privacy, thereby accounting for both dense and sparse research opportunities.
• The design explicitly separates agent strategy from execution, employing process metrics like diversity, cost, and step success rate to measure academic contribution and search efficiency.

FML-Bench, also written as FML-bench in the literature, is a benchmark framework for evaluating automatic machine learning research agents: systems that autonomously propose research ideas, modify codebases, run experiments, and refine hypotheses from empirical feedback. The benchmark was introduced as a suite centered on eight diverse and fundamental machine learning research problems and was later extended into a controlled-study framework spanning 18 fundamental tasks across 10 domains, with an explicit separation between agent strategy and execution infrastructure. Across both formulations, the central objective is to measure scientific research capability rather than narrow engineering proficiency, using real research repositories, protected evaluation procedures, and multi-dimensional process metrics rather than final scores alone (Zou et al., 12 Oct 2025, Zou et al., 17 May 2026).

1. Origin, motivation, and problem setting

FML-Bench was motivated by limitations in prior benchmarks for autonomous ML research systems. The initial formulation argues that existing benchmarks overemphasized engineering-oriented, “Kaggle-style” tasks such as feature pipelines and standard model training rather than core ML research problems such as representation learning, causality, and fairness. It also identifies limited task diversity, a high coding barrier, and weak academic rigor, with performance gains often arising from hyperparameter fiddling or script manipulation rather than algorithmic innovation (Zou et al., 12 Oct 2025).

The benchmark therefore targets agents that operate as automatic research systems rather than as coding assistants. In the initial formulation, the stated goals were to focus explicitly on eight fundamental ML research problems, instantiate each task on an existing end-to-end GitHub repository with a working baseline, provide a unified I/O interface, offer a five-dimensional evaluation protocol, and make the benchmark easily extensible by adding adapters for new repositories. The later controlled-study formulation preserves the same broad agenda but reframes the benchmark as an instrument for analyzing search dynamics: it aims to isolate agent “strategy” from “infrastructure” so that performance differences can be attributed to search topology, memory, selection rules, and prompt templates rather than code-editing or experiment-running machinery (Zou et al., 12 Oct 2025, Zou et al., 17 May 2026).

This design situates FML-Bench at the intersection of AutoML, program repair, code agents, and research automation. A plausible implication is that the benchmark treats the research process itself as the object of evaluation: not only whether an agent improves a baseline, but how it searches, how reliably it executes, how costly its exploration is, and whether its edits constitute academic contributions rather than superficial modifications.

2. Benchmark composition and task coverage

The initial benchmark comprises eight tasks, each tied to a publicly available research repository and a task-specific baseline. These are: Generalization with DomainBed / ERM and out-of-domain accuracy; Data Efficiency with Easy-Few-Shot-Learning / ProtoNets and few-shot accuracy; Representation Learning with Lightly / MoCo and linear-probe accuracy; Continual Learning with Continual-Learning / Synaptic Intelligence and average accuracy over tasks; Causality with CausalML / DragonNet and $|ATE - estimate\ error|$; Robustness & Reliability with ART / dp-instahide and a defense score defined as $$\mathrm{HarmonicMean}(\mathrm{clean\ acc}, \mathrm{backdoor\ acc})$$; Privacy with PrivacyMeter / Wide-ResNet-28-2 and membership-inference AUC; and Fairness & Bias with AIF360 / Adversarial Debiasing and AvgOddsDiff while maintaining accuracy (Zou et al., 12 Oct 2025).

The later formulation expands FML-Bench to 18 tasks across 10 high-level ML domains. These domains are Generalization, Data Efficiency, Representation, Continual Learning, Causality, Robustness, Privacy, Fairness, Unlearning, and Federated Learning. The individual tasks include DomainBed-CM and DomainBed-OH for generalization; EasyFSL and USB for data efficiency; Lightly and Solo-learn for representation; Cont.-Learn and PyCIL for continual learning; CausalML and gCastle for causality; ART and OpenOOD for robustness; PrivacyMeter and Opacus for privacy; AIF360 and Fairlearn for fairness; Unlearning for forgetting on TOFU/Llama3.2; and PFLlib for federated learning (Zou et al., 17 May 2026).

A central feature of the expanded benchmark is its annotation of “improvement opportunity” structure. Tasks are categorized as either dense or sparse depending on whether small tweaks often help or most tweaks fail. In the reported task overview, Continual Learning with split-MNIST, gCastle, ART, PrivacyMeter, Opacus, AIF360, Fairlearn, Unlearning, and PFLlib are dense, whereas the DomainBed, EasyFSL, USB, Lightly, Solo-learn, PyCIL, CausalML, and OpenOOD tasks are sparse. This classification becomes central to the later analysis of which search strategies succeed under which task regimes (Zou et al., 17 May 2026).

3. Interface, execution model, and extensibility

FML-Bench reduces engineering overhead by placing agents inside working research repositories rather than giving them blank-slate problems. In the original formulation, the benchmark gives agents a standard input consisting of task description, full codebase, modifiable file suggestions, protected evaluation files, a command list, baseline metrics, and a target improvement. The agent must edit the repository, rerun the provided commands, and output improved code together with a final metrics JSON that is post-processed into a fixed format (Zou et al., 12 Oct 2025).

The benchmark’s execution interface is repository-agnostic. To add a new repository, the benchmark requires a COMMAND_LIST, a designation of read-only paths for evaluation integrity, and a lightweight adapter that parses final logs or JSON into the standard output format. The protected segments mechanism ensures that agents cannot modify READONLY_PATHS, while the unified post-processing layer can ingest metric name/value pairs from JSON, text logs, or CSV outputs (Zou et al., 12 Oct 2025).

The controlled-study version further standardizes infrastructure. It defines a shared evaluation pipeline in which the benchmark feeds each agent the same task description, baseline code, and validation feedback; strictly controls all code editing and experiment runs via a shared toolchain; and logs only the agent’s choice of “what to try next.” The pipeline is organized as a repeated loop of Propose, Modify, and Execute, with embeddings and validation metrics recorded for process analysis. After 100 steps, the best-validated code is selected and evaluated on the final test metric (Zou et al., 17 May 2026).

This infrastructure design is meant to prevent a common confound in agent benchmarking: superior performance due to better editor integration, better patch application, or better metric parsing rather than better research search logic. The later paper makes that separation explicit and treats it as a primary design objective (Zou et al., 17 May 2026).

4. Evaluation framework and process metrics

The first FML-Bench paper formalizes agent research progress over $T$ iterations with an objective that combines five elements: Utility, Academic Contribution Rate, Cost, Step Success Rate, and Diversity. Utility is defined as the empirical gain from a modification, $U(m,C)=\mathrm{perf}(C\oplus m)-\mathrm{perf}(C)$. Diversity is defined from CodeGraphBERT embeddings of code diffs as the average Euclidean distance of each embedding from the centroid. Academic Contribution Rate measures the fraction of successful steps that are classified as academic edits. Step Success Rate is the fraction of executed steps without errors. Cost is measured as token consumption and wall-clock time, both per step and in aggregate (Zou et al., 12 Oct 2025).

The later framework expands this into 12 process-level behavioral metrics distributed across five dimensions. For exploration, it defines $D_{\mathrm{spread}}$, $D_{\mathrm{reach}}$, $U$ as cluster count normalized by number of valid steps, and $D_{\mathrm{eff}}$ from the eigenvalue spectrum of the sample covariance of GraphCodeBERT embeddings. For generalization, it measures the validation–test gap. For reliability, it defines $R_{\mathrm{valid}} = n/T$. For efficiency, it records AUC-over-steps, First-improvement step, Best-improvement step, and Late-gain. For cost, it tracks total token usage and wall-clock time in hours (Zou et al., 17 May 2026).

The controlled-study paper also introduces a normalized test improvement score,

$$\hat p = \max\Bigl(0,\;\frac{p_{\rm agent} - p_{\rm baseline}}{\lvert p_{\rm best}-p_{\rm worst}\rvert}\Bigr),$$

with a sign adjustment for lower-is-better tasks, together with a Pairwise Win-Rate over agents. These definitions allow final task outcomes to be compared across heterogeneous objectives such as OOD accuracy, MAE, AUROC, or fairness metrics (Zou et al., 17 May 2026).

Taken together, the two versions of FML-Bench move from a five-dimensional benchmark oriented toward academic research quality to a richer process-level analysis of search behavior. This suggests an evolution from benchmarking capability to benchmarking capability plus search dynamics.

5. Agent families and experimental protocols

In the initial empirical study, FML-Bench evaluates three classes of automatic research agents. TheAIScientist is characterized as “broad-shallow” and tested with GPT-5 and Gemini-2.5-Pro backends. AIDE / Weco is characterized as “tree-search” balancing exploration and exploitation and is likewise tested with GPT-5 and Gemini-2.5-Pro. Claude Code is characterized as “narrow-deep” and evaluated using Opus-4.1. Each agent is run for three independent trials, each with $$\mathrm{HarmonicMean}(\mathrm{clean\ acc}, \mathrm{backdoor\ acc})$$0 modification-execution iterations, and the reported result for each agent is the best final metric among the three trials (Zou et al., 12 Oct 2025).

The controlled-study paper ports six representative agent strategies to a shared framework so that only search logic differs. These are Autoresearch as a greedy hill-climber, The AI Scientist v1 as parallel linear chains, The AI Scientist v2 as best-first tree search, AIDE as solution-space tree search with debugging, AIRA as MCTS with UCT, and OpenEvolve as MAP-Elites island evolution. In the main experiments, all 18 tasks are run for $$\mathrm{HarmonicMean}(\mathrm{clean\ acc}, \mathrm{backdoor\ acc})$$1 valid steps under a 40 minute per-validation cap on one A100 GPU, with GPT-5.4 fixed as the backbone LLM and three independent random seeds per agent-task pair, for a total of $$\mathrm{HarmonicMean}(\mathrm{clean\ acc}, \mathrm{backdoor\ acc})$$2 runs (Zou et al., 17 May 2026).

The later paper also introduces AdaptiveSearch, an agent that begins as a greedy hill-climber and switches to broader exploration if the best-so-far curve shows no meaningful gain for $$\mathrm{HarmonicMean}(\mathrm{clean\ acc}, \mathrm{backdoor\ acc})$$3 steps with slope $$\mathrm{HarmonicMean}(\mathrm{clean\ acc}, \mathrm{backdoor\ acc})$$4. This switching rule is presented as an explicit response to the empirical structure of dense and sparse opportunity regimes (Zou et al., 17 May 2026).

6. Empirical findings, interpretation, and naming distinctions

The initial FML-Bench study reports that agents employing broad research exploration strategies outperform those focusing on narrow but deep exploration. On the eight-task benchmark, TheAIScientist with Gemini-2.5-Pro wins 4/8 tasks, and the paper highlights that Gemini-2.5-Pro often outperforms GPT-5. Claude Code’s narrow, linear refinement occasionally matches the best utility, as in Generalization, but “fails on average.” The same study reports a strong positive Pearson correlation between Diversity and final Utility, with an example of $$\mathrm{HarmonicMean}(\mathrm{clean\ acc}, \mathrm{backdoor\ acc})$$5 in Continual Learning, and average secondary metrics showing that TheAIScientist with Gemini-2.5-Pro has the highest mean diversity and academic contribution rate while maintaining a Step Completion Rate of 1.00 (Zou et al., 12 Oct 2025).

The controlled-study paper qualifies that conclusion. Across 18 tasks, TAS v2 attains mean $$\mathrm{HarmonicMean}(\mathrm{clean\ acc}, \mathrm{backdoor\ acc})$$6 with win-rate 58.5%, while Autoresearch attains mean $$\mathrm{HarmonicMean}(\mathrm{clean\ acc}, \mathrm{backdoor\ acc})$$7 with win-rate 57.4%, leading to the claim that strategy complexity alone does not guarantee strong performance. The post-hoc opportunity-density analysis then reports that Autoresearch ranks first on dense tasks but falls to rank six on sparse tasks, while broader strategies reverse that trend. AdaptiveSearch, which switches from greedy hill-climbing to multi-branch exploration when improvement stagnates, reaches mean $$\mathrm{HarmonicMean}(\mathrm{clean\ acc}, \mathrm{backdoor\ acc})$$8 and win-rate 58.6% in the seven-agent comparison (Zou et al., 17 May 2026).

The process-level correlations in the later paper further sharpen the interpretation. AUC-over-steps has pooled Spearman $$\mathrm{HarmonicMean}(\mathrm{clean\ acc}, \mathrm{backdoor\ acc})$$9 with $T$0, First-improvement step has $T$1 with $T$2, Exploration Reach has $T$3 with $T$4, and Effective dimension has $T$5 with $T$6. By contrast, “solution diversity” and “compute cost” are reported as not significantly associated with final performance (Zou et al., 17 May 2026). This creates an important nuance: the earlier paper associates breadth of exploration with stronger outcomes relative to narrow incremental refinement, whereas the later controlled study argues that diversity by itself is not a reliable predictor once infrastructure is controlled and task opportunity structure is taken into account. A plausible implication is that effective search is neither merely broad nor merely deep; it is contingent on whether the task offers dense local improvements or requires wider branching to escape sparse opportunity landscapes.

A recurrent source of confusion is nomenclature. FML-Bench concerns automatic ML research agents, whereas FLBench is a separate federated learning benchmark suite covering medical, financial, and AIoT domains, with a different problem formulation centered on communication, privacy-preserving mechanisms, data heterogeneity, and cooperation strategy (Liang et al., 2020). The names are similar, but the benchmarks address different research areas and evaluation criteria.