Auto-Bench: Automated Benchmark for AI Scientists

• Auto-Bench is an automated benchmark for scientific discovery, challenging LLMs to infer hidden causal graphs through iterative hypothesis formation and controlled interventions.
• The framework evaluates performance in discrete chemistry and social network settings, using repeated interactions with an oracle to simulate experimental research.
• Experimental results highlight sharp performance degradation with increased graph complexity, revealing current LLMs’ limits in structured reasoning and long-term causal tracking.

Auto-Bench is an automated benchmark for scientific discovery in LLMs that evaluates whether LLM agents can conduct human-like research by formulating hypotheses, designing experiments, updating beliefs, uncovering hidden causal structure, and making optimal decisions with valid justifications. It is introduced as the first end-to-end benchmark explicitly designed to stress-test LLMs as “AI scientists” in two domains—one natural, simulated chemistry, and one social, simulated social networks—through iterative causal-graph discovery. The benchmark is organized around repeated interaction with an oracle that knows, but does not reveal, the ground-truth graph; models must infer that hidden structure under a fixed intervention budget, with performance measured by success rate and average iterations to correctness (Chen et al., 21 Feb 2025).

1. Motivation and benchmark scope

Auto-Bench is motivated by the question of whether LLMs can conduct human-like scientific research, specifically by formulating hypotheses, designing experiments, updating beliefs, and ultimately uncovering hidden causal structure. The benchmark is proposed in response to the absence of a standardized benchmark specifically designed for scientific discovery in LLM agents, despite the fact that scientific discovery is an iterative process that demands efficient knowledge updating and encoding, environmental understanding, hypothesis identification, and action reasoning (Chen et al., 21 Feb 2025).

The benchmark covers two task classes. In the chemistry setting, a directed acyclic graph governs molecules whose node states are discrete. In the social-network setting, an undirected graph encodes symmetric friendships among people. In both cases, the model interacts repeatedly with an oracle and must reconstruct the hidden graph from intervention-induced observations. This design makes the benchmark explicitly interactive rather than purely static, and it ties scientific-discovery capability to causal-graph inference under controlled interventions.

A central feature of the benchmark is that it treats discovery as a closed loop: hypothesis formation, intervention choice, observation accumulation, and hypothesis revision are all part of the evaluated behavior. This distinguishes Auto-Bench from benchmarks that only test answer generation on fixed inputs. The benchmark therefore operationalizes “AI scientist” capability as iterative causal discovery rather than as isolated reasoning over pre-specified facts.

2. Task formulation in chemistry and social networks

Let $N$ denote the number of nodes, and let the ground-truth adjacency be

$H \in \{0,1\}^{N \times N},$

where $H_{i,j}=1$ iff there is an edge $i \to j$ in chemistry or $i-j$ in the social setting. At cycle $t$, the LLM maintains a hypothesis adjacency

$K^{(t)} \in \{0,1\}^{N \times N}.$

The two benchmark settings differ in graph structure, intervention semantics, and correctness criteria.

Setting	Hidden structure	Intervention effect
Chemistry	A directed acyclic graph over molecules	Intervening on node $k$ resets all downstream descendants to fresh random states
Social network	An undirected graph over people	Intervening on person $k$ increments the state of $k$ and all neighbors by $H \in \{0,1\}^{N \times N},$0

In the chemistry setting, each node $H \in \{0,1\}^{N \times N},$1 has a discrete state $H \in \{0,1\}^{N \times N},$2. If the model intervenes on node $H \in \{0,1\}^{N \times N},$3, then each node $H \in \{0,1\}^{N \times N},$4 reachable from $H \in \{0,1\}^{N \times N},$5 is assigned a fresh random state, while the intervened node itself remains unchanged. The target is recovery of the adjacency matrix $H \in \{0,1\}^{N \times N},$6, but correctness is judged up to causal-equivalence in terms of reachability rather than exact adjacency.

In the social-network setting, the hidden graph is symmetric. If the model intervenes on person $H \in \{0,1\}^{N \times N},$7, then the state of $H \in \{0,1\}^{N \times N},$8 and all its neighbors is incremented by one. From these state-change observations, the model must recover a symmetric adjacency $H \in \{0,1\}^{N \times N},$9 exactly. The task is therefore structurally simpler in graph type but exacting in its matching rule.

This task design places both natural and social sciences under a common causal-discovery abstraction. A plausible implication is that the benchmark is intended to separate general-purpose language competence from structured experimental reasoning over hidden interactions.

3. Interactive oracle protocol and iterative hypothesis revision

At each cycle $H_{i,j}=1$0, the LLM receives a description of the task, all past interventions $H_{i,j}=1$1 and observations, and its current hypothesis $H_{i,j}=1$2, which is initially uniform or empty. It must then output both a new hypothesis $H_{i,j}=1$3 and a new intervention. The intervention vector is

$H_{i,j}=1$4

where $H_{i,j}=1$5 indicates intervention on node $H_{i,j}=1$6. The oracle returns a new state vector

$H_{i,j}=1$7

and the observation matrix after $H_{i,j}=1$8 cycles is

$H_{i,j}=1$9

The protocol is defined explicitly. At Step 0, the oracle samples a random hidden graph $i \to j$0 and an initial node-state vector $i \to j$1, and the LLM is prompted with the task description and asked for its first intervention and hypothesis. At each later step, the model outputs a binary $i \to j$2 matrix and an intervention index, the oracle executes the intervention and returns the new state vector, and the model appends that observation to memory before refining its hypothesis and choosing the next intervention.

Termination occurs when the hypothesis is correct under the benchmark’s criterion. In chemistry, the protocol terminates successfully if the model’s inferred graph matches the oracle graph up to reachability equivalence. In the social setting, it terminates only under exact adjacency match. If the model remains incorrect at $i \to j$3, the round is counted as a failure, with

$i \to j$4

This protocol makes intervention planning part of the task rather than a fixed evaluation wrapper. The benchmark therefore tests not only whether a model can interpret observations, but whether it can choose information-bearing experiments under a limited budget.

4. Scoring, reachability, and complexity control

Auto-Bench records two primary metrics for each model and configuration: success rate and average iterations. Success rate is the number of successful rounds divided by the total number of rounds. Average iterations is the average cycle index at which the model first attains correctness, computed over successful rounds only (Chen et al., 21 Feb 2025).

Because multiple directed adjacency matrices can induce identical downstream behavior, chemistry is scored through reachability. For any binary matrix $i \to j$5, let $i \to j$6 denote the $i \to j$7-fold matrix product, and define

$i \to j$8

Then the chemistry hypothesis is correct iff

$i \to j$9

In the social setting, by contrast, the requirement is exact equality:

$i-j$0

Complexity varies along two axes. Graph size is set to $i-j$1, and in chemistry the state-space size is set to $i-j$2. The benchmark states that larger $i-j$3 or $i-j$4 implies exponentially more possible graphs and therefore higher task difficulty. This makes the complexity control explicit and interpretable in combinatorial terms.

The paper also introduces a long-term trajectory tracking extension. Here the input is a matrix

$i-j$5

and the model must output

$i-j$6

for $i-j$7 and $i-j$8. Two additional metrics are defined:

$i-j$9

and per-step trajectory accuracy

$t$0

These definitions extend the benchmark beyond graph recovery to long-horizon temporal change tracking. This suggests that Auto-Bench treats persistent structured state maintenance as part of scientific-discovery competence, not merely a peripheral auxiliary skill.

5. Experimental results and performance degradation

The benchmark evaluates GPT-4o, Qwen2.5, Gemini, Llama, and Claude. In the social-network setting with $t$1, GPT-4o and Qwen2.5 each achieve $t$2, Gemini $t$3, Llama $t$4, and Claude $t$5. At $t$6, GPT-4o and Qwen2.5 remain at $t$7 success, with average iterations $t$8 and $t$9 respectively, while Gemini falls to $K^{(t)} \in \{0,1\}^{N \times N}.$0, Llama to $K^{(t)} \in \{0,1\}^{N \times N}.$1, and Claude to $K^{(t)} \in \{0,1\}^{N \times N}.$2. At $K^{(t)} \in \{0,1\}^{N \times N}.$3, GPT-4o records $K^{(t)} \in \{0,1\}^{N \times N}.$4 and all other models fail completely (Chen et al., 21 Feb 2025).

In the chemistry setting, performance is high at small scale and degrades sharply at larger scale. For $K^{(t)} \in \{0,1\}^{N \times N}.$5, GPT-4o and Qwen2.5 each achieve $K^{(t)} \in \{0,1\}^{N \times N}.$6, Gemini $K^{(t)} \in \{0,1\}^{N \times N}.$7, Llama $K^{(t)} \in \{0,1\}^{N \times N}.$8, and Claude $K^{(t)} \in \{0,1\}^{N \times N}.$9. For $k$0, GPT-4o and Qwen2.5 remain at $k$1, Gemini reaches $k$2, Llama $k$3, and Claude $k$4. For $k$5, GPT-4o and Gemini each achieve $k$6 success, with average iterations $k$7 and $k$8, and the remaining models fail.

The long-term trajectory extension shows a further breakdown in performance as sequence length grows. With simple zero-shot prompts and no chain-of-thought, even GPT-4o’s $k$9 drops from approximately $k$0 at $k$1 to $k$2 by $k$3. Chain-of-thought recovers perfect performance for GPT-4o up to $k$4, but by $k$5 performance again collapses. Qwen2.5 exhibits more modest chain-of-thought gains, peaking at approximately $k$6 at small $k$7.

Across both core tasks and the trajectory extension, the reported pattern is consistent: sharp degradation as node count increases beyond $k$8, slightly easier behavior for directed chemical graphs than undirected social graphs at small sizes, and severe difficulty in long-horizon temporal reasoning. The benchmark characterizes this as a gap between current machine performance and human scientific reasoning.

6. Interpretation, open problems, and terminological ambiguity

Auto-Bench identifies three major deficits in current LLMs: maintaining and updating structured beliefs over many variables, planning multi-step information-maximizing experiments, and tracking long causal or temporal chains in context windows (Chen et al., 21 Feb 2025). The paper contrasts these limitations with human scientists, who routinely manage much larger networks of hypotheses and recall experiment sequences without catastrophic forgetting. The benchmark’s central empirical claim is therefore not merely that accuracy decreases with scale, but that current LLMs lack robust mechanisms for scientific-discovery-style iterative reasoning under combinatorial and temporal pressure.

The paper lists several open challenges. These include extending the benchmark beyond discrete DAGs to continuous, probabilistic causal systems and unstructured observations such as textual lab reports; incorporating external knowledge retrieval and symbolic solvers during hypothesis construction; designing specialized architectures or memory mechanisms for long experiment trajectories; exploring active-learning strategies that maximize information gain under limited intervention budgets; and bridging matrix-based causal discovery with richer domain-specific hypothesis generation such as mechanistic chemical pathways or social diffusion models.

A recurrent misconception is that “Auto-Bench” denotes a single benchmark family across the broader LLM evaluation literature. In fact, the name is shared by multiple unrelated systems. Distinct examples include “AutoBench: Automating LLM Evaluation through Reciprocal Peer Assessment” (Loi et al., 26 Oct 2025), “AutoBench: Automatic Testbench Generation and Evaluation Using LLMs for HDL Design” (Qiu et al., 2024), “AutoBench-V: Can Large Vision-LLMs Benchmark Themselves?” (Bao et al., 2024), and the AUTO-BENCH substrate described within “An Executable Benchmarking Suite for Tool-Using Agents” (Zhong et al., 10 May 2026). The scientific-discovery benchmark discussed here is specifically “Auto-Bench: An Automated Benchmark for Scientific Discovery in LLMs” (Chen et al., 21 Feb 2025).

Within that narrower sense, Auto-Bench functions as a transparent and reproducible platform for quantifying “AI scientist” capability through causal discovery, intervention planning, and iterative hypothesis revision. Its results indicate that high performance on small causal systems does not presently extrapolate to larger graphs or longer trajectories, and the benchmark is designed precisely to expose that failure regime rather than to obscure it.