SciAgentArena: AI Scientific Benchmark

Updated 5 July 2026

SciAgentArena is a benchmark that evaluates AI agents as scientific collaborators in heterogeneous biomedical research, covering tasks from molecules to population genetics.
It categorizes tasks into Data Analysis, Optimization, Discovery, and Validity, using stepwise verification and domain-specific metrics to assess performance under realistic workflow conditions.
Its interactive, agent-agnostic framework allows agents to run in diverse environments, emphasizing robust multi-step workflows and precise scientific validation.

SciAgentArena is a systematic benchmark for evaluating AI agents in real-world scientific research scenarios drawn from emerging needs across multiple domains. Rather than testing simplified question answering, static coding exercises, or narrowly scoped domain tasks, it evaluates whether agents can function as useful scientific collaborators across multiple levels of biomedical research, from molecules to cells, tissues, patients, and population genetics. The benchmark comprises approximately 200 tasks, with Figure 1 indicating 198 tasks, and couples those tasks to stepwise verification and an interactive, agent-agnostic environment for assessing diverse AI agents (Liu et al., 10 Jun 2026).

1. Conceptual scope and motivation

SciAgentArena is built around a two-sided critique of prior evaluation. On one side, many AI-agent benchmarks evaluate mathematics, logic, software engineering, programming, or general web and terminal behavior, and the paper specifically contrasts its setting with AIME, Folio, SciCode, SWE-bench, and TerminalBench. On the other side, many scientific benchmarks focus on question answering, static prediction, coding for one narrowly defined task, or a single scientific modality, and the paper cites GPQA, ScienceQA, and BioML-bench as examples of this limitation. The central claim is that real scientific work is heterogeneous, tool-heavy, multi-stage, and often requires agents to determine whether a task is scientifically valid before attempting it (Liu et al., 10 Jun 2026).

The benchmark organizes this problem space into four recurring capability categories: Data Analysis, Optimization, Discovery, and Validity. Data Analysis asks whether an agent can execute established multi-step workflows correctly. Optimization asks whether it can optimize a solution, select a method well, or design a better solution under constraints. Discovery is used as a category for tasks that require hypothesis or idea generation, although the paper’s strongest empirical evidence comes from analysis, optimization, and validity. Validity asks whether an agent can detect that a task or claim is infeasible, unsupported, or scientifically invalid. This design makes the benchmark less a static test set than a structured probe of scientific behavior under realistic workflow conditions (Liu et al., 10 Jun 2026).

A second organizing principle is evaluation across scales. SciAgentArena spans molecular-scale problems in drug discovery, cellular-scale problems in single-cell omics, tissue-scale problems in spatial omics, patient-scale problems in electronic health records, and population-level problems in genetics, with additional cross-domain tasks such as eQTL computation, drug target identification, and synthetic lethality prediction. This “across scales” framing is meant to test whether current agents can generalize across distinct scientific regimes rather than succeed only in one well-documented niche (Liu et al., 10 Jun 2026).

2. Interactive and agent-agnostic benchmark architecture

A central systems contribution of SciAgentArena is the separation between the runtime framework and the evaluation framework. Each agent is allowed to run in its own compatible environment and toolchain, while outputs are sent to a unified judge backend. The paper presents this split as necessary because different agents may depend on different package versions, wrappers, or execution assumptions. Agents may receive file paths, datasets, prompts specifying scientific objectives, and expected output schemas; they may return Python code, data files, structured JSON, action sequences, rankings, molecular proposals, diagnoses, or full pipelines (Liu et al., 10 Jun 2026).

The environment is interactive in the practical sense that agents must inspect files, execute code, call tools, process intermediate outputs, and sustain multi-step workflows. The benchmark is also explicitly tool-using. Depending on the task family, agents may need cheminformatics software such as RDKit, omics packages such as Scanpy or Squidpy, statistical-genetics software, FHIR query actions, dataframe manipulation, figure interpretation, and code execution with debugging. Intermediate verification is a first-class design principle: many tasks check substeps rather than only final outputs, and examples include checkpointing pipeline stages, verifying files or variables, comparing action sequences to ground truth, and testing whether an agent correctly rejected an invalid request (Liu et al., 10 Jun 2026).

SciAgentArena is described as agent-agnostic and community-extensible. The released resources include task definitions, datasets, benchmark framework code, front-end evaluation pages, submission support, and leaderboard-style infrastructure. The paper explicitly states that it aims to become a kind of “Leetcode” for scientific research. At the execution level, the runtime framework imposes a 24-hour maximum run limit for agents, while molecule-optimization tasks add a 30-minute CPU-only limit and a 100 oracle-call budget (Liu et al., 10 Jun 2026).

3. Domain structure and task families

The benchmark spans five core biomedical domains and a cross-domain layer. The task families are heterogeneous: some are single benchmark items, others are case sets, and others are decomposed into subtasks. The total benchmark size is therefore best understood as a multi-family evaluation suite whose aggregate size is about 198 task items or evaluations (Liu et al., 10 Jun 2026).

Domain	Representative tasks	Notable counts
Computational Drug Discovery	preprocessing, data analysis, molecule optimization, safety assessment, claim validation	78 tasks across five categories
Single-Cell Omics	preprocessing pipeline, method selection, trajectory-preserving integration, perturbation prediction, validity	12 main tasks + 15 validity questions
Spatial Omics	preprocessing, neighborhood graph and enrichment analysis, method selection, validity	11 main tasks + 15 validity questions
EHR Modeling	FHIR workflows, stepwise workflows, rare disease diagnosis, drug recommendation, causal analysis	5 task types covering 505 evaluable patient cases
Genetics	single-ancestry PRS, Mendelian randomization, multi-ancestry PRS, validation	14 + 15 + 18 subtasks + 10 validation questions
Cross-domain	eQTL computation, TargetID, synthetic lethality	paper mentions 4 cross-domain task families

In computational drug discovery, the benchmark covers chemical data preprocessing, chemical data analysis, molecule optimization, chemical safety assessment, and chemical claim validation. Preprocessing tasks include molecular property calculation, substructure filtering, similarity ranking, format conversion, target identification, and formal charge measurement. Data-analysis tasks involve SAR analysis, dimensionality reduction, dataframe cleaning and harmonization, multi-step joins, pipeline repair, visual interpretation, external database lookups, assay harmonization, and ADMET enrichment. Molecule-optimization tasks include rediscovery, isomer enumeration, binary-classifier objectives, narrow-band property targeting, scaffold hopping, similarity-constrained improvement, and multi-property optimization. Claim-validation tasks are intentionally flawed and ask the agent to identify the failure mode, assign conclusion status, and recommend sensible follow-up rather than overclaiming (Liu et al., 10 Jun 2026).

Single-cell omics contains seven preprocessing pipeline tasks—READ, QC, FILTER, DOUBLET, NORM, HVG+PCA, and NEG+VIS—tested in both step-wise and pipeline execution modes, plus three method-selection tasks for clustering, batch-effect correction, and differential expression detection, and two optimization tasks for trajectory-preserving integration and perturbation-effect prediction on unseen perturbations. Spatial omics parallels this structure but adds SNG and NEA, and includes optimization tasks for batch-effect correction, spatially aware domain detection, and spatially variable gene identification. In both domains, validity question sets test whether the agent can determine whether a requested analysis is actually executable or biologically meaningful (Liu et al., 10 Jun 2026).

EHR modeling comprises five task types. T1 translates natural-language clinical questions into structured FHIR action sequences. T2 evaluates stepwise clinical workflows over 455 cases. T3 covers rare disease diagnosis across 20 cases. T4 asks for open-ended inpatient drug recommendation. T5 is a target-trial-style causal inference task on MIMIC-IV Demo. Genetics is similarly decomposed into single-ancestry PRS, Mendelian randomization, multi-ancestry PRS, and validation. The cross-domain layer includes eQTL computation, 20 TargetID tasks, and 20 synthetic lethality tasks, and the text states a total of 4 cross-domain task families (Liu et al., 10 Jun 2026).

4. Scoring, verification, and task-specific metrics

SciAgentArena is not governed by a single benchmark-wide mathematical reward function. The paper explicitly states that there are no central mathematical benchmark-defining equations and that the benchmark is primarily procedural and empirical. Instead, each task family uses domain-specific metrics and rule-based evaluation protocols designed by domain experts (Liu et al., 10 Jun 2026).

Drug-discovery preprocessing tasks execute scripts with timeouts and compare outputs to references using numerical agreement, set overlap, or identifier matching. Chemical data-analysis tasks run code in a fresh Jupyter kernel, check successful execution, and then verify required variables, dataframes, figures, and correctness. Molecule optimization uses a success-rate definition based on executability, chemical validity, and satisfaction of task-specific formal criteria such as similarity thresholds, property windows, or structural pattern matches, under the CPU and oracle-call limits already noted. Safety-assessment tasks use metrics such as accuracy, F1, and AUROC, while claim-validation tasks score whether the agent identified the failure mode, assigned the correct conclusion status, and recommended sensible follow-up (Liu et al., 10 Jun 2026).

In single-cell and spatial omics, analysis tasks use pass@1 checkpointing. Batch-effect correction uses an scIB-inspired score $S_{total}$ that combines batch-effect reduction and biological conservation. Single-cell clustering reports Adjusted Rand Index, Adjusted Mutual Information, Normalized Mutual Information, Homogeneity, Completeness, V-measure, Fowlkes-Mallows Index, and Purity. Differential-expression evaluation compares outputs to marker references using Jaccard similarity, overlap coefficient, precision, recall, and F1. Perturbation prediction uses Pearson correlation for all genes, Pearson correlation for the top 20 DEGs, and mean squared error, computed after subtracting the mean expression of control cells. Spatial domain detection and spatially variable gene detection use analogous metric families, including SAD_Global_Moran_I for spatial domains and Jaccard, overlap coefficient, precision, recall, and F1 for SVG detection (Liu et al., 10 Jun 2026).

EHR tasks T1 through T3 are evaluated with action-level F1 obtained by parsing predicted responses into lists of actions, matching predicted and ground-truth actions by bidirectional substring matching, and macro-averaging across cases. T4 uses the same logic but adds fuzzy string matching for drug names with a SequenceMatcher threshold of ratio $\ge 0.82$ . T5 is the most explicitly structured scoring regime in the benchmark: it evaluates eight dimensions—D1 estimand specification, D2 cohort construction, D3 temporal validity, D4 confounder coverage, D5 method correctness, D6 artifact reproducibility, D7 result accuracy, and D8 robustness or uncertainty reporting—on an ordinal scale $\{0,1,2\}$ for a maximum total of 200 points. D3 is a hard gate against post-treatment leakage, D6 recomputes the point estimate from the submitted cohort file, D1 and D2 are graded by GPT-4o-mini, and the remaining criteria are rule-based (Liu et al., 10 Jun 2026).

Genetics tasks are scored at the subtask level based on successful execution, valid formatting, correct processing, and agreement with references or accepted checks. eQTL computation uses F1, Jaccard, and $PCC_{\beta}$ , with means and standard deviations reported across 100 selected genes. Cross-domain TargetID and synthetic lethality use accuracy. Across the benchmark, the emphasis is on verifiable intermediate states and domain-relevant metrics rather than on a single scalar arena score (Liu et al., 10 Jun 2026).

5. Agents benchmarked and empirical performance

SciAgentArena evaluates 18 agents and models: GPT 5.2, Gemini 3. Pro, Claude Sonnet 4.6, ToolUniverse, Codex, Claude Code, CellForge, STELLA, AutoBA, Biomni, TxAgent, Medea, CACTUS, ChemToolAgent, DrugAgent, LIDDiA, DELTA, and MRAgent. The benchmark includes general LLMs, general coding agents, domain-specialized biomedical agents, chemistry agents, clinical agents, genetics-specific agents, and multimodal LLMs. Agents are run under their maximal functional settings and, where possible, with the most suitable backbone LLMs (Liu et al., 10 Jun 2026).

The paper’s strongest overall empirical claim is that no single agent dominates everything. Claude Code often leads on coding robustness and executable workflow tasks; STELLA, especially STELLA(mem), is particularly strong on structured clinical workflows; ToolUniverse is often strong where tool use maps cleanly to the task; MRAgent is best on Mendelian randomization; Medea leads some cross-domain biomedical tasks; and CACTUS performs strongly on chemical claim validation. The benchmark therefore portrays scientific-agent ability as fragmented and domain-contingent rather than converged into one generally best system (Liu et al., 10 Jun 2026).

The clearest successes occur on well-specified data-analysis workflows. In single-cell preprocessing, Claude Code, ToolUniverse, and STELLA(mem) can pass all preprocessing tests in some settings. In spatial preprocessing, GPT-5.2 is the only agent reported to pass all tests in the step-wise setting. In EHR T1, STELLA(mem) reaches F1 = 0.913 and Gemini 3. Pro reaches F1 = 0.901. In EHR T2, STELLA(mem) reaches F1 = 0.855 on the 435-case set and F1 = 0.891 on the curated 20-case set. In genetics, Claude Code and STELLA(mem) are perfect on both single-ancestry PRS and multi-ancestry PRS, while MRAgent is the only agent to implement all 13 Mendelian-randomization methods and achieves perfect MR performance (Liu et al., 10 Jun 2026).

Performance deteriorates on open-ended optimization and long-horizon tasks. In single-cell perturbation prediction, only Claude Code and STELLA(mem) produce runnable code that beats both the reference method scLAMBDA and the average-expression baseline pert-mean. In EHR T4, Gemini 3. Pro is best but only at F1 = 0.311, indicating that inpatient drug recommendation remains difficult for all agents. In EHR T5, Claude Sonnet 4.6 has the highest total at 88%, followed by GPT-5.2 at 68%, while artifact reproducibility is highlighted as highly discriminative. In drug discovery, agents can often solve direct single-objective optimization tasks but struggle sharply on multi-objective molecular optimization and hard structural constraints, and no agent solves the Valsartan SMARTS task (Liu et al., 10 Jun 2026).

Cross-domain tasks reinforce this unevenness. In eQTL computation, Claude Code identifies more overlapping eQTLs, while GPT-5.2 has the highest $PCC_{\beta}$ , yet several agents cannot generate runnable code and most default to OLS without covariates. Medea leads on synthetic lethality, while ToolUniverse leads on TargetID. These results support the benchmark’s central conclusion that current agents are already useful in structured scientific workflows but are not yet reliable autonomous scientists (Liu et al., 10 Jun 2026).

6. Failure modes and scientific implications

SciAgentArena is designed not only to rank agents but to diagnose recurring failure modes. One major pattern is conservative convergence or popularity bias. Across domains, agents repeatedly choose familiar and heavily documented methods instead of adapting to task-specific constraints: Leiden for clustering, Harmony and then scVI for batch correction, Wilcoxon for differential expression, Leiden-like clustering on spatial graphs, Global Moran’s I for SVGs, PRS-CS or PRS-CSx for PRS, and local-search optimizers such as genetic algorithms, beam search, or hill-climbing in chemistry. The paper interprets this as defaulting toward documentation frequency rather than task-sensitive scientific judgment (Liu et al., 10 Jun 2026).

Another central failure is tool or API hallucination, especially in specialized scientific software. The paper highlights hallucinated functions such as spatial_louvain() in Squidpy and reports that 48.6% of spatial errors are non-existent code. Closely related is runtime fragility: agents often fail to maintain file names, variables, schemas, paths, and intermediate states, producing cross-stage contract drift in long workflows. The benchmark also identifies a more subtle problem: scientifically incomplete but executable code. Biomni’s behavior in scRNA-seq label handling is used as a case where code runs but fails the scientific intent because it simply reuses labels rather than performing robust analysis (Liu et al., 10 Jun 2026).

Validity checking is a particularly important weakness. The benchmark contains many tasks where the correct response is refusal or problem reformulation, yet agents often proceed as though every user request must be feasible. Examples include ranking compounds despite mixed IC50 units, inferring CNVs across healthy pancreas endocrine cells, using unsupported Scanpy or Squidpy functionality, doing causal inference from association-only evidence, or accepting contradictory premises. The paper characterizes this as a form of excessive compliance or sycophancy. It also notes over-generation and under-generation in clinical workflows: some models produce clinically plausible but unnecessary actions, while others under-produce crucial steps. Together, these patterns explain the benchmark’s larger conclusion that agents remain weak at novelty generation, self-directed exploration, and robust solution formulation for open-ended research questions (Liu et al., 10 Jun 2026).

The paper’s implications are therefore largely architectural. Future scientific agents will need stronger grounding, package and API verification, premise checking, explicit refusal behavior, better persistent state tracking, and improved optimization under constraints. The authors also argue that future systems should support more human interaction rather than guessing through ambiguity, especially in settings such as genetics where clarification about paths, covariates, phenotypes, or PRS setup may be essential (Liu et al., 10 Jun 2026).

7. Position within arena-style scientific evaluation

SciAgentArena belongs to a broader family of arena-style and benchmark-style systems for scientific AI, but its emphasis is distinct. SciArena is an open and collaborative platform for evaluating foundation models on scientific literature tasks through community voting on pairwise comparisons, and is centered on literature-grounded long-form responses rather than executable workflows (Zhao et al., 1 Jul 2025). PaperArena evaluates tool-augmented agentic reasoning on scientific literature and provides a modular execution platform for cross-paper, multi-tool reasoning, but it is focused on scientific reading and synthesis over papers rather than on biomedical workflow execution across scales (Wang et al., 13 Oct 2025). SC-Arena formalizes a “Virtual Cell” abstraction and a knowledge-augmented LLM-as-a-judge framework for single-cell biology, but it is a domain-specific natural-language evaluation framework rather than an interactive multi-domain agent benchmark (Zhao et al., 26 Feb 2026). EinsteinArena demonstrates an agent-native platform for open distributed research and discovery with verifiers, leaderboards, and public discussion for mathematical problems, but its domain is persistent decentralized discovery in mathematics rather than heterogeneous biomedical analysis pipelines (Bianchi et al., 9 Jun 2026).

Against that backdrop, SciAgentArena’s distinctive contribution is the combination of real-world biomedical workflows, stepwise verification, interactive execution, agent-agnostic evaluation, and explicit validity testing. Its released resources—the website, GitHub repository, and Hugging Face dataset—reinforce that it is intended as a practical framework for measuring progress in AI agents for science and for guiding the design of future agents capable of addressing complex scientific challenges (Liu et al., 10 Jun 2026).