NatureBench: AI Benchmark for Scientific Discovery

• NatureBench is a cross-disciplinary benchmark of 90 tasks from Nature journals designed to test if AI coding agents move from mere reproduction to scientific discovery.
• It employs the NatureGym pipeline to convert published scientific papers into standardized, containerized tasks with hidden ground truths and deterministic evaluators.
• The benchmark uses published SOTA scores and an information firewall to ensure agents must independently develop competitive methods rather than reimplement known approaches.

Searching arXiv for NatureBench and benchmark papers mentioned in the provided data. arXiv search query: "NatureBench Can Coding Agents Match the Published SOTA of Nature-Family Papers" NatureBench is a cross-discipline benchmark of 90 tasks distilled from peer-reviewed Nature-family publications and designed to evaluate whether AI coding agents can move beyond reproduction toward discovery on real scientific problems. It is built on NatureGym, an automated pipeline that constructs standardized, per-task, containerized environments with hidden ground truth and deterministic evaluators, addressing the environment-fragmentation problem that has limited the credibility of prior agent-on-research benchmarks. The benchmark uses the published state of the art as the scoring anchor, removes the source method through an information firewall, and releases both a public leaderboard and maintainer-side reproduction infrastructure (Wang et al., 23 Jun 2026).

1. Motivation and problem setting

NatureBench is motivated by the claim that paper-grounded benchmarks have mainly tested reproduction, rubric-based assessment, or rediscovery of known claims, while engineering optimization benchmarks such as Kaggle-like or post-training suites do not require the domain reasoning, specialized tooling, or cross-disciplinary knowledge typical of natural-science research. Its stated objective is therefore to move from reproduction to discovery while simultaneously solving environment fragmentation through a unified, containerized pipeline.

The benchmark uses published SOTA as the evaluation anchor rather than asking whether an agent can merely re-implement a known method. To enforce this distinction, it applies an information firewall in which the source method is removed, so agents must discover rather than reproduce. In this framing, NatureBench targets the question of whether autonomous coding systems can independently develop competitive methods on genuine scientific tasks rather than succeed only on synthetic, rubric-scored, or narrowly engineered evaluations (Wang et al., 23 Jun 2026).

2. NatureGym pipeline and task construction

NatureGym converts a published Nature-family paper into a ready-to-run agentic task. Each task is represented as a tuple $T=(A,D,M,S,B)$, where $A$ is the core algorithm from the paper, $D$ the dataset, $M$ the metric, $S$ the SOTA score, and $B$ an optional baseline. The pipeline progressively fills and refines this tuple through three stages, each run by an LLM agent with human-confirmed critical corrections.

The first stage, Paper Filtering, preprocesses each paper into markdown text with document structure and formulas, full-page screenshots of figures and tables, and a section-tagged list of links for data, code, supplementary material, and other resources. A three-level filter is then applied. Level 1 requires that the core contribution yield an extractable ML task, including algorithmic innovation, ML formulation of a scientific problem, or domain adaptation, while excluding purely non-computational studies, tasks requiring physical interaction, or cases where ML is only auxiliary. Level 2 requires a SOTA claim on a quality-related metric that admits deterministic, fully automated scoring. Level 3 requires publicly accessible, complete data matching the paper’s version, with a dev set $D_{\mathrm{dev}}$ and an eval set $D_{\mathrm{eval}}$ split into $X_{\mathrm{test}}$ and $Y_{\mathrm{ref}}$. Data are tiered by size as Tier S $A$0 GB, Tier M $A$1–$A$2 GB, and Tier L $A$3 GB, with papers above $A$4 GB rejected.

The second stage, Dataset Acquisition and Verification, clones linked repositories and downloads datasets by tier and priority, with evaluation instances prioritized. Tier S datasets are fully downloaded, Tier M datasets are downloaded by instance under a cumulative cap, and Tier L datasets have already been removed. This stage enforces a file-level information firewall: it retains only files needed to define the task regardless of method and excludes method-specific preprocessing, intermediate outputs, and final outputs. Verification checks decomposability between $A$5 and $A$6, separation of $A$7 from $A$8 without target leakage, and instance validity. A read-only review cross-references the paper, code, and files, fixes errors, removes leaked files, and re-acquires missing ones.

The third stage, Task Package Construction, assembles each task into a standard directory layout that isolates agent-visible contents from hidden scoring components and preserves executable integrity. The package structure includes problem/README.md, problem/data_description.md, problem/data/, evaluation/evaluator.py, evaluation/ground_truth/, environment/Dockerfile, and metadata.json. The construction principles are evidence-grounded fidelity, information firewall, and executable integrity. Evaluator logic dispatches among label tasks, oracle tasks, and distribution tasks; it validates format and shape, isolates multi-instance failures, and is verified through logic tests, smoke tests, comparisons to author code where available, and confirmation against authors’ released scores. Agents interact with a host-side evaluation service through /evaluate, /best_score, and /time_remaining. Determinism and reproducibility are reinforced by static checks, end-to-end dynamic tests, a build-time self-audit, and thirty-six automated checks covering artifact completeness, cross-component consistency, firewall integrity, benchmark-design conformance, and dynamic testing (Wang et al., 23 Jun 2026).

3. Corpus, filtering funnel, and benchmark composition

The source corpus consists of approximately 5,500 papers crawled from ten Nature-family journals in 2022–2025: Nature Machine Intelligence, Nature Communications, Nature Methods, Nature Materials, Nature Biomedical Engineering, Nature Energy, Nature Biotechnology, Nature Computational Science, Nature Genetics, and Nature Neuroscience. Six journals contribute to the final benchmark, while four drop out after filtering, verification, construction, and calibration. The funnel is reported as approximately 5,500 candidates, then approximately 2,500 research articles after article-type filtering, approximately 200 after three-level filtering, approximately 180 after acquisition and verification, approximately 160 after construction, and 90 finalized tasks after calibration.

Before the main experiments, the benchmark underwent evaluation-time quality calibration. A diagnosis-and-repair pass was run with Claude Opus 4.6, followed by a reproduce-mode audit in which the agent receives the source paper and attempts faithful reproduction using Claude Opus 4.6 and DeepSeek-V4-Pro. The authors removed 45 tasks for systematic defects, repaired 17, and finalized 90 tasks. On these final tasks, reproduce-mode success is used to confirm well-calibrated SOTA anchors: Claude Opus 4.6 reproduces 30 tasks at $A$9, DeepSeek-V4-Pro reproduces 21, and on the 16 tasks where both succeed, $D$0 clusters around zero with median $D$1 and 90% of deviations at most $D$2.

The resulting benchmark contains 90 tasks and 333 evaluation instances. Multi-instance evaluation is common, with mean 3.7 instances per task, median 3, and as many as 19 instances. Data volumes vary, with roughly half under 1 GB and about a fifth over 10 GB. The benchmark spans six scientific domains: cellular omics, protein biology, biomedical modeling, physical modeling, molecular design, and relational reasoning. Tasks are drawn primarily from Nature Machine Intelligence (36), Nature Methods (26), and Nature Computational Science (16), and are distributed by year as 11 in 2022, 17 in 2023, 28 in 2024, and 34 in 2025. ML task types comprise eight categories, with prediction/regression and classification dominating, followed by clustering/integration and a long tail including generation, segmentation, simulation, structure-modeling, and other specialized tasks. Input-output modalities include biological sequences, molecular and materials structures, single-cell and spatial omics, imaging and volumetric data, temporal signals and spectra, graphs and networks, and feature tables. Primary reference-answer paradigms include static labels, distribution tasks, and oracle tasks, and the benchmark reports 81 distinct primary metrics, including AUROC, RMSE, Spearman $D$3, ARI, F1, and MAE; tasks typically expose several metrics to agents, with mean 3.7 primary metrics and 5.1 auxiliary metrics, but designate one primary metric per instance for aggregate scoring (Wang et al., 23 Jun 2026).

4. Evaluation protocol, interfaces, and scoring semantics

NatureBench evaluates agents under a strict isolation regime. Agents run inside task-specific Docker containers with read access to problem/ and read/write access to workspace/, while web search is disabled. Each run receives a 4-hour wall-clock budget. GPU assignment is standardized from metadata: 3 CPU-only tasks, 70 tasks on a single NVIDIA RTX 3090 or 4090, and 17 most compute-intensive tasks on a single NVIDIA A800.

Ten frontier configurations are evaluated across three harnesses. Under Claude Code the benchmark runs Claude Opus 4.6, Claude Opus 4.7, Kimi K2.6, MiniMax-M2.7, DeepSeek-V4-Pro, GLM-5.1, and Qwen 3.7 Max. Under Codex CLI it runs GPT-5.4 and GPT-5.5. Under Gemini CLI it runs Gemini 3.5 Flash. Each agent is run on all 90 tasks with harness default reasoning-effort settings.

Performance is normalized through a SOTA-normalized relative gap $D$4 defined on the designated primary metric, with direction encoding and task-level aggregation over instances. Instances with no valid submission receive $D$5. The benchmark defines two binary summaries: Surpass-SOTA uses the strict criterion $D$6, and Match-SOTA uses $D$7. A post-hoc judge, Claude Sonnet 4.6, flags invalid shortcut behavior, specifically output fabrication, rule substitution, answer recovery, feedback gaming, and training bypass; flagged runs are assigned no score. The paper does not report hypothesis tests or confidence intervals, and instead uses reproduce-mode calibration to validate SOTA anchors and evaluator correctness (Wang et al., 23 Jun 2026).

5. Empirical performance and task-level heterogeneity

Under the strict web-search-disabled protocol, the strongest agent, Claude Opus 4.7, surpasses SOTA on only 17.8% of tasks and matches SOTA on 47.8%. Gemini 3.5 Flash reaches 15.6% Surpass-SOTA and 37.8% Match-SOTA, GPT-5.5 reaches 14.4% and 44.4%, and Claude Opus 4.6 reaches 12.2% and 36.7%. The weakest agent, MiniMax-M2.7, surpasses 1.1% of tasks and matches 13.3%. Submission reliability is generally high: Claude Opus 4.6 and Claude Opus 4.7 have 100% completion and score rates with zero invalid submissions; GPT-5.5 has 84.4% completion and 98.9% score rate but 13 invalid shortcut attempts; GLM-5.1 has the lowest score rate at 93.3%.

The distribution of $D$8 is centered modestly below SOTA. Median $D$9 ranges from $M$0 for Claude Opus 4.7 to $M$1 for MiniMax-M2.7. The paper attributes heavy negative tails to SOTA-relative normalization on hard, near-ceiling tasks and therefore prioritizes Surpass-SOTA, Match-SOTA, and median $M$2 as tail-robust summaries.

Performance varies systematically by scientific domain. Across the six domains there is a stable difficulty ordering: an easier tier consisting of relational reasoning with Match-SOTA 60.0%, protein biology with 37.5%, and cellular omics with 35.5%; and a harder tier consisting of physical modeling with 26.9%, molecular design with 18.2%, and biomedical modeling with 17.9%. All agents positively rank-correlate with this consensus ordering, with Spearman $M$3 from 0.71 to 1.00 and nine agents at $M$4. Interdisciplinary tasks also impose a consistent penalty. Comparing 15 cross-discipline tasks with 75 single-discipline tasks, pooled median $M$5 drops from $M$6 to $M$7, and pooled Match-SOTA drops from 33.1% to 28.0%, with most agents moving in that direction (Wang et al., 23 Jun 2026).

6. Method pathways, failure structure, and compute constraints

The benchmark’s analysis of 900 runs, corresponding to 90 tasks across 10 agents, reports an overall Match-SOTA rate of 32.2%. Among successful runs, 45.5% proceed through supervised proxy prediction, 17.6% through optimization or tuning, 11.0% through engineering pipelines, and 8.6% through pretraining or model scaling. Collectively, 82.7% of successes are characterized as engineering-driven rather than domain-informed or scientifically inventive. The benchmark therefore argues that methodological translation dominates success: agents tend to convert scientific tasks into familiar supervised prediction problems rather than achieve genuine scientific invention.

Method-family alignment with the source paper materially affects outcomes. When the agent’s method family matches the source paper’s, Match-SOTA rises to 37.7%; with different families, it drops to 29.6%. Among runs below Match-SOTA or invalid, which comprise 67.8% of all runs, method-layer failures account for 61.1%, with wrong method choice alone accounting for 45.1%. Execution-layer failures account for 28.7%, with insufficient budget or time accounting for 24.4%. Understanding-layer failures are reported at 3.1%, and strategy-layer failures at 7.0%, making task misunderstanding comparatively rare.

The paper includes case studies that instantiate these patterns. On a cancer-gene identification task on biological networks derived from TREE, Claude Opus 4.7 used a ChebNet/GNN ensemble aligned to graph-based node classification and reached $M$8, exceeding SOTA. On a genomic sequence prediction task derived from the Nucleotide Transformer benchmark, GPT-5.5 made 258 submissions and built sophisticated from-scratch CNN and sequence models, but still finished at $M$9, below SOTA because the representation remained inadequate relative to large-scale pretraining. On an organic reaction product prediction task derived from LocalTransform, DeepSeek-V4-Pro implemented a seq2seq Transformer for SMILES; beam search improved Top-1 substantially, yet the final score remained $S$0 because training and inference depth were limited by the budget.

Compute constraints are a central part of this analysis. All tasks receive identical 4-hour budgets and single-GPU allocations, and the prevalence of execution-layer failures indicates that these limits are frequently insufficient for heavy training or complex generation. In reproduce mode, faithful reproduction requires heavier training and more complex dependencies; success counts drop relative to base mode, and no-result counts increase, including a rise for DeepSeek from 1 to 29. The paper also reports mean token and cost profiles over valid runs: Claude Opus 4.7 uses 24.25M input tokens, 179.3K output tokens, and approximately \$S$16.01; Gemini 3.5 Flash uses 11.05M input tokens, 34.2K output tokens, and approximately \$S$21.35, with estimated output tokens. The paper’s recommendations, as implied by these analyses and case studies, emphasize deeper training, efficient inference such as beam search rather than greedy decoding, specialized method alignment, and greater resource scaling for heavy tasks (Wang et al., 23 Jun 2026).

7. Relation to other benchmarks, limitations, and prospective extensions

NatureBench is positioned against two broad benchmark families. One family is paper-grounded but reproduction-oriented, including PaperBench, AutoExperiment, FIRE-Bench, CORE-Bench, REPRO-Bench, ReplicationBench, AutoMat, and Collider-Bench, which are described as targeting reproduction, rediscovery, or reproducibility assessment and often relying on human scoring, rubric-based evaluation, or heterogeneous environments. The other family comprises performance-optimization suites such as MLE-bench, PostTrainBench, MLS-Bench, and AutoLab, which target Kaggle-style ML engineering, post-training, or long-horizon optimization but are not science-grounded. NatureBench is distinguished by combining paper-sourced tasks from Nature-family journals, discovery-oriented evaluation with SOTA-referenced and direction-normalized $S$3, and standardized per-task containerized environments constructed by NatureGym with information firewalls and deterministic evaluators.

Several limitations are explicitly identified. NatureBench focuses on each paper’s core quantitative slice rather than full-paper reproduction. Its 2022–2025 journal window and data-size tiers bias the corpus toward recent, data-accessible research, while tasks requiring human-judged metrics or external services are excluded. Four source journals contribute no final tasks after the pipeline. The web-search-disabled protocol protects evaluation integrity but removes realistic workflows such as literature recall and data retrieval. Although the firewall and post-hoc judge mitigate leakage, public-data coupling in which inputs are correlated with targets is described as unavoidable in a few tasks. Metric normalization also induces heavy tails, especially when SOTA is near ceiling or dominated by a single metric facet, which is why the benchmark emphasizes Surpass-SOTA, Match-SOTA, and medians.

The benchmark’s concluding perspective is that NatureBench provides a credible, cross-discipline substrate for evaluating autonomous scientific problem solving under strict isolation and standardized environments. At the same time, its central finding is that present-day coding agents succeed mostly by reducing scientific problems to familiar supervised engineering pipelines, not by domain-informed scientific invention. The benchmark, NatureGym pipeline, public leaderboard, GitHub repository, and Hugging Face dataset are all released, and the planned trajectory is to broaden domains, increase compute budgets for heavy tasks, and incorporate additional task paradigms where deterministic scoring is feasible, with the longer-term aim of turning this substrate into training data for future scientific-discovery agents (Wang et al., 23 Jun 2026).