NatureBench: Can Coding Agents Match the Published SOTA of Nature-Family Papers?

Abstract: We introduce NatureBench, a cross-discipline benchmark of 90 tasks distilled from peer-reviewed Nature-family publications, designed to evaluate whether AI coding agents can move beyond reproduction toward discovery on real scientific problems. NatureBench is built on NatureGym, an automated pipeline that constructs a standardized, per-task containerized environment from a source paper, addressing the environment-fragmentation problem that has limited the credibility of prior agent-on-research benchmarks. Evaluating ten frontier agent configurations under a strict web-search-disabled protocol, we find that the strongest model surpasses SOTA on only 17.8% of tasks under the g>0.1 criterion. Analysis of method pathways reveals that agents succeed primarily through methodological translation, converting scientific tasks into familiar supervised prediction problems, rather than through genuine scientific invention. Failures are dominated by wrong method choice and insufficient compute budget, not by task misunderstanding. We release the benchmark, the NatureGym pipeline, and a public leaderboard with maintainer-side reproduction. Code: https://github.com/FrontisAI/NatureBench

• The paper introduces NatureBench, a benchmark evaluating AI coding agents on 90 scientific discovery tasks from Nature journals, emphasizing reproducibility and SOTA alignment.
• It details the NatureGym pipeline that converts heterogeneous paper formats into containerized, reproducible tasks with rigorous information firewalls and automated evaluators.
• Experimental results reveal that agents mainly perform methodological translation with significant gaps from published SOTA, highlighting limits in autonomous scientific discovery.

NatureBench: Evaluating Coding Agents on Scientific Discovery Tasks

Benchmark Motivation and Design

NatureBench addresses the critical lack of standardized evaluation for AI coding agents tasked with scientific discovery, moving beyond reproduction toward genuine problem-solving across scientific domains. Instead of relying on paper-based reproduction (such as PaperBench, CORE-Bench) or engineering-optimization benchmarks (e.g., MLE-bench, PostTrainBench), NatureBench uniquely combines paper-sourced tasks from scientific domains with an optimization-oriented evaluation anchored to published SOTA claims.

The benchmark is constructed from 90 tasks distilled from peer-reviewed Nature-family publications covering six domains: cellular omics, protein biology, biomedical modeling, physical modeling, molecular design, and relational reasoning. The tasks are generated by NatureGym, an automated pipeline that converts heterogeneous paper formats into containerized, reproducible task packages. Each package provides a task brief, dataset, held-out test set, and automated evaluator, with an information firewall removing source methods to ensure agents engage in discovery—rather than mere reproduction. The core metric for evaluation is the SOTA-normalized relative gap $g$, ensuring scale-free and direction-normalized comparison across tasks with heterogeneous metrics.

Figure 1: NatureBench covers six scientific task domains with diverse ML task types and sources, shown here alongside the leaderboard for ten evaluated models.

NatureGym Pipeline and Task Construction

NatureGym forms the backbone of NatureBench, implementing several stages: paper filtering, dataset acquisition and verification, and task package construction. Filtering ensures extractable ML tasks, automated evaluation, and data completeness, followed by adversarial review. Only paper components definitional to the task—irrespective of method—are retained, establishing a strict information firewall.

Datasets are acquired and verified at the file level, re-confirming boundaries and decomposability for development and evaluation splits. Task packages are formatted with agent-visible documents (problem/README.md, data_description.md), data folders, and hidden evaluation logic. Each evaluator dispatches on reference-answer types (label, oracle, distribution), with rigorous self-audit and environment verification via Docker builds. This meticulous review process guarantees fidelity and reproducibility, addressing persistent credibility issues in earlier scientific-agent benchmarks.

Figure 2: The three-stage NatureGym pipeline transforms a Nature-family paper into a disciplined, containerized task package guarded by an information firewall.

Benchmark Composition, Quality, and Calibration

NatureBench's 90 tasks were distilled from $\sim$5,500 papers (2022–2025) from ten Nature-family journals via five pipeline phases. Final tasks are highly heterogeneous, spanning eight ML types (classification, regression, clustering, generation, segmentation, simulation, structure modeling), numerous data modalities (sequences, images, graphs, tables), and evaluation paradigms (static label, distribution, oracle).

Benchmark quality is safeguarded by multi-layer calibration. Agents (Claude Opus 4.6, DeepSeek-V4-Pro) are run in base and reproduce modes; defects (leakage, shortcutting, inconsistent metrics, environment failures) are diagnosed, repaired, or excluded. Tasks are only retained if reproducibility relative to SOTA is confirmed, as shown by tight reproducibility metrics clustering around zero in successful cases.

Figure 3: NatureBench's broad coverage and heterogeneity in scientific domains, ML task families, data, and contribution types.

Figure 4: Reproduce-mode calibration probes task quality, attributing discrepancies to agent limits rather than package defects.

Experimental Evaluation and Results

Ten frontier coding agents were evaluated, including Claude Opus 4.7, Gemini 3.5 Flash, GPT-5.5, among others. Agents were run in isolated, web-search-disabled containers with identical wall-clock and compute budgets, ensuring uniformity. Main results demonstrate that clear improvements over published SOTA ($g > 0.1$) are rare: Claude Opus 4.7 surpasses SOTA on only 17.8% and matches it on 47.8% of tasks, with other models trailing further.

Completion and validity rates are uniformly high, with minimal shortcut behavior (invalid submissions). The score distribution is concentrated in modest sub-SOTA ranges, indicating most agents are runnable but not competitive with published scientific methods.

Figure 5: NatureBench's evaluation protocol isolates agent-visible task descriptions and data from evaluation components, reinforcing method independence and scoring integrity.

Figure 6: Gap distributions reveal that most agent runs perform below SOTA, with a heavy negative tail for weaker models.

Mechanistic Analysis and Failure Modes

Detailed annotation of all 900 agent-task runs reveals that successful agents primarily engage in methodological translation—reducing scientific tasks to familiar ML pipelines (supervised prediction, engineering optimization)—rather than scientific invention. Only 8.3% of successes reflect genuine domain reasoning. Failures are dominated by wrong method choice (45.1%) and insufficient compute budget (24.4%), not misunderstanding or malformed output.

Figure 7: Solution mechanism breakdown shows methodological translation dominates successes, while method selection and execution depth drive failures.

Domain and Interdisciplinary Difficulty

Difficulty varies systematically by domain. Relational Reasoning, Protein Biology, and Cellular Omics form an easier tier (up to 60% Match-SOTA), while Physical Modeling, Molecular Design, and Biomedical Modeling are significantly harder (down to 18%). Consistency in difficulty gradients across all agents demonstrates persistent domain-specific gaps in capability. Interdisciplinary tasks widen the gap: single-discipline tasks show higher performance ($\tilde{g}_{\text{all} = -0.13$) compared to cross-discipline tasks ($\tilde{g}_{\text{all} = -0.21$).

Figure 8: Domain- and interdisciplinary-performance decompositions reveal a stable difficulty gradient and larger gap for cross-discipline tasks.

Case Studies

Agent trajectories illustrate the spectrum of success and failure. Method-aligned graph modeling (ChebNet ensemble) in cancer-gene identification matched SOTA; extensive iteration for genomic sequence prediction still fell short due to limited representation strength; plausible sequence-to-sequence modeling for reaction product prediction was stymied by training/inference budget constraints.

Figure 9: Agent trajectories show varying submission counts and score improvements, highlighting iteration dynamics and bottlenecks across tasks.

Figure 10: Best submission breakdowns show significant per-instance variability in gap for multi-instance tasks.

Benchmark Validity and Implications

The SOTA-normalized gap metric reliably measures agent performance, with extreme scores reflecting metric normalization rather than task error. Task coverage is bounded to quantifiable problem slices, and protocol safeguards (information firewall, validity judge, web search disablement) constrain leakage and feedback risks. NatureBench creates a credible substrate for AI-native scientific discovery evaluation.

Practical and Theoretical Implications

NatureBench shows conclusively that frontier coding agents remain far from SOTA on cross-disciplinary scientific discovery, with their dominant success mode being methodological translation rather than invention. This suggests that future advances will depend not only on scaling compute and improving code synthesis, but also on method-agile, domain-reasoned solution generation. NatureBench establishes rigorous infrastructure to support future AI scientist training, benchmarking, and leaderboard-driven cumulative progress.

Conclusion

NatureBench, supported by the NatureGym pipeline, demonstrates that current coding agents are limited in their capacity for autonomous scientific discovery, largely due to method selection and execution depth bottlenecks. By providing containerized, reproducible, cross-domain benchmarks tied to published SOTA claims, NatureBench forms a foundational standard for measuring genuine algorithmic progress in AI-for-Science, with implications for both practical agent deployment and theoretical advancement in cross-disciplinary reasoning and scientific automation (2606.24530).