ScientistTwo: Pioneering the Human Knowledge Frontier with Autonomous AI

Abstract: Scientific discovery is defined by the ability to identify the boundaries of existing knowledge and venture into unexplored territory. The ultimate vision for AI in science is problem-driven autonomous research: given a fundamental challenge by a human expert, the AI independently navigates the scientific landscape, uncovers theoretical and empirical bottlenecks, and systematically expands the frontier of knowledge. In this paper, we introduce ScientistTwo, a fully autonomous multi-agent framework designed to realize this vision. Specifically, ScientistTwo takes an initial problem as input, establishes state-of-the-art baselines, formulates novel hypotheses, and coordinates specialized agents to orchestrate an end-to-end discovery cycle without human intervention. Moreover, the framework rigorously conducts experiments using diverse datasets and metrics, refines methodologies through automated ablation studies, and validates research findings via a closed-loop simulated peer-review rebuttal engine. To evaluate ScientistTwo's capabilities against the highest standards of human scientific achievement, we benchmark it across papers accepted at top-tier conferences such as ICLR, ICML, and NeurIPS. As a result, ScientistTwo autonomously generates expert-level, publishable papers and fully verified, executable codebases. Its solutions consistently outperform human state-of-the-art models, and achieve higher average review ratings than human-authored papers under automated AI review agents. These results show that ScientistTwo is not merely an assistive tool but an autonomous scientific pioneer capable of pushing the frontiers of human discovery. Project website: https://scientist-two.github.io/

• The paper describes ScientistTwo, an autonomous AI system that autonomously conducts problem-driven scientific research, resulting in empirical evaluations, ablations, and conference-ready manuscripts.
• ScientistTwo achieved an 80.4% success rate on 107 tasks, with an average 25.2% improvement over human-based benchmarks, demonstrating significant gains in efficiency and effectiveness in empirical research.
• The system's multi-agent architecture, with specialized critics at each stage, enhances the quality of research by ensuring that only robust, validated artifacts advance through the pipeline.

Problem formulation and contribution

ScientistTwo presents an autonomous multi-agent system for problem-driven empirical research. Given a scientific problem and an existing state-of-the-art implementation, the system is intended to generate a new methodology, executable codebase, empirical evaluation, ablation analysis, and conference-format manuscript without human intervention (2609.19644). The human contribution is restricted to specifying the initial problem; ScientistTwo autonomously selects and evaluates research directions.

The paper’s central claim is stronger than automated machine-learning optimization. Existing systems often optimize a single scalar metric on a fixed benchmark or search within a pre-existing implementation space. ScientistTwo instead treats research as a sequence of interdependent decisions: identify limitations in the baseline, formulate hypotheses, implement them, test them across multiple datasets and metrics, isolate the causal source of improvements, respond to simulated peer review, and verify the resulting artifacts.

The evaluation uses 107 research problems derived from accepted ICLR, ICML, and NeurIPS papers. These tasks span optimization, reinforcement learning, LLMs, interpretability, robustness, privacy, time-series forecasting, and related areas. The headline result is that ScientistTwo reports successful advances on 86 of 107 tasks, corresponding to an 80.4% success rate, with an average relative improvement of 25.2% over the human-designed baselines. These figures establish broad empirical coverage, although they do not by themselves establish that every improvement constitutes a scientifically novel contribution: the system’s own ablation and integrity criteria are therefore important to the interpretation of the result.

System architecture

ScientistTwo is organized as a staged pipeline of specialized agents rather than a single monolithic research agent. Each stage produces an artifact that is evaluated by a critic agent. The critic can accept the artifact, reject it, or request refinement. This produces a recurring generate–evaluate–refine control structure throughout the system.

Figure 1: ScientistTwo’s end-to-end pipeline for limitation analysis, idea generation, empirical validation, ablation, drafting, and simulated peer review.

The initial stage extracts actionable limitations from the target paper and implementation. A Limitation Extractor proposes weaknesses, while a Limitation Verifier assesses whether the extracted set is sufficient to guide improvement. ScientistTwo then generates seed hypotheses and ranks them according to an LLM-based novelty assessment. Only a subset of the highest-ranked ideas is initially implemented, which limits the cost of experimentation but makes the quality of the novelty ranking consequential.

Candidate ideas are evaluated using a subset-to-full-benchmark strategy. ScientistTwo first reproduces the baseline on a representative subset, tests candidate modifications on that subset, and discards ideas that fail substantially. Ideas that show promise are subjected to engineering refinement and then evaluated on the full benchmark. This is computationally pragmatic, but it introduces a selection assumption: subset performance must be sufficiently predictive of full-benchmark performance. The paper does not provide a systematic analysis of cases in which the subset filter eliminates an idea that would have succeeded at scale.

Figure 2: Subset screening, engineering refinement, and full-benchmark validation of candidate ideas.

The idea-refinement stage combines exploitation and exploration. An Idea Evolver uses successful and failed execution traces to propose new hypotheses, while unevaluated seed ideas are retained to prevent premature convergence around an early local optimum. The system terminates when it obtains a target number of successful ideas or reaches a fixed refinement budget, after which a Selector Agent chooses the strongest candidate using full-benchmark results and execution logs.

Ablation studies are treated as part of the discovery process rather than as post hoc documentation. An Ablation Planner constructs component-level experiments, and an Ablation Critic determines whether the proposed components are responsible for the observed gains. If the ablation exposes redundant or counterproductive components, ScientistTwo modifies the method and reruns the evaluation. Importantly, a refinement is retained only if it strictly improves the current best result. This acceptance rule constrains some forms of reward hacking, although it does not eliminate the possibility that the benchmark or metric remains an incomplete proxy for scientific value.

Manuscript generation and review loops

After selecting a method, ScientistTwo drafts a complete paper from the method description, benchmark results, and ablations. A Peer-Reviewer Agent then assigns a numerical score and generates strengths, weaknesses, and targeted questions. If the score is below a predefined threshold, a Rebuttal Planner converts reviewer concerns into supplementary experiments. A Rebuttal Coding Agent executes those experiments, and a Paper Enhancer revises the manuscript.

The system also includes a Meta-Review Agent. Unlike the ordinary review loop, meta-review can trigger revision of the underlying method rather than merely modifying exposition. If the meta-review identifies a substantive methodological weakness, ScientistTwo modifies the idea, verifies that the revised result is strictly better, reruns ablations, and redrafts the paper. This design encodes the paper’s view that peer review should function as an empirical feedback mechanism rather than as a purely editorial stage.

The review experiments provide strong evidence for the utility of this loop, while also revealing an evaluation dependency. Without rebuttal, the ScientistTwo variant receives average scores of 5.2 from ScholarPeer and 5.6 from the Stanford Agentic Reviewer. After one review–rebuttal iteration, the scores rise to 6.9 and 5.8; after two iterations, they reach 7.6 and 5.7, respectively. ScholarPeer acceptance increases from 46.9% without rebuttal to 93.9% after two rounds, while held-out Stanford acceptance increases from 49.0% to 69.4%.

The held-out evaluator is particularly relevant because ScholarPeer is used during development. However, the improvement is asymmetric: repeated optimization improves ScholarPeer substantially, whereas Stanford’s average score slightly decreases between the first and second rounds. This pattern is consistent with partial evaluator overfitting, even though the held-out acceptance rate remains high. The paper appropriately notes that the in-distribution result should not be interpreted as independent evidence of generalization.

Empirical performance

ScientistTwo substantially outperforms the autonomous research baselines included in the study. Against ScientistOne, its ScholarPeer score increases from 3.8 to 7.5 and its ScholarPeer acceptance rate from 14.3% to 91.9%. Under the Stanford Agentic Reviewer, ScientistTwo obtains an average score of 5.7 and a 72.1% acceptance rate, whereas every listed baseline has a zero acceptance rate.

System	ScholarPeer score	ScholarPeer acceptance	Stanford score	Stanford acceptance
ScientistOne	3.8	14.3%	4.1	0.0%
ScientistTwo	7.5	91.9%	5.7	72.1%

The comparison with accepted human-authored papers is more informative than the comparison with weaker autonomous baselines. ScientistTwo-generated papers exceed the average scores of accepted ICLR 2026 and NeurIPS 2025 papers under both automated reviewers. For ICLR-derived tasks, ScientistTwo obtains ScholarPeer 7.0 versus 6.8 for accepted papers and Stanford 5.4 versus 5.2. For NeurIPS-derived tasks, it obtains ScholarPeer 7.3 versus 6.2 and Stanford 5.6 versus 5.5. It does not match the average scores of ICML 2026 Spotlight papers, particularly under Stanford’s evaluator, where it scores 5.7 versus 6.1.

These results support the narrower claim that ScientistTwo can generate manuscripts that satisfy automated acceptance criteria at the level of ordinary top-tier conference papers. They do not establish equivalence with the strongest human research, since spotlight status is a stricter criterion and because automated reviewers are imperfect proxies for expert assessment.

The comparison with AutoSOTA clarifies the distinction between metric optimization and autonomous methodological development. Across successful tasks, ScientistTwo reports a median gain of 7.7% and an average gain of 25.2%, compared with AutoSOTA’s 2.7% median and 7.5% average. On NeurIPS tasks, the corresponding averages are 13.9% and 8.5%. ScientistTwo performs worse on the ICLR subset, with a 3.8% average gain versus AutoSOTA’s 7.2%.

The systems optimize different objectives, so these numbers are not a controlled head-to-head comparison. AutoSOTA often changes configurations within an existing code path, whereas ScientistTwo seeks new mechanisms and requires component-level attribution. The paper’s detailed comparison reports several cases in which AutoSOTA improves a headline metric through configuration changes, while ScientistTwo rejects apparently successful changes because the gain is attributable to generic training controls rather than the proposed contribution. This is a consequential methodological distinction, but it also means that the systems’ success criteria are not directly interchangeable.

Ablation findings and artifact integrity

The idea-improvement ablation shows that gains are largest in early refinement rounds and diminish subsequently. Strong ideas also tend to emerge early, while later rounds increasingly select evolved hypotheses when initial seeds are weak. The result supports the use of both seed exploration and trace-based evolution: seed generation supplies high-quality initial proposals, while evolution provides recovery when those proposals fail.

The review-driven idea-refinement experiment offers a concrete example. Starting from the Engram method for machine unlearning, ScientistTwo first produces LFR-Engram, which improves the overall score from 0.705 for the human baseline to 0.897. After meta-review identifies remaining weaknesses, the system develops FCD-Engram, raising the overall score to 0.916 and improving memory, utility, privacy, exact-match, and forget-quality measures relative to LFR-Engram. This result demonstrates that review feedback can alter the algorithmic proposal rather than simply improve its presentation.

The paper also evaluates artifact integrity using four checks: score reproducibility, specification compliance, reference verification, and method–code alignment. The complete ScientistTwo system passes all four checks on 49 of 49 audited tasks. In contrast, variants without the corresponding refinement agents exhibit failures, including specification violations, hallucinated references, and mismatches between method descriptions and implementations. These audits are among the paper’s more important contributions because autonomous research systems can otherwise produce plausible manuscripts whose reported results are not supported by executable artifacts.

The integrity result should nevertheless be interpreted as a bounded audit result. It verifies the tested repositories and claims under the specified audit procedures; it does not guarantee semantic correctness of every scientific claim, statistical validity of every experiment, or reproducibility under independent hardware and software environments.

Case study: DynaSpec-RAG

The DynaSpec-RAG case study illustrates the intended behavior of the system in a concrete forecasting problem. ScientistTwo identifies three limitations of retrieval-augmented time-series forecasting: discontinuities at the observed–forecast boundary, entanglement of long-term trends with short-term fluctuations, and degradation caused by irrelevant or noisy retrieved trajectories.

DynaSpec-RAG addresses these issues with boundary anchoring, real-FFT decomposition into macro-trends and seasonal components, frequency-specific gating, and a safety switch that suppresses retrieval when it does not improve validation performance. The method operates on frozen time-series foundation models and introduces only 0.27 million trainable parameters.

Figure 3: Architecture of DynaSpec-RAG for frequency-aware, dynamically gated retrieval augmentation.

Across seven datasets, DynaSpec-RAG obtains an average MSE of 0.1892 and MAE of 0.2488, compared with 0.1940 and 0.2492 for TS-RAG. It achieves the best MSE on six of seven datasets and the best MAE on four, while maintaining competitive performance on Electricity, where TS-RAG is slightly better. The results substantiate the claim that the system can produce parameter-efficient methodological changes that generalize across datasets without retraining the backbone.

The case study is also useful because its contribution is structurally interpretable: each component corresponds to a diagnosed failure mode, and the safety fallback explicitly prevents retrieval from causing harm. However, one case study cannot establish that ScientistTwo routinely produces similarly coherent mechanisms across all 86 successful tasks.

Cost, human evaluation, and iterative discovery

ScientistTwo requires approximately 2.5 days and $3,765 per task on average. Idea refinement, peer review, and meta-review dominate the computational budget because they repeatedly execute experiments and generate long contexts. This cost is not incidental: the framework’s empirical rigor depends on running many candidate implementations, ablations, and supplementary experiments. Consequently, its success rate is inseparable from substantial proprietary-model and virtual-machine expenditure.

Human evaluation of 33 generated papers by nine experienced reviewers yields positive standalone ratings across all dimensions. The overall scientific-maturity score is 3.7 on a five-point scale. In pairwise comparison with accepted human papers, ScientistTwo is preferred for benchmark breadth, ablation quality, and experiment analysis, while human papers retain a slight advantage in methodological rigor. Overall, reviewers report parity rather than a clear advantage for either group. This result moderates the stronger automated-review claims and indicates that the remaining gap is concentrated in conceptual and methodological depth.

The sequential frontier-expansion experiment reports compounding gains on incremental BPE tokenization. ScientistTwo first improves the human baseline by 10.9% with VD-STrans, then improves VD-STrans by 9.6% with BXT-Transducer, and finally improves BXT-Transducer by 8.2% with SBR-Transducer. These are conditional relative gains at successive stages, not a single aggregate improvement over the original baseline. Their significance is therefore that the system can reuse its own outputs as research context, although the reported review scores vary across iterations and the experiment concerns one problem family.

Limitations and open questions

The paper’s evaluation is broad but not fully independent. The benchmark consists of existing ML papers and their associated implementations, so ScientistTwo operates in a setting with accessible code, established metrics, and relatively well-defined experimental protocols. It remains open whether the same pipeline can handle problems requiring substantial theoretical invention, new data collection, laboratory instrumentation, or expert judgment that cannot be reduced to executable benchmark evaluation.

The reliance on automated reviewers is another limitation. ScholarPeer is used to refine the papers and therefore cannot serve as an independent evaluator. Stanford Agentic Reviewer is held out, but it is still an LLM-based judge and may reward stylistic or structural properties correlated with acceptance rather than scientific validity. Human evaluation mitigates this concern but covers only 33 papers and reports parity rather than superiority in overall scientific maturity.

The headline 80.4% success rate also conditions on the system’s operational definition of success. Failed tasks are excluded from the average improvement calculation, and the paper does not report a confidence interval or a task-level distribution of gains. Results may therefore be influenced by heterogeneous task difficulty and survivorship among the 86 successful cases. The comparison with AutoSOTA similarly uses different acceptance rules, metrics, and baselines, so its numerical superiority should be regarded as descriptive rather than causal.

Finally, ScientistTwo does not consistently produce spotlight- or oral-level work, and its average cost is approximately $3,800 per task. The paper explicitly concedes that the system currently emphasizes local algorithmic refinement over new theoretical formulations. A specific open question is whether its limitation extraction, ablation, and review loops can generate genuinely new theoretical objects rather than increasingly elaborate modifications of existing empirical methods.

Conclusion

ScientistTwo develops a technically substantial autonomous research pipeline that integrates hypothesis generation, executable experimentation, evolutionary refinement, ablation-based attribution, artifact auditing, and simulated peer review. Its evaluation reports advances on 86 of 107 tasks, a 25.2% average relative improvement over human baselines, high automated acceptance rates, and complete integrity-audit success on the audited benchmark. The strongest evidence concerns empirical ML research with reproducible code and established evaluation protocols. The paper’s results support the claim that multi-agent research systems can perform substantially more than single-metric code optimization, while its own human evaluations, cost analysis, and inability to consistently reach spotlight-level quality delimit the scope of that claim (2609.19644).