---
title: 'ScientistTwo: Autonomous Research Framework'
url: https://www.emergentmind.com/topics/scientisttwo
type: topic
---

# ScientistTwo: Autonomous Research Framework

ScientistTwo is a fully autonomous, multi-agent framework for problem-driven scientific research. Introduced in “ScientistTwo: Pioneering the Human Knowledge Frontier with Autonomous AI” [2609.19644], it accepts an initial scientific problem, typically represented by an existing paper, benchmark, method, and codebase, and autonomously performs limitation extraction, hypothesis generation, implementation, experimentation, ablation, manuscript drafting, simulated peer review, rebuttal, meta-review, and integrity auditing. Its stated output is an improved manuscript and a corresponding reproducible codebase. The system is primarily evaluated on computational machine-learning research problems derived from accepted ICLR, ICML, and NeurIPS papers.

## 1. Scientific objective and conceptual scope

ScientistTwo is designed around **problem-driven autonomous scientific discovery**. A human supplies a scientific challenge, while the system independently analyzes the relevant state of the art, identifies theoretical, methodological, empirical, or implementation limitations, proposes hypotheses, tests them computationally, and iteratively refines successful methods.

The framework distinguishes this operating mode from narrower automated machine-learning systems that optimize a scalar metric on a single dataset. Its intended research lifecycle is:

$$
(\mathcal{P}^{+},\mathcal{C}^{+})=\mathcal{A}(\mathcal{G}),
$$

where $\mathcal{G}$ is the initial scientific problem, $\mathcal{A}$ is the autonomous system, $\mathcal{P}^{+}$ is the improved manuscript, and $\mathcal{C}^{+}$ is the corresponding reproducible codebase.

ScientistTwo’s claimed contribution is the integration of several mechanisms:

- **Holistic evaluation** across multiple datasets and metrics rather than single-metric optimization.
- **Subset-first experimentation** to eliminate weak hypotheses before full-scale evaluation.
- **Iterative hypothesis evolution** using both successful and failed experimental traces.
- **Automated component-level ablations** for method refinement and causal attribution.
- **Closed-loop peer review and rebuttal**, where criticism generates additional experiments.
- **Meta-review-driven algorithmic refinement**, which can modify the method rather than only the prose.
- **Integrity auditing** for scores, specification compliance, references, and paper-code alignment.

The system is therefore intended to automate a broad research workflow rather than merely perform code generation or hyperparameter search. Its evidence, however, concerns primarily computational machine-learning research and benchmarked algorithmic refinement. The supplied account does not establish general superiority over human scientists in selecting important questions, developing paradigm-changing theories, conducting physical experiments, or producing independently validated knowledge.

## 2. Architecture and agent communication

ScientistTwo is decomposed into specialized agents:

$$
\mathcal{A}=\{\mathcal{A}_1,\mathcal{A}_2,\ldots,\mathcal{A}_{N_a}\}.
$$

Agents are assigned to research phases including limitation extraction, ideation, coding, empirical validation, ablation, drafting, reviewing, rebuttal, and auditing. The framework is not described as a monolithic model: foundation models provide reasoning and coding capabilities, while orchestration logic controls bounded loops, artifact comparison, filtering, and promotion.

Agents communicate through candidate artifacts, execution traces, categorical decisions, review feedback, and persistent state. Artifacts include ideas, code, results, manuscripts, and review reports. Decisions include `Good`, `Bad`, `Engineer`, `Accept`, and `Refine`.

A generic refinement stage is represented by a bounded critic–refine loop:

```python
def stage(candidate, critic, refine, max_rounds):
    for _ in range(max_rounds):
        verdict, feedback = critic(candidate)
        if verdict == "accept":
            return candidate
        elif verdict == "refine":
            candidate = refine(candidate, feedback)
        else:
            return None
    return None
```

The reported default configuration uses **Gemini 3.6 Flash** for most agents and **Claude Code with Opus 4.8** for the Idea Experiment Coding Agent, Ablation Study Agent, Rebuttal Agent, and Draft Enhancer. A generalization test replaces Claude Code with **Antigravity**, powered by Gemini 3.8 Flash. Novelty checking retrieves two reference papers using Google Search. Paper drafting follows the ICLR 2025 format and uses PaperOrchestra. Peer review uses ScholarPeer during development and Stanford Agentic Reviewer as an external or held-out evaluator.

The architecture is organized around staged candidate production and criticism. A hypothesis is not promoted solely because it is novel or because it produces an isolated numerical gain. It must generally survive implementation, full-set evaluation, ablation, comparison, review, and integrity checks.

## 3. End-to-end research workflow

### Initial analysis and hypothesis generation

A **Limitation Extractor** identifies weaknesses in the reference method. A **Limitation Verifier** determines whether the extracted limitations are actionable for generating improvements. If they are insufficient, the extraction process is repeated, with a reported maximum of 16 rounds.

A **Novelty Checker** scores candidate hypotheses and ranks them by novelty. The initial seed pool is denoted by $\mathcal{H}_0$. An **Idea Generator Agent** adds candidate ideas, after which only the highest-priority candidates are initially implemented.

ScientistTwo combines exploration and exploitation. Exploration evaluates previously unevaluated seed ideas, while exploitation generates evolved ideas from prior experimental evidence. At refinement round $k$, the Idea Evolver examines earlier traces:

$$
\mathbf{R}_{<k}=\bigcup_{i=0}^{k-1}\mathcal{R}_i.
$$

Each trace contains a hypothesis, execution results, codebase, decision, and feedback or rationale:

$$
(h,\mathcal{E}^{h},\mathcal{C}^{h},d^{h},r^{h}).
$$

The evolved candidate set is $\mathcal{I}_k$, and the total candidate pool is:

$$
\mathcal{H}_k=\mathcal{I}_k\cup\mathcal{H}_0^{(k)}.
$$

Under the reported configuration, two candidates are evaluated per round: one seed idea and one evolved idea. At most four refinement rounds are run, and the process can stop early after four successful ideas.

### Baseline reproduction and subset-first experimentation

Before evaluating novel hypotheses, a **Baseline Coding Agent** reproduces the reference method on a representative benchmark subset. The resulting baseline execution evidence and runnable code become comparison artifacts.

ScientistTwo then applies a subset-first strategy:

1. Reproduce the baseline on a subset.
2. Test candidate ideas on that subset.
3. Discard weak ideas at relatively low computational cost.
4. Scale promising ideas to the complete benchmark.

For a candidate hypothesis $h$, a **Subset Coding Agent** produces modified code and execution results. A **Subset Critic Agent** assigns one of three decisions:

- **Bad**: the candidate is substantially inferior and is discarded.
- **Good**: the candidate is consistently better and is promoted.
- **Engineer**: the candidate is promising but requires implementation or hyperparameter refinement.

An **Engineering Agent** responds to `Engineer` feedback for a bounded number of iterations. Candidates that do not achieve a `Good` decision within the budget are pruned. Promising candidates proceed to full-set implementation and validation through corresponding coding, critic, and engineering agents.

### Selection and ablation

A **Selector Agent** compares successful candidates using full-benchmark metrics, execution traces, and code artifacts. The selected method is not necessarily the most novel hypothesis; selection depends on empirical performance and the quality of the complete evaluation.

ScientistTwo treats ablation as a central scientific mechanism. An **Ablation Planner Agent** decomposes the selected method into components and proposes executable plans. An **Ablation Coding Agent** removes or modifies components, while an **Ablation Critic Agent** evaluates whether the reported improvement is attributable to the proposed mechanism rather than incidental engineering, generic training controls, or an evaluation artifact.

If ablation evidence is inadequate, a **Full-Set Engineering Agent** revises the method. The revised method is retained only if a **Result Comparison Agent** finds it strictly superior to the prior best method. Under the reported configuration, ablation-based refinement is allowed at most once.

This requirement differentiates ScientistTwo from systems that accept any numerical improvement. In the reported case of DMC-TeCh, a candidate that exceeded the baseline on five of six metrics was rejected because its ablation critic attributed the gain to EMA and label smoothing rather than the proposed multi-core mechanism.

### Drafting, peer review, and rebuttal

An **Initial Drafter Agent**, incorporating PaperOrchestra, receives the selected method, main results, ablation results, code, and execution logs. It generates a conference-formatted manuscript.

A **Peer-Reviewer Agent**, implemented using ScholarPeer, evaluates the manuscript for strengths, weaknesses, questions, and targeted criticisms and assigns a review score on a 1–10 scale. If the score is below the nominal acceptance threshold of 8, a **Rebuttal Planner Agent** converts criticism into supplementary experimental tasks. A **Rebuttal Coding Agent** executes those tasks, and a **Paper Enhancer Agent** revises claims, tables, figures, limitations, and method descriptions.

The review cycle continues until the score reaches 8 or the iteration budget is exhausted. The reported configuration allows at most two review rounds.

A **Meta-Review Agent** then returns either `Accept` or `Refine`. `Accept` exports the manuscript and code. `Refine` sends the criticism back to the engineering stage, where the underlying scientific method—not merely its presentation—is changed. The revised method must again outperform the previous best and pass ablation, drafting, review, rebuttal, and meta-review stages.

## 4. Reproducibility and integrity mechanisms

ScientistTwo incorporates a post-hoc **Chain-of-Evidence Integrity Audit** with four principal checks.

**Score verification** reruns generated code and compares reported scores with reproduced scores. Coding agents are prompted to provide self-contained scripts, execution instructions, reproducible outputs, and complete code artifacts.

**Specification compliance** checks whether the implementation follows task rules and avoids reward hacking, including modifying the evaluation protocol, omitting required datasets, or exploiting an unmeasured metric.

**Reference verification** uses a search-augmented LLM to check bibliography entries against live search results. Unsupported or hallucinated references are corrected or removed.

**Method-code alignment** compares the repository with the manuscript and generates an audit report identifying discrepancies between the described algorithm and the implementation.

In the final reported configuration, the audit results were:

- Score verification: $49/49$.
- Specification violations: $0/49$.
- Reference-verification failures: none reported in the final row.
- Method-code alignment: $49/49$.

Variants with refinement or auditing mechanisms removed exhibited failures in specification compliance, reference verification, and method-code alignment. These results indicate that ScientistTwo treats reproducibility controls as structural components of the pipeline rather than merely as prompt instructions.

The audits do not prove scientific validity. Correct score reproduction and paper-code alignment do not establish that a research question is important, that a causal interpretation is correct, or that the method generalizes beyond the benchmark.

## 5. Evaluation design and reported performance

### Benchmark composition

ScientistTwo is evaluated on 107 research problems derived from:

- 38 NeurIPS 2025 accepted papers.
- 5 ICLR 2026 accepted papers.
- 64 ICML 2026 Spotlight papers.

The tasks cover large language models, interpretability, time-series forecasting, optimization, reinforcement learning, robustness, privacy, causal inference, robotics and scene generation, neuroscience and EEG, speech and multimodal learning, game theory, graph learning, and generative modeling.

Because the tasks are heterogeneous, ScientistTwo retains the original paper-specific metrics. Aggregate measures include relative performance gain, median and average gain, review ratings, acceptance rates, human reviewer ratings, and integrity-audit pass rates.

The headline results are:

- 86 of 107 papers improved.
- An 80.4% success rate.
- A reported 25.2% average relative improvement over human state-of-the-art methods.
- Publication-quality and executable paper–code outputs according to the paper.
- Higher average automated-review ratings than several compared autonomous systems.

The 86 successful papers are those for which complete ScientistTwo outputs were generated; the success rate is calculated over all 107 input tasks.

### Comparison with autonomous systems

| Framework | Papers | ScholarPeer rating | Stanford rating | Stanford acceptance |
|---|---:|---:|---:|---:|
| ScientistOne | 21 | $3.8\pm1.2$ | $4.1\pm0.7$ | 0.0% |
| AI Scientist-v2 | 3 | $2.0\pm1.0$ | $2.5\pm0.2$ | 0.0% |
| DeepScientist | 3 | $3.0\pm0.0$ | $4.1\pm0.6$ | 0.0% |
| ScientistTwo | 86 | $7.5\pm1.3$ | $5.7\pm0.6$ | 72.1% |

ScientistTwo’s reported ScholarPeer acceptance rate is 91.9%, while its Stanford acceptance rate is 72.1%. Comparisons are affected by different numbers of papers, different task sets, differences in paper-generation pipelines, and the fact that ScholarPeer is used inside ScientistTwo’s development loop.

Compared with AutoSOTA, ScientistTwo reports a median gain of 7.7% and an average gain of 25.2% over 86 tasks, whereas AutoSOTA reports a median gain of 2.7% and an average gain of 7.5% over 105 tasks. The comparison is not perfectly controlled because the systems may use different tasks, metrics, and objectives. AutoSOTA primarily changes configurations or hyperparameters; ScientistTwo is intended to introduce algorithmic mechanisms and require component-level attribution.

### Human evaluation

Nine experienced human reviewers evaluated 33 NeurIPS-derived ScientistTwo papers. Mean standalone ratings on a 1–5 scale were:

- Introduction and related work: 4.2.
- Method soundness and algorithmic rigor: 4.1.
- Experimental baselines and benchmarks: 4.0.
- Ablations: 4.3.
- Experimental insight and limitation analysis: 4.0.
- Overall scientific maturity and venue readiness: 3.7.

In pairwise comparison with human-authored papers, ScientistTwo was favored in introduction and related work, experimental breadth, ablations, and experimental insight; humans were favored in method soundness and rigor; the overall comparison was parity. These findings support competitiveness, especially in experimental breadth, but do not establish superiority in methodological rigor.

## 6. Case studies and methodological significance

### Incremental BPE tokenization

For “Incremental BPE Tokenization,” the reported sequence was:

1. Human baseline based on the method of Jiang and Gong.
2. VD-STrans: a 10.9% relative improvement over the human baseline.
3. BXT-Transducer: an additional 9.6% over VD-STrans.
4. SBR-Transducer: an additional 8.2% over BXT-Transducer.

The sequence is presented as evidence that ScientistTwo can use prior discoveries as context for subsequent research cycles.

### AI Engram unlearning

On the TOFU `forget10` split with Llama-3.2-1B-Instruct, FCD-Engram, produced after meta-review refinement, achieved an overall score of 0.916, compared with 0.897 for LFR-Engram and 0.705 for the human Engram baseline. The case is used to show that meta-review can alter the underlying algorithm rather than only improve manuscript wording.

### DynaSpec-RAG

ScientistTwo developed DynaSpec-RAG, a zero-shot time-series forecasting method using smooth anchoring of retrieved trajectories, real-FFT decomposition into macro-trends and seasonal components, fine-grained frequency-band gating, and a safety switch that suppresses harmful retrieval. It has 0.27M trainable parameters and does not retrain the frozen time-series foundation model.

Across ETTh1, ETTh2, ETTm1, ETTm2, Weather, Electricity, and Exchange, DynaSpec-RAG reported average MSE of 0.1892 and average MAE of 0.2488. It had the best average MSE and MAE in the reported table, although TS-RAG was slightly better on Electricity MSE and MAE.

### Additional algorithmic examples

On Pinet, ScientistTwo’s ANSE introduced amortized Stiefel-nullspace reparameterization, reduced-space Douglas–Rachford, and log-barrier feasibility flow. The reported effects included lower running statistics on three of four DC3 sets, constraint violation reduced from $4\times10^{-4}$ to $2\times10^{-14}$, and $3.0\times$ faster training.

On DMSQD, LC-FTT added a dimension-adaptive smoothness penalty, with reported improvements of up to $+536$ QD and $+2.08$ percentage points of coverage on 10D Sphere.

On Temporal Sparse Autoencoders, Sheaf-SAE introduced sheaf Dirichlet energy, a causal Koopman operator, and straight-through support projection, reporting a $3.4\%\pm0.2\%$ improvement on semantic and context probes across three suites for Pythia-160m and Gemma-2-2b.

These examples illustrate the framework’s intended mode of operation: generate a mechanistic proposal, implement it, compare it on a task-specific benchmark, assess component attribution, and retain it only if the complete evidence supports the change.

## 7. Limitations, interpretation, and broader context

ScientistTwo’s evidence is strongest for autonomous, reproducible, benchmarked machine-learning research improvement. The system demonstrates integration of literature analysis, coding, experimentation, ablation, manuscript production, simulated review, and integrity auditing. It does not establish broad autonomous scientific discovery in the sense of independently selecting major scientific questions, conducting physical or expensive real-world experiments, developing paradigm-changing theories, or producing knowledge validated independently of automated evaluators.

The principal limitations include the following.

**Evaluation dependence on automated reviewers**: ScholarPeer is part of the development loop, so its scores and acceptance rates are not independent. Stanford Agentic Reviewer is more informative as a held-out evaluator, but it remains an automated judge rather than conventional expert review.

**Benchmark selection**: The evaluation favors computational research problems with available code, reproducible datasets, and automatable metrics. It is not equivalent to open-ended discovery from an unknown scientific frontier.

**Heterogeneous aggregate metrics**: The 25.2% average relative gain aggregates different metrics, datasets, and task types. The reported account does not provide a formal normalization procedure across tasks, confidence intervals, statistical significance tests, or corrections for multiple comparisons.

**Limited human evaluation**: The human study includes nine reviewers and 33 papers. The comparison with human-authored papers is not described as a blinded, matched, randomized protocol. Its principal implication is competitiveness in experimental breadth, not general superiority.

**Cost and proprietary dependence**: On 33 NeurIPS-derived tasks, the average full research cycle required approximately 2–3 days and cost approximately \$3,765. Major costs arose from idea refinement, repeated benchmark execution, dynamic peer review, meta-review, long prompts, and generated context. The framework depends on proprietary models, search services, and substantial computing resources.

**Scientific and ethical risks**: Potential risks include incorrect scientific claims, benchmark overfitting, reward hacking, misattribution of novelty, weak or fabricated citations, unsafe deployment of unverified methods, publication flooding, unclear authorship, and concentration of research capacity in organizations with substantial computational resources. Integrity audits reduce some risks but cannot guarantee scientific truth.

The framework’s relationship to earlier research automation systems is also relevant. “Prediction of next career moves from scientific profiles” [1802.04830] demonstrates that scientific activity can be represented through recent career history, scientific environment, and collaboration-network structure, while “A Two-Dimensional Approach to Evaluate the Scientific Production of Countries” [1304.2698] represents national scientific production through simultaneous quantity and quality coordinates. These studies concern scientific mobility and bibliometric evaluation rather than autonomous discovery, but they provide contextual precedents for modeling research as a structured process involving trajectories, environments, collaboration, and multidimensional performance.

“MirrorMind: Empowering OmniScientist with the Expert Perspectives and Collective Knowledge of Human Scientists” [2511.16997] offers a complementary architecture centered on episodic, semantic, and persona memory, domain concept graphs, and interdisciplinary multi-agent coordination. ScientistTwo instead emphasizes autonomous hypothesis search, executable experimentation, ablation, review-driven refinement, and artifact verification. A plausible implication is that a future system could combine MirrorMind-style individual and domain memory with ScientistTwo’s closed-loop experimentation and integrity auditing; such an integration is not established by the reported work.

ScientistTwo’s most defensible characterization is therefore as an autonomous framework for end-to-end computational research improvement. Its architectural significance lies in treating scientific automation as a bounded loop of hypothesis generation, implementation, empirical testing, criticism, revision, and verification rather than as one-shot idea generation or metric optimization. Its stronger claim to be an autonomous scientific pioneer remains narrower than the evidence: the reported results support substantial automation and competitive benchmark performance, but not yet general replacement of human scientific judgment or independently validated frontier discovery.

Source: https://www.emergentmind.com/topics/scientisttwo