ScientistOne System: Verifiable Research Automation

• ScientistOne system is an autonomous research platform that enforces mechanistic verifiability on every claim using a rigorous Chain-of-Evidence paradigm.
• It employs a structured three-stage pipeline integrating literature grounding, ideation, and claim-based paper writing to ensure comprehensive auditability.
• Benchmark evaluations reveal state-of-the-art performance with near-perfect claim provenance and minimal evidentiary errors.

ScientistOne is an end-to-end autonomous research system built to enforce mechanistic verifiability on every claim presented in a scientific manuscript. Unlike prior AI-driven research agents, which frequently exhibit evidence-chain failures such as hallucinated references, unreproducible results, and method-code misalignment, ScientistOne is architected around the Chain-of-Evidence (CoE) paradigm, ensuring every statement in the output paper can be reliably traced to concrete supporting artifacts (e.g., code, logs, or original literature). ScientistOne demonstrates that enforcing comprehensive claim auditability can not only match but sometimes improve solution quality across diverse research benchmarks, achieving state-of-the-art (SOTA) performance in multiple competitive domains (Meng et al., 25 May 2026).

1. Foundational Principles: Chain-of-Evidence (CoE)

ScientistOne operationalizes the CoE framework, drawing an analogy to database ACID properties, by treating verifiability as intrinsic to the scientific output. The CoE requirement stipulates that every claim $c \in C$ in a paper must possess a traceable, acyclic evidence path

$$c \;\rightarrow\; c_1 \;\rightarrow\; c_2 \;\rightarrow \cdots \;\rightarrow\; e,$$

where $e \in E$ is a primary evidence artifact (PDF, code, log, evaluator output). The explicit $\mathsf{Chain}: C \to \mathrm{Paths}(C \cup E)$ mapping is preserved at every generation stage, rendering any claim auditable back to its root.

Four claim types are explicitly handled:

• Citation claims: Must correspond to canonical literature in an academic repository. LLM or NLI-based entailment is used to determine if the cited paper supports the claim.
• Numerical claims: Linked to experiment logs or outputs, and must fall within a tolerance $\mathsf{tol} = \max(1\%,\, 3\sigma/|\bar{s}|)$ of a canonical re-run.
• Methodological claims: Anchored to the precise implementation snippet(s) in submitted code.
• Conclusion claims: Derived logically or arithmetically from validated subclaims.

This protocol guarantees that each claim's provenance can be navigated systematically, enabling automated and human verification.

2. System Architecture and End-to-End Workflow

ScientistOne consists of a strict three-stage pipeline, with structured provenance metadata carried through all phases such that the finalized output assigns each claim a $\{\texttt{source}:\dots\}$ tag.

Stage 1: Literature Grounding

Seeded by keywords or existing papers, the Problem Investigator agent:

1. Crawls the SemanticScholar API to a two-hop depth, collecting 2K–5K paper candidates.
2. Uses LLM-based relevance scoring (“problem-alignment”, “method-relevance”) to filter to $\sim$500 promising entries.
3. Executes three focused agent-driven PDF reading rounds, extracting structured content per paper.
4. Audits coverage for baseline gaps, performing additional micro-crawls if needed.
5. Synthesizes a structured Experiment Brief, referencing 25–40 papers, each reference tagged with PDF origin (page/line).

Stage 2: Discovery (Ideation & Parallel Explore-Exploit)

Starting from the Experiment Brief, ScientistOne:

• Proposes 5–10 candidate solver approaches, scoring for novelty and feasibility.
• Employs a Parallel Explore-Exploit (PEE) search over $B$ branches and $I$ iterations: $$c \;\rightarrow\; c_1 \;\rightarrow\; c_2 \;\rightarrow \cdots \;\rightarrow\; e,$$0
• Prunes solutions with specification violations, selects the best performer post-ablation.
• Aggregates and records: best code, evaluator logs, headline and ablation scores.

Stage 3: Claim-Grounded Paper Writing

A five-step Claim-Grounded pipeline:

1. Conceive: LLM-generated Markdown narrative, each fact annotated with inline source tags (e.g., {source:log_123}).
2. Ground: Deterministic verification of source citation and numerical tags; unsupported claims labelled.
3. Critic: LLM-based holistic audit for misalignment, overclaims, and missing baselines.
4. Resolve: LLM rewriting to ablate, rephrase, or drop unsupported claims; iterated until convergence.
5. Compose: Conversion of grounded Markdown to LaTeX, preserving all evidence tags.

A “Claim Verifier” performs final mechanistic checks by claim type. Only drafts with zero blocking flags pass to publication.

3. CoE Integrity Audit and Verifiability Metrics

ScientistOne introduces a rigorous post-hoc integrity audit (CoE Audit), applicable to any completed artifact bundle (paper PDF/TeX, code, references). The audit performs four independent integrity checks:

Integrity Check	Description	Passing Criteria
I1: Score Verification	Extracts and reruns reported headline metric; checks $|s_{\rm paper} - s_{\rm rerun}| \leq \mathsf{tol}$	Score falls within tolerance
I2: Spec Violation	Detects code “cheats” (evaluator imports, test leaks)	Majority (5x) LLM votes "clean"
I3: Reference Verification	Confirms bibliographic existence; flags hallucinations	All entries must match canonical
I4: Method–Code Alignment	Judges if Methods section matches submitted code	Acceptable simplifications only

ScientistOne is the only system natively emitting source tags for quantitative claims, achieving a native Claim Provenance Rate (CPR) of 98–99%.

4. Comparative Experimental Results

ScientistOne was evaluated on the ADRS benchmark suite, comprising five competitive systems optimization tasks: Prism, Cloudcast, EPLB, LLM-SQL, and TXN. Results against four agentic baselines (Sakana AI-Scientist v2, AutoResearchClaw, DeepScientist, AI-Researcher) under identical constraints—three seeds per task, 20 solver iterations, two-hour code limit—demonstrate the unique verifiability and performance profile of ScientistOne:

• CoE Audit Summary (Sample):
  • Score Verification: 100% (12/12), compared to 42–92% for baselines.
  • Spec Violations: 0 flagged in 15; others: up to 10/15 flagged.
  • Hallucinated References: 0/337, while baselines range up to 20.9%.
  • Method–Code Alignment: 93% (14/15), outperforming all comparators.
• Solver Quality:
  • Prism: 26.26 vs. human 21.89 (SOTA).
  • Cloudcast: 618.08 (lower is better) vs. human 626.24.
  • EPLB: 0.1461 vs. human 0.1265 (higher is better).
  • LLM-SQL: 0.7299, surpassing human 0.6920.
  • TXN: 3906 vs. human 2724.8 (higher is better).

Verifiability enforcement does not degrade solver performance and may enhance solution quality in specific settings.

• Automated Peer Review: Using the ScholarPeer LLM reviewer, ScientistOne obtains a mean rating of 4.5/10 with a 40% accept rate (tripling the best baseline), with best runs achieving up to 6.6/10 and 4/5 tasks accepted.

5. Generalization Across Domains

ScientistOne demonstrates robust out-of-domain generalization without task-specific modifications, as evidenced on MLE-Bench (medical imaging, fine-grained categorization, 3D perception) and Parameter Golf (LLM training under strict model size constraints):

• MLE-Bench Results:
  • 3D Object Detection: 0.1763 (Gold)
  • RSNA Brain Tumor: 0.6518 (Gold)
  • iMet 2020 FGVC7: 0.6791 (Silver)
  • iNaturalist 2019 FGVC6: 0.2445 (Silver)
  • AI4Code: 0.8356 (above median)
• Parameter Golf: Achieves SOTA 1.0600 under 16 MB constraints, while the baseline DeepScientist is rendered invalid (submission too large). ScientistOne produces novel quantization algorithms (Hessian-weighted SVD initialization, GPTQ-driven ALS refinement), demonstrating effective agent-driven scientific discovery.

These results highlight that strong claim-level verifiability and auditability generalize beyond the benchmark of origin.

6. Limitations and Future Directions

ScientistOne and the CoE protocol currently depend upon deterministic benchmark evaluators, limiting applicability in open-ended scientific domains (e.g. wet-lab, theoretical ML), which may require domain-specific or qualitative integrity checks. Reference verification in CoE Audit (I3) confirms bibliographic existence, but claim-level scholarly NLI-based entailment remains an open challenge. The framework effectively enforces citation, numerical, and method claim types, but verifying qualitative claims or true novelty is an open research problem. ADRS benchmark tasks simplify actual research workflows—true multi-benchmark or multi-dataset scenarios are targets for future expansion. Audit failure rates represent lower bounds; more subtle, uncaptured evidence-chain discontinuities may exist.

ScientistOne establishes a new evidence-centric research automation paradigm, integrating provenance at every pipeline step and enabling mechanistic integrity evaluation without loss in solution quality (Meng et al., 25 May 2026).