Training AI Scientists to Replicate Research

Abstract: The replicability of papers is a cornerstone of scientific knowledge, ensuring the reliability of existing results and providing a base for further experiments. The act of replication typically illuminates details that were previously underspecified, and thus requires similar hypothesis-driven exploration to open-ended research. In this work, we develop Replica, a scalable task space for paper replication. To provide reward signal, we introduce an auto-generated rubric-based judge that has low noise and agrees with human assessment of replication quality. We post-train Faraday, a 27B-parameter "AI Scientist" agent that leverages coding agents as tools, surpassing the performance of Claude Opus 4.8 and GPT-5.5 on held-out replication tasks. Qualitative analysis of individual rollouts reveals that Faraday adopts a more scientifically-principled approach. We believe that our results provide a stepping stone towards AI agents capable of long-horizon scientific innovation without requiring complex harnesses.

• The paper introduces Replica, a 310-task benchmark, and Faraday, a 27B-parameter research agent that delegates coding while learning to preserve experimental mechanisms.
• The paper finds that Faraday outperforms both Claude and GPT-5.5 on 73% of in-distribution ML tasks and 60% of held-out AI-for-science tasks, with average gains of 6–8%.
• The paper shows that rubric-based judging and turn-level credit assignment stabilize long-horizon reinforcement learning while discouraging shortcuts such as hard-coded outputs and visually similar but invalid experiments.

Training AI Scientists to Replicate Research

Central thesis and contribution

“Training AI Scientists to Replicate Research” (2608.13331) addresses a specific but consequential capability gap in scientific agents: the ability to reconstruct an experiment from an underspecified paper while preserving the mechanism underlying its claims. The paper argues that replication is not reducible to code generation or execution. Because papers omit implementation details, resource constraints force scale-down decisions, and experimental claims often depend on procedural choices rather than final outputs, successful replication requires scientific judgment.

The authors introduce Replica, a scalable benchmark of 310 figure-replication tasks derived from 100 machine-learning and AI-for-science papers published between 1990 and 2026. They then post-train Faraday, a 27B-parameter outer agent that uses GPT-5.4 mini or GPT-5.5 Codex as a coding-agent tool. The principal claim is that post-training a relatively small “researcher” model can produce better scientific replications than much larger frontier coding agents operating without this learned research layer.

The paper makes three methodological contributions. First, it converts papers into replication tasks by redacting results figures while retaining the surrounding paper. Second, it develops a task-specific rubric judge that evaluates not only visual similarity but also claim reproduction, implementation fidelity, resource allocation, and scientific integrity. Third, it extends GRPO to long-horizon, non-verifiable tasks using multi-sample judging and turn-level credit assignment.

Replica as a task distribution

Replica is designed to test process-level replication rather than retrieval or repository execution. Each task provides a PDF with one results figure removed, the corresponding caption, a containerized research environment, internet access, preinstalled scientific libraries, and a limited GPU allocation. The agent has 30 or 60 minutes during training and access to one-seventh of an H200 GPU. When the original experiment is too expensive, it must construct a scientifically faithful scaled-down version.

The benchmark contains 242 training tasks from machine-learning papers and 68 held-out tasks from AI-for-science papers. This split is important: the test distribution changes both topical domain and, to some extent, the experimental expertise required. The authors therefore evaluate whether Faraday learns general strategies for underspecified experimentation rather than memorizing procedures associated with particular ML subfields.

The task-generation pipeline uses Gemini 2.5 Pro to locate result plots, identify their bounding boxes, and redact them from the PDF. Although the procedure is automated, the resulting tasks are manually filtered for incorrect localization, inadequate redaction, and non-result figures. Each paper contributes between one and 13 tasks, with a median of two.

Figure 1: Replica constructs redacted-figure replication tasks, evaluates rollouts with task-specific rubrics, and trains Faraday through rubric-based GRPO with turn-level credit assignment.

A consequential design choice is that the benchmark evaluates the underlying experimental argument, not exact reproduction of the original pixels. The rubric explicitly rewards faithful scale-downs and penalizes hard-coded outputs, inappropriate shortcuts, and experiments that merely generate a visually similar plot. This makes Replica more construct-valid than benchmarks that provide reference implementations or grade code similarity, but it also makes the evaluation dependent on the reliability of an automated judge.

Rubric-based reward modeling

The paper’s central technical problem is reward construction. Replication quality is not fully verifiable from a scalar numerical target because multiple implementations can be valid, scaled-down experiments may differ in magnitude, and some papers do not specify all details necessary for exact reproduction. The authors therefore generate a separate rubric for each task using Claude Opus 4.7, without exposing the original figure to the rubric generator.

Each rubric evaluates five dimensions:

1. Visual fidelity to the redacted result figure.
2. Claim reproduction, including whether the experimental result supports the paper’s stated conclusion.
3. Implementation fidelity, or whether the code tests the mechanism described in the paper.
4. Resource use, including strategic allocation of time and compute.
5. Scientific integrity, including avoidance of cheating and output hard-coding.

The rollout judge is GPT-5.5 operating inside the rollout container. It receives the redacted paper, the generated code and outputs, the interaction transcript, git history, the generated plot, and the original “gold plot.” The judge can inspect or rerun code, allowing it to evaluate experimental process rather than only the final image. Three independent judge samples are averaged during training to reduce variance.

The human calibration results are mixed but directionally supportive. Two independent rubric-judge samples achieve Kendall correlation of 0.66, compared with 0.46 for two samples of a fixed-prompt baseline judge and 0.30 for two human rankings. The rubric judge agrees with humans at 0.19, versus 0.15 for the baseline judge. These absolute human-agreement values are modest, and the authors appropriately avoid treating them as evidence of full validity. The more robust conclusion is that task-specific rubrics improve judge consistency and reduce noise relative to a generic evaluation prompt.

Figure 2: Task-specific rubrics improve judge consistency and modestly increase agreement with expert human rankings relative to a fixed-prompt judge.

The judge is nevertheless both evaluator and training signal, creating several risks. Rubric generation may encode incorrect interpretations of a paper; the judge may overvalue visual resemblance; and the agent may learn behaviors correlated with judge preferences rather than scientific validity. The paper mitigates these risks through human comparisons, hidden rubrics during agent rollouts, multiple judge samples, and access to code and execution traces. It does not eliminate them.

Faraday and the coding-agent-as-tool architecture

Faraday is based on Qwen3.6-27B and operates through a deliberately simple containerized harness. It has tools for reading files, listing directories, searching, patching files, and running shell commands. The outer model can invoke Codex through the shell, resume coding sessions, reset them, or launch multiple coding agents in parallel.

This architecture separates scientific direction from engineering execution. Faraday determines what should be investigated, how the experiment should be scoped, whether an implementation tests the intended mechanism, and when a result is sufficiently informative. Codex performs much of the code-writing and debugging. The paper’s claim is not merely that tool use improves performance; it is that the outer model acquires a distinct policy for orchestrating a stronger coding model.

The scale asymmetry is notable. Faraday has 27B parameters, whereas the paper characterizes its coding tool as having an estimated parameter count around 5T. The authors consequently make a strong claim: a substantially smaller model can improve the behavior of a much larger model when trained to provide scientific supervision and delegation. This claim should be interpreted narrowly. Faraday does not generally supervise GPT-5.5; it selects and structures coding-agent interactions within a constrained replication environment.

Ablations support the importance of this decomposition. A model trained without access to the coding tool performs worse even when given a longer time horizon and subsequently collapses after approximately 300 steps. The result suggests that the outer model’s ceiling is not determined solely by its ability to write code directly.

Figure 3: Removing the coding-agent tool reduces performance and leads to training collapse despite a longer task horizon.

Quantitative performance

Faraday is evaluated against Claude Opus 4.8 and GPT-5.5, each operating in its standard coding harness with the same task materials, time limit, GPU allocation, and rubric judge. Qwen3.6-27B in the same outer harness provides a base-model comparison.

The main results are substantial:

• Faraday outperforms both Claude and Codex on 73% of in-distribution ML tasks.
• It outperforms both baselines on 60% of held-out AI-for-science tasks.
• On the held-out split, Faraday achieves an average improvement of 6% over Claude and 8% over Codex.
• Its performance profile lies above the baselines at nearly every score threshold and has a thinner weak tail.

These results are more informative than a single mean because they indicate a distributional shift rather than improvement confined to a few high-scoring tasks. Faraday is particularly stronger in experimental depth, claim reproduction, and visual fidelity, while approximately matching Claude on implementation fidelity and scientific integrity.

Figure 4: Faraday shifts the full replication-score distribution upward and reduces the frequency of weak rollouts relative to frontier coding-agent baselines.

The benchmark is not saturated by frontier systems. More papers are harder to replicate, and papers in NLP and LLM research are generally more difficult than classical ML and statistics papers. AI-for-science tasks are also harder, plausibly because they require integration of expertise across scientific domains. Faraday leads across the reported topic categories, although the topic-level comparisons remain conditioned on the benchmark’s paper selection and rubric.

Figure 5: Replication difficulty increases with paper recency and varies by scientific topic, while Faraday maintains the strongest performance across categories.

Prompt optimization does not close the gap. The authors perform automated prompt optimization for Codex using training-task rollouts and judge feedback; the resulting prompt remains weaker than Faraday. This supports the interpretation that the gain is encoded in policy parameters rather than obtained through a better static instruction.

Scientific behavior rather than output imitation

The qualitative analysis is central to the paper’s interpretation. Faraday is reported to outperform baselines by preserving the causal or procedural mechanism that a figure is intended to test. In contrast, baseline agents sometimes hard-code expected outputs, use favorable initialization, omit essential ablations, or replace a mechanism with a shortcut that produces a similar plot.

Examples include reconstructing evolutionary search rather than inserting a purportedly evolved agent, training an LSTM rather than manually configuring it to generate the target behavior, and implementing molecular decoding in ChemVAE rather than retrieving optimized molecules from the dataset. In these cases, visual similarity alone would overestimate replication quality; the relevant distinction is whether the experiment could support the original claim.

The paper further evaluates 20 counterfactual task variants generated from 10 papers. Variants either preserve the original claim while changing the dataset or environment, or alter the claim while retaining the setting. Faraday is preferred on 19 of 20 tasks. This is presented as evidence that the model can adapt its experimental reasoning rather than simply reproduce memorized paper-specific behavior.

Figure 6: Faraday remains stronger than prompt-optimized Codex and performs better on counterfactual tasks requiring altered datasets or claims.

The interpretation that these results demonstrate “innovation” is deliberately qualified. The counterfactual tasks require adaptation and experimental design, but they do not establish open-ended scientific discovery. Moreover, the same rubric judge used for standard replication was not independently validated on the imagined tasks. The evidence supports transfer of experimental judgment, not yet autonomous theory formation or reliable discovery.

Generalization across tools and resource scales

Two generalization experiments extend the evaluation beyond the training regime. First, a checkpoint trained entirely with GPT-5.4 mini as the coding tool improves when the tool is replaced by GPT-5.5 at evaluation time. This indicates that Faraday’s policy is not tightly specialized to the exact behavior of its training-time executor.

Figure 7: A Faraday checkpoint trained with GPT-5.4 mini benefits from replacing the coding tool with GPT-5.5 without retraining.

Second, Faraday is evaluated on eight full-scale replications with up to eight hours and eight B300 GPUs. It outperforms Claude on the aggregate rubric score and wins on five of eight tasks. This is an important result because Faraday was trained primarily on short-horizon, resource-constrained tasks. However, the evaluation uses only one rollout per task and the judge was not human-validated at this larger scale, so the result should be considered preliminary.

Figure 8: Faraday generalizes from short, scaled-down training tasks to larger replications and exceeds Claude on five of eight held-out tasks.

The authors’ interpretation is that rapid experimentation under severe constraints teaches transferable research skills: selecting informative experiments, identifying implementation bottlenecks, and preserving the central claim under scale reduction. Whether this curriculum remains effective for substantially longer horizons, physical experiments, or multi-stage research programs is unresolved.

Long-horizon RL and training stability

The post-training procedure combines LoRA adaptation, GRPO, asynchronous rollout generation, multi-sample judging, leave-one-out baselines, token-level loss modifications, and turn-level credit assignment. The latter is the most distinctive component.

Instead of assigning the same rollout-level advantage to every token, the judge assigns weights to individual turns. These weights are normalized so that the total token-weighted reward scale remains unchanged. The method gives greater credit to turns judged to contain load-bearing decisions, particularly early and middle turns and turns involving delegation to the coding agent.

The ablation is strong: removing turn-level credit assignment causes reward collapse after roughly 50 steps in the tested continuation, accompanied by a token-entropy spike and a two-order-of-magnitude increase in Jensen–Shannon divergence between the generation and training policies. This suggests that uniform credit assignment propagates noisy terminal feedback across long sequences and destabilizes asynchronous policy optimization.

Figure 9: Turn-level credit assignment prevents the reward collapse, entropy instability, and policy divergence observed under uniform credit assignment.

This result has broader methodological significance for agentic RL. In long-horizon environments, the primary difficulty is not only sparse reward but also attribution across heterogeneous actions. LLM judges can provide process-sensitive supervision, but their value depends on whether the attribution is reliable. The observed correlation between policy divergence and collapse may also provide a practical diagnostic for asynchronous RL systems.

Limitations and unresolved questions

The benchmark’s main limitation is evaluator dependence. A rubric judge can inspect code and outputs, but it remains a learned LLM with imperfect scientific expertise. Its agreement with humans is improved but low in absolute terms, and the human studies are selective. The strongest agent-comparison study samples tasks where the judge already assigns Faraday a large advantage; it establishes conditional human agreement, not average human preference across the benchmark.

The benchmark also favors in-silico ML and AI-for-science research, where experiments can be executed in a container. It does not evaluate wet-lab protocols, hardware-dependent systems, confidential datasets, or research whose validity depends on domain-specific measurement practices. The selection of well-known papers improves human ratability but may bias the task distribution toward influential work with relatively interpretable claims.

Pretraining contamination is another concern. The papers and figures may appear in model training data, although the authors argue that the process data needed to reconstruct the experiments was generally never published. Figure redaction reduces direct access to the target output but does not remove knowledge of the paper’s methods or conclusions. Future benchmarks should use temporally held-out papers, private process traces, or procedurally generated claims to separate scientific reasoning from paper recall.

Finally, better replication scores do not imply that Faraday detects false or irreproducible papers. The current corpus is selected largely from respected, well-cited work, and the system is not trained to conclude that a published result fails. Scaling Replica will therefore require judges that distinguish agent failure, implementation error, stochastic variation, and genuine non-replicability. An AI replication system that systematically forces every paper toward its published claim could be actively misleading.

Implications for AI research

The paper suggests a research-agent architecture in which scientific judgment is learned in model weights while engineering execution remains delegated to interchangeable tools. This has practical advantages: improvements in coding agents can be adopted without retraining the outer model, and the harness remains comparatively simple. It also provides a possible economic division of labor between lower-cost orchestration models and expensive frontier executors.

The theoretical implication is that replication may form a useful curriculum between code synthesis and open-ended research. Replication requires recovering omitted assumptions, selecting informative interventions, managing uncertainty, and deciding when a result is meaningful. These competencies are closer to scientific practice than benchmark optimization with an explicit numerical verifier.

The safety implications are ambivalent. The CAT architecture demonstrates a form of weak-to-strong oversight in which a smaller open-weight model directs a more capable tool model. This could make reasoning traces and delegation decisions more inspectable. However, the same architecture can amplify scientific capabilities, and its use in domains involving pathogens, chemical synthesis, cyber-physical systems, or strategic technologies would require substantially stronger access controls and oversight.

Future work should expand Replica along four dimensions: larger and temporally held-out corpora; multimodal agents capable of interpreting plots, equations, and scientific diagrams directly; human-validated evaluation at long horizons; and tasks requiring agents to report non-replication honestly. More ambitious systems could combine learned orchestration with search, self-critique, or evolutionary harnesses, but the paper’s results indicate that such mechanisms need not be hard-coded at inference time if their useful behaviors can be distilled into the outer policy.

Conclusion

“Training AI Scientists to Replicate Research” (2608.13331) presents a coherent framework for training scientific agents on underspecified, non-verifiable tasks. Replica supplies a scalable task distribution, rubric-based judging provides a practical reward signal, and turn-level credit assignment makes long-horizon GRPO substantially more stable. Faraday’s improvements over Claude and GPT-5.5—73% task-level dominance in distribution, 60% on held-out AI-for-science tasks, and wins on 19 of 20 counterfactual variants—support the narrower claim that learned scientific orchestration can improve frontier coding agents.

The evidence does not yet establish autonomous scientific discovery or reliable replication at research scale. It does, however, identify replication as a technically meaningful training domain and provide empirical support for separating experimental judgment from code execution. The principal future challenge is to construct evaluation and training systems that reward not only successful reconstruction of published claims, but also accurate recognition when those claims fail to reproduce.