Training AI Scientists to Replicate Research
This presentation examines a fundamental challenge in scientific AI: teaching agents to reconstruct experiments from underspecified papers. It introduces Replica, a benchmark of 310 figure-replication tasks, and Faraday, a 27-billion-parameter model that orchestrates frontier coding agents to perform scientific replications. The work demonstrates that learned scientific judgment can outperform much larger models, achieving 73% task-level dominance on machine-learning papers and 60% on held-out AI-for-science tasks, while introducing turn-level credit assignment to stabilize long-horizon reinforcement learning.Script
Scientific papers omit the procedural details needed to replicate their experiments. Bridging that gap requires scientific judgment, not just code generation, because resource constraints, underspecified methods, and claim-dependent designs force researchers to make decisions the original authors never documented.
The researchers built Replica, a benchmark of 310 replication tasks where agents receive a paper with one results figure redacted, a containerized environment, and severely limited compute. The agent must reconstruct a scientifically faithful version of the experiment, which often means strategically scaling down while preserving the mechanism the original claim depends on.
Faraday uses a 27-billion-parameter outer model to orchestrate GPT-5 point 5 as a coding tool. This architecture separates scientific direction from engineering execution. The smaller model decides what to investigate, how to scope experiments, and whether implementations test the intended mechanism, while delegating code-writing to a much larger executor.
Faraday outperforms both Claude Opus and GPT-5 point 5 on 73% of machine-learning replication tasks and 60% of held-out AI-for-science papers. It shifts the entire score distribution upward, not just the average, reducing weak replications while improving experimental depth and claim reproduction.
Training stability depends critically on turn-level credit assignment. When the system assigns uniform credit across all actions, training collapses after just 50 steps, accompanied by entropy spikes and two-order-of-magnitude policy divergence. Weighting credit by each turn's causal importance prevents this collapse and enables stable learning across long experimental horizons.
A smaller model can learn to supervise a larger one when trained to provide scientific judgment rather than raw coding capability. The work demonstrates that replication serves as a practical curriculum between code synthesis and open-ended discovery. To explore how these techniques apply to other scientific domains and create your own research video summaries, visit EmergentMind dot com.
