---
title: 'ReplicatorBench: Replication Benchmark'
url: https://www.emergentmind.com/topics/replicatorbench
type: topic
---

# ReplicatorBench: Replication Benchmark

Searching arXiv for ReplicatorBench and closely related benchmarking papers.
ReplicatorBench is a benchmark for evaluating whether large language model agents can carry out an end-to-end research replication workflow in the social and behavioral sciences, rather than merely rerun published code or solve isolated computational subtasks. It is built from expert-documented replication studies from the SCORE project and is designed to test whether an agent can identify a focal claim, locate the new data needed for replication, design and execute the computational study, and interpret the outcome well enough to decide whether the claim is replicable [2602.11354]. In the benchmark’s framing, the central distinction is between *reproduction*—obtaining the same findings using the original data and methods—and *replication*—testing whether the finding holds on a new, independently obtained sample. This distinction is operationally central because ReplicatorBench treats data retrieval and staged process evaluation as first-class parts of the benchmark, not ancillary setup work [2602.11354].

## 1. Scope, provenance, and benchmark intent

ReplicatorBench is explicitly positioned as an end-to-end benchmark for *replicability*, not just reproducibility. Its domain is the social and behavioral sciences (SBS), and its instances are derived from human replication studies conducted under SCORE, the DARPA/Center for Open Science effort “Systematizing Confidence in Open Research and Evidence” [2602.11354]. SCORE produced end-to-end human replication records, including preregistration documents, descriptions of how new data were found, code and data artifacts, and final replication reports; ReplicatorBench filters these materials into benchmarkable agent tasks [2602.11354].

The benchmark keeps only cases where the focal claim can be tested with a single quantitative check and clear pass/fail criteria, where the original study is observational so that the replication sample can in principle be assembled from online or archival data, and where core replication materials are available, including preregistration or replication plan, data access, and code documentation [2602.11354]. This yields 19 studies spanning six SBS subject areas: Health 6, Economics 4, Political Science 4, Education 2, Sociology 2, and Public Administration 1 [2602.11354].

Each benchmark instance corresponds to one paper and one focal claim. A claim is treated as replicable or non-replicable according to the human SCORE replication result, with the binary outcome defined as whether the preregistered replication criteria are satisfied on the new sample: “criteria met” or “criteria unmet” [2602.11354]. The paper contains an unresolved inconsistency about class counts: one table reports 14 criteria-met and 5 criteria-unmet cases, while the evaluation section states 4 criteria-met and 15 criteria-unmet studies. The text does not resolve this discrepancy [2602.11354]. What is clear is that the benchmark is intentionally not all-positive, and therefore evaluates whether an agent can identify both successful and unsuccessful replications [2602.11354].

This design contrasts with computational-science replication benchmarks that center on reconstruction of known results from released data and code. A plausible implication is that ReplicatorBench targets a more ecologically faithful approximation of the workflow of human replicators in SBS, especially because the acquisition of a suitable new sample is part of the task rather than a hidden precondition [2602.11354].

## 2. Benchmark structure and staged decomposition

A major contribution of ReplicatorBench is its decomposition of replication into three stages—Extraction, Generation, and Interpretation—with 1,568 gradable checkpoints overall [2602.11354]. Table 1(b), as summarized in the source text, breaks these out into 456 extraction-information checkpoints, 86 web-search checkpoints, 570 generation-design checkpoints, 190 generation-execution checkpoints, and 247 interpretation checkpoints [2602.11354].

The staged design is meant to localize failure. Rather than score only a final binary verdict, the benchmark assigns intermediate credit so that retrieval failures, planning failures, execution failures, and interpretation failures can be distinguished [2602.11354]. This is an important methodological choice because real replication workflows can fail before any model fitting occurs.

| Stage | Main function | Checkpoints |
|---|---|---:|
| Extraction | Recover claim structure and replication resources | 456 information + 86 web-search |
| Generation | Design and execute the replication study | 570 design + 190 execution |
| Interpretation | Judge whether preregistered criteria are met | 247 |

In the **Extraction** stage, the agent receives the original paper PDF plus the focal claim and must produce a structured “post-registration” of the original study as a JSON document with 24 gradable information fields across five dimensions: focal claim, data, method, results, and metadata [2602.11354]. The schema includes the hypothesis, claim statement and location, study type, data source, wave or subset, sample size, unit of analysis, access details, procedural steps, models, variables, controls, tools or software, numerical results with p-values and confidence intervals, and metadata such as title/DOI and links to original code or data [2602.11354].

The second Extraction subtask is open web retrieval of replication resources. The agent must output URLs in a structured format with fields for `url`, `kind`, `resource_name`, and `why_needed`, plus a `missing` list with search queries for unresolved resources [2602.11354]. The benchmark treats replication-data retrieval as an information retrieval problem: agents are not given candidate URLs and must search the web for the new data documented by human replicators [2602.11354].

In the **Generation** stage, the input includes extracted materials and replication data, and the agent must first design a replication plan and then execute it [2602.11354]. The Design step asks for a structured JSON with 30 gradable fields across seven dimensions: hypothesis, study type, data plan, proposed methodology, codebase, environment specifications, and analysis steps [2602.11354]. The Execution step then requires the agent to process the data, run models, produce tables/figures/statistics, and iteratively debug failures, yielding a structured execution report containing commands, runtime environment, result summaries with coefficients, p-values, standard errors, and saved outputs [2602.11354].

In the **Interpretation** stage, the agent must inspect the execution output and logs together with prior artifacts and determine whether the focal claim replicated [2602.11354]. The final structured report has 13 gradable fields across summary, fidelity assessment, results comparison, replication report, and failure handling, including an “overall answer” about whether the preregistered criteria were satisfied and recommendations if replication failed because of data, code, method, or result issues [2602.11354].

## 3. Inputs, schemas, and evaluation protocol

ReplicatorBench uses structured outputs but open-ended content, and its scoring is stage-specific [2602.11354]. For Extraction information fields and Interpretation fields, an LLM-as-a-judge system, LLMEval, assigns 0–3 per field, where 3 means an exact or nearly exact semantic match and 0 means no match [2602.11354]. For Generation-Design, the judge uses 0 or 3 per field against the human preregistration, and for Generation-Execution it assigns 0 or 1 for each of the 10 computational checkpoints [2602.11354]. Stage scores are macro averages over their fields or checkpoints [2602.11354].

On a subset of extraction fields, LLMEval using GPT-4o had the strongest reported correlation with human judgment: Spearman 86.18 and Kendall’s tau 78.43, versus 74.66/60.57 for ROUGE-L and 72.38/57.97 for BERTScore [2602.11354]. This does not eliminate concerns about judge reliability, but it does ground the benchmark’s scoring strategy in an explicit comparison against human raters [2602.11354].

The web-search subtask is evaluated separately by alias matching because valid resources may have multiple accepted entry points or landing pages [2602.11354]. The benchmark reports macro and micro precision, recall, and F1, plus hit@any and hit@all [2602.11354]. Hit@any means at least one required resource was found, and hit@all means all required resources were found [2602.11354]. For the final replication verdict, the benchmark reports binary precision, recall, and F1 against the human conclusion, with macro aggregation used to treat the two classes equally [2602.11354]. If the agent is inconclusive because execution failed or outputs were not found, that counts as incorrect [2602.11354].

The inferential criterion for a successful replication is also stated explicitly: the final decision is based on whether “the inference criteria for the focal claim are satisfied (i.e., a statistically significant effect ($\alpha = 0.05$, two-tailed) in the same pattern as the original study)” [2602.11354]. This is the closest the paper comes to a central formal criterion. The stage sizes are fixed at 24 fields for Extraction information, 30 for Generation-Design, 10 binary checkpoints for Generation-Execution, and 13 fields for Interpretation [2602.11354].

A notable feature is the heavy use of structured schemas. The extraction schema asks for exact locations in the paper where the hypothesis and claim are stated; the design schema requires the agent to justify why the new dataset is similar enough to the original and also how it differs enough to count as a new sample; the interpretation schema requires a fidelity assessment and explicit failure-handling recommendations [2602.11354]. This suggests that ReplicatorBench operationalizes replication as disciplined process documentation as much as computational execution.

## 4. ReplicatorAgent and benchmark execution environment

To establish baselines, the paper introduces **ReplicatorAgent**, a ReAct-style agentic framework that alternates reasoning and tool use and is designed explicitly to support iterative debugging rather than stopping after first failure [2602.11354]. Its tool set is intentionally simple and motivated by the benchmark’s observed failure modes: recursive file and directory inspection, targeted file readers, dataset inspection utilities for columns and basic summaries, minimal-diff file editing via `edit_file`, and constrained `write_file` operations that require explicit overwrite flags [2602.11354].

The execution environment is a sandboxed container orchestrated by a lightweight runner that builds an image, mounts code and data into a standardized directory structure, executes the analysis entrypoint, and exports logs and artifacts [2602.11354]. The benchmark allows internet access only during the Extraction stage; the Generation and Interpretation stages occur in a closed environment [2602.11354]. The system also has an optional human-approval function for commands and file writes, though the benchmark itself does not conceptually require human intervention, and the paper reports that no significant unsafe actions were observed [2602.11354].

ReplicatorAgent’s workflow is stage-dependent. In Extraction, it reads the paper and claim, fills the post-registration template, and performs web search for new data and code resources [2602.11354]. In Design, it creates `replication_info.json`, inspects available replication data and code, and prepares the computational plan [2602.11354]. In Execution, it follows a multi-phase loop: generate Dockerfile, build image, start container, preview the entrypoint, execute the analysis, and, if there are errors, inspect logs, patch files or dependencies, and retry [2602.11354]. In Interpretation, it inspects `execution_results.json`, logs, discovered outputs, and any unreported files before generating the final assessment [2602.11354].

The paper also studies several design variants. It evaluates ReplicatorAgent atop GPT-4o, GPT-5, GPT-5-mini, and o3 [2602.11354]. It compares **Native** execution mode, where the agent runs the original replication package in its native language, with **Python** mode, where non-Python scripts must be translated to Python [2602.11354]. It also compares a standard setting with access to human-written replication code against a **data-only** setting in which the agent must implement the method from the paper and the replication data alone [2602.11354].

A practical detail with methodological significance is log truncation: if raw logs exceed context limits, the system keeps the first 2000 lines and prompts the agent to rewrite the analysis entrypoint so that key coefficients and p-values appear early in a compact summary [2602.11354]. This suggests the benchmark designers treat context-window management as a real execution constraint rather than an incidental engineering issue.

## 5. Empirical findings and characteristic failure modes

The main empirical conclusion is that current LLM agents are much better at computation and interpretation than at retrieving the right replication data [2602.11354]. In Extraction-information scoring, human annotators achieve 72.14 LLMEval in leave-one-out extraction, while the best model, GPT-5, reaches 66.57; GPT-5-mini gets 63.75, GPT-4o 63.23, and o3 61.27 [2602.11354]. This indicates that agents can read papers and extract structured claim details reasonably well, but still lag expert humans [2602.11354].

Web search is the weakest stage by a wide margin. Among the base models equipped with search tools, GPT-4o has the best macro precision and F1 at 21.75 precision and 19.49 F1, while GPT-5 has the highest recall at 30.62 and the best coverage with hit@any 63.16 and hit@all 15.79 [2602.11354]. o3 reaches macro F1 16.68, and GPT-5-mini 10.56 [2602.11354]. Even specialized search-tuned systems remain weak: the best such model, o3-deep-research, reaches macro F1 23.26 and hit@any 52.63 [2602.11354]. The paper’s explanation is that successful replication-data retrieval requires long-context reasoning, iterative query reformulation, and mapping a claim to canonical data sources, whereas search-tuned models may return plausible but non-canonical pages that are penalized under alias/domain grading [2602.11354].

Generation is substantially stronger. In Python mode, Design scores range from 83.18 for GPT-5-mini to 97.97 for o3, and Execution scores from 67.62 for GPT-5-mini to 96.27 for o3 and 96.20 for GPT-5 [2602.11354]. Interpretation is also relatively strong when execution succeeds: GPT-5 reaches 93.35 LLMEval, o3 87.57, GPT-4o 75.60, and GPT-5-mini 54.55 [2602.11354]. On the final binary replication outcome, GPT-5 performs best with accuracy 78.95, macro precision 77.78, macro recall 85.71, and macro F1 77.38 [2602.11354].

Yet the benchmark’s central finding is the mismatch between stage-wise strength and final correctness. An agent can execute code successfully and still fail at the actual replication judgment because it misinterprets results, deviates from the preregistered plan, or silently implements the wrong analysis [2602.11354]. The paper emphasizes that a benchmark focused only on “did the code run?” would overestimate agent utility [2602.11354].

The runtime error analysis sharpens this diagnosis. The appendix reports total runtime errors of 193 for GPT-4o, 38 for GPT-5, 78 for GPT-5-mini, and 35 for o3 [2602.11354]. By category, GPT-4o has 35 setup, 124 input-data, 33 implementation, and 1 timeout errors; GPT-5 has 21 setup, 6 input-data, 11 implementation, and 0 timeout; GPT-5-mini has 4 setup, 0 input-data, 74 implementation, and 0 timeout; o3 has 9 setup, 1 input-data, 25 implementation, and 0 timeout [2602.11354]. The broad takeaway is that newer large models are much more stable than GPT-4o in navigating messy data environments, while smaller models such as GPT-5-mini still struggle with coding logic and syntax [2602.11354].

The native-versus-Python comparison illustrates a common silent-failure mode. Python mode can improve execution by avoiding native-language dependency and compatibility failures, but translation can introduce hallucinations or subtle semantic drift [2602.11354]. The paper’s concrete example is an R-to-Python translation that first crashed because `data_clean` was undefined, then “fixed” the issue by loading an existing file but hallucinated the filename `data_clean.rds` instead of the correct `data_clean_5pct.rds`; execution succeeded, but the numerical results were wrong [2602.11354]. This suggests that iterative debugging may repair executability without repairing methodological fidelity.

## 6. Relation to adjacent benchmark literature and limitations

ReplicatorBench is best understood within a broader shift toward paper-scale and workflow-scale evaluation of AI agents. In astrophysics, “ReplicationBench” evaluates whether agents can replicate entire papers from manuscripts, datasets, and execution metadata, emphasizing faithfulness to original methodology and correctness of outputs [2510.24591]. In AI research, “PaperBench” evaluates replication of 20 ICML 2024 papers from scratch using hierarchical rubrics and fresh-environment reruns of `reproduce.sh` [2504.01848]. In collider physics, “Collider-Bench” evaluates whether agents can reconstruct LHC analyses from papers and public scientific software, with quantitative scoring against hidden signal-yield references [2605.13950]. ReplicatorBench differs from these benchmarks in a specific way: it centers *replication on new data* and uses SCORE-derived human replication records as ground truth, rather than focusing primarily on reconstruction from original data or code [2602.11354].

This suggests that ReplicatorBench occupies a distinct methodological niche. Static question-answer benchmarks do not capture the retrieval and workflow orchestration burden of real replication studies, while reproduction-oriented code benchmarks often assume that the essential data and sometimes code are already available [2602.11354]. ReplicatorBench instead models the documented workflow of human replicators: post-registration of the original claim, retrieval of new data, preregistration of the replication plan, execution with debugging, and interpretation against preregistered criteria [2602.11354].

The benchmark nevertheless has clear limitations. It is small, with only 19 studies, a tradeoff the authors present as deliberate in order to prioritize process fidelity and expert-verified ground truth over scale [2602.11354]. It is limited to observational SBS studies where replication data can be web-retrieved; it does not cover experimental replications requiring new primary data collection [2602.11354]. Generation and Interpretation happen in a controlled sandbox and the replication data are curated, so the benchmark abstracts away some real-world friction even as it is more realistic than prior work [2602.11354]. The rubric-based LLMEval scores are acknowledged as approximations rather than absolute measures of replication competence [2602.11354]. Finally, the class-count inconsistency in the paper’s own description is an unresolved caveat [2602.11354].

Even with those limitations, the central empirical message is narrow and robust: current frontier LLM agents are already fairly capable at the computational core of replication—especially environment setup, iterative debugging, and producing structured outputs—but remain weak at one of the most characteristically human parts of the workflow, namely finding the right new data and reasoning from those data to a trustworthy replicability judgment [2602.11354].

Source: https://www.emergentmind.com/topics/replicatorbench