ReplicatorBench: Replication Benchmark

• ReplicatorBench is a benchmark that assesses large language model agents' ability to replicate social and behavioral research by following a structured, multi-step workflow.
• It decomposes the replication process into Extraction, Generation, and Interpretation stages, each with defined checkpoints to evaluate data retrieval, study design, and result fidelity.
• Empirical results indicate that while LLM agents perform well in computational execution, they face challenges in accurately retrieving new data and making reliable replicability judgments.

Searching arXiv for ReplicatorBench and closely related benchmarking papers. ReplicatorBench is a benchmark for evaluating whether LLM agents can carry out an end-to-end research replication workflow in the social and behavioral sciences, rather than merely rerun published code or solve isolated computational subtasks. It is built from expert-documented replication studies from the SCORE project and is designed to test whether an agent can identify a focal claim, locate the new data needed for replication, design and execute the computational study, and interpret the outcome well enough to decide whether the claim is replicable (Nguyen et al., 11 Feb 2026). In the benchmark’s framing, the central distinction is between reproduction—obtaining the same findings using the original data and methods—and replication—testing whether the finding holds on a new, independently obtained sample. This distinction is operationally central because ReplicatorBench treats data retrieval and staged process evaluation as first-class parts of the benchmark, not ancillary setup work (Nguyen et al., 11 Feb 2026).

1. Scope, provenance, and benchmark intent

ReplicatorBench is explicitly positioned as an end-to-end benchmark for replicability, not just reproducibility. Its domain is the social and behavioral sciences (SBS), and its instances are derived from human replication studies conducted under SCORE, the DARPA/Center for Open Science effort “Systematizing Confidence in Open Research and Evidence” (Nguyen et al., 11 Feb 2026). SCORE produced end-to-end human replication records, including preregistration documents, descriptions of how new data were found, code and data artifacts, and final replication reports; ReplicatorBench filters these materials into benchmarkable agent tasks (Nguyen et al., 11 Feb 2026).

The benchmark keeps only cases where the focal claim can be tested with a single quantitative check and clear pass/fail criteria, where the original study is observational so that the replication sample can in principle be assembled from online or archival data, and where core replication materials are available, including preregistration or replication plan, data access, and code documentation (Nguyen et al., 11 Feb 2026). This yields 19 studies spanning six SBS subject areas: Health 6, Economics 4, Political Science 4, Education 2, Sociology 2, and Public Administration 1 (Nguyen et al., 11 Feb 2026).

Each benchmark instance corresponds to one paper and one focal claim. A claim is treated as replicable or non-replicable according to the human SCORE replication result, with the binary outcome defined as whether the preregistered replication criteria are satisfied on the new sample: “criteria met” or “criteria unmet” (Nguyen et al., 11 Feb 2026). The paper contains an unresolved inconsistency about class counts: one table reports 14 criteria-met and 5 criteria-unmet cases, while the evaluation section states 4 criteria-met and 15 criteria-unmet studies. The text does not resolve this discrepancy (Nguyen et al., 11 Feb 2026). What is clear is that the benchmark is intentionally not all-positive, and therefore evaluates whether an agent can identify both successful and unsuccessful replications (Nguyen et al., 11 Feb 2026).

This design contrasts with computational-science replication benchmarks that center on reconstruction of known results from released data and code. A plausible implication is that ReplicatorBench targets a more ecologically faithful approximation of the workflow of human replicators in SBS, especially because the acquisition of a suitable new sample is part of the task rather than a hidden precondition (Nguyen et al., 11 Feb 2026).

2. Benchmark structure and staged decomposition

A major contribution of ReplicatorBench is its decomposition of replication into three stages—Extraction, Generation, and Interpretation—with 1,568 gradable checkpoints overall (Nguyen et al., 11 Feb 2026). Table 1(b), as summarized in the source text, breaks these out into 456 extraction-information checkpoints, 86 web-search checkpoints, 570 generation-design checkpoints, 190 generation-execution checkpoints, and 247 interpretation checkpoints (Nguyen et al., 11 Feb 2026).

The staged design is meant to localize failure. Rather than score only a final binary verdict, the benchmark assigns intermediate credit so that retrieval failures, planning failures, execution failures, and interpretation failures can be distinguished (Nguyen et al., 11 Feb 2026). This is an important methodological choice because real replication workflows can fail before any model fitting occurs.

Stage	Main function	Checkpoints
Extraction	Recover claim structure and replication resources	456 information + 86 web-search
Generation	Design and execute the replication study	570 design + 190 execution
Interpretation	Judge whether preregistered criteria are met	247

In the Extraction stage, the agent receives the original paper PDF plus the focal claim and must produce a structured “post-registration” of the original study as a JSON document with 24 gradable information fields across five dimensions: focal claim, data, method, results, and metadata (Nguyen et al., 11 Feb 2026). The schema includes the hypothesis, claim statement and location, study type, data source, wave or subset, sample size, unit of analysis, access details, procedural steps, models, variables, controls, tools or software, numerical results with p-values and confidence intervals, and metadata such as title/DOI and links to original code or data (Nguyen et al., 11 Feb 2026).

The second Extraction subtask is open web retrieval of replication resources. The agent must output URLs in a structured format with fields for url, kind, resource_name, and why_needed, plus a missing list with search queries for unresolved resources (Nguyen et al., 11 Feb 2026). The benchmark treats replication-data retrieval as an information retrieval problem: agents are not given candidate URLs and must search the web for the new data documented by human replicators (Nguyen et al., 11 Feb 2026).

In the Generation stage, the input includes extracted materials and replication data, and the agent must first design a replication plan and then execute it (Nguyen et al., 11 Feb 2026). The Design step asks for a structured JSON with 30 gradable fields across seven dimensions: hypothesis, study type, data plan, proposed methodology, codebase, environment specifications, and analysis steps (Nguyen et al., 11 Feb 2026). The Execution step then requires the agent to process the data, run models, produce tables/figures/statistics, and iteratively debug failures, yielding a structured execution report containing commands, runtime environment, result summaries with coefficients, p-values, standard errors, and saved outputs (Nguyen et al., 11 Feb 2026).

In the Interpretation stage, the agent must inspect the execution output and logs together with prior artifacts and determine whether the focal claim replicated (Nguyen et al., 11 Feb 2026). The final structured report has 13 gradable fields across summary, fidelity assessment, results comparison, replication report, and failure handling, including an “overall answer” about whether the preregistered criteria were satisfied and recommendations if replication failed because of data, code, method, or result issues (Nguyen et al., 11 Feb 2026).

3. Inputs, schemas, and evaluation protocol

ReplicatorBench uses structured outputs but open-ended content, and its scoring is stage-specific (Nguyen et al., 11 Feb 2026). For Extraction information fields and Interpretation fields, an LLM-as-a-judge system, LLMEval, assigns 0–3 per field, where 3 means an exact or nearly exact semantic match and 0 means no match (Nguyen et al., 11 Feb 2026). For Generation-Design, the judge uses 0 or 3 per field against the human preregistration, and for Generation-Execution it assigns 0 or 1 for each of the 10 computational checkpoints (Nguyen et al., 11 Feb 2026). Stage scores are macro averages over their fields or checkpoints (Nguyen et al., 11 Feb 2026).

On a subset of extraction fields, LLMEval using GPT-4o had the strongest reported correlation with human judgment: Spearman 86.18 and Kendall’s tau 78.43, versus 74.66/60.57 for ROUGE-L and 72.38/57.97 for BERTScore (Nguyen et al., 11 Feb 2026). This does not eliminate concerns about judge reliability, but it does ground the benchmark’s scoring strategy in an explicit comparison against human raters (Nguyen et al., 11 Feb 2026).

The web-search subtask is evaluated separately by alias matching because valid resources may have multiple accepted entry points or landing pages (Nguyen et al., 11 Feb 2026). The benchmark reports macro and micro precision, recall, and F1, plus hit@any and hit@all (Nguyen et al., 11 Feb 2026). Hit@any means at least one required resource was found, and hit@all means all required resources were found (Nguyen et al., 11 Feb 2026). For the final replication verdict, the benchmark reports binary precision, recall, and F1 against the human conclusion, with macro aggregation used to treat the two classes equally (Nguyen et al., 11 Feb 2026). If the agent is inconclusive because execution failed or outputs were not found, that counts as incorrect (Nguyen et al., 11 Feb 2026).

The inferential criterion for a successful replication is also stated explicitly: the final decision is based on whether “the inference criteria for the focal claim are satisfied (i.e., a statistically significant effect ($\alpha = 0.05$, two-tailed) in the same pattern as the original study)” (Nguyen et al., 11 Feb 2026). This is the closest the paper comes to a central formal criterion. The stage sizes are fixed at 24 fields for Extraction information, 30 for Generation-Design, 10 binary checkpoints for Generation-Execution, and 13 fields for Interpretation (Nguyen et al., 11 Feb 2026).

A notable feature is the heavy use of structured schemas. The extraction schema asks for exact locations in the paper where the hypothesis and claim are stated; the design schema requires the agent to justify why the new dataset is similar enough to the original and also how it differs enough to count as a new sample; the interpretation schema requires a fidelity assessment and explicit failure-handling recommendations (Nguyen et al., 11 Feb 2026). This suggests that ReplicatorBench operationalizes replication as disciplined process documentation as much as computational execution.

4. ReplicatorAgent and benchmark execution environment

To establish baselines, the paper introduces ReplicatorAgent, a ReAct-style agentic framework that alternates reasoning and tool use and is designed explicitly to support iterative debugging rather than stopping after first failure (Nguyen et al., 11 Feb 2026). Its tool set is intentionally simple and motivated by the benchmark’s observed failure modes: recursive file and directory inspection, targeted file readers, dataset inspection utilities for columns and basic summaries, minimal-diff file editing via edit_file, and constrained write_file operations that require explicit overwrite flags (Nguyen et al., 11 Feb 2026).

The execution environment is a sandboxed container orchestrated by a lightweight runner that builds an image, mounts code and data into a standardized directory structure, executes the analysis entrypoint, and exports logs and artifacts (Nguyen et al., 11 Feb 2026). The benchmark allows internet access only during the Extraction stage; the Generation and Interpretation stages occur in a closed environment (Nguyen et al., 11 Feb 2026). The system also has an optional human-approval function for commands and file writes, though the benchmark itself does not conceptually require human intervention, and the paper reports that no significant unsafe actions were observed (Nguyen et al., 11 Feb 2026).

ReplicatorAgent’s workflow is stage-dependent. In Extraction, it reads the paper and claim, fills the post-registration template, and performs web search for new data and code resources (Nguyen et al., 11 Feb 2026). In Design, it creates replication_info.json, inspects available replication data and code, and prepares the computational plan (Nguyen et al., 11 Feb 2026). In Execution, it follows a multi-phase loop: generate Dockerfile, build image, start container, preview the entrypoint, execute the analysis, and, if there are errors, inspect logs, patch files or dependencies, and retry (Nguyen et al., 11 Feb 2026). In Interpretation, it inspects execution_results.json, logs, discovered outputs, and any unreported files before generating the final assessment (Nguyen et al., 11 Feb 2026).

The paper also studies several design variants. It evaluates ReplicatorAgent atop GPT-4o, GPT-5, GPT-5-mini, and o3 (Nguyen et al., 11 Feb 2026). It compares Native execution mode, where the agent runs the original replication package in its native language, with Python mode, where non-Python scripts must be translated to Python (Nguyen et al., 11 Feb 2026). It also compares a standard setting with access to human-written replication code against a data-only setting in which the agent must implement the method from the paper and the replication data alone (Nguyen et al., 11 Feb 2026).

A practical detail with methodological significance is log truncation: if raw logs exceed context limits, the system keeps the first 2000 lines and prompts the agent to rewrite the analysis entrypoint so that key coefficients and p-values appear early in a compact summary (Nguyen et al., 11 Feb 2026). This suggests the benchmark designers treat context-window management as a real execution constraint rather than an incidental engineering issue.

5. Empirical findings and characteristic failure modes

The main empirical conclusion is that current LLM agents are much better at computation and interpretation than at retrieving the right replication data (Nguyen et al., 11 Feb 2026). In Extraction-information scoring, human annotators achieve 72.14 LLMEval in leave-one-out extraction, while the best model, GPT-5, reaches 66.57; GPT-5-mini gets 63.75, GPT-4o 63.23, and o3 61.27 (Nguyen et al., 11 Feb 2026). This indicates that agents can read papers and extract structured claim details reasonably well, but still lag expert humans (Nguyen et al., 11 Feb 2026).

Web search is the weakest stage by a wide margin. Among the base models equipped with search tools, GPT-4o has the best macro precision and F1 at 21.75 precision and 19.49 F1, while GPT-5 has the highest recall at 30.62 and the best coverage with hit@any 63.16 and hit@all 15.79 (Nguyen et al., 11 Feb 2026). o3 reaches macro F1 16.68, and GPT-5-mini 10.56 (Nguyen et al., 11 Feb 2026). Even specialized search-tuned systems remain weak: the best such model, o3-deep-research, reaches macro F1 23.26 and hit@any 52.63 (Nguyen et al., 11 Feb 2026). The paper’s explanation is that successful replication-data retrieval requires long-context reasoning, iterative query reformulation, and mapping a claim to canonical data sources, whereas search-tuned models may return plausible but non-canonical pages that are penalized under alias/domain grading (Nguyen et al., 11 Feb 2026).

Generation is substantially stronger. In Python mode, Design scores range from 83.18 for GPT-5-mini to 97.97 for o3, and Execution scores from 67.62 for GPT-5-mini to 96.27 for o3 and 96.20 for GPT-5 (Nguyen et al., 11 Feb 2026). Interpretation is also relatively strong when execution succeeds: GPT-5 reaches 93.35 LLMEval, o3 87.57, GPT-4o 75.60, and GPT-5-mini 54.55 (Nguyen et al., 11 Feb 2026). On the final binary replication outcome, GPT-5 performs best with accuracy 78.95, macro precision 77.78, macro recall 85.71, and macro F1 77.38 (Nguyen et al., 11 Feb 2026).

Yet the benchmark’s central finding is the mismatch between stage-wise strength and final correctness. An agent can execute code successfully and still fail at the actual replication judgment because it misinterprets results, deviates from the preregistered plan, or silently implements the wrong analysis (Nguyen et al., 11 Feb 2026). The paper emphasizes that a benchmark focused only on “did the code run?” would overestimate agent utility (Nguyen et al., 11 Feb 2026).

The runtime error analysis sharpens this diagnosis. The appendix reports total runtime errors of 193 for GPT-4o, 38 for GPT-5, 78 for GPT-5-mini, and 35 for o3 (Nguyen et al., 11 Feb 2026). By category, GPT-4o has 35 setup, 124 input-data, 33 implementation, and 1 timeout errors; GPT-5 has 21 setup, 6 input-data, 11 implementation, and 0 timeout; GPT-5-mini has 4 setup, 0 input-data, 74 implementation, and 0 timeout; o3 has 9 setup, 1 input-data, 25 implementation, and 0 timeout (Nguyen et al., 11 Feb 2026). The broad takeaway is that newer large models are much more stable than GPT-4o in navigating messy data environments, while smaller models such as GPT-5-mini still struggle with coding logic and syntax (Nguyen et al., 11 Feb 2026).

The native-versus-Python comparison illustrates a common silent-failure mode. Python mode can improve execution by avoiding native-language dependency and compatibility failures, but translation can introduce hallucinations or subtle semantic drift (Nguyen et al., 11 Feb 2026). The paper’s concrete example is an R-to-Python translation that first crashed because data_clean was undefined, then “fixed” the issue by loading an existing file but hallucinated the filename data_clean.rds instead of the correct data_clean_5pct.rds; execution succeeded, but the numerical results were wrong (Nguyen et al., 11 Feb 2026). This suggests that iterative debugging may repair executability without repairing methodological fidelity.

6. Relation to adjacent benchmark literature and limitations

ReplicatorBench is best understood within a broader shift toward paper-scale and workflow-scale evaluation of AI agents. In astrophysics, “ReplicationBench” evaluates whether agents can replicate entire papers from manuscripts, datasets, and execution metadata, emphasizing faithfulness to original methodology and correctness of outputs (Ye et al., 28 Oct 2025). In AI research, “PaperBench” evaluates replication of 20 ICML 2024 papers from scratch using hierarchical rubrics and fresh-environment reruns of reproduce.sh (Starace et al., 2 Apr 2025). In collider physics, “Collider-Bench” evaluates whether agents can reconstruct LHC analyses from papers and public scientific software, with quantitative scoring against hidden signal-yield references (Faroughy et al., 13 May 2026). ReplicatorBench differs from these benchmarks in a specific way: it centers replication on new data and uses SCORE-derived human replication records as ground truth, rather than focusing primarily on reconstruction from original data or code (Nguyen et al., 11 Feb 2026).

This suggests that ReplicatorBench occupies a distinct methodological niche. Static question-answer benchmarks do not capture the retrieval and workflow orchestration burden of real replication studies, while reproduction-oriented code benchmarks often assume that the essential data and sometimes code are already available (Nguyen et al., 11 Feb 2026). ReplicatorBench instead models the documented workflow of human replicators: post-registration of the original claim, retrieval of new data, preregistration of the replication plan, execution with debugging, and interpretation against preregistered criteria (Nguyen et al., 11 Feb 2026).

The benchmark nevertheless has clear limitations. It is small, with only 19 studies, a tradeoff the authors present as deliberate in order to prioritize process fidelity and expert-verified ground truth over scale (Nguyen et al., 11 Feb 2026). It is limited to observational SBS studies where replication data can be web-retrieved; it does not cover experimental replications requiring new primary data collection (Nguyen et al., 11 Feb 2026). Generation and Interpretation happen in a controlled sandbox and the replication data are curated, so the benchmark abstracts away some real-world friction even as it is more realistic than prior work (Nguyen et al., 11 Feb 2026). The rubric-based LLMEval scores are acknowledged as approximations rather than absolute measures of replication competence (Nguyen et al., 11 Feb 2026). Finally, the class-count inconsistency in the paper’s own description is an unresolved caveat (Nguyen et al., 11 Feb 2026).

Even with those limitations, the central empirical message is narrow and robust: current frontier LLM agents are already fairly capable at the computational core of replication—especially environment setup, iterative debugging, and producing structured outputs—but remain weak at one of the most characteristically human parts of the workflow, namely finding the right new data and reasoning from those data to a trustworthy replicability judgment (Nguyen et al., 11 Feb 2026).