ReproBench: Benchmark for Reproducibility

• ReproBench is a benchmark that assesses the reproducibility of social science papers by evaluating AI’s ability to match paper claims with reproduced outcomes.
• It comprises 112 real-world task instances featuring diverse datasets, programming languages, and data formats to mimic realistic replication challenges.
• Benchmark metrics focus on accuracy, applicability, and cost, highlighting procedural issues like file management, multi-language execution, and cross-artifact reasoning.

Searching arXiv for the most relevant papers on ReproBench and closely related reproducibility-assessment work. ReproBench is a benchmark for automated reproducibility assessment of social science research papers, designed to evaluate whether agentic AI systems can judge the reproducibility of a paper from the original paper PDF and its reproduction package rather than merely rerunning provided code. In its canonical form, REPRO-Bench comprises 112 task instances, each tied to a real social science paper with a publicly available reproduction report, and frames the task as assigning a reproducibility score on a four-point ordinal scale. The benchmark was introduced to address limitations in earlier reproduction-oriented benchmarks that focused on code execution alone, simplified real-world workflows, or lacked diversity in programming languages and data formats (Hu et al., 25 Jul 2025). Subsequent work has used ReproBench as an external evaluation target for document-centric reproducibility-assessment systems, notably ARA, which reports substantially higher accuracy than previously tested agents on this benchmark (Riehl et al., 4 May 2026).

1. Definition and scope

ReproBench, introduced as REPRO-Bench, is a collection of end-to-end evaluation tasks for assessing the reproducibility of social science papers under conditions intended to be comparable to real-world assessment practice (Hu et al., 25 Jul 2025). Each task corresponds to a paper paired with a reproduction package and a human-generated reproduction report. The benchmark is explicitly aimed at assessing whether an automated system can determine reproducibility in a setting where the relevant evidence is distributed across paper text, code, data, and documentation, rather than encoded in a simplified execution script.

The benchmark’s central distinction from prior work is methodological. Earlier benchmarks such as SciCode and CORE-Bench were described as assuming that papers are reproducible and focusing on code and data execution rather than on whether reproduced results are consistent with the claims in the paper. ReproBench instead requires a system to compare outputs from the reproduction package against the scientific claims articulated in the paper itself (Hu et al., 25 Jul 2025).

This suggests that ReproBench occupies a boundary position between software-oriented benchmark design and meta-scientific evaluation. It is not primarily a benchmark of model execution competence alone, nor purely a document understanding benchmark. Rather, it binds together document interpretation, environment interaction, multi-language code handling, and judgment against externally validated reproducibility outcomes.

2. Dataset composition and curation

ReproBench contains 112 tasks, each corresponding to a real social science paper and its reproduction package (Hu et al., 25 Jul 2025). The tasks are drawn from four sources: mass social science replication projects, Institute for Replication, Retraction Watch, and social media. The benchmark therefore aggregates cases from multiple reproducibility-assessment channels rather than from a single replication initiative.

Each task provides three primary artifacts to the agent:

1. the original paper PDF,
2. the reproduction package consisting of code, data, and documentation,
3. a list of major findings from the paper (Hu et al., 25 Jul 2025).

The benchmark is intentionally heterogeneous in implementation conditions. The underlying papers and packages span multiple programming languages, including R, Stata, MATLAB, and Python, and multiple data formats, including .csv, .dta, and .sav (Hu et al., 25 Jul 2025). Packages average 4.2 GB, while papers are about 29 pages long, which makes the tasks nontrivial in terms of context length, file-system exploration, and execution management (Hu et al., 25 Jul 2025).

A key structural feature is the reproducibility label schema. Each paper is annotated with a reproducibility score from 1 to 4 using established social science criteria:

Score	Meaning
1	Major findings irreproducible
2	Minor code inconsistencies/errors
3	Minor reporting/display issues
4	Major findings fully reproducible

The score distribution is balanced, with 56 instances assigned scores 1 or 2 and 56 assigned scores 3 or 4 (Hu et al., 25 Jul 2025). The ARA study further characterizes ReproBench as using human-assigned labels derived from the actual outcome of full replication efforts and notes inclusion controls such as papers with a DOI, available code and data, and a credible human-written reproduction report (Riehl et al., 4 May 2026).

3. Task formulation and evaluation protocol

The ReproBench task is formulated as an agentic assessment problem. The agent receives the three benchmark artifacts and must output a reproducibility score in a specific JSON format in a designated directory (Hu et al., 25 Jul 2025). The environment includes the software required for realistic execution, such as Stata, MATLAB, and LaTeX, and agents are allowed to read files, write files, execute scripts, and perform file-system operations (Hu et al., 25 Jul 2025).

The required assessment process consists of four broad operations:

1. reading and analyzing the paper PDF to extract relevant claims and results,
2. exploring and executing the reproduction package,
3. comparing produced results with those reported in the paper,
4. assigning a reproducibility score according to the benchmark criteria (Hu et al., 25 Jul 2025).

The official evaluation metrics are accuracy, applicability, and cost. Accuracy is defined as the percentage of tasks for which the agent’s score matches the ground truth; applicability is the percentage of tasks for which the agent produces valid output in the required format; cost is the API usage cost per task (Hu et al., 25 Jul 2025). The reported accuracy formula is

$$\text{Accuracy} = \frac{1}{N}\sum_{i=1}^N \mathbf{1}(S_i^{\text{agent}} = S_i^{\text{true}})$$

where $N$ is the total number of tasks (Hu et al., 25 Jul 2025).

ARA later evaluates on ReproBench using a somewhat broader reporting set: accuracy, F1 score, score distance, and absolute score distance (Riehl et al., 4 May 2026). That study uses ReproBench to test whether a document-based workflow-graph method can recover labels aligned with human reproducibility judgments, even when using only publication text rather than the full reproduction package.

4. Baseline agents and empirical findings

The initial REPRO-Bench study evaluates three representative AI agents: AutoGPT, CORE-Agent, and SWE-Agent, all using GPT-4o as the core LLM (Hu et al., 25 Jul 2025). Agents are terminated if API cost exceeds \$4 per task. Performance is poor across the board, which is one of the benchmark’s central empirical conclusions.

The principal results are as follows:

Agent	Accuracy	Applicability	Cost (\$)
AutoGPT	20.5%	60.7%	2.03
CORE-Agent	21.4%	46.4%	2.00
SWE-Agent	1.8%*	1.8%*	1.20

*Adjusted values reported in parentheses in the source are 10.7% accuracy and 19.6% applicability for SWE-Agent when outputs placed in wrong directories are considered (Hu et al., 25 Jul 2025).

The best-performing baseline, CORE-Agent, reaches only 21.4% accuracy, which is below the 25% expected by random guessing over four classes (Hu et al., 25 Jul 2025). AutoGPT obtains 20.5% accuracy with somewhat higher applicability. SWE-Agent performs substantially worse. These results indicate that benchmark difficulty is not an artifact of narrow scoring criteria; the agents are frequently unable even to return valid outputs, and when they do, their judgments often fail to match human assessments.

The paper reports several error patterns. Agents perform better at identifying fully reproducible papers with score 4 than papers with intermediate issues scored 2 or 3. They perform better on R-based tasks than on Stata-based tasks, are significantly challenged by multi-language packages, and do not show a significant degradation when handling multiple data formats (Hu et al., 25 Jul 2025). The authors also report that benchmark-level feature correlations with reproducibility scores have $|\rho| < 0.1$ using Spearman correlation, suggesting that coarse metadata such as package size or number of formats do not trivially determine the labels (Hu et al., 25 Jul 2025).

These findings support a common interpretation of ReproBench as a hard benchmark, but the paper’s own evidence is more specific: the difficulty appears to arise from weak planning, insufficient directory and environment handling, limited multi-language robustness, and inadequate comparison between reproduced outputs and paper claims.

5. REPRO-Agent and benchmark-driven improvement

The benchmark was also used to derive an improved agent, REPRO-Agent, from empirical analysis of failure cases on ReproBench (Hu et al., 25 Jul 2025). The design changes include a structured workflow template, a fallback or dummy prediction to ensure applicability, and few-shot or in-context examples tailored to common failure patterns such as Stata log handling and directory-structure issues.

REPRO-Agent achieves 36.6% accuracy and 92.9% applicability (Hu et al., 25 Jul 2025). Relative to the best baseline, CORE-Agent at 21.4% accuracy, this is reported as a 71% relative improvement in accuracy. Applicability improves by 53% relative over the highest baseline (Hu et al., 25 Jul 2025).

The importance of this result is twofold. First, it shows that the benchmark is not saturated; simple agent substitutions do not solve it, but benchmark-specific workflow engineering can substantially improve outcomes. Second, it indicates that a large share of failure is procedural rather than purely inferential. The gains are driven not by new ground-truth labels or relaxed scoring, but by better sequencing of file analysis, code inspection, execution, and results comparison (Hu et al., 25 Jul 2025).

A plausible implication is that ReproBench functions effectively as a diagnostic benchmark for agent architecture, not just as a scoreboard. The initial performance ceiling is low, but targeted changes in planning and error recovery materially change outcomes.

6. ReproBench in subsequent research

ReproBench has already been adopted as a comparative benchmark for later reproducibility-assessment systems. In particular, ARA evaluates on ReproBench and reports 60.71% accuracy, compared with 36.84% for ReplicatorAgent on the same benchmark (Riehl et al., 4 May 2026). The ARA paper reports the following ReproBench results:

System	ACC (%)	F1 (%)	Score Distance	Abs. Score Distance
ARA	60.71	13.32	0.67	1.19
ReplicatorAgent	36.84	22.67	0.63	0.98

ARA’s setting differs from the original REPRO-Bench experimental regime in an important respect: the paper states that ARA operates only on the publication text, producing a workflow graph linking sources, methods, experiments, and outputs, and then inferring reconstructability (Riehl et al., 4 May 2026). The ARA authors therefore use ReproBench as an external target for document-based reproducibility inference, asking whether paper-internal structure is sufficient to approximate the human judgments established through deeper execution-based replication.

The ARA study also offers a critical reading of ReproBench as a benchmark. It emphasizes its strengths as a gold-standard reference with labels backed by completed human replication attempts and its four-point ordinal scale, which preserves nuance beyond binary reproducible/non-reproducible classification (Riehl et al., 4 May 2026). At the same time, it notes that the labels reflect execution outcomes that may depend on factors not detectable from paper text alone, such as repository issues or post-publication data availability (Riehl et al., 4 May 2026).

This suggests an important distinction in how ReproBench can be interpreted. For execution-capable agents, it is a benchmark of end-to-end reproducibility assessment. For text-only systems, it becomes a benchmark of alignment with human reproducibility judgments under partial observability.

7. Significance, limitations, and interpretive issues

ReproBench’s primary significance lies in shifting benchmark design from code rerunability to reproducibility judgment. It evaluates whether an agent can synthesize evidence from papers, data, code, documentation, and execution results in order to assign a structured assessment. This places it in a different category from benchmarks that merely test software execution, static claim extraction, or checklist completion (Hu et al., 25 Jul 2025).

Several limitations are explicit in the source literature. The benchmark is domain-specific, focusing on social science, which constrains direct generalization to natural science, engineering, or mathematically dense fields (Riehl et al., 4 May 2026). The label space is ordinal and involves distinctions such as minor coding inconsistencies versus minor reporting discrepancies, which may be difficult even for experts to separate cleanly in all cases (Hu et al., 25 Jul 2025, Riehl et al., 4 May 2026). In addition, the benchmark’s realism creates a mismatch between accessible evidence and ground truth: a system restricted to paper text cannot observe all factors that determine the final human reproduction outcome (Riehl et al., 4 May 2026).

There are also methodological misconceptions that the benchmark helps correct. One is the assumption that reproducibility assessment reduces to successful code execution. ReproBench explicitly requires comparison against claims in the paper and can assign lower scores even when artifacts run, if inconsistencies remain (Hu et al., 25 Jul 2025). Another is the assumption that diversity of file formats is the main obstacle. The benchmark results indicate that multiple data formats were less problematic than multi-language execution and weak cross-artifact reasoning (Hu et al., 25 Jul 2025).

The REPRO-Bench paper concludes that more advanced AI agents should be developed for real-world reproducibility assessment (Hu et al., 25 Jul 2025). ARA, in turn, presents evidence that structured workflow extraction from documents can substantially improve alignment with human judgments on ReproBench, while also underscoring that such systems complement rather than replace actual replication (Riehl et al., 4 May 2026). Taken together, these results position ReproBench as both a benchmark and a methodological probe: it measures system performance, but it also reveals which components of agentic scientific reasoning remain underdeveloped.