---
title: 'ReproRepo: Scalable Reproducibility Audit'
url: https://www.emergentmind.com/topics/reprorepo
type: topic
---

# ReproRepo: Scalable Reproducibility Audit

ReproRepo is a scalable framework for reproducibility evaluation that uses human-raised GitHub issues as naturally occurring supervision on realistic reproduction blockers in paper-linked repositories. Rather than constructing expert-curated tasks, injected failures, or manually scored rubrics, it converts paper–repository pairs into static audit tasks and asks LLM agents to recover hidden human-reported blockers from frozen code snapshots and paper PDFs. In its published instantiation, the framework covers 1,149 recent machine learning papers from major conferences and 7,553 reproducibility-related GitHub issues, with leakage control through pre-fix and post-fix repository snapshots and automated scoring based on exact and semantic issue alignment [2606.18237].

## 1. Conceptual basis and problem formulation

ReproRepo was introduced to address a scaling problem in reproducibility benchmarking. Earlier evaluations of agentic reproducibility assistance were described as difficult to scale because they depended on substantial manual artifact preparation, expert-curated tasks, injected failures, rubric design, and human evaluation. ReproRepo replaces that supervision source with organically occurring GitHub issues attached to paper-linked repositories. These issues record reproduction blockers that users actually encountered, including broken imports, missing checkpoints, undocumented data preparation, and paper–code mismatches [2606.18237].

The framework is deliberately a **static, blind audit** rather than an execution benchmark. Each audit run provides an agent with a local workspace containing a frozen repository snapshot and the paper PDF, while withholding internet access, code execution, environment setup, GPU compute, issue histories, pull requests, and external retrieval. The task is to emit an ordered list of concrete, GitHub-issue-style blockers that a reproducer would likely encounter. This makes ReproRepo a benchmark for **reproducibility auditing**, not for end-to-end rerunning of experiments [2606.18237].

A common misconception is that ReproRepo measures whether a paper is reproducible in the strict experimental sense. It does not. Its operational question is narrower: whether an agent can recover hidden human-reported blockers from the observable paper–repository pair. The distinction matters because the published results show a consistent gap between **semantic region identification** and **exact localization**. Agents often identify the right workflow, artifact family, or failure surface, while missing the precise trigger or root cause [2606.18237].

## 2. Corpus construction, snapshotting, and issue taxonomy

The published corpus spans five conference splits: NeurIPS 2022 main, NeurIPS 2022 datasets and benchmarks, NeurIPS 2024 main, NeurIPS 2024 datasets and benchmarks, and ICLR 2026. Paper metadata and GitHub URLs were collected from Paper Copilot and conference metadata. Non-GitHub links and papers without a usable repository were removed. Across the full collection process, 12,686 accepted papers were examined, and the final benchmark retained 1,149 papers that had at least one accepted benchmark issue after filtering [2606.18237].

Issue collection included both open and closed GitHub issues, with a preprocessing filter that removed repositories with more than 100 issues to stay within GitHub API limits. An LLM judge filtered those issues for reproducibility relevance, retaining cases such as installation or environment failures, missing artifacts, documentation gaps, paper–repository mismatches, training or evaluation failures, and metric discrepancies. The resulting benchmark contains 7,553 human-reported issues, comprising 3,267 open issues and 4,286 closed issues. Among the closed issues, 150 were identified as having clear code patches that fixed the problem [2606.18237].

Leakage control is central to the design. For open issues, ReproRepo uses the default-branch head at collection time. For closed issues, it reconstructs an **issue-time pre-fix snapshot** immediately before the issue was opened, and, when patch evidence exists, also constructs a **post-fix snapshot**. The post-fix snapshot is then used to estimate a conservative upper bound on false positives: if an agent still reports a blocker that matches a fixed issue on the post-fix state, that paper is counted as a false positive [2606.18237].

Human issues are organized along two axes—**crash vs. silent** and **setup vs. late-stage**—which yield four categories: **crash immediate**, **crash late**, **silent wrong setup**, and **silent wrong number**. Across venues and tracks, silent wrong setup is the dominant category, accounting for 52–59% of issues. The issue distribution is long-tailed, with most papers having only a few issues and a small number of repositories having many [2606.18237].

## 3. Audit protocol, output schema, and evaluation metrics

The agent’s output is a single `findings.json` file containing ranked, atomic findings and a brief reproducibility assessment. Each finding is expected to resemble a well-formed GitHub issue report: it should include the symptom, trigger, likely root cause, static evidence such as files, paths, commands, or configurations, impact on reproduction, affected claims or metrics, and a category label. The allowed category labels include `environment_setup`, `dependency_version`, `data_access`, `model_weights`, `training_reproduction`, `evaluation_reproduction`, `runtime_error`, `hardware_backend`, `api_service`, `result_mismatch`, `documentation_gap`, and `paper_repo_mismatch` [2606.18237].

Scoring is performed by a separate LLM judge that sees the hidden human issues for the same snapshot but not the agent during generation. For each hidden issue, the judge compares it to the agent’s top-\(k\) findings and assigns one of three labels: **exact match (EM)**, **semantic match (SM)**, or **none**. The comparison is dimension-aware: it examines user-facing symptom, trigger path or workflow, root cause, involved artifact such as file, command, config, dataset, model, or metric, and impact on reproducing the paper [2606.18237].

The reported metrics are defined at both issue and paper levels. Let \(F_p(k)\) denote the top-\(k\) findings for paper \(p\), and let \(y_i^{(k)} \in \{\mathrm{EM}, \mathrm{SM}, \mathrm{None}\}\) be the best label assigned to hidden issue \(i\). ReproRepo reports:

$$
\mathrm{Issue\_EM@k} = \frac{\left|\{ i \in I : y_i^{(k)} = \mathrm{EM} \}\right|}{|I|}
$$

$$
\mathrm{Issue\_SM@k} = \frac{\left|\{ i \in I : y_i^{(k)} \in \{\mathrm{EM}, \mathrm{SM}\} \}\right|}{|I|}
$$

At the paper level, the benchmark distinguishes **any-issue** and **all-issue** variants. The any-issue version counts a paper as matched if at least one of its issues is recovered; the all-issue version requires all issues for that paper to be recovered. False positive rate is then estimated on the post-fix subset at the paper level: a paper is counted as a false positive if any finding matches an already-fixed issue in its post-fix snapshot [2606.18237].

This protocol implies that ReproRepo evaluates ordered triage quality as well as coverage. The prompt explicitly instructs the model to order findings by the likelihood that real users would report them, and the published ablation shows that this ranking instruction improves early-budget performance without changing asymptotic coverage as \(k\) increases [2606.18237].

## 4. Empirical results, reliability, and behavioral patterns

The framework was evaluated with four frontier model–agent configurations: DeepSeek-V4-Pro with Claude Code, Claude Opus 4.7 with Claude Code, GPT-5.4-Mini with Codex, and GPT-5.5 with Codex. On ICLR 2026 at \(k=10\), GPT-5.5 with Codex was the strongest overall configuration, reaching **Issue EM@10 = 25.4%**, **Issue SM@10 = 58.1%**, **Paper any EM@10 = 59.7%**, **Paper any SM@10 = 89.7%**, **Paper all EM@10 = 34.8%**, **Paper all SM@10 = 65.6%**, with **FPR = 3.50%** and average cost **\$1.26** per run [2606.18237].

| Configuration | Paper any SM@10 | FPR / cost |
|---|---:|---:|
| DeepSeek-V4-Pro + Claude Code | 89.0% | 2.80% / \$0.29 |
| Claude Opus 4.7 + Claude Code | 86.5% | 2.10% / \$4.07 |
| GPT-5.4-Mini + Codex | 82.1% | 1.40% / \$0.22 |
| GPT-5.5 + Codex | 89.7% | 3.50% / \$1.26 |

Several patterns are structurally important. First, the benchmark shows **high semantic coverage without execution**. With the best configuration, agents surface at least one semantically related hidden issue for roughly 90% of papers, despite having no ability to run code or inspect issue histories. Second, there is a persistent **EM–SM gap**. On ICLR 2026, GPT-5.5 with Codex achieves 58.1% Issue SM@10 but only 25.4% Issue EM@10, indicating that agents frequently identify the correct semantic region while falling short of precise localization [2606.18237].

Third, performance is stable across venues, but datasets-and-benchmarks tracks are harder. For GPT-5.5 with Codex at \(k=10\), paper-any semantic match rates range from **87.6%** on NeurIPS 2024 datasets and benchmarks to **96.6%** on NeurIPS 2022 datasets and benchmarks, while issue-level exact match varies between **12.2%** and **25.4%**. Those tracks also consume more tokens, consistent with larger and more complex artifacts [2606.18237].

Fourth, the paper demonstrates that the **paper itself materially improves auditing quality**. In an ICLR 2026 ablation with GPT-5.5, adding the paper to the repository raises Paper any EM@10 from **50.7%** under code-only input to **59.7%** under paper-plus-code input, a **+8.98 percentage-point** gain. Semantic rates improve more modestly, from **87.6%** to **89.7%**. This indicates that paper context mainly helps convert generic repository risk detection into paper-grounded localization [2606.18237].

The authors also report several reliability studies. Three independent runs of the full pipeline show small variance; for GPT-5.5 on ICLR 2026, **Paper any SM@10 = 89.7% ± 0.73%** and **Issue EM@10 = 25.0% ± 0.50%**. A human validation over 150 judge-labeled finding–issue pairs yields **85.3%** agreement on matched versus unmatched, with judge precision **84.0%** and recall **93.3%** on human-labeled matches. This does not eliminate judge noise, but it indicates that the alignment layer is not purely anecdotal [2606.18237].

A further nuance concerns unmatched top-ranked findings. For 89 ICLR 2026 papers with exactly one human issue where the top-ranked GPT-5.5 finding did not match that issue, manual review found concrete static evidence in all 89 cases. The dominant categories were documentation–repository mismatch, dependency or environment gap, missing or underspecified artifacts, missing evaluation or reproduction workflow, and paper–code protocol mismatch. This suggests that unmatched findings often represent parallel, plausible reproducibility risks rather than unsupported hallucinations [2606.18237].

## 5. Position within the reproducibility-tool landscape

ReproRepo belongs to a broader ecosystem of repositories, platforms, and assessment systems, but its role is unusually specific: it evaluates **static reproducibility auditing** at scale. This differs from platforms designed to host reproductions, package executable environments, or execute experiments directly.

| System | Primary function | Distinguishing feature |
|---|---|---|
| ReproducedPapers.org [2012.01172] | Teaching and sharing ML reproductions | Aspect-badged, low-barrier, multiple reproductions per paper |
| REP [1510.00624] | Reproducible computational experiment platform | Python, notebooks, Git, Docker, IPykee |
| Repro [2204.13848] | Running published research code | Per-paper Docker images behind a lightweight Python API |
| SciRep [2503.07080] | Cross-domain reproducibility capsules | Packages code, data, dependencies, databases, and commands |
| ReproScore [2605.13275] | Repository-scale reproducibility assessment | Separates static readiness from execution outcome |
| RRP [2512.06039] | Unified open-science execution environment | Integrates RDMS, Git, containerized environments, and shareable projects |

ReproducedPapers.org is a low-barrier venue for machine-learning reproductions, including lightweight efforts such as partial results, hyperparameter sweeps, and algorithmic variants, and it organizes contributions through aspect badges such as **Replicated**, **Reproduced**, and **Hyperparams check** [2012.01172]. REP and SciRep are execution-oriented infrastructures: REP centers on notebooks, Git, Docker, and comparable model pipelines [1510.00624], while SciRep builds self-contained reproducibility capsules that package code, data, dependencies, optional databases, and commands for re-execution on any computer [2503.07080]. Repro standardizes access to original paper code by wrapping each supported paper in a Docker image with a lightweight Python API [2204.13848]. ReproScore, by contrast, offers a two-tier scoring architecture that separates static **reproducibility readiness** from execution-based **reproducibility outcome**, which is conceptually adjacent to ReproRepo’s distinction between visible static evidence and full execution behavior [2605.13275]. RRP unifies research data management with containerized environments and modular, shareable projects spanning proposal to publication [2512.06039].

This suggests that ReproRepo occupies an **audit layer** rather than a repository-execution layer. It is closest in spirit to large-scale evaluation frameworks such as ReproScore, but instead of scoring repository structure or running controlled probes, it uses naturally occurring GitHub issues as hidden supervision for agentic discovery of real-world blockers. It is therefore complementary to execution and packaging systems rather than a replacement for them [2606.18237].

## 6. Limitations, interpretation, and future directions

The framework’s main limitations derive from its supervision source and non-execution design. GitHub issues are noisy and incomplete: users may report only the first blocker they encounter, some valid risks are never filed, and a few reports reflect user-specific environments. The benchmark mitigates these effects through LLM filtering, pre-fix and post-fix snapshotting, and false-positive evaluation, but it does not eliminate reporting bias [2606.18237].

Selection bias is also explicit. The corpus is conference-driven, recent, and centered on machine learning, which may underrepresent long-tail domains or older repository cultures. ReproRepo’s scale is large relative to earlier reproducibility evaluations, but it remains tied to venues with established paper–code linking practices and public GitHub use [2606.18237].

A second major limitation is that static auditing cannot fully diagnose late-stage crashes, environment-specific runtime bugs, or metric discrepancies that only appear after training or evaluation. The published category analysis reflects this boundary: agents recover **silent wrong setup** and **crash immediate** more reliably than **crash late** and **silent wrong number**. In other words, the benchmark is strongest on blockers that leave visible static traces in README files, configs, paths, commands, and paper–repository inconsistencies [2606.18237].

The authors explicitly present future work as movement toward hybridization rather than replacement. Proposed directions include combining static audits with partial execution sandboxes, automatic triage from semantic to exact matching, richer provenance checks, and extension beyond machine learning. Ethical handling is also defined narrowly: the release uses only public artifacts, avoids unnecessary personal information, and treats findings as **triage signals**, not definitive judgments on papers, repositories, or authors [2606.18237].

In that sense, ReproRepo represents a change in how reproducibility evaluation can be scaled. It does not solve reproducibility by itself, and it does not certify that published results can be rerun. Instead, it operationalizes a large, realistic, and refreshable benchmark for whether agents can detect the kinds of blockers that human users actually file. For a field increasingly interested in repository-level LLM assistance, that narrower objective is technically distinct and methodologically significant [2606.18237].

Source: https://www.emergentmind.com/topics/reprorepo