ReproRepo: Scalable Reproducibility Audit
- ReproRepo is a scalable framework for reproducibility auditing that uses frozen code snapshots and paper PDFs to recover real-world blockers.
- It replaces manual expert curation with naturally occurring GitHub issues, identifying errors like broken imports and documentation gaps.
- The framework employs static audits and LLM agents to rank blockers effectively, demonstrating robust performance across diverse ML conferences.
ReproRepo is a scalable framework for reproducibility evaluation that uses human-raised GitHub issues as naturally occurring supervision on realistic reproduction blockers in paper-linked repositories. Rather than constructing expert-curated tasks, injected failures, or manually scored rubrics, it converts paper–repository pairs into static audit tasks and asks LLM agents to recover hidden human-reported blockers from frozen code snapshots and paper PDFs. In its published instantiation, the framework covers 1,149 recent machine learning papers from major conferences and 7,553 reproducibility-related GitHub issues, with leakage control through pre-fix and post-fix repository snapshots and automated scoring based on exact and semantic issue alignment (Li et al., 16 Jun 2026).
1. Conceptual basis and problem formulation
ReproRepo was introduced to address a scaling problem in reproducibility benchmarking. Earlier evaluations of agentic reproducibility assistance were described as difficult to scale because they depended on substantial manual artifact preparation, expert-curated tasks, injected failures, rubric design, and human evaluation. ReproRepo replaces that supervision source with organically occurring GitHub issues attached to paper-linked repositories. These issues record reproduction blockers that users actually encountered, including broken imports, missing checkpoints, undocumented data preparation, and paper–code mismatches (Li et al., 16 Jun 2026).
The framework is deliberately a static, blind audit rather than an execution benchmark. Each audit run provides an agent with a local workspace containing a frozen repository snapshot and the paper PDF, while withholding internet access, code execution, environment setup, GPU compute, issue histories, pull requests, and external retrieval. The task is to emit an ordered list of concrete, GitHub-issue-style blockers that a reproducer would likely encounter. This makes ReproRepo a benchmark for reproducibility auditing, not for end-to-end rerunning of experiments (Li et al., 16 Jun 2026).
A common misconception is that ReproRepo measures whether a paper is reproducible in the strict experimental sense. It does not. Its operational question is narrower: whether an agent can recover hidden human-reported blockers from the observable paper–repository pair. The distinction matters because the published results show a consistent gap between semantic region identification and exact localization. Agents often identify the right workflow, artifact family, or failure surface, while missing the precise trigger or root cause (Li et al., 16 Jun 2026).
2. Corpus construction, snapshotting, and issue taxonomy
The published corpus spans five conference splits: NeurIPS 2022 main, NeurIPS 2022 datasets and benchmarks, NeurIPS 2024 main, NeurIPS 2024 datasets and benchmarks, and ICLR 2026. Paper metadata and GitHub URLs were collected from Paper Copilot and conference metadata. Non-GitHub links and papers without a usable repository were removed. Across the full collection process, 12,686 accepted papers were examined, and the final benchmark retained 1,149 papers that had at least one accepted benchmark issue after filtering (Li et al., 16 Jun 2026).
Issue collection included both open and closed GitHub issues, with a preprocessing filter that removed repositories with more than 100 issues to stay within GitHub API limits. An LLM judge filtered those issues for reproducibility relevance, retaining cases such as installation or environment failures, missing artifacts, documentation gaps, paper–repository mismatches, training or evaluation failures, and metric discrepancies. The resulting benchmark contains 7,553 human-reported issues, comprising 3,267 open issues and 4,286 closed issues. Among the closed issues, 150 were identified as having clear code patches that fixed the problem (Li et al., 16 Jun 2026).
Leakage control is central to the design. For open issues, ReproRepo uses the default-branch head at collection time. For closed issues, it reconstructs an issue-time pre-fix snapshot immediately before the issue was opened, and, when patch evidence exists, also constructs a post-fix snapshot. The post-fix snapshot is then used to estimate a conservative upper bound on false positives: if an agent still reports a blocker that matches a fixed issue on the post-fix state, that paper is counted as a false positive (Li et al., 16 Jun 2026).
Human issues are organized along two axes—crash vs. silent and setup vs. late-stage—which yield four categories: crash immediate, crash late, silent wrong setup, and silent wrong number. Across venues and tracks, silent wrong setup is the dominant category, accounting for 52–59% of issues. The issue distribution is long-tailed, with most papers having only a few issues and a small number of repositories having many (Li et al., 16 Jun 2026).
3. Audit protocol, output schema, and evaluation metrics
The agent’s output is a single findings.json file containing ranked, atomic findings and a brief reproducibility assessment. Each finding is expected to resemble a well-formed GitHub issue report: it should include the symptom, trigger, likely root cause, static evidence such as files, paths, commands, or configurations, impact on reproduction, affected claims or metrics, and a category label. The allowed category labels include environment_setup, dependency_version, data_access, model_weights, training_reproduction, evaluation_reproduction, runtime_error, hardware_backend, api_service, result_mismatch, documentation_gap, and paper_repo_mismatch (Li et al., 16 Jun 2026).
Scoring is performed by a separate LLM judge that sees the hidden human issues for the same snapshot but not the agent during generation. For each hidden issue, the judge compares it to the agent’s top- findings and assigns one of three labels: exact match (EM), semantic match (SM), or none. The comparison is dimension-aware: it examines user-facing symptom, trigger path or workflow, root cause, involved artifact such as file, command, config, dataset, model, or metric, and impact on reproducing the paper (Li et al., 16 Jun 2026).
The reported metrics are defined at both issue and paper levels. Let denote the top- findings for paper , and let be the best label assigned to hidden issue . ReproRepo reports:
At the paper level, the benchmark distinguishes any-issue and all-issue variants. The any-issue version counts a paper as matched if at least one of its issues is recovered; the all-issue version requires all issues for that paper to be recovered. False positive rate is then estimated on the post-fix subset at the paper level: a paper is counted as a false positive if any finding matches an already-fixed issue in its post-fix snapshot (Li et al., 16 Jun 2026).
This protocol implies that ReproRepo evaluates ordered triage quality as well as coverage. The prompt explicitly instructs the model to order findings by the likelihood that real users would report them, and the published ablation shows that this ranking instruction improves early-budget performance without changing asymptotic coverage as increases (Li et al., 16 Jun 2026).
4. Empirical results, reliability, and behavioral patterns
The framework was evaluated with four frontier model–agent configurations: DeepSeek-V4-Pro with Claude Code, Claude Opus 4.7 with Claude Code, GPT-5.4-Mini with Codex, and GPT-5.5 with Codex. On ICLR 2026 at , GPT-5.5 with Codex was the strongest overall configuration, reaching Issue EM@10 = 25.4%, Issue SM@10 = 58.1%, Paper any EM@10 = 59.7%, Paper any SM@10 = 89.7%, Paper all EM@10 = 34.8%, Paper all SM@10 = 65.6%, with FPR = 3.50% and average cost \$1.26 per run (Li et al., 16 Jun 2026).
| Configuration | Paper any SM@10 | FPR / cost |
|---|---|---|
| DeepSeek-V4-Pro + Claude Code | 89.0% | 2.80% / \$0.29 |
| Claude Opus 4.7 + Claude Code | 86.5% | 2.10% / \$4.07 |
| GPT-5.4-Mini + Codex | 82.1% | 1.40% / \$0.22 |
| GPT-5.5 + Codex | 89.7% | 3.50% / \$1.26 |
Several patterns are structurally important. First, the benchmark shows high semantic coverage without execution. With the best configuration, agents surface at least one semantically related hidden issue for roughly 90% of papers, despite having no ability to run code or inspect issue histories. Second, there is a persistent EM–SM gap. On ICLR 2026, GPT-5.5 with Codex achieves 58.1% Issue SM@10 but only 25.4% Issue EM@10, indicating that agents frequently identify the correct semantic region while falling short of precise localization (Li et al., 16 Jun 2026).
Third, performance is stable across venues, but datasets-and-benchmarks tracks are harder. For GPT-5.5 with Codex at 0, paper-any semantic match rates range from 87.6% on NeurIPS 2024 datasets and benchmarks to 96.6% on NeurIPS 2022 datasets and benchmarks, while issue-level exact match varies between 12.2% and 25.4%. Those tracks also consume more tokens, consistent with larger and more complex artifacts (Li et al., 16 Jun 2026).
Fourth, the paper demonstrates that the paper itself materially improves auditing quality. In an ICLR 2026 ablation with GPT-5.5, adding the paper to the repository raises Paper any EM@10 from 50.7% under code-only input to 59.7% under paper-plus-code input, a +8.98 percentage-point gain. Semantic rates improve more modestly, from 87.6% to 89.7%. This indicates that paper context mainly helps convert generic repository risk detection into paper-grounded localization (Li et al., 16 Jun 2026).
The authors also report several reliability studies. Three independent runs of the full pipeline show small variance; for GPT-5.5 on ICLR 2026, Paper any SM@10 = 89.7% ± 0.73% and Issue EM@10 = 25.0% ± 0.50%. A human validation over 150 judge-labeled finding–issue pairs yields 85.3% agreement on matched versus unmatched, with judge precision 84.0% and recall 93.3% on human-labeled matches. This does not eliminate judge noise, but it indicates that the alignment layer is not purely anecdotal (Li et al., 16 Jun 2026).
A further nuance concerns unmatched top-ranked findings. For 89 ICLR 2026 papers with exactly one human issue where the top-ranked GPT-5.5 finding did not match that issue, manual review found concrete static evidence in all 89 cases. The dominant categories were documentation–repository mismatch, dependency or environment gap, missing or underspecified artifacts, missing evaluation or reproduction workflow, and paper–code protocol mismatch. This suggests that unmatched findings often represent parallel, plausible reproducibility risks rather than unsupported hallucinations (Li et al., 16 Jun 2026).
5. Position within the reproducibility-tool landscape
ReproRepo belongs to a broader ecosystem of repositories, platforms, and assessment systems, but its role is unusually specific: it evaluates static reproducibility auditing at scale. This differs from platforms designed to host reproductions, package executable environments, or execute experiments directly.
| System | Primary function | Distinguishing feature |
|---|---|---|
| ReproducedPapers.org (Yildiz et al., 2020) | Teaching and sharing ML reproductions | Aspect-badged, low-barrier, multiple reproductions per paper |
| REP (Likhomanenko et al., 2015) | Reproducible computational experiment platform | Python, notebooks, Git, Docker, IPykee |
| Repro (Deutsch et al., 2022) | Running published research code | Per-paper Docker images behind a lightweight Python API |
| SciRep (Costa et al., 10 Mar 2025) | Cross-domain reproducibility capsules | Packages code, data, dependencies, databases, and commands |
| ReproScore (Samuel et al., 13 May 2026) | Repository-scale reproducibility assessment | Separates static readiness from execution outcome |
| RRP (Cuny et al., 4 Dec 2025) | Unified open-science execution environment | Integrates RDMS, Git, containerized environments, and shareable projects |
ReproducedPapers.org is a low-barrier venue for machine-learning reproductions, including lightweight efforts such as partial results, hyperparameter sweeps, and algorithmic variants, and it organizes contributions through aspect badges such as Replicated, Reproduced, and Hyperparams check (Yildiz et al., 2020). REP and SciRep are execution-oriented infrastructures: REP centers on notebooks, Git, Docker, and comparable model pipelines (Likhomanenko et al., 2015), while SciRep builds self-contained reproducibility capsules that package code, data, dependencies, optional databases, and commands for re-execution on any computer (Costa et al., 10 Mar 2025). Repro standardizes access to original paper code by wrapping each supported paper in a Docker image with a lightweight Python API (Deutsch et al., 2022). ReproScore, by contrast, offers a two-tier scoring architecture that separates static reproducibility readiness from execution-based reproducibility outcome, which is conceptually adjacent to ReproRepo’s distinction between visible static evidence and full execution behavior (Samuel et al., 13 May 2026). RRP unifies research data management with containerized environments and modular, shareable projects spanning proposal to publication (Cuny et al., 4 Dec 2025).
This suggests that ReproRepo occupies an audit layer rather than a repository-execution layer. It is closest in spirit to large-scale evaluation frameworks such as ReproScore, but instead of scoring repository structure or running controlled probes, it uses naturally occurring GitHub issues as hidden supervision for agentic discovery of real-world blockers. It is therefore complementary to execution and packaging systems rather than a replacement for them (Li et al., 16 Jun 2026).
6. Limitations, interpretation, and future directions
The framework’s main limitations derive from its supervision source and non-execution design. GitHub issues are noisy and incomplete: users may report only the first blocker they encounter, some valid risks are never filed, and a few reports reflect user-specific environments. The benchmark mitigates these effects through LLM filtering, pre-fix and post-fix snapshotting, and false-positive evaluation, but it does not eliminate reporting bias (Li et al., 16 Jun 2026).
Selection bias is also explicit. The corpus is conference-driven, recent, and centered on machine learning, which may underrepresent long-tail domains or older repository cultures. ReproRepo’s scale is large relative to earlier reproducibility evaluations, but it remains tied to venues with established paper–code linking practices and public GitHub use (Li et al., 16 Jun 2026).
A second major limitation is that static auditing cannot fully diagnose late-stage crashes, environment-specific runtime bugs, or metric discrepancies that only appear after training or evaluation. The published category analysis reflects this boundary: agents recover silent wrong setup and crash immediate more reliably than crash late and silent wrong number. In other words, the benchmark is strongest on blockers that leave visible static traces in README files, configs, paths, commands, and paper–repository inconsistencies (Li et al., 16 Jun 2026).
The authors explicitly present future work as movement toward hybridization rather than replacement. Proposed directions include combining static audits with partial execution sandboxes, automatic triage from semantic to exact matching, richer provenance checks, and extension beyond machine learning. Ethical handling is also defined narrowly: the release uses only public artifacts, avoids unnecessary personal information, and treats findings as triage signals, not definitive judgments on papers, repositories, or authors (Li et al., 16 Jun 2026).
In that sense, ReproRepo represents a change in how reproducibility evaluation can be scaled. It does not solve reproducibility by itself, and it does not certify that published results can be rerun. Instead, it operationalizes a large, realistic, and refreshable benchmark for whether agents can detect the kinds of blockers that human users actually file. For a field increasingly interested in repository-level LLM assistance, that narrower objective is technically distinct and methodologically significant (Li et al., 16 Jun 2026).