ReproRepo: Scaling Reproducibility Audits with GitHub Repository Issues

Abstract: Reproducing research results from papers and released code is central to scientific progress. Existing works have introduced benchmarks to evaluate whether LLM agents can assist with reproducibility, but they are difficult to scale due to their reliance on substantial manual effort for data curation and evaluation. We introduce ReproRepo, a scalable framework for reproducibility evaluation that leverages human-raised GitHub issues as naturally occurring supervision on realistic reproduction blockers. We instantiate ReproRepo on 1,149 recent machine learning papers from major conferences and evaluate four frontier model-agent configurations. Our results show that LLM agents, even without executing code, can identify many real-world reproducibility problems from paper-repository pairs: the best agent in our study, namely Codex with GPT-5.5, surfaces at least one semantically related human-reported blocker for ~90% of papers in the study. Further analysis shows that agents are particularly effective for surfacing visible failures and identifying the right semantic region, but may still be insufficient in exact localization. ReproRepo can serve as a reusable, scalable framework for future evaluations of LLM agents on real-world reproducibility auditing. Our code is released at https://github.com/LithiumDA/ReproRepo.

• The paper introduces a scalable benchmark using GitHub issues as supervisory signals to perform static reproducibility audits with LLM agents.
• It systematically links papers to repositories and curates 7,553 real-world reproducibility issues from 1,149 ML papers.
• Empirical results show high semantic match rates (≈90%) for visible failures and minimal false positives (under 3.5%) across audits.

ReproRepo: A Scalable Benchmark for Static Paper-Code Reproducibility Auditing with LLM Agents

Motivation and Context

Reproducibility remains a central challenge in scientific machine learning due to the complexity of typical paper-code-data-experiment pipelines. Prior LLM-based reproducibility benchmarks exhibit limited coverage, constrained by labor-intensive, expert-driven manual curation, annotation, and evaluation, typically offering fewer than 100 tasks and quickly becoming obsolete as the field develops. Addressing evaluative scalability, "ReproRepo: Scaling Reproducibility Audits with GitHub Repository Issues" (2606.18237) introduces a new paradigm: leveraging naturally occurring, human-reported GitHub issues as supervisory signals for benchmarking LLM agents on realistic paper-code reproducibility blockers.

Framework and Methodology

The ReproRepo pipeline systematically develops an issue-grounded benchmark at scale by (i) linking conference papers to corresponding GitHub repositories, (ii) collecting open and closed user-raised GitHub issues associated with reproduction efforts, and (iii) annotating those issues relevant to reproducibility using an LLM judge. Individual benchmarks are built from paper-repository pairs and sets of matched reproduction issues, representing an accurate slice of real-world user pain-points.

Agents receive, as input, a static snapshot of the paper PDF and repository files (with git history stripped and internet access disabled). They are prompted to audit for concrete, user-facing reproduction blockers—encompassing missing artifacts, misconfigured dependencies, training/evaluation workflow gaps, result provenance ambiguities, and more—without any code execution or environment setup. Crucially, agents must rank potential issues by their estimated likelihood to be encountered by a typical reproducer (Figure 1).

Figure 1: The ReproRepo pipeline constructs issue-grounded benchmarks from paper-repository pairs, enabling static audits by LLM agents whose output is compared to hidden, human-reported reproducibility blockers.

Agent findings are compared post hoc to the withheld ground-truth issue set using a separate LLM judge, yielding both exact match (EM) and semantic match (SM) alignment metrics. False positive rates are estimated by rerunning agents on post-fix snapshots associated with resolved, patch-backed issues.

Dataset Scale and Statistical Overview

ReproRepo aggregates 1,149 accepted ML papers from NeurIPS 2022/2024 (main and datasets/benchmarks tracks) and ICLR 2026, yielding 7,553 unique, curated reproducibility issues. This represents an order-of-magnitude increase over prior benchmarks. The per-paper issue count distribution is long-tailed: most repositories present a handful of blockers, while a minority accumulate many user reports. Papers span diverse research foci, with coverage across vision, audio, foundational models, and more. Across venues, four principal failure archetypes dominate: silent wrong setup, crash immediate, silent wrong number, and crash late, with silent wrong setup accounting for over half of all annotated issues (Figure 2).

Figure 2: Distributional analyses reveal most papers contribute a few issues with a long tail of high-issue repositories; research area diversity is broad, and the dominant failure mode across venues is silent wrong setup.

Main Empirical Results

Evaluation spans four (Claude Code, Codex) × (DeepSeek-V4-Pro, Claude Opus 4.7, GPT-5.4-Mini, GPT-5.5) agent configurations. The main focus is on compute-light, static audit, budgeted to top-$k$=10 findings per agent-paper interaction.

Key findings include:

• High semantic coverage: Codex+GPT-5.5 semantically surfaces at least one human-reported reproduction blocker for $\sim 90\%$ of audited papers ($\mathrm{SM}@10$), with exact match rates up to 60%. Results are consistent across venues and years, ruling out collection artifacts and information leakage.
• Precision, not verbosity: Most semantic matches occur within the top 4 ranked findings with minimal gain for larger $k$; long agent reports do not drive the result (Figure 3, left/right).
• Minimal hallucination: The false positive rate (incidence of re-identifying a resolved and patched blocker in the post-fix snapshot) is under 3.5% for all configurations.
• Visible failures most recovered: Static audit is especially effective for surfacing silent wrong setup and crash immediate categories, but performs less well on crash late or metric deviation failures, which generally require execution (Figure 3, right).

Figure 3: Left: Semantic match rates rise steeply at small $k$ and plateau quickly, indicating that highly ranked agent findings recover most reproduction blockers. Right: GPT-5.5 achieves substantially higher recall on visible failure categories during static inspection.

An ablation validates the necessity of providing both paper and code as input: code-only audits significantly underperform on exact matching, demonstrating that paper-repository consistency is pivotal for localizing real blockers.

Taxonomic and Qualitative Analyses

Category-level recall analysis demonstrates ReproRepo benchmarks not only repository-level failures, but also mismatches between paper claims and repository artifacts. Agents are proficient in identifying defects in setup, documentation, artifact availability, and reproducible evaluation workflow, but are limited in diagnosing post-install crashes or silent deviation failures without code execution. Manual analysis of non-matching high-ranked agent findings reveals that the majority represent valid but alternative reproducibility risks that users may have locally circumvented without filing issues.

Implications for LLM-Based Artifact Auditing

ReproRepo establishes that state-of-the-art LLM agents can, through static inspection alone, anticipate and surface a large fraction of user-observed reproducibility blockers at scale. The direct supervision signal—real user-filed GitHub issues—yields realistic, updatable, and diverse evaluation settings that align closely with the lived experience of paper reproduction. However, static auditing is not sufficient for complete localization, especially for late-stage or silent numerical failures; dynamic, execution-grounded validation remains necessary for comprehensive coverage.

From a practical perspective, ReproRepo offers an efficient, continually extensible protocol for conference artifact evaluation, reviewer assistance, and repository triage. On the theoretical side, the methodology introduced outperforms manual curation in scale, freshness, and ecological validity, yet naturally incorporates the noise, incompleteness, and redundancy intrinsic to real user reports.

Limitations and Future Directions

The supervision signal exclusively depends on human-raised GitHub issues; unreported or user-corrected blockers are omitted, and some issue reports may reflect user/environment-specific or transient failures. Complementary expert curation or hybrid pipelines could enrich coverage. Current experiments exclusively analyze static, no-execution auditing; real-world triage may benefit from integrating static and dynamic analysis, especially as benchmarks shift toward harder, less visible failure types. The extensibility of ReproRepo allows benchmarking on future, newly published artifacts with minimal manual overhead.

Conclusion

ReproRepo redefines the landscape for LLM-based reproducibility auditing by harnessing naturally occurring, richly contextual user feedback at scale. The results unambiguously demonstrate the feasibility of static, agentic triage for a significant fraction of reproducibility blockers in modern ML research. The approach presents an extensible foundation for subsequent advancements in agentic auditing workflows, scalable artifact evaluation, and robust reproducibility science.