From Industry Claims to Empirical Reality: An Empirical Study of Code Review Agents in Pull Requests

Abstract: Autonomous coding agents are generating code at an unprecedented scale, with OpenAI Codex alone creating over 400,000 pull requests (PRs) in two months. As agentic PR volumes increase, code review agents (CRAs) have become routine gatekeepers in development workflows. Industry reports claim that CRAs can manage 80% of PRs in open source repositories without human involvement. As a result, understanding the effectiveness of CRA reviews is crucial for maintaining developmental workflows and preventing wasted effort on abandoned pull requests. However, empirical evidence on how CRA feedback quality affects PR outcomes remains limited. The goal of this paper is to help researchers and practitioners understand when and how CRAs influence PR merge success by empirically analyzing reviewer composition and the signal quality of CRA-generated comments. From AIDev's 19,450 PRs, we analyze 3,109 unique PRs in the commented review state, comparing human-only versus CRA-only reviews. We examine 98 closed CRA-only PRs to assess whether low signal-to-noise ratios contribute to abandonment. CRA-only PRs achieve a 45.20% merge rate, 23.17 percentage points lower than human-only PRs (68.37%), with significantly higher abandonment. Our signal-to-noise analysis reveals that 60.2% of closed CRA-only PRs fall into the 0-30% signal range, and 12 of 13 CRAs exhibit average signal ratios below 60%, indicating substantial noise in automated review feedback. These findings suggest that CRAs without human oversight often generate low-signal feedback associated with higher abandonment. For practitioners, our results indicate that CRAs should augment rather than replace human reviewers and that human involvement remains critical for effective and actionable code review.

• The paper shows that CRA-only reviews yield a 45.20% merge rate compared to 68.37% for human reviews, challenging industry claims.
• The paper quantifies review feedback quality using a two-tier keyword framework, revealing over 60% of CRA-only PRs have low actionable content.
• The paper advocates for mixed-review workflows, where human oversight improves merge outcomes and reduces PR abandonment rates.

Empirical Assessment of Code Review Agents in Pull Requests

Introduction

Automated code review agents (CRAs) have become integral to contemporary software development workflows, especially with the proliferation of autonomous coding agents capable of generating high volumes of pull requests (PRs). Industry narratives suggest that CRAs can efficiently manage the majority of PRs without human input, purportedly achieving up to 80% agent-only handling rates. However, this claim has lacked rigorous empirical substantiation regarding its effect on merge rates and code quality within collaborative, open-source environments. The paper "From Industry Claims to Empirical Reality: An Empirical Study of Code Review Agents in Pull Requests" (2604.03196) delivers a comprehensive quantitative and qualitative evaluation of CRAs, centering on their impact on PR merge outcomes and the actionable content of their feedback.

Data Set and Methodological Overview

The study is grounded in the AIDev dataset, comprising 19,450 PRs and focusing on a filtered set of 3,109 unique PRs within open-source GitHub projects that were reviewed either by humans, CRAs, or a mix. The PR analysis is differentiable by reviewer type—CRA-only, human-only, and mixed—and further stratified by review states (Commented, Approved, Changes Requested, Dismissed). The primary methodological innovations are:

• Operationalization of reviewer composition and outcome using structured merge/enrichment of GitHub metadata.
• Quantification of reviewer quality through a two-tier keyword framework (identifying critical and important signals) for signal-to-noise ratio analysis in review comments.
• Statistical validation using chi-squared tests to assess the association between reviewer type and PR outcomes.

Key Findings: Merge Rates and Reviewer Effectiveness

The analysis reveals a pronounced disparity between agent-only and human-involved review workflows. Within the 'Commented' review state—the only condition permitting CRA-exclusive reviews—CRA-only reviewed PRs display a 45.20% merge rate, which is 23.17 percentage points lower than the 68.37% merge rate for human-only reviewed PRs. Additionally, CRA-only reviews suffer a 34.88% abandonment rate, compared to 21.60% for human-only, confirming a statistically significant association ($\chi^2 = 83.0319$, $p < 0.001$) between reviewer composition and PR fate.

This pattern is consistent across reviewer mixtures: as human participation increases in the review, merge rates rise and abandonment rates decrease, indicating a clear additive effect of human oversight on positive review outcomes. The findings directly challenge prevailing industry claims regarding the sufficiency of CRAs operating autonomously.

Signal-to-Noise Ratio in CRA-Only Reviews

A granular analysis of 98 closed CRA-only PRs elucidates the underlying cause for high abandonment rates: a predominance of noisy, non-actionable review feedback. Employing a keyword-based classification to compute signal-to-noise ratio, the distribution is heavily concentrated in the lowest signal range.

Figure 1: Signal-to-noise ratio distribution across closed CRA-only PRs, illustrating the prevalence of low-actionable-content feedback in agent-only reviews.

60.2% of closed CRA-only PRs exhibit a signal ratio between 0–30%, and 92.31% of individual CRAs analyzed achieve average signal ratios below 60%. Only isolated cases, such as specialized agents focused on narrow domains, demonstrate high precision in actionable comment generation. CRAs such as Copilot and github-advanced-security[bot]—despite high usage—return signal ratios of just 19.79% and 27.62%, respectively, underlining the systemic nature of noise generation.

Comment volume does not correlate with feedback quality; high-signal comments are both concise and sparse, while noisy feedback tends to be verbose but unhelpful. This high noise floor in CRA-only reviews directly burdens developers, increases cognitive effort, and is statistically associated with an uptick in PR abandonment.

Theoretical and Practical Implications

This study empirically falsifies the notion that current CRAs, in their default autonomous configuration, can supplant human reviewers for end-to-end PR approval in open-source workflows. Human review not only improves the merge rate but also ensures that code review discussions yield actionable, context-aware feedback. The observed correlation between low CRA signal-to-noise ratios and PR abandonment rates introduces a practical imperative: CRAs must augment, rather than replace, human oversight.

On a theoretical level, these findings prompt a re-evaluation of trust, context modeling, and reviewer accountability mechanisms in agentic software collaboration. Automated reviewer frameworks must evolve to incorporate contextual understanding, minimize noisy outputs, and develop strategies for adaptive human-in-the-loop escalation based on signal content and outcome prediction models.

Practically, the paper advocates the constraining of CRAs to tightly scoped, precision-gated tasks—such as semantic security checks or style enforcement—where their feedback achieves high actionable density. Human reviewers remain essential for architectural reasoning, nuanced bug detection, and integration of contextual project knowledge. Mixed-review workflows, in which CRAs serve as first-pass filters with required human follow-up, present a scalable compromise for large-scale projects.

Future Directions and Open Challenges

Building on the demonstrated limitations of agentic reviewers, key directions for future inquiry include:

• Rigorous development of new metrics to evaluate the actionable quality of automated review comments beyond binary merge outcomes.
• Integration of reviewer effort estimation (via comment volume, redundancy, and change impact metrics) to optimize CRA-human collaboration strategies.
• Investigation of predictive models capable of triaging which PRs require elevated human oversight, leveraging both historical agent performance and dynamic project context signals.

Progress in these areas may facilitate the engineering of empirically grounded, adaptive review systems that achieve workflow efficiency while securing review integrity.

Conclusion

"From Industry Claims to Empirical Reality: An Empirical Study of Code Review Agents in Pull Requests" (2604.03196) delivers a comprehensive, quantitative refutation of exaggerated industry claims concerning the capability of CRAs to autonomously govern code review processes. The dominant pattern in the data is that CRAs, when operating without human involvement, produce predominantly noisy feedback, resulting in significantly higher abandonment rates and inferior merge outcomes compared to human or mixed-reviewer workflows. This necessitates a paradigm in which CRAs are harnessed as precision tools within human-in-the-loop frameworks, rather than as holistic substitutes for expert code review in collaborative software development.