Is Agentic Code Review Helpful? Mining Developers' Feedback to CodeRabbit Reviews in the Wild

Abstract: Agentic code review, where autonomous agents provide code review comments on pull requests, is increasingly integrated into development workflows, yet there is limited empirical evidence on how developers respond to such comments in practice. In this paper, we present an empirical study of agentic code reviews using CodeRabbit as a case study. Through an empirical study of 31,073 pairs of code reviews and developer feedback from 10,191 pull requests across 239 GitHub repositories, our results show that agentic reviews receive mixed reception: 36.4% were accepted and 7.3% triggered discussion, while 56.3% were rejected. Rejections were primarily associated with invalid suggestions that were false positives, redundant, or out of scope, as well as misalignment with developer intent and coding practices. We further found that agentic reviews tend to focus more on functional concerns than evolvability-related comments, yet they were more likely to be invalid. To improve effectiveness in review practices, we explored various LLM-based approaches for predicting review rejection. We found that lightweight learning-based methods achieve up to 76% F1 score, suggesting learnable patterns exist between code reviews and their corresponding feedback. Our results highlight the current state of CodeRabbit's agentic code reviews, showing opportunity gaps for improvement, as well as shortcomings hindering its effectiveness.

• The paper empirically evaluates CodeRabbit’s agentic code review using 31,073 review-comment pairs, revealing a 36.4% acceptance rate and a 12.5% improvement over time.
• The paper employs a combination of stratified manual annotation and GPT-5.1-driven labeling, achieving high inter-annotator agreement and up to 76% F1 score with lightweight LLM-based rejection prediction.
• The paper highlights practical implications by demonstrating that automated quality gating can pre-screen non-helpful reviews while addressing challenges in aligning with evolving developer intent.

Empirical Evaluation of Agentic Code Review Helpfulness: Insights from CodeRabbit in Practice

Introduction and Motivation

Agentic code review, wherein autonomous agents powered by LLMs evaluate code changes and generate inline review comments, constitutes an emerging paradigm in software engineering workflows. The operational promise of systems like CodeRabbit lies in accelerated feedback cycles, exhaustive coverage, and potential reduction in human reviewer fatigue. Despite rapid uptake in industry, empirical data on actual developer interactions with—and responses to—agentic reviews remains limited. This study delivers a large-scale, systematic analysis of CodeRabbit usage on GitHub, quantifying not only acceptance and rejection rates but also elucidating the qualitative dynamics underlying developer feedback.

CodeRabbit System Architecture

CodeRabbit adopts a context-enriched agentic architecture, incorporating project-level analysis, integration of static analyzers (over 25 tools), and bidirectional knowledge flow via a repository-level knowledge base.

Figure 1: Architecture diagram of CodeRabbit, illustrating multi-source context aggregation and feedback loops between autonomous agent, diagnostics, developer feedback, and external knowledge.

The context enrichment pipeline includes AST- and path-based config ingestion, static analysis output aggregation, and cross-repository learning. The system differentiates between agentic and user-generated learnings, scaffolding historical developer feedback for continuous refinement. Real-time external querying further underpins the agent’s recommendations, though the architecture still faces limitations in resolving scope and developer intent.

Dataset and Methodology

The analysis targets 31,073 pairs of CodeRabbit review comments and subsequent developer feedback, mining 10,191 pull requests across 239 open-source projects spanning the ten most prevalent programming languages on GitHub. Feedback categorization leveraged a combination of stratified manual annotation (10% of data) and subsequent GPT-5.1-driven annotation, yielding substantial inter-annotator agreement. Developer responses were clustered into acceptance, rejection, and discussion. Rejection rationales were further subdivided into intentional design trade-offs, developer preference, false positives, redundancy, and out-of-scope suggestions.

Developer Feedback Patterns and Temporal Trends

Accepted suggestions comprised 36.4% of agentic reviews, with 7.3% serving as catalysts for inter-developer discussion. Notably, 56.3% of reviews were rejected.

Figure 2: Example of a real-world developer response to a CodeRabbit agentic review, illustrating explicit rejection based on upstream implementation context.

Rejection rationales split into misalignment with coding practices (42%), encompassing intended design trade-offs and personal preferences, versus invalid suggestions (58%), dominated by false positives (43.3%), redundancy, and out-of-scope critiques. Temporal analysis using the Mann-Kendall trend test demonstrates improvement: acceptance rates rose by 12.5% over the study period, with a concurrent decline in invalid suggestions, though rejections due to misalignment with practices rose, highlighting growing technical acuity but limited context personalization.

Concern Type Distribution and Response Correlates

Agentic reviews exhibited a marked emphasis on functionality: 75.9% targeted immediate code defects (e.g., missing functionality, logical/process errors), while 24.1% focused on evolvability aspects such as solution approach, documentation, and code organization.

Figure 3: Distribution of agentic review comment types by CodeRabbit, highlighting the predominance of functional issues versus evolvability-related recommendations.

CodeRabbit increasingly prioritized detection of functional defects over time (+17.5% proportional shift). Functional reviews, despite their prevalence, were rejected at similar rates (56.2%) to evolvability reviews (56.7%), with invalidity dominating rejections in the former and misalignment in the latter. Subcategory analysis reveals functional-defect reviews frequently failed due to misinterpretation of code semantics or external system context. Evolvability feedback, even when technically valid, was often dismissed due to evolving or implicit project conventions and personal preference.

Predicting Rejection: Lightweight Model Approaches

The study evaluated three LLM-based strategies for early prediction of developer rejection of agentic reviews: direct zero-shot prompting, full-parameter fine-tuning (ModernBERT), and low-rank adaptation (LoRA) on compact model variants (≤ 4B parameters). Direct prompting with frontier models (e.g., GPT-5.1, Llama-3-70B) yielded low F1 scores—maxing at 64%. Training-based approaches achieved significantly higher effectiveness: LoRA on Qwen3-4B reached a peak F1 of 76%, with ModernBERT and other small LLMs consistently surpassing 66% for this task. These results demonstrate the feasibility of practical, inference-time rejection gating via lightweight quality filters that mine learnable patterns from prior developer-agent interactions.

Practical and Theoretical Implications

Shortcomings: High false-positive rates for functional critiques, redundancy, and scope missteps reflect incomplete agentic comprehension of system-wide architecture, concurrent development activities, and implicit developer intent. Even as technical validity increases, agentic reviews' lack of contextual personalization results in persistent friction with evolving engineering norms.

Opportunity gaps: Underutilized but highly accepted review types—such as naming conventions, readability, and test coverage—suggest missed practical value in maintenance-oriented feedback, especially when personalization and context-awareness are enhanced.

Human-AI modality: The divergence between typical human review patterns (rare functional defect reports, majority evolvability recommendations [7202946]) and CodeRabbit’s functional emphasis presents a trade-off: increased defect-finding complements human reviews but at significant triage cost due to false positives.

Gating for adoption: The strong discriminatory performance of compact, learning-based quality gates has important infrastructure implications. They can filter non-helpful reviews before developer exposure and adapt dynamically to repository-specific feedback patterns, potentially boosting long-term trust and perceived reliability of agentic code review platforms.

Limitations

The study’s external validity is constrained by exclusive focus on CodeRabbit and open-source repositories with permissive licensing. Manual and LLM-driven annotations, despite high inter-annotator agreement, introduce some label noise. Trends may differ in proprietary enterprise environments with distinct coding standards and organizational conventions.

Conclusion

Real-world empirical evaluation of agentic code review with CodeRabbit reveals a nuanced deployment landscape: while over one third of reviews are accepted and acceptance rates are rising, a majority are still ultimately rejected mainly due to invalidity or misalignment with coding practices. CodeRabbit’s technical reasoning is improving, but adaptation to fine-grained developer intent remains limited. Lightweight LLM-based gating systems leveraging feedback data deliver strong predictive performance (up to 76% F1), and represent a viable path for increasing the quality and developer acceptance of automated review comments. Further research should focus on enhanced architectural modeling, richer integration of process context, and more personalized adaptive feedback to maximize the net benefit of agentic code review systems.