Understanding Dominant Themes in Reviewing Agentic AI-authored Code

Abstract: While prior work has examined the generation capabilities of Agentic AI systems, little is known about how reviewers respond to AI-authored code in practice. In this paper, we present a large-scale empirical study of code review dynamics in agent-generated PRs. Using a curated subset of the AIDev dataset, we analyze 19,450 inline review comments spanning 3,177 agent-authored PRs from real-world GitHub repositories. We first derive a taxonomy of 12 review comment themes using topic modeling combined with LLM-assisted semantic clustering and consolidation. According to this taxonomy, we then investigate whether zero-shot prompts to LLM can reliably annotate review comments. Our evaluation against human annotations shows that open-source LLM achieves reasonably high exact match (78.63%), macro F1 score (0.78), and substantial agreement with human annotators at the review comment level. At the PR level, the LLM also correctly identifies the dominant review theme with 78% Top-1 accuracy and achieves an average Jaccard similarity of 0.76, indicating strong alignment with human judgments. Applying this annotation pipeline at scale, we find that apart from functional correctness and logical changes, reviews of agent-authored PRs predominantly focus on documentation gaps, refactoring needs, styling and formatting issues, with testing and security-related concerns. These findings suggest that while AI agents can accelerate code production, there remain gaps requiring targeted human review oversight.

• The paper develops a 12-themed review taxonomy and applies an LLM annotation pipeline to analyze 19,450 review comments, revealing core implementation and documentation as the most dominant themes and finding that testing, security, and build/rejection themes lead to PRs being rejected and proposes two main improvements for agentic architecture.
• 11 thematic categories cover a comprehensive identification of development topics, including new AI-centric categories, to create a complete repository of agentic AI workflows review themes.
• The findings suggest that improved documentation and styling can improve acceptance rates, while addressing security and building issues can decrease rejection rates.

Motivation and research questions

Autonomous coding agents now produce pull requests (PRs) end-to-end, yet the review side of this workflow remains poorly characterized. Prior work reports that roughly 70% of agent-authored PRs experience longer review times, go unreviewed, or are rejected, making review a bottleneck for human-AI collaboration (Hassan et al., 7 Sep 2025). This paper addresses that gap with a large-scale empirical study of inline review comments on agent-authored PRs, guided by three questions: (RQ1) how accurately can LLMs annotate review comment themes; (RQ2) which themes dominate reviews of agentic PRs; and (RQ3) which themes distinguish accepted from rejected PRs.

Dataset

The study uses version 2 of the curated AIDev subset (Li et al., 20 Jul 2025), restricted to repositories with more than 100 GitHub stars. After joining the pull_request, pr_reviews, and pr_review_comments tables, the analysis corpus comprises 19,450 inline review comments across 3,177 agent-authored PRs, produced by agents including OpenAI Codex, Devin, GitHub Copilot, Cursor, and Claude Code.

Deriving a 12-category review theme taxonomy

Because existing taxonomies target commit messages or human-centric reviews, the authors derive a review-specific taxonomy by concatenating each PR's review comments into documents and applying BERTopic (Grootendorst, 2022). The 49 initial clusters were consolidated—using GPT-5-assisted semantic grouping while preserving distinctive keywords—into 42 fine-grained topics and finally 12 thematic categories aligned with conventional commit specification tags: security (secu), testing (test), styling (style), documentation (docs), dependency management (chore), build/configuration (build), CI/CD (ci), core features (feat), refactoring (refactor), command-line tools (cmd), reverts/rejections (undo), and performance (perf). Two design choices stand out: fixes are not given their own category since they span multiple themes, and a new undo category captures the abundant revert/rollback requests specific to agentic workflows.

RQ1: LLM annotation reliability

The authors evaluate Gemma 3:12B (selected after experiments with Llama 3.2:3B, Llama 3.1:8B, and DeepSeek-R1:14B; proprietary models were excluded for cost and reproducibility) against human annotations on 571 review comments from 100 randomly sampled PRs.

Metric	Comment level	PR level
Exact match / Top-1 accuracy	78.63%	78%
Macro precision	0.8691	0.9236
Macro recall	0.7308	0.8730
Macro F1	0.7756	0.8819
Cohen's $\kappa$ / Jaccard similarity	0.7348 ($\kappa$)	0.8142 (Jaccard)

Substantial agreement at the comment level and high set overlap at the PR level indicate that an open-weight, zero-shot LLM is a viable proxy annotator at scale. The caveat is that these figures rest on a small validation set annotated primarily by one author, so inter-rater reliability is not established.

RQ2: Dominant themes

Applying the pipeline to 19,007 validly annotated comments across 3,162 PRs yields a clear hierarchy:

• Core implementation/features (feat) dominates at both levels: 38.5% of comments and 46.5% of PR-dominant themes.
• Refactoring (14% comments, 10.4% PRs) and documentation (11.4%, 10.5%) follow.
• Styling/formatting (10.3%, 8.5%) and reverts/undo (10%, 8.7%) form the next tier.
• Testing consistently ranks fifth at over 5%; security accounts for 2.9% of comments and 3.8% of PR-level themes.
• CI/CD is rare (0.6%), and cmd never becomes a dominant PR theme, suggesting it could be merged into build or ci.

Beyond functional correctness, reviewer attention concentrates on documentation gaps, refactoring needs, formatting, reverts, and testing/security concerns—evidence that agents generate substantial non-functional "noise" requiring targeted human oversight.

RQ3: Accepted versus rejected PRs

Rejected PRs are operationalized as those with a valid closed_at but null merged_at (483 PRs, 2,558 comments); accepted PRs have both timestamps (2,035 PRs, 12,191 comments). Chi-square tests reveal statistically significant contrasts:

• Documentation is markedly more prevalent in accepted PRs (12.97% vs 10.05% of comments; 12.48% vs 6.42% at PR level; $p < 0.001$), as is styling ($p < 0.05$). These appear to be constructive hurdles reviewers will iterate on when core logic is sound.
• Testing issues predict rejection (6.53% vs 5.44%; $p < 0.05$).
• At the PR level, security concerns (5.59% vs 3.05%; $p < 0.01$) and build/configuration problems (3.11% vs 1.57%; $p < 0.05$) are strong rejection drivers.
• Undo comments trend higher in rejected PRs at both levels but fall just short of significance ($p \approx 0.057$), hinting that unnecessary agent-initiated reverts contribute to rejection.

The practical implication is asymmetric: documentation and style feedback accompanies merge success, whereas testing, security, and build defects block it—suggesting agentic architectures need stronger internal validation loops specifically for these categories.

Limitations

The authors acknowledge several constraints. LLM labeling may misread nuanced or project-specific terminology, biasing theme distributions. Findings derive solely from AIDev's popular open-source repositories and may not generalize to enterprise settings. Validation relied on a single primary annotator over a small sample, limiting inter-rater reliability claims. Rejection defined by missing merge timestamps may conflate technical failure with repository archival or maintenance closures. Finally, uncorrected chi-square tests leave borderline results (notably undo) vulnerable to Type I error.

Conclusion

This study provides the first large-scale characterization of what human reviewers actually say about agent-authored code, combining BERTopic-driven taxonomy construction with a validated zero-shot LLM annotation pipeline achieving roughly 78% agreement with humans at both comment and PR granularity. Its central empirical finding—that documentation and styling comments accompany acceptance while testing, security, and build failures drive rejection—offers concrete targets for improving agent self-correction and for building fine-tuning datasets oriented toward automated review and repair. Open questions include whether the taxonomy and rejection patterns hold under multi-annotator validation, larger samples, and non-open-source development contexts.