---
title: Code Review Agents in AI-Driven Reviews
url: https://www.emergentmind.com/topics/code-review-agents-cras
type: topic
---

# Code Review Agents in AI-Driven Reviews

Code Review Agents (CRAs) are automated or AI-based reviewers that inspect code changes, pull requests, or repository state and produce review feedback such as inline comments, vulnerability reports, suggested patches, or structured review artifacts. In recent literature, the term covers both bot-based pull-request reviewers and more explicitly agentic systems that can understand code context, plan review actions, interact with development environments, and maintain memory across steps. The resulting research area spans automated review comment generation, secure pre-commit review, repository-aware reasoning, workflow orchestration across the pull-request lifecycle, and human-supervised review pipelines [2603.15911, 2601.19138, 2605.17548].

## 1. Definition and historical trajectory

Code review has long been treated as a primary quality gate in software development; one recent position paper explicitly traces this lineage to Fagan’s formalisation of code inspection in 1976 and argues that review has historically served four overlapping goals: defect detection, style and standards enforcement, knowledge transfer, and team awareness [2606.13175]. CRA research inherits those goals but re-specifies the reviewer as a software agent rather than a purely human participant.

The scope of the term varies by paper. In large-scale GitHub studies, CRAs are AI review bots such as GitHub Copilot, CodeRabbit, Devin, Gemini Code Assist, and Claude Code that participate in inline review conversations much like human reviewers do [2603.15911]. In secure review work, a CRA is an autonomous code review agent operating in a diff-centric CLI environment with repository navigation, explicit tool use, and security-focused memory [2601.19138]. Vision papers further broaden the notion into an end-to-end, AI-powered, multi-agent, human-in-the-loop workflow spanning PR Creation, PR Augmentation, Reviewer Selection, AI-Assisted Code Review, and PR Retrospective [2605.17548].

The historical transition from review assistance to more agentic review is visible in the literature. RAID, a refactoring-aware diff tool, did not itself act as an autonomous reviewer, but it augmented GitHub diffs with refactoring metadata and reduced the median number of lines to be reviewed from 14.5 to 2 for move refactorings and from 113 to 55 for extractions, thereby lowering cognitive effort in refactoring-heavy review [2103.11453]. Later work such as CodeAgent made the reviewers themselves multi-agent LLM systems, explicitly modelling review as collaborative interaction rather than one-shot generation [2402.02172].

## 2. Architectural patterns

A recurring CRA design pattern is role decomposition. CodeAgent organises review into four phases—Basic Info Sync, Code Review, Code Alignment, and Document—and assigns six characters, namely User, CEO, CPO, CTO, Reviewer, and Coder. Its distinctive mechanism is QA-Checker, a supervisory agent that iteratively repairs prompts when an answer is not sufficiently aligned with the original review question. On vulnerability analysis, the full system reported 449 confirmed vulnerabilities with a Rate\(_{cr}\) of 92.96%, whereas CodeAgent\(_{w/o}\) reported 413 confirmed vulnerabilities with a Rate\(_{cr}\) of 73.23%, indicating that conversational supervision materially affected precision [2402.02172].

A second pattern is issue-oriented specialisation. RevAgent decomposes review comment generation into a Generation Stage, where five category-specific commentator agents draft candidate comments; a Discrimination Stage, where a critic agent selects the most appropriate issue-comment pair; and a Training Stage, where commentators and critic are fine-tuned on curated category-specific data. Its taxonomy includes Refactoring, Bugfix, Testing, Logging, and Documentation, while Others is treated as noise and excluded from training. Averaged across four LLM backbones, RevAgent improved over the best baseline by 12.90\% on BLEU, 10.87\% on ROUGE-L, 6.32\% on METEOR, and 8.57\% on SBERT, and human evaluation rated it highest on readability, accuracy, and category-matching [2511.00517].

A third pattern is repository-aware, tool-using agency. AgenticSCR formalises secure review as
$$
\text{AgenticSCR} = \langle \mathcal{E}, \mathcal{A}, \mathcal{G}, \mathcal{M}, \mathcal{T}, \Pi \rangle,
$$
where \(\mathcal{E}\) is the CLI/repository environment, \(\mathcal{A}=\{a_d,a_v\}\) are two specialist subagents, \(\mathcal{G}\) are the review goals, \(\mathcal{M}\) is the memory system, \(\mathcal{T}\) is the toolset, and \(\Pi\) is the coordination policy. Its workflow is explicitly detector \(\rightarrow\) validator: the detector localises suspicious lines, generates a security review comment, and predicts a CWE-based type, while the validator checks whether the finding is a real security issue and filters false positives. The design is grounded by security-focused semantic memory containing SAST/CodeQL rules for the detector and a CWE tree derived from the CWE-1000 taxonomy for the validator [2601.19138].

More programmatic visions extend CRA architecture from a single review event to the full pull-request lifecycle. The five-stage framework proposed in “Rethinking Code Review in the Age of AI” specifies a PR Creation Agent, PR Augmentation Agent, Reviewer Suggestion Tool, PR Review Agent, and retrospective summarisation machinery, while preserving humans at explicit quality gates such as PR author verification, reviewer assignment, approval of automated findings, and final merge authority [2605.17548]. This lifecycle view shifts the unit of design from “comment generation” to “review process orchestration.”

## 3. Benchmarks and evaluation logic

CRA evaluation has diversified because review is not well captured by single-reference text similarity. CRScore explicitly frames code review as a one-to-many problem and replaces reference matching with a reference-free metric grounded in claims, implications, and code smells extracted from the diff [2409.19801]. CR-Bench instead asks whether a review comment is a BUG_HIT, a VALID_SUGGESTION, or NOISE and makes signal-to-noise ratio a first-class metric [2603.11078]. c-CRAB converts human review intent into executable tests, so the final oracle is behavioural rather than textual [2603.23448]. SCRBench targets repository-aware, line-level immature vulnerabilities at the pre-commit stage [2601.19138].

| Benchmark or metric | Evaluation unit | Key reported property |
|---|---|---|
| CRScore [2409.19801] | Review comment quality | Reference-free; Spearman correlation 0.5431 with human relevance |
| CR-Bench / CR-Evaluator [2603.11078] | Review comments on real PRs | BUG_HIT / VALID_SUGGESTION / NOISE; SNR |
| c-CRAB [2603.23448] | PR review plus review-guided revision | Test-based; 184 PRs and 234 validated comments/tests |
| SCRBench [2601.19138] | Pre-commit secure review | 144 code changes, 107 CVEs, 33 CWE types |

CRScore measures three dimensions. Conciseness is the fraction of review sentences aligned above threshold with some pseudo-reference, comprehensiveness is the fraction of pseudo-references covered by the review, and relevance is the harmonic mean of the two. Using pseudo-references grounded in LLM-generated claims and static-analysis smells, CRScore Rel reached a Spearman correlation of 0.5431 with human relevance Likert scores, substantially above BLEU, ROUGE-L, chrF, BERTScore, and other reference-based metrics [2409.19801].

CR-Bench and CR-Evaluator make a different point: defect-detection recall alone is insufficient because false positives directly affect developer productivity. They therefore report Recall, Precision, F1, Usefulness Rate, and
$$
\text{SNR} = \frac{\text{Total Bug Hits} + \text{Total Valid Suggestions}}{\text{Total Noise Count}}.
$$
This framing exposes a hidden frontier between issue resolution and spurious findings [2603.11078].

c-CRAB shifts evaluation to executable consequences. For each instance \(i\), with validated tests \(T_i\) and a patch \(\hat{P}_i\) produced after a coding agent is guided by a CRA’s reviews, the score is
$$
s_i=\frac{|\{t \in T_i \mid t(\hat{P}_i)=PASS\}|}{|T_i|}.
$$
The benchmark was curated from 671 PRs and 1,313 comments, then filtered to 184 PRs and 234 validated comments/tests spanning 67 repositories [2603.23448].

## 4. Empirical behaviour in pull-request workflows

Large-scale observational evidence consistently shows that present-day CRAs behave differently from human reviewers. In 278,790 inline code review conversations across 300 open-source GitHub projects, AI-agent reviews were narrower and more verbose, with over 95% of AI comments concentrated in Code Improvement and Defect Detection. Human reviewers supplied broader feedback, including Understanding, Testing, Knowledge Transfer, and Social comments. Understanding feedback alone accounted for 31% of comments when humans reviewed human-written code and 17% when humans reviewed agent-generated code, and human reviewers exchanged 11.8% more rounds when reviewing AI-generated code than human-written code. AI suggestions were adopted at 16.6%, compared with 56.5% for human suggestions, and when adopted they produced larger increases in code complexity and code size than human suggestions [2603.15911].

Reviewer composition also correlates with outcome quality. In the AIDev-based study of CRA-only versus human-only reviewing, the main comparison was restricted to Commented PRs because that is where CRA-only reviewing appears. Among those PRs, CRA-only review achieved a 45.20% merge rate, whereas human-only review achieved 68.37%; CRA-only PRs also had a 34.88% closed rate versus 21.60% for human-only PRs. The association between reviewer type and PR outcome was statistically significant with \(\chi^2 = 83.0319\), \(df = 8\), \(p < 0.001\). In a signal analysis of 98 closed CRA-only PRs, 59 PRs (60.2%) fell into the 0–30% signal range, and 12 of 13 CRAs had average signal ratios below 60% [2604.03196].

Benchmark studies reveal the same precision–noise tension under controlled conditions. On CR-Bench-verified, a single-shot GPT-5.2 reviewer reached Recall 27.01%, Precision 3.56%, F1 6.30%, Usefulness 83.63%, and SNR 5.11, while Reflexion increased Recall to 32.76% but lowered Usefulness to 47.72% and SNR to 1.95. GPT-5-mini under Reflexion reached Recall 27.59% but SNR 0.91, illustrating that aggressive bug-finding can degrade signal integrity [2603.11078]. c-CRAB yields a related conclusion at the review-to-repair level: Claude Code achieved 32.1% overall, Devin 24.8%, PR-Agent 23.1%, and Codex 20.1%, while the union across all four tools solved 97 out of 234 tests, about 41.5%, leaving a substantial gap between current CRA output and human review intent as encoded by executable tests [2603.23448].

## 5. Security-critical and domain-specific review agents

Security review is one of the most technically ambitious CRA subfields. AgenticSCR targets immature vulnerabilities in pre-commit review, where changes are small, context is incomplete, and latency matters. On SCRBench, AgenticSCR generated 211 comments and achieved 17.5% overall correctness (LRT), whereas a static LLM-based secure reviewer generated 652 comments but only 6.9% LRT. CodeQL achieved 0.6% LRT, Semgrep 3.7%, and Snyk 0.9%. The paper’s headline claim is that AgenticSCR achieves at least 153% relatively higher percentage of correct code review comments than the static LLM baseline. Performance varied by high-level vulnerability category: Injection 23.4%, Authorization 21.4%, Information 6.2%, Resource 16.7%, and Control 0.0%. The ablation study showed a Base agent at 7.3% LRT, Base + SAST Rules at 13.0%, and Base + SAST Rules + CWE at 17.5%, isolating the contribution of structured security memory [2601.19138].

Compiler optimisation review shows a different but equally specialised CRA design. Archer, described as the first automated agentic code review tool for compiler optimisations, constrains review from both ends: obligations guide analysis and a deterministic validation guard admits only findings backed by executable evidence. Evaluated on 70 open PRs and 328 closed PRs in LLVM from the last two months, Archer found that 21% of open PRs and 11% of closed PRs were buggy. Across the real-world dataset it reported 51 semantic bugs, including 34 miscompilations and 17 crashes. On the regression dataset, direct LLM review found no bugs, mini-SWE-agent found 1 bug, and Optimuzz found 3 bugs, all also covered by Archer, underscoring the value of domain-specific obligations and proof-or-execution-backed validation [2607.01808].

At the same time, secure CRA systems themselves create new attack surfaces. CoTDeceptor targets CoT-enhanced LLM detectors and vulnerability-review agents by exploiting their reasoning chains and semantic abstraction processes. Its headline result is evasion of 14 out of 15 vulnerability categories, compared to only 2 bypassed by prior methods. The paper argues that reasoning traces can leak decision boundaries, turning interpretability into a double-edged property for security review pipelines embedded in CI/CD or repository auditing [2512.21250].

## 6. Governance, reviewability, and competing visions

Recent work increasingly treats CRA effectiveness as a governance problem rather than only a model-capability problem. “Software Delegation Contracts” defines delegated coding work as a tuple \((T, A, W, C)\) consisting of Task, Authority, Returned work package, and Acceptance context. In a controlled pilot study with 64 agent executions and 192 reviews, explicit contracts did not improve objective task outcomes—every run passed hidden acceptance checks, with zero scope violations—but they improved reviewability: evidence sufficiency improved in 22 of 30 paired comparisons and worsened in none, with a mean gain of +0.83 on a 5-point scale (\(p < 0.0001\), Cliff’s delta = 0.66). The overhead was +13% agent tokens and +38% wall-clock time, indicating that review-facing evidence can be purchased at measurable cost [2606.17099].

Governance also appears in triage models for agent-authored PRs. “Early-Stage Prediction of Review Effort in AI-Generated Pull Requests” analysed 33,707 agent-authored PRs from 2,807 repositories and identified a two-regime behavioural pattern: 28.3% of all PRs were instant merges, defined as less than 1 minute from creation to merge, while the remainder entered iterative review cycles with phenomena such as agentic ghosting. The proposed Circuit Breaker triage model used only static structural features at creation time; a LightGBM model reached AUC 0.9571 with 95% CI [0.955, 0.962] on a temporal split, and at a 20% review budget intercepted 69% of total review effort. This reframes CRA deployment as review-effort governance rather than purely automated judgement [2601.00753].

The field is also divided over whether CRAs should augment or replace human reviewers. “The End of Code Review: Coding Agents Supersede Human Inspection” argues that coding agents have crossed a threshold at which traditional human code review is no longer a necessary component of a software quality pipeline, and proposes structured agent sign-off in place of a human approval checkbox for routine changes [2606.13175]. Empirical work points in the opposite direction: human–AI collaboration studies conclude that human oversight remains critical for contextual feedback, validation of suggestion quality, and final confirmatory judgement, while PR-outcome studies conclude that CRAs should augment rather than replace human reviewers because standalone CRA feedback is often low-signal and associated with higher abandonment [2603.15911, 2604.03196]. The lifecycle framework in “Rethinking Code Review in the Age of AI” therefore retains humans at key decision points throughout PR Creation, PR Augmentation, Reviewer Selection, AI-Assisted Code Review, and PR Retrospective, treating HITL quality gates as safeguards against hallucination, context loss, cascading errors, bias, privacy risk, and automation bias [2605.17548].

Taken together, the literature presents CRAs not as a single tool category but as a family of review systems with different operational assumptions: some act as inline conversational reviewers, some as specialist comment generators, some as secure pre-commit agents, and some as full pull-request workflow orchestrators. The strongest empirical findings so far are mixed. CRAs can scale defect screening, exploit repository context, use tools, and integrate structured knowledge; yet they also remain vulnerable to noise, mislocalisation, context loss, low adoption, and adversarial manipulation. Current evidence therefore supports a technically ambitious but operationally constrained view: CRA research is advancing rapidly, but robust deployment still depends on benchmark design, evidence-grounded evaluation, domain-specific scaffolding, and carefully placed human oversight.

Source: https://www.emergentmind.com/topics/code-review-agents-cras