Automated Code Review

Updated 28 July 2025

Automated Code Review is a technique that leverages machine learning, deep learning, and information retrieval to evaluate source code for quality, maintainability, and compliance.
ACR systems employ transformer models, Bi-LSTM architectures, and graph-based approaches to generate review comments, predict defects, and recommend suitable reviewers.
The approach enhances efficiency by rapidly identifying code issues and supporting human-in-the-loop processes, balancing automation with expert oversight in industrial pipelines.

Automated Code Review (ACR) encompasses the set of methods, systems, and workflows aimed at partially or fully automating the assessment of source code for correctness, quality, maintainability, and adherence to project or organizational standards. Leveraging advances in information retrieval, machine learning, and deep learning—especially LLMs—ACR spans techniques for recommending reviewers, assessing code changes, generating review comments, predicting or generating revised code, and evaluating review quality. The field sits at the intersection of software engineering and machine learning, and has evolved rapidly, reflecting both the scale of modern software projects and the increasing maturity of automated intelligence for code and text.

1. Major Tasks in Automated Code Review

Automated Code Review is a multi-faceted area encompassing a wide spectrum of tasks, which have been systematically categorized in recent literature (Tufano et al., 12 Mar 2025). Typical automated code review tasks include:

Review Comment Generation and Code Refinement: Generation-based systems automatically produce natural language feedback or code suggestions, often treating ACR as code-to-text or code-to-code translation or generation (Tufano et al., 2021, Zhou et al., 2023, Li et al., 2022).
Review Quality Assessment and Comment Classification: Techniques classify review feedback by usefulness, relevance, or conversational tone, addressing the need for more meaningful review analytics and prioritization (Turzo et al., 2023, Naik et al., 29 Sep 2024, Jiang et al., 9 Jan 2025).
Reviewer Recommendation and Assignment: Algorithms predict the most suitable reviewers for a code change, using factors such as code ownership, change history, and past review outcomes (Tufano et al., 12 Mar 2025).
Defect Detection and Code Quality Prediction: Classifiers and deep models flag code changes for potential defects or quality violations, sometimes leveraging community knowledge from online platforms or static/dynamic analysis results (Sodhi et al., 2018, Lu et al., 23 May 2025, Tang et al., 3 Feb 2024).
Pull Request Acceptance Prediction: Predicting merge likelihood or review-induced change characteristics based on code and review meta-data.
Review Analytics and Sentiment Analysis: Automatic mining of trends, bottlenecks, toxic comments, or sentiment to guide process improvements.
Automation of Code Modification: Models generate revised code prior to review or in response to a human comment, bridging reviewer intent and actual code modifications (Tufano et al., 2021, Lin et al., 20 Mar 2025).

A systematic review (Tufano et al., 12 Mar 2025) reports 34 automated code review task types grouped into assessment, classification, analysis, retrieval, quality prediction, time management, and sentiment analysis.

2. Architectures and Methodologies

Multiple technical approaches underlie ACR systems, ranging from heuristic and information retrieval methods to deep learning–based models. Notable architectural patterns and methods include:

Information Retrieval and Document Fingerprinting: Early solutions (for example, leveraging Stack Overflow content (Sodhi et al., 2018)) used document fingerprinting (Winnowing algorithm) to efficiently match code fragments in review with similar community-sourced code, aggregating defectiveness signals via hash-based similarity.
Multi-Level Embedding and Neural Models: Systems like CORE (Siow et al., 2019) combine word-level and character-level embeddings fed to Bi-LSTMs with attention mechanisms, allowing representations robust to rare tokens and domain-specific vocabulary.
Transformer Models for Code and Comments: Encoder–decoder transformer architectures (e.g., T5, CodeT5) dominate generative and recommendation-based pipelines. These models are optimized using tasks specific to code review workflows, such as multi-task objectives for code diff tagging, denoising, and review comment generation (Li et al., 2022).
Graph-Based and Structural Models: Graph convolutional networks (GCNs) applied to simplified abstract syntax trees (ASTs) have demonstrated improved modeling of code structure and semantics, outperforming sequence-based models in approval/rejection prediction and ACR classification (Wu et al., 2022, Wu et al., 2022).
Contrastive Learning in Multi-Modal ACR: Multi-modal architectures combine code and text representations (from SimAST-GCN, RoBERTa) pre-trained via contrastive loss with minimal augmentation, supporting robust classification of code review outcomes (Wu et al., 2022).
Multi-Agent LLM Frameworks: Recent systems (e.g., CodeAgent (Tang et al., 3 Feb 2024) and practical defect-focused frameworks (Lu et al., 23 May 2025)) introduce collaborative multi-role agent architectures, employing specialized LLM roles (reviewer, meta-reviewer, validator) coordinated via chain-of-thought (CoT) reasoning and quality control modules (e.g., QA-Checker) for iterative refinement and decision tracing.
Prompt Engineering/Role-Aware Design: For efficient human-in-the-loop deployment, prompt design includes line-aware formatting of review comments, explicit association with code lines, and localization cues to optimize developer interaction and reduce cognitive burden (Lu et al., 23 May 2025).

3. Datasets, Evaluation, and Metrics

ACR research is supported by several large-scale, task-specific datasets and benchmarks:

Repository-Mined Datasets: The Apache Automatic Code Review (AACR) (Wu et al., 2022) and Multi-Modal Apache Automatic Code Review (MACR) (Wu et al., 2022) datasets contain method-level code changes, review comments, and approval labels, spanning multiple Apache projects and capturing real-world class imbalances.
CodeReview Dataset: Large-scale pull request–based collections supporting multilingual evaluation for change quality, review comment generation, and code refinement (Li et al., 2022).
Benchmarks for Generative and Semantic Assessment: GradedReviews (Jiang et al., 9 Jan 2025) enables human-aligned evaluation of generated code reviews, with explicit manual scoring for over 5,000 generated reviews and their references.
Comprehension-Oriented Probing: CodeReviewQA (Lin et al., 20 Mar 2025) assesses LLM capability beyond text generation using MCQA probes on decomposed reasoning subtasks (change type recognition, localization, and solution identification) across 900 manually curated review cases.
Evaluation Metrics: Traditional metrics include BLEU and ROUGE-L for comment/code similarity, accuracy/F1 for classification, and Mean Reciprocal Rank/Recall@k for retrieval (Li et al., 2022, Siow et al., 2019, Wu et al., 2022). However, these suffer from inherent limitations in coverage and alignment with human judgment (Jiang et al., 9 Jan 2025, Zhou et al., 2023).
- Semantic and Reference-Free Metrics: CRScore (Naik et al., 29 Sep 2024) adopts a neuro-symbolic approach combining LLM-generated claims, static analyzer output, and semantic textual similarity (STS) to compute conciseness, comprehensiveness, and relevance, aligning better with human annotation than reference-based metrics.
- Edit Progress (EP): Quantifies incremental improvement of code revision relative to the ground truth, rewarding partial success (Zhou et al., 2023).
- Comprehensive Performance Index (CPI), Key Bug Inclusion (KBI), False Alarm Rate (FAR): Purpose-built to measure defect-inclusive performance and filter overgeneration in real-world settings (Lu et al., 23 May 2025).

4. Industrial Deployment, Practical Impact, and Limitations

Industrial studies reveal the nuanced effects of integrating ACR systems into development workflows:

Positive Impacts: LLM-based review tools, such as Qodo PR Agent (Cihan et al., 24 Dec 2024), provide meaningful code quality improvements, enhanced bug detection, and adherence to best practices. High developer resolution rates for automated comments (around 70–80%) indicate relevance and acceptability. Increases in code quality have been observed through post-merge analyses correlating higher engagement with defect reduction.
Drawbacks and Challenges: Increased pull request closure times (from approximately 6 to 8.5 hours on average) have been documented (Cihan et al., 24 Dec 2024), attributed to the extra layer of review and discussion induced by automated comments. Unnecessary or irrelevant suggestions require developer triage, sometimes lowering efficiency. In controlled experiments, automated reviews increase low-severity issue detection but do not improve high-severity defect identification or reduce review time; reviewers' focus is narrowed to tool-flagged locations, which can result in missed issues elsewhere (Tufano et al., 18 Nov 2024).
Developer Perceptions: Practitioners view automated review tools as valuable secondary checks, aiding knowledge transfer, but emphasize their role as complements to, rather than replacements for, human judgment. Automated suggestions influence reviewer behavior, often reducing exploration outside the scope of flagged concerns.
Realization in Code Review Pipelines: Industry deployments leverage combinations of LLM comment generation, reviewer assignment automation, and integration with pull request dashboards. Practical pipelines emphasize language-agnostic design (abstract syntax tree–based context extraction), chain-of-thought prompting, and ergonomic presentation (line-aware comments, prioritization of actionable feedback) (Lu et al., 23 May 2025).

5. Human Factors, Cognitive Models, and Decision Support

Recent research frames code review as a cognitive and decision-making process to inform the development of AI or semi-automated support tools (Heander et al., 13 Jul 2025):

CRDM Model: Code review comprises an orientation phase (establishing context, rationale; e.g., understanding task urgency, author intent) and an analytical phase (implementation assessment, risk analysis, iterative action selection), drawing on recognition-primed decision-making—a paradigm where experience and "mental simulation" quickly guide choice of next steps.
Support and Augmentation (not Replacement): Automated systems should enhance, not displace, the interpersonal benefits of code review, such as knowledge transfer and collaborative learning. Tool design should focus on assisting orientation (context gathering, rationale summarization) and analytical navigation (highlighting code areas for scrutiny, surfacing historic review outcomes), leaving final judgment to humans.
Agentic and Integrated Decision Support: Embedding AI agents as context-aware decision support (offering "next-action" recommendations or second-opinion cues) mirrors integrated decision support systems in adjacent fields, potentially improving efficiency and reviewer satisfaction without eroding team ownership or the human dimension of review (Heander et al., 13 Jul 2025).

6. Open Challenges and Future Directions

Despite substantial advances, ACR faces multiple challenges and research frontiers:

Evaluation and Human Alignment: Existing metrics, especially BLEU and surface similarity, are insufficient for robust evaluation; there is a continued need for semantic and reference-free metrics closely following human judgment (Jiang et al., 9 Jan 2025, Naik et al., 29 Sep 2024). Benchmark construction, such as GradedReviews and CodeReviewQA, and probes for comprehension, localization, and intent mapping, are critical for model diagnosis and improvement.
Robustness and Generalizability: Many techniques struggle with transfer across diverse projects or programming languages and may overfit to training distributions (e.g., reviewer recommendation biases, over-relying on historic data) (Tufano et al., 12 Mar 2025).
Integration and Usability: Industrial deployments encounter trade-offs between code quality gains and process bottlenecks (longer PR cycles, irrelevant comment noise). Human-in-the-loop refinements, adaptive filtering, and ergonomic prompt design remain active areas for tool improvement (Cihan et al., 24 Dec 2024, Lu et al., 23 May 2025).
Scaling and Cost: Real-world use must address the computational and monetary expense of training, inference, and maintenance for large models. Recommendations include exploring parameter-efficient fine-tuning, prompt engineering, and modular language-agnostic frameworks (Tufano et al., 12 Mar 2025, Lu et al., 23 May 2025).
Preserving Human and Interpersonal Aspects: Fully automating reviews risks undercutting knowledge transfer and shared ownership. Cognitive models point toward support systems that augment orientation and analysis phases, delivering scalable assistance without sacrificing interpersonal or organizational benefits (Heander et al., 13 Jul 2025).

7. Summary Table: Representative Approaches and Metrics

Method/Task	Architecture/Metric	Core Strength
CORE (Siow et al., 2019)	Multi-level Bi-LSTM with attention	High Recall@10, robust to OOV
SimAST-GCN (Wu et al., 2022)	Bi-GRU + GCN + attention	Strong on structure, F1, AUC
CodeReviewer (Li et al., 2022)	Multi-task Transformer	Handles multilingual, multi-task
CLMN (Wu et al., 2022)	SimAST-GCN + RoBERTa, contrastive	True multi-modal, robust F1/MCC
CodeAgent (Tang et al., 3 Feb 2024)	Multi-agent LLM + QA-Checker	Strong recall/F1, collaborative
CRScore (Naik et al., 29 Sep 2024)	Neuro-symbolic, pseudo-reference	Superior alignment w/ human eval
GradedReviews (Jiang et al., 9 Jan 2025)	EmbeddingSim, LLM Eval	Semantic alignment, human-grounded
Defect-Focused (Lu et al., 23 May 2025)	Code slicing, multi-role LLM	2× KBI, language-agnostic