AACR-Bench: Automated Code Review Benchmark

• AACR-Bench is a benchmark for Automated Code Review that provides multi-level context—diff, file, and repository—for robust evaluation across ten major programming languages.
• It employs an AI-assisted, expert-verified annotation pipeline to expand defect coverage by approximately 285%, ensuring high-quality ground truth beyond raw PR comments.
• The benchmark contrasts direct prompting with agent frameworks, highlighting trade-offs in precision and recall while emphasizing model, language, and retrieval method dependencies.

Searching arXiv for the specified benchmark paper and related benchmark context. AACR-Bench is a benchmark for Automated Code Review (ACR) designed to evaluate LLMs under holistic repository-level context rather than only diff-level or file-level inputs. It was introduced to address two stated deficiencies in prior ACR benchmarks: incomplete ground truth derived from raw Pull Request (PR) comments, and limited support for multi-language, repository-level reasoning. AACR-Bench provides PR metadata, diff hunks, full-file context, and entire repository context across ten programming languages, and it uses an “AI-assisted, Expert-verified” annotation pipeline that raises defect coverage from 391 original PR comments to 1,505 final comments, corresponding to an approximately 285% increase (Zhang et al., 27 Jan 2026).

1. Benchmark motivation and problem setting

AACR-Bench is positioned against prior ACR benchmarks that, according to its definition, suffer from two critical limitations. The first is incomplete ground truth: benchmarks such as CodeReviewer, SWR-Bench, and CodeFuse-CR-Bench simply use raw PR comments as “gold” annotations. In AACR-Bench’s framing, such comments are noisy, often under-report latent defects, and therefore cannot faithfully measure a model’s ability to surface hidden issues (Zhang et al., 27 Jan 2026).

The second limitation is narrow context scope. Benchmarks with cross-file or repository-level context are described as being almost always restricted to Python, whereas benchmarks supporting multiple languages stop at file- or diff-level. AACR-Bench therefore targets a combined setting in which both linguistic diversity and repository-wide reasoning are present simultaneously. This suggests that the benchmark is not only a dataset expansion, but also an attempt to alter the unit of evaluation from localized patch inspection to mixed local-global review reasoning.

A central implication of this design is methodological: ACR performance can no longer be interpreted only as a function of model quality. In the benchmark’s own conclusions, performance depends critically on the underlying LLM, the retrieval method and its granularity, the programming language, and whether an Agent paradigm is employed (Zhang et al., 27 Jan 2026). In that sense, AACR-Bench defines evaluation as a systems problem rather than a single-model inference problem.

2. Corpus composition and context hierarchy

AACR-Bench covers 50 popular GitHub repositories, with 5 repositories per language, spanning the top 10 languages of the 2025 StackOverflow survey: JavaScript, Python, TypeScript, Java, C#, C++, C, PHP, Go, and Rust (Zhang et al., 27 Jan 2026). The benchmark contains 200 PRs and 1,505 total comments / defects.

The context supplied for each PR is explicitly stratified. AACR-Bench delivers the PR title and description, all diff hunks extracted via Git, the full-file context for each diff hunk, and the entire repo context, including cross-file dependencies and other modules. Correspondingly, comment annotations are tagged by one of three context levels:

• Diff-level: issues resolvable within the hunk
• File-level: issues requiring the entire file
• Repo-level: issues requiring cross-file context or PR metadata

This hierarchy is integral to the benchmark’s purpose. Rather than treating context as an undifferentiated token budget, AACR-Bench operationalizes context granularity as an evaluable variable. That design enables analyses of when local reasoning suffices and when repository-wide dependencies become necessary.

The benchmark’s headline corpus statistics are as follows:

Item	Value
Repositories	50
Pull Requests	200
Programming Languages	10
Total Comments / Defects	1,505
Security Vulnerabilities	53
Code Defects	709
Maintainability / Readability Issues	626
Performance Issues	117

These category counts indicate that AACR-Bench is not restricted to correctness bugs. It also includes maintainability, readability, performance, and security-oriented findings, which broadens the evaluation target from narrow defect detection to more general review competence (Zhang et al., 27 Jan 2026).

3. Annotation pipeline and defect coverage expansion

AACR-Bench uses an “AI-assisted, Expert-verified” annotation pipeline intended to uncover latent defects beyond the comments found in the original PR discussions (Zhang et al., 27 Jan 2026). The benchmark defines defect coverage gain as the relative increase from the number of original PR comments to the number of final comments. With 391 original PR comments and 1,505 final comments, the reported CoverageIncrease is approximately 285\%.

The workflow is described in six stages.

First, PR Extraction & Filtering collected 12,715 PRs from Dec 2024–Dec 2025. These were filtered by English description, $\leq 1{,}000$ changed lines, $\geq 2$ inline comments ($\geq 1$ accepted), language consistency, and semantic validity, yielding 573 PRs.

Second, Stratified Sampling reduced the pool to 200 PRs, covering 10 languages and 50 repos.

Third, Human-Augmented Review Summaries applied LLM-based deep semantic analysis of multi-turn GitHub review threads to produce 391 “Augmented Review Comments”, with 95\% manual-verified accuracy.

Fourth, Machine-Generated Reviews were produced by six LLMs—Claude-4.5-Sonnet, Qwen3-Coder-480B, GPT-5.2, Deepseek-V3.2, GLM-4.7, Gemini-3-Pro—each using two frameworks: an internal system and the Claude Code agent. The resulting comments were processed with semantic de-duplication via pairwise LLM comparison.

Fifth, Expert Verification involved 80 senior engineers in a three-round, double-blind annotation process. Rounds 1 and 2 used two independent annotations per comment, and Round 3 used a core expert team to resolve disagreements. Annotators validated correctness, categorized issue type, and labeled context scope.

Sixth, the Final Ground Truth comprised 391 human-augmented plus 1,114 LLM-augmented comments, for a total of 1,505 fine-grained review comments.

This pipeline is significant because it redefines benchmark ground truth as a curated synthesis rather than a direct copy of repository discussion artifacts. A plausible implication is that AACR-Bench evaluates models against a denser and more normalized target than benchmarks that treat raw review threads as complete supervision.

4. Evaluation protocol and formal matching criteria

AACR-Bench evaluates five LLMs under two paradigms: Direct Prompting (non-Agent) and Agent Architecture (Zhang et al., 27 Jan 2026). The non-Agent setting includes Qwen3-Coder-480B-A35B, Deepseek-V3.2, GLM-4.7, GPT-5.2, and Claude-4.5-Sonnet. The Agent setting uses Claude Code, with the above base models integrated via a tool-calling agent framework.

The benchmark defines three context granularity levels:

• Diff: only the diff hunk plus PR title and description
• File: the full file containing the hunk plus PR metadata
• Repo: the entire repository code plus PR metadata

For non-Agent runs, four retrieval conditions are defined:

1. No context as baseline
2. BM25, retrieving the top-3 code snippets by text-similarity
3. Embedding, retrieving the top-3 snippets via Qwen3-Embedding-8B vector similarity
4. Agent, in which Claude Code autonomously selects contexts

The protocol also specifies that the Agent always sees full repo and metadata, that similarity methods retrieve exactly 3 chunks, and that all methods receive PR title and description. These details make the retrieval setup tightly controlled, especially in separating the effects of context volume from context selection.

The evaluation metrics are Precision, Recall, and $F_1$, with $G$ denoting the set of ground-truth comments, $M$ the set of model-generated comments, and $TP$ the subset of generated comments that correctly match ground-truth comments. Comment matching requires both line-range overlap and semantic equivalence, where semantic equivalence is verified with an LLM match prompt. This dual criterion is important because it avoids treating topical but mislocalized comments as correct matches.

5. Reported results

AACR-Bench reports $F_1$-scores for the five evaluated models across Agent, No Context, BM25, and Embedding conditions. The benchmark highlights the following comparative results (Zhang et al., 27 Jan 2026).

Method	Notable results
Agent	Claude-4.5: 16.1; Deepseek-V3.2: 6.7; GLM-4.7: 6.7; GPT-5.2: 4.6; Qwen-480B: 6.8
No Context	Claude-4.5: 14.5; Deepseek-V3.2: 9.7; GLM-4.7: 16.0; GPT-5.2: 12.2; Qwen-480B: 14.0
BM25	Claude-4.5: 10.0; Deepseek-V3.2: 15.6; GLM-4.7: 14.7; GPT-5.2: 14.6; Qwen-480B: 11.7
Embedding	Claude-4.5: 13.5; Deepseek-V3.2: 8.9; GLM-4.7: 15.6; GPT-5.2: 11.7; Qwen-480B: 14.4

Among non-Agent methods, the benchmark identifies Deepseek-V3.2 with BM25 as best overall at $F_1 = 15.59\%$ and GLM-4.7 with Embedding at $F_1 = 15.63\%$. GPT-5.2 peaks at $\geq 2$0 with BM25, compared with $\geq 2$1 under direct prompting. Claude-4.5-Sonnet in Agent mode is reported to excel in precision but to suffer from low recall, with precision = 39.9\% and recall = 10.1\%.

The benchmark also analyzes context granularity. For non-Agent methods, Recall decays as context scope widens, following Diff $\geq 2$2 File $\geq 2$3 Repo. The provided example is Qwen-480B, No context, where recall falls from 33.8\% at Diff to 22.6\% at File and 17.6\% at Repo. For the Agent method, the trend can sometimes reverse, with better performance on Repo than Diff due to multi-turn context exploration; however, absolute recall remains very low. The example given is Qwen-480B Agent, with Diff = 4.49\% and Repo = 5.94\%.

A further finding concerns retrieval noise. The benchmark states that providing additional context through BM25 or Embedding often hurts precision for strong models. The specific example is Claude-4.5, whose $\geq 2$4 drops 31\% under BM25 relative to No Context. This result directly challenges the assumption that more retrieved evidence monotonically improves review quality.

6. Interpretation, deployment implications, and common misconceptions

AACR-Bench presents several explicit findings that are framed as deployment recommendations rather than only benchmark observations (Zhang et al., 27 Jan 2026). The first is a precision–recall trade-off: generative (direct) methods maximize recall but hallucinate, whereas Agent frameworks deliver high precision but only surface a small fraction of defects. The benchmark therefore recommends that future ACR systems hybridize both paradigms.

The second is adaptive context awareness. AACR-Bench states that “More context is not always better.” It further reports that the optimal retrieval strategy is model- and language-specific, giving examples such as Deepseek favoring BM25, Qwen favoring Embedding, and Claude favoring Agent. The stated recommendation is that systems require meta-reasoning to decide context granularity at runtime. A plausible implication is that repository-level context should be treated as a selectively invoked capability rather than as a default input configuration.

The third is language-specific bias. The benchmark reports high $\geq 2$5 on Python, Java, Go, and C#, and low $\geq 2$6 on TypeScript, PHP, and Rust. It attributes this to language features, specifically contrasting strong typing and explicit namespaces with pointer macros. This indicates that cross-language ACR is not merely a matter of multilingual token exposure; it is entangled with language-level semantic structure and review affordances.

The fourth is the need to unify local and global reasoning. AACR-Bench states that RAG approaches help global dependencies but introduce noise that obscures local defects, while Agents catch complex cross-file issues but miss obvious diff-level problems, a pattern labeled context-tunnel. The benchmark recommends dynamic attention mechanisms that allocate reasoning focus between diff, file, and repo context.

These conclusions also clarify several common misconceptions. One misconception is that repository-level access is inherently superior to localized review. AACR-Bench reports the opposite for many non-Agent settings, where widening context is associated with lower recall. Another misconception is that agentic review uniformly dominates direct prompting. The results instead show a split regime: agentic systems can improve precision, but often at the cost of very low recall. A third misconception is that benchmark performance on raw PR-comment datasets fully characterizes ACR ability. AACR-Bench argues that earlier assessments may have misjudged or only partially captured model capabilities because of data limitations in ground truth and context design (Zhang et al., 27 Jan 2026).

Taken together, AACR-Bench defines ACR evaluation as a benchmark of context selection, ground-truth completeness, language heterogeneity, and usage paradigm. Its accompanying code, data, and evaluation artifacts are available at https://github.com/alibaba/aacr-bench.