---
title: 'AACR-Bench: Automated Code Review Benchmark'
url: https://www.emergentmind.com/topics/aacr-bench
type: topic
---

# AACR-Bench: Automated Code Review Benchmark

Searching arXiv for the specified benchmark paper and related benchmark context.
AACR-Bench is a benchmark for Automated Code Review (ACR) designed to evaluate Large Language Models (LLMs) under holistic repository-level context rather than only diff-level or file-level inputs. It was introduced to address two stated deficiencies in prior ACR benchmarks: incomplete ground truth derived from raw Pull Request (PR) comments, and limited support for multi-language, repository-level reasoning. AACR-Bench provides PR metadata, diff hunks, full-file context, and entire repository context across ten programming languages, and it uses an “AI-assisted, Expert-verified” annotation pipeline that raises defect coverage from 391 original PR comments to 1,505 final comments, corresponding to an approximately 285% increase [2601.19494].

## 1. Benchmark motivation and problem setting

AACR-Bench is positioned against prior ACR benchmarks that, according to its definition, suffer from two critical limitations. The first is **incomplete ground truth**: benchmarks such as CodeReviewer, SWR-Bench, and CodeFuse-CR-Bench simply use raw PR comments as “gold” annotations. In AACR-Bench’s framing, such comments are noisy, often under-report latent defects, and therefore cannot faithfully measure a model’s ability to surface hidden issues [2601.19494].

The second limitation is **narrow context scope**. Benchmarks with cross-file or repository-level context are described as being almost always restricted to Python, whereas benchmarks supporting multiple languages stop at file- or diff-level. AACR-Bench therefore targets a combined setting in which both linguistic diversity and repository-wide reasoning are present simultaneously. This suggests that the benchmark is not only a dataset expansion, but also an attempt to alter the unit of evaluation from localized patch inspection to mixed local-global review reasoning.

A central implication of this design is methodological: ACR performance can no longer be interpreted only as a function of model quality. In the benchmark’s own conclusions, performance depends critically on the underlying LLM, the retrieval method and its granularity, the programming language, and whether an Agent paradigm is employed [2601.19494]. In that sense, AACR-Bench defines evaluation as a systems problem rather than a single-model inference problem.

## 2. Corpus composition and context hierarchy

AACR-Bench covers **50 popular GitHub repositories**, with **5 repositories per language**, spanning the top 10 languages of the 2025 StackOverflow survey: **JavaScript, Python, TypeScript, Java, C\#, C++, C, PHP, Go, and Rust** [2601.19494]. The benchmark contains **200 PRs** and **1,505 total comments / defects**.

The context supplied for each PR is explicitly stratified. AACR-Bench delivers the **PR title and description**, **all diff hunks extracted via Git**, the **full-file context for each diff hunk**, and the **entire repo context**, including cross-file dependencies and other modules. Correspondingly, comment annotations are tagged by one of three context levels:

- **Diff-level**: issues resolvable within the hunk  
- **File-level**: issues requiring the entire file  
- **Repo-level**: issues requiring cross-file context or PR metadata

This hierarchy is integral to the benchmark’s purpose. Rather than treating context as an undifferentiated token budget, AACR-Bench operationalizes context granularity as an evaluable variable. That design enables analyses of when local reasoning suffices and when repository-wide dependencies become necessary.

The benchmark’s headline corpus statistics are as follows:

| Item | Value |
|---|---:|
| Repositories | 50 |
| Pull Requests | 200 |
| Programming Languages | 10 |
| Total Comments / Defects | 1,505 |
| Security Vulnerabilities | 53 |
| Code Defects | 709 |
| Maintainability / Readability Issues | 626 |
| Performance Issues | 117 |

These category counts indicate that AACR-Bench is not restricted to correctness bugs. It also includes maintainability, readability, performance, and security-oriented findings, which broadens the evaluation target from narrow defect detection to more general review competence [2601.19494].

## 3. Annotation pipeline and defect coverage expansion

AACR-Bench uses an **“AI-assisted, Expert-verified” annotation pipeline** intended to uncover latent defects beyond the comments found in the original PR discussions [2601.19494]. The benchmark defines **defect coverage gain** as the relative increase from the number of original PR comments to the number of final comments. With **391 original PR comments** and **1,505 final comments**, the reported **CoverageIncrease** is approximately **285\%**.

The workflow is described in six stages.

First, **PR Extraction & Filtering** collected **12,715 PRs** from **Dec 2024–Dec 2025**. These were filtered by **English description**, **\(\leq 1{,}000\) changed lines**, **\(\geq 2\) inline comments (\(\geq 1\) accepted)**, **language consistency**, and **semantic validity**, yielding **573 PRs**.

Second, **Stratified Sampling** reduced the pool to **200 PRs**, covering **10 languages** and **50 repos**.

Third, **Human-Augmented Review Summaries** applied **LLM-based deep semantic analysis of multi-turn GitHub review threads** to produce **391 “Augmented Review Comments”**, with **95\% manual-verified accuracy**.

Fourth, **Machine-Generated Reviews** were produced by **six LLMs**—**Claude-4.5-Sonnet, Qwen3-Coder-480B, GPT-5.2, Deepseek-V3.2, GLM-4.7, Gemini-3-Pro**—each using **two frameworks**: an **internal system** and the **Claude Code agent**. The resulting comments were processed with **semantic de-duplication via pairwise LLM comparison**.

Fifth, **Expert Verification** involved **80 senior engineers** in a **three-round, double-blind annotation** process. Rounds 1 and 2 used **two independent annotations per comment**, and Round 3 used a **core expert team** to resolve disagreements. Annotators **validated correctness**, **categorized issue type**, and **labeled context scope**.

Sixth, the **Final Ground Truth** comprised **391 human-augmented** plus **1,114 LLM-augmented** comments, for a total of **1,505 fine-grained review comments**.

This pipeline is significant because it redefines benchmark ground truth as a curated synthesis rather than a direct copy of repository discussion artifacts. A plausible implication is that AACR-Bench evaluates models against a denser and more normalized target than benchmarks that treat raw review threads as complete supervision.

## 4. Evaluation protocol and formal matching criteria

AACR-Bench evaluates **five LLMs** under **two paradigms**: **Direct Prompting (non-Agent)** and **Agent Architecture** [2601.19494]. The non-Agent setting includes **Qwen3-Coder-480B-A35B, Deepseek-V3.2, GLM-4.7, GPT-5.2, and Claude-4.5-Sonnet**. The Agent setting uses **Claude Code**, with the above base models integrated via a **tool-calling agent framework**.

The benchmark defines three context granularity levels:

- **Diff**: only the diff hunk plus PR title and description  
- **File**: the full file containing the hunk plus PR metadata  
- **Repo**: the entire repository code plus PR metadata  

For non-Agent runs, four retrieval conditions are defined:

1. **No context** as baseline  
2. **BM25**, retrieving the **top-3 code snippets by text-similarity**  
3. **Embedding**, retrieving the **top-3 snippets via Qwen3-Embedding-8B vector similarity**  
4. **Agent**, in which **Claude Code autonomously selects contexts**  

The protocol also specifies that the **Agent always sees full repo and metadata**, that **similarity methods retrieve exactly 3 chunks**, and that **all methods receive PR title and description**. These details make the retrieval setup tightly controlled, especially in separating the effects of context volume from context selection.

The evaluation metrics are **Precision**, **Recall**, and **\(F_1\)**, with \(G\) denoting the set of ground-truth comments, \(M\) the set of model-generated comments, and \(TP\) the subset of generated comments that correctly match ground-truth comments. Comment matching requires both **line-range overlap** and **semantic equivalence**, where semantic equivalence is **verified with an LLM match prompt**. This dual criterion is important because it avoids treating topical but mislocalized comments as correct matches.

## 5. Reported results

AACR-Bench reports \(F_1\)-scores for the five evaluated models across Agent, No Context, BM25, and Embedding conditions. The benchmark highlights the following comparative results [2601.19494].

| Method | Notable results |
|---|---|
| Agent | Claude-4.5: 16.1; Deepseek-V3.2: 6.7; GLM-4.7: 6.7; GPT-5.2: 4.6; Qwen-480B: 6.8 |
| No Context | Claude-4.5: 14.5; Deepseek-V3.2: 9.7; GLM-4.7: 16.0; GPT-5.2: 12.2; Qwen-480B: 14.0 |
| BM25 | Claude-4.5: 10.0; Deepseek-V3.2: 15.6; GLM-4.7: 14.7; GPT-5.2: 14.6; Qwen-480B: 11.7 |
| Embedding | Claude-4.5: 13.5; Deepseek-V3.2: 8.9; GLM-4.7: 15.6; GPT-5.2: 11.7; Qwen-480B: 14.4 |

Among non-Agent methods, the benchmark identifies **Deepseek-V3.2 with BM25** as best overall at **\(F_1 = 15.59\%\)** and **GLM-4.7 with Embedding** at **\(F_1 = 15.63\%\)**. **GPT-5.2** peaks at **\(F_1 = 14.64\%\)** with BM25, compared with **\(F_1 = 12.19\%\)** under direct prompting. **Claude-4.5-Sonnet** in Agent mode is reported to excel in precision but to suffer from low recall, with **precision = 39.9\%** and **recall = 10.1\%**.

The benchmark also analyzes context granularity. For **non-Agent methods**, **Recall decays as context scope widens**, following **Diff \(>\) File \(>\) Repo**. The provided example is **Qwen-480B, No context**, where recall falls from **33.8\% at Diff** to **22.6\% at File** and **17.6\% at Repo**. For the **Agent method**, the trend can sometimes reverse, with better performance on Repo than Diff due to **multi-turn context exploration**; however, absolute recall remains very low. The example given is **Qwen-480B Agent**, with **Diff = 4.49\%** and **Repo = 5.94\%**.

A further finding concerns **retrieval noise**. The benchmark states that providing additional context through **BM25** or **Embedding** often hurts precision for strong models. The specific example is **Claude-4.5**, whose **\(F_1\)** drops **31\% under BM25** relative to **No Context**. This result directly challenges the assumption that more retrieved evidence monotonically improves review quality.

## 6. Interpretation, deployment implications, and common misconceptions

AACR-Bench presents several explicit findings that are framed as deployment recommendations rather than only benchmark observations [2601.19494]. The first is a **precision–recall trade-off**: **generative (direct) methods maximize recall but hallucinate**, whereas **Agent frameworks deliver high precision but only surface a small fraction of defects**. The benchmark therefore recommends that future ACR systems **hybridize both paradigms**.

The second is **adaptive context awareness**. AACR-Bench states that **“More context is not always better.”** It further reports that the optimal retrieval strategy is **model- and language-specific**, giving examples such as **Deepseek favoring BM25**, **Qwen favoring Embedding**, and **Claude favoring Agent**. The stated recommendation is that systems require **meta-reasoning to decide context granularity at runtime**. A plausible implication is that repository-level context should be treated as a selectively invoked capability rather than as a default input configuration.

The third is **language-specific bias**. The benchmark reports **high \(F_1\)** on **Python, Java, Go, and C\#**, and **low \(F_1\)** on **TypeScript, PHP, and Rust**. It attributes this to **language features**, specifically contrasting **strong typing** and **explicit namespaces** with **pointer macros**. This indicates that cross-language ACR is not merely a matter of multilingual token exposure; it is entangled with language-level semantic structure and review affordances.

The fourth is the need to **unify local and global reasoning**. AACR-Bench states that **RAG approaches help global dependencies but introduce noise that obscures local defects**, while **Agents catch complex cross-file issues but miss obvious diff-level problems**, a pattern labeled **context-tunnel**. The benchmark recommends **dynamic attention mechanisms** that allocate reasoning focus between **diff**, **file**, and **repo** context.

These conclusions also clarify several common misconceptions. One misconception is that repository-level access is inherently superior to localized review. AACR-Bench reports the opposite for many non-Agent settings, where widening context is associated with lower recall. Another misconception is that agentic review uniformly dominates direct prompting. The results instead show a split regime: agentic systems can improve precision, but often at the cost of very low recall. A third misconception is that benchmark performance on raw PR-comment datasets fully characterizes ACR ability. AACR-Bench argues that earlier assessments may have **misjudged or only partially captured model capabilities** because of data limitations in ground truth and context design [2601.19494].

Taken together, AACR-Bench defines ACR evaluation as a benchmark of **context selection**, **ground-truth completeness**, **language heterogeneity**, and **usage paradigm**. Its accompanying code, data, and evaluation artifacts are available at **https://github.com/alibaba/aacr-bench**.

Source: https://www.emergentmind.com/topics/aacr-bench