---
title: 'SWRBench: A Benchmark for Automated Code Review'
url: https://www.emergentmind.com/topics/swrbench
type: topic
---

# SWRBench: A Benchmark for Automated Code Review

Searching arXiv for the cited SWRBench paper and closely related automated code review benchmark context.
SWRBench is a benchmark for automated code review (ACR) designed to evaluate systems in a more realistic pull-request setting than earlier code review benchmarks. It is defined as a dataset of **1,000 manually verified GitHub pull requests**, organized around **PR-centric review with full project context** and an **objective LLM-based evaluation** that checks whether generated reviews semantically cover a structured ground truth of review-worthy issues rather than relying on lexical overlap or subjective holistic scoring [2509.01494].

## 1. Definition and benchmark scope

SWRBench, expanded in the source as “Software Review Benchmark,” is explicitly constructed for **PR-centric review** rather than method-level or diff-hunk-level review. Each evaluation instance is a whole pull request, and the benchmark includes both the pull request changes and enough repository information to reconstruct the full project state around that pull request. This design is intended to support evaluation of whether an ACR system can identify what in the PR deserves attention, localize issues, and articulate review comments under conditions closer to real pull-request review practice [2509.01494].

The benchmark’s three defining design goals are: **PR-centric review**, **comprehensive context**, and **objective LLM-based evaluation**. In concrete terms, each instance includes the PR source data, structured ground-truth change-point data for Change-PRs, and codebase checkpoint information so the complete PR codebase can be reconstructed. This distinguishes SWRBench from benchmarks centered on isolated snippets or local diff units [2509.01494].

A central feature of the benchmark is its balanced composition of **500 Change-PRs** and **500 Clean-PRs**. The source states that this balanced design is deliberate, especially for evaluating **false positive behavior**, which is presented as a major concern for automated code review tools. This suggests that SWRBench is intended not only to measure issue recall but also to expose over-reporting tendencies that would be costly in practical review settings [2509.01494].

## 2. Motivation and departures from earlier benchmarks

The benchmark is motivated by three limitations attributed to prior ACR benchmarks. First, representative earlier datasets such as **Trans-ReviewData, AutoTransformData, T5-ReviewData, and CodeReviewer** operate at overly fine-grained units such as methods, triplets involving method/comment/revised method, or single diff hunks. Second, those benchmarks usually lack the **full project context** needed to judge cross-file effects, project-wide consistency, outdated tests or documentation, or interactions with unchanged code. Third, common evaluation metrics such as **Exact-Match**, **BLEU / CodeBLEU**, and simple **LLM-as-judge ratings** are criticized for failing to measure whether a generated review actually found the issues that mattered in the pull request [2509.01494].

Within this framing, SWRBench differs from prior automated code review benchmarks along several axes: the evaluation unit is the **whole PR** rather than a method or diff hunk; the context includes full PR data plus a **full project codebase checkpoint**; the ground truth is a structured set of **change-points** rather than raw human comments or text pairs; and evaluation is based on **issue matching** against structured ground truth rather than lexical similarity or subjective quality ratings [2509.01494].

The benchmark is therefore not defined primarily as a comment-generation dataset. Its task objective is to detect **review-worthy issues for an entire PR**, allowing multiple valid review phrasings while grounding evaluation in whether the underlying issues were identified. This makes the benchmark closer to end-to-end pull-request review than to localized snippet commenting [2509.01494].

## 3. Construction pipeline and dataset composition

The construction pipeline has four main stages. It begins with **12 popular and actively maintained open-source Python GitHub projects**, described as aligned with those used in **SWE-Bench**. Using the GitHub API, the authors collected PR titles, descriptions, commit histories, diffs, review comments, discussion threads, and final PR status. They then filtered out PRs without review comments, PRs with large-scale changes beyond predefined thresholds on changed files or lines of code, and PRs containing rebase operations. After this first filtering step, **21,350 PRs** remained [2509.01494].

The second stage defines and extracts **change-points**, described as verifiable cases where a reviewer suggestion led to a subsequent code modification by the developer. For extraction, the annotation process uses **Gemini-2.5-Pro** with **three independent requests per PR**. A PR is retained only if all three runs produce **consistent change-point extraction results**. PRs are then partitioned into **Change-PRs**, which contain at least one change-point, and **Clean-PRs**, which contain none [2509.01494].

The third stage improves dataset quality in two ways. First, the authors apply the **SZZ algorithm** to remove PRs in which issues were not caught during original review but were fixed later. Second, they resample Clean-PRs to align their coarse statistics with Change-PRs, including lines changed and number of commits. The stated purpose is to prevent systems from separating Change-PRs and Clean-PRs by cheap heuristics instead of actual code understanding [2509.01494].

The fourth stage is **manual verification and refinement**. After automated filtering and sampling, about **3,500 PRs** remained. From these, **1,000 Change-PRs** and **1,000 Clean-PRs** were sampled for human verification by **five experienced graduate students** in computer science, with each PR independently annotated by at least two annotators. The manual review served both to verify LLM-identified change-points and to remove trivial PRs such as simple documentation typos or minor formatting changes. The final released benchmark contains **500 Change-PRs** and **500 Clean-PRs**, for a total of **1,000 PRs** [2509.01494].

The paper reports the following dataset statistics.

| PR Type | Count | Avg. Commits | Avg. Files | Avg. Lines + | Avg. Lines - | Avg. Change Points |
|---|---:|---:|---:|---:|---:|---:|
| Change-PR | 500 | 4.05 | 6.29 | 123.64 | 60.69 | 1.90 |
| Clean-PR | 500 | 3.25 | 6.85 | 112.60 | 67.22 | 0.00 |
| All | 1000 | 3.65 | 6.57 | 118.12 | 63.96 | 0.95 |

A further reported statistic concerns change-type distribution: before manual verification and filtering, **functional changes were under 15%** of raw identified changes; after filtering out trivial PRs, the share of functional changes in SWRBench rose to **31.8%**. The source interprets this as making the benchmark better reflect review-relevant functional issues rather than being dominated by superficial documentation or style changes [2509.01494].

## 4. Ground truth, taxonomy, and task formulation

The benchmark’s core supervision unit is the **change-point** rather than raw review text. For Change-PRs, each ground-truth change-point includes **type**, **description**, and **relevant commit SHA**. The benchmark’s task can be summarized as follows: given a pull request and its project context, generate a review report that identifies review-worthy change-points [2509.01494].

The taxonomy has two top-level categories: **Evolvability (E)** and **Functional (F)**. Under Evolvability, the taxonomy includes **E.1 Documentation**, with **E.1.1 Textual** and **E.1.2 Language Supported**; **E.2 Visual Representation**; and **E.3 Structure**, with **E.3.1 Organization** and **E.3.2 Solution Approach**. Under Functional, it includes **F.1 Interface**, **F.2 Logic**, **F.3 Resource**, **F.4 Check**, **F.5 Support**, and **F.6 Larger Defects**. The source notes that one passage says “11 predefined change-point types” and another mentions “13 types in Table,” but that the taxonomy table itself shows **11 fine-grained leaf types** under the two top-level families [2509.01494].

The benchmark output is an **unstructured natural-language review report** containing one or more review comments or issues. However, evaluation does not occur at the sentence level or hunk level. Instead, the scoring unit is the **predicted change-point** extracted from the generated review by the evaluator. This design aligns the benchmark with the practical question of whether the system identified distinct review-worthy issues, regardless of their exact wording [2509.01494].

Because the benchmark supports full repository reconstruction, it is relevant not only to prompt-based systems that consume PR metadata and diffs, but also to **agent-based systems** that inspect files outside the diff, navigate the repository, or execute tools. This suggests that SWRBench is positioned as a benchmark for realistic ACR workflows rather than only text-conditioned review generation [2509.01494].

## 5. Evaluation methodology and metric definitions

The evaluation pipeline is one of SWRBench’s primary contributions. The source presents its central idea as objective semantic matching: the evaluator does not rate how good a review sounds, but instead checks whether each ground-truth issue was successfully **“hit”** by the generated review. To do this, the evaluation LLM first parses the generated review into predicted change-points, assigning each a **change type** and a **severity score**; it then compares those predicted change-points with the PR’s structured ground-truth change-points and determines whether each ground-truth item was covered [2509.01494].

The paper defines the benchmark’s primary metrics as follows:

$$
Precision=\frac{TP}{TP+FP}
$$

$$
Recall=\frac{TP}{TP+FN}
$$

$$
F1=2\times\frac{Precision\times Recall}{Precision+Recall}
$$

Here, **True Positives (TP)** are ground-truth change-points successfully hit by at least one predicted change-point, **False Positives (FP)** are predicted change-points that do not hit any ground-truth change-point, and **False Negatives (FN)** are ground-truth change-points not hit by any predicted change-points [2509.01494].

In addition to overall Precision, Recall, and F1, the paper reports **functional Precision, Recall, and F1**, restricted to predicted change-points typed as functional by the evaluator and ground-truth change-points whose type is functional. It also reports **Avg. Count**, the average number of predicted change-points per PR, and **Avg. Severity**, the average evaluator-assigned severity score, though these are described as descriptive rather than primary benchmark objectives [2509.01494].

The evaluator’s reliability is validated through a human study over **100 generated code review reports** from the RQ2 experiments. These reports were independently evaluated by **3 human experts** and **2 LLMs**—**Gemini-2.5-Pro** and **Gemini-2.5-Flash**—using the same task definitions and prompt. For the key **hit** decision, agreement between any two evaluators ranged from **89.2% to 94.9%**. For type assignment, human–LLM consistency ranged from **69.4% to 94.8%**, while inter-human agreement ranged from **67.4% to 83.6%**. Severity agreement was substantially lower, including among humans. Because Gemini-2.5-Pro and Gemini-2.5-Flash had similar agreement, the later large-scale experiments used **Gemini-2.5-Flash**, with the paper reporting that evaluating all **1,000 PRs** costs approximately **\$1.57** [2509.01494].

## 6. Experimental findings and observed failure modes

The benchmark evaluates several ACR systems: **LLM-Review**, **SWR-Agent**, **CR-Agent**, **Hybrid-Review**, and **PR-Review**. The experimental setup spans both limited-context and richer-context conditions. For example, **LLM-Review** uses basic PR information such as title, description, and commit diff; **SWR-Agent** can explore the codebase and execute snippets; **Hybrid-Review** augments PR information with a static analysis report; and **PR-Review** uses prompt engineering that consolidates diffs across commits, prioritizes files under context limits, and asks for structured categories such as test-related or security-related issues. All tools use official default configurations, and for the two baselines built by the authors the inference temperature is **0.2** [2509.01494].

The strongest overall result in the main tool comparison is **PR-Review + Gemini-2.5-Pro**, with **Precision 16.65**, **Recall 23.18**, **F1 19.38**, **Avg. Count 1.32**, and **Avg. Severity 5.64**. Mean performance by tool shows **PR-Review** as the strongest framework overall, with mean **Precision 15.39**, **Recall 24.06**, and **F1 18.73**, as well as functional mean **Precision 16.55**, **Recall 43.89**, and **F1 23.87**. The source presents the central conclusion that current systems perform poorly on SWRBench and that **precision is the main bottleneck**, since most approaches generate too many false positives [2509.01494].

A prominent empirical finding is that ACR systems are better at detecting **functional** issues than **evolvability** issues. Averaged **PR-Review** results by change type show higher F1 for functional categories such as **F.2 Logic: 26.20**, **F.3 Resource: 24.26**, and **F.6 Larger Defects: 27.65** than for evolvability categories such as **E.1.2 Language Features: 7.85**, **E.2 Visual Representation: 6.05**, and **E.3.1 Organization: 16.45**. The paper attributes this to the greater subjectivity of many evolvability issues and the greater objectivity of functional defects [2509.01494].

Using **PR-Review** as the strongest framework, the paper also compares multiple LLM backends. Strong overall F1 values include **Gemini-2.5-Pro: 19.38**, **GPT-o3: 18.13**, **DeepSeek-R1: 18.58**, **Claude-3.7-Sonnet: 18.23**, and **GPT-4o: 18.73**. Within the Qwen family, reasoning-enhanced variants often outperform their non-reasoning counterparts, for example **Qwen-2.5-14B: 9.01** versus **Qwen-2.5-R1-14B: 15.95**, although **Qwen-2.5-R1-7B: 7.51** is noted as an exception due to output formatting issues. The source interprets this as evidence that **reasoning-enhancement training** is generally beneficial for code review [2509.01494].

Several recurring failure modes are emphasized: **high false positive rates / low precision**, **weak handling of evolvability issues**, **instability across runs and across models**, poor outcomes when static-analysis augmentation is naively integrated, and the fact that **multi-agent approaches are not automatically better**. In the reported results, **Hybrid-Review** has very low precision, while **CR-Agent** underperforms, which the source attributes to coordination overhead and error propagation [2509.01494].

## 7. Multi-Review aggregation, significance, and limitations

The paper’s proposed improvement strategy is **Multi-Review**, motivated by overlap analyses showing instability and complementary coverage across runs and models. The study reports that across different LLMs, only **36** successfully identified change-points overlapped, and across **five runs of the same LLM**, only **27** successfully identified change-points overlapped. This is interpreted as evidence that single-pass reviews are noisy and that different runs or models capture partly complementary issues [2509.01494].

Multi-Review operates by running **PR-Review** or another ACR system multiple times on the same pull request, collecting \(n\) review reports, and then feeding them to an LLM aggregator that synthesizes the reports, filters valid points, discards erroneous suggestions, and outputs a final unified review. The paper distinguishes **Multi-Agg**, which aggregates reports from multiple distinct LLMs, and **Self-Agg**, which aggregates multiple executions of the same LLM. In the reported experiments, **Gemini-2.5-Flash** is used as the representative model with \(n \in \{0,1,3,5,10\}\), where \(n=0\) corresponds to baseline PR-Review without aggregation [2509.01494].

The strongest highlighted result is **Gemini-2.5-Flash + Self-Agg (\(n=10\))**, which reaches **Overall F1 21.91%**, a **43.67% increase**, and **Overall Recall 30.44%**, a **118.83% increase**. The paper also reports gains for smaller open-source models, including **Qwen-Chat-7B Self-Agg (\(n=10\))**, which improves overall F1 by **26.13%** to **14.67%**, and **Qwen-Chat-32B Self-Agg (\(n=10\))**, which improves overall F1 by **19.25%** to **16.41%**. The source states that the larger gain is in recall, while precision remains the next frontier for improvement [2509.01494].

The paper identifies several threats to validity. Under **internal validity**, the stated threats are implementation bugs and inaccuracy in manual verification, mitigated by code review of the implementation and public release of code. Under **external validity**, the threats are the choice of LLMs, ACR tools, and selected projects, mitigated by choosing representative tools and **12 popular, well-maintained open-source GitHub projects**. Under **construct validity**, the central threat is the reliability of the objective LLM-based evaluator, mitigated by the reported human validation study showing high consistency with human experts, especially for hit detection [2509.01494].

Taken together, SWRBench defines a shift in automated code review evaluation from local comment generation toward repository-context-rich PR review. Its distinguishing features are the use of **whole pull requests**, **full project context**, **manually verified change-points**, and **reference-based semantic evaluation**. The reported experimental results indicate that current ACR systems remain weak on this task, particularly because of **low precision**, while the Multi-Review results suggest that aggregation can materially improve performance without changing the underlying benchmark definition [2509.01494].

Source: https://www.emergentmind.com/topics/swrbench