SWRBench: A Benchmark for Automated Code Review

• SWRBench is a PR-centric benchmark for automated code review that uses complete project context to evaluate whole pull requests.
• It employs a balanced dataset of 500 Change-PRs and 500 Clean-PRs to effectively measure both issue detection and false positive rates.
• The benchmark relies on an objective LLM-based evaluation that matches predicted change-points against a structured ground truth for review-worthy issues.

Searching arXiv for the cited SWRBench paper and closely related automated code review benchmark context. SWRBench is a benchmark for automated code review (ACR) designed to evaluate systems in a more realistic pull-request setting than earlier code review benchmarks. It is defined as a dataset of 1,000 manually verified GitHub pull requests, organized around PR-centric review with full project context and an objective LLM-based evaluation that checks whether generated reviews semantically cover a structured ground truth of review-worthy issues rather than relying on lexical overlap or subjective holistic scoring (Zeng et al., 1 Sep 2025).

1. Definition and benchmark scope

SWRBench, expanded in the source as “Software Review Benchmark,” is explicitly constructed for PR-centric review rather than method-level or diff-hunk-level review. Each evaluation instance is a whole pull request, and the benchmark includes both the pull request changes and enough repository information to reconstruct the full project state around that pull request. This design is intended to support evaluation of whether an ACR system can identify what in the PR deserves attention, localize issues, and articulate review comments under conditions closer to real pull-request review practice (Zeng et al., 1 Sep 2025).

The benchmark’s three defining design goals are: PR-centric review, comprehensive context, and objective LLM-based evaluation. In concrete terms, each instance includes the PR source data, structured ground-truth change-point data for Change-PRs, and codebase checkpoint information so the complete PR codebase can be reconstructed. This distinguishes SWRBench from benchmarks centered on isolated snippets or local diff units (Zeng et al., 1 Sep 2025).

A central feature of the benchmark is its balanced composition of 500 Change-PRs and 500 Clean-PRs. The source states that this balanced design is deliberate, especially for evaluating false positive behavior, which is presented as a major concern for automated code review tools. This suggests that SWRBench is intended not only to measure issue recall but also to expose over-reporting tendencies that would be costly in practical review settings (Zeng et al., 1 Sep 2025).

2. Motivation and departures from earlier benchmarks

The benchmark is motivated by three limitations attributed to prior ACR benchmarks. First, representative earlier datasets such as Trans-ReviewData, AutoTransformData, T5-ReviewData, and CodeReviewer operate at overly fine-grained units such as methods, triplets involving method/comment/revised method, or single diff hunks. Second, those benchmarks usually lack the full project context needed to judge cross-file effects, project-wide consistency, outdated tests or documentation, or interactions with unchanged code. Third, common evaluation metrics such as Exact-Match, BLEU / CodeBLEU, and simple LLM-as-judge ratings are criticized for failing to measure whether a generated review actually found the issues that mattered in the pull request (Zeng et al., 1 Sep 2025).

Within this framing, SWRBench differs from prior automated code review benchmarks along several axes: the evaluation unit is the whole PR rather than a method or diff hunk; the context includes full PR data plus a full project codebase checkpoint; the ground truth is a structured set of change-points rather than raw human comments or text pairs; and evaluation is based on issue matching against structured ground truth rather than lexical similarity or subjective quality ratings (Zeng et al., 1 Sep 2025).

The benchmark is therefore not defined primarily as a comment-generation dataset. Its task objective is to detect review-worthy issues for an entire PR, allowing multiple valid review phrasings while grounding evaluation in whether the underlying issues were identified. This makes the benchmark closer to end-to-end pull-request review than to localized snippet commenting (Zeng et al., 1 Sep 2025).

3. Construction pipeline and dataset composition

The construction pipeline has four main stages. It begins with 12 popular and actively maintained open-source Python GitHub projects, described as aligned with those used in SWE-Bench. Using the GitHub API, the authors collected PR titles, descriptions, commit histories, diffs, review comments, discussion threads, and final PR status. They then filtered out PRs without review comments, PRs with large-scale changes beyond predefined thresholds on changed files or lines of code, and PRs containing rebase operations. After this first filtering step, 21,350 PRs remained (Zeng et al., 1 Sep 2025).

The second stage defines and extracts change-points, described as verifiable cases where a reviewer suggestion led to a subsequent code modification by the developer. For extraction, the annotation process uses Gemini-2.5-Pro with three independent requests per PR. A PR is retained only if all three runs produce consistent change-point extraction results. PRs are then partitioned into Change-PRs, which contain at least one change-point, and Clean-PRs, which contain none (Zeng et al., 1 Sep 2025).

The third stage improves dataset quality in two ways. First, the authors apply the SZZ algorithm to remove PRs in which issues were not caught during original review but were fixed later. Second, they resample Clean-PRs to align their coarse statistics with Change-PRs, including lines changed and number of commits. The stated purpose is to prevent systems from separating Change-PRs and Clean-PRs by cheap heuristics instead of actual code understanding (Zeng et al., 1 Sep 2025).

The fourth stage is manual verification and refinement. After automated filtering and sampling, about 3,500 PRs remained. From these, 1,000 Change-PRs and 1,000 Clean-PRs were sampled for human verification by five experienced graduate students in computer science, with each PR independently annotated by at least two annotators. The manual review served both to verify LLM-identified change-points and to remove trivial PRs such as simple documentation typos or minor formatting changes. The final released benchmark contains 500 Change-PRs and 500 Clean-PRs, for a total of 1,000 PRs (Zeng et al., 1 Sep 2025).

The paper reports the following dataset statistics.

PR Type	Count	Avg. Commits	Avg. Files	Avg. Lines +	Avg. Lines -	Avg. Change Points
Change-PR	500	4.05	6.29	123.64	60.69	1.90
Clean-PR	500	3.25	6.85	112.60	67.22	0.00
All	1000	3.65	6.57	118.12	63.96	0.95

A further reported statistic concerns change-type distribution: before manual verification and filtering, functional changes were under 15% of raw identified changes; after filtering out trivial PRs, the share of functional changes in SWRBench rose to 31.8%. The source interprets this as making the benchmark better reflect review-relevant functional issues rather than being dominated by superficial documentation or style changes (Zeng et al., 1 Sep 2025).

4. Ground truth, taxonomy, and task formulation

The benchmark’s core supervision unit is the change-point rather than raw review text. For Change-PRs, each ground-truth change-point includes type, description, and relevant commit SHA. The benchmark’s task can be summarized as follows: given a pull request and its project context, generate a review report that identifies review-worthy change-points (Zeng et al., 1 Sep 2025).

The taxonomy has two top-level categories: Evolvability (E) and Functional (F). Under Evolvability, the taxonomy includes E.1 Documentation, with E.1.1 Textual and E.1.2 Language Supported; E.2 Visual Representation; and E.3 Structure, with E.3.1 Organization and E.3.2 Solution Approach. Under Functional, it includes F.1 Interface, F.2 Logic, F.3 Resource, F.4 Check, F.5 Support, and F.6 Larger Defects. The source notes that one passage says “11 predefined change-point types” and another mentions “13 types in Table,” but that the taxonomy table itself shows 11 fine-grained leaf types under the two top-level families (Zeng et al., 1 Sep 2025).

The benchmark output is an unstructured natural-language review report containing one or more review comments or issues. However, evaluation does not occur at the sentence level or hunk level. Instead, the scoring unit is the predicted change-point extracted from the generated review by the evaluator. This design aligns the benchmark with the practical question of whether the system identified distinct review-worthy issues, regardless of their exact wording (Zeng et al., 1 Sep 2025).

Because the benchmark supports full repository reconstruction, it is relevant not only to prompt-based systems that consume PR metadata and diffs, but also to agent-based systems that inspect files outside the diff, navigate the repository, or execute tools. This suggests that SWRBench is positioned as a benchmark for realistic ACR workflows rather than only text-conditioned review generation (Zeng et al., 1 Sep 2025).

5. Evaluation methodology and metric definitions

The evaluation pipeline is one of SWRBench’s primary contributions. The source presents its central idea as objective semantic matching: the evaluator does not rate how good a review sounds, but instead checks whether each ground-truth issue was successfully “hit” by the generated review. To do this, the evaluation LLM first parses the generated review into predicted change-points, assigning each a change type and a severity score; it then compares those predicted change-points with the PR’s structured ground-truth change-points and determines whether each ground-truth item was covered (Zeng et al., 1 Sep 2025).

The paper defines the benchmark’s primary metrics as follows:

$Precision=\frac{TP}{TP+FP}$

$Recall=\frac{TP}{TP+FN}$

$$F1=2\times\frac{Precision\times Recall}{Precision+Recall}$$

Here, True Positives (TP) are ground-truth change-points successfully hit by at least one predicted change-point, False Positives (FP) are predicted change-points that do not hit any ground-truth change-point, and False Negatives (FN) are ground-truth change-points not hit by any predicted change-points (Zeng et al., 1 Sep 2025).

In addition to overall Precision, Recall, and F1, the paper reports functional Precision, Recall, and F1, restricted to predicted change-points typed as functional by the evaluator and ground-truth change-points whose type is functional. It also reports Avg. Count, the average number of predicted change-points per PR, and Avg. Severity, the average evaluator-assigned severity score, though these are described as descriptive rather than primary benchmark objectives (Zeng et al., 1 Sep 2025).

The evaluator’s reliability is validated through a human study over 100 generated code review reports from the RQ2 experiments. These reports were independently evaluated by 3 human experts and 2 LLMs—Gemini-2.5-Pro and Gemini-2.5-Flash—using the same task definitions and prompt. For the key hit decision, agreement between any two evaluators ranged from 89.2% to 94.9%. For type assignment, human–LLM consistency ranged from 69.4% to 94.8%, while inter-human agreement ranged from 67.4% to 83.6%. Severity agreement was substantially lower, including among humans. Because Gemini-2.5-Pro and Gemini-2.5-Flash had similar agreement, the later large-scale experiments used Gemini-2.5-Flash, with the paper reporting that evaluating all 1,000 PRs costs approximately $1.57 (Zeng et al., 1 Sep 2025).

6. Experimental findings and observed failure modes

The benchmark evaluates several ACR systems: LLM-Review, SWR-Agent, CR-Agent, Hybrid-Review, and PR-Review. The experimental setup spans both limited-context and richer-context conditions. For example, LLM-Review uses basic PR information such as title, description, and commit diff; SWR-Agent can explore the codebase and execute snippets; Hybrid-Review augments PR information with a static analysis report; and PR-Review uses prompt engineering that consolidates diffs across commits, prioritizes files under context limits, and asks for structured categories such as test-related or security-related issues. All tools use official default configurations, and for the two baselines built by the authors the inference temperature is 0.2 (Zeng et al., 1 Sep 2025).

The strongest overall result in the main tool comparison is PR-Review + Gemini-2.5-Pro, with Precision 16.65, Recall 23.18, F1 19.38, Avg. Count 1.32, and Avg. Severity 5.64. Mean performance by tool shows PR-Review as the strongest framework overall, with mean Precision 15.39, Recall 24.06, and F1 18.73, as well as functional mean Precision 16.55, Recall 43.89, and F1 23.87. The source presents the central conclusion that current systems perform poorly on SWRBench and that precision is the main bottleneck, since most approaches generate too many false positives (Zeng et al., 1 Sep 2025).

A prominent empirical finding is that ACR systems are better at detecting functional issues than evolvability issues. Averaged PR-Review results by change type show higher F1 for functional categories such as F.2 Logic: 26.20, F.3 Resource: 24.26, and F.6 Larger Defects: 27.65 than for evolvability categories such as E.1.2 Language Features: 7.85, E.2 Visual Representation: 6.05, and E.3.1 Organization: 16.45. The paper attributes this to the greater subjectivity of many evolvability issues and the greater objectivity of functional defects (Zeng et al., 1 Sep 2025).

Using PR-Review as the strongest framework, the paper also compares multiple LLM backends. Strong overall F1 values include Gemini-2.5-Pro: 19.38, GPT-o3: 18.13, DeepSeek-R1: 18.58, Claude-3.7-Sonnet: 18.23, and GPT-4o: 18.73. Within the Qwen family, reasoning-enhanced variants often outperform their non-reasoning counterparts, for example Qwen-2.5-14B: 9.01 versus Qwen-2.5-R1-14B: 15.95, although Qwen-2.5-R1-7B: 7.51 is noted as an exception due to output formatting issues. The source interprets this as evidence that reasoning-enhancement training is generally beneficial for code review (Zeng et al., 1 Sep 2025).

Several recurring failure modes are emphasized: high false positive rates / low precision, weak handling of evolvability issues, instability across runs and across models, poor outcomes when static-analysis augmentation is naively integrated, and the fact that multi-agent approaches are not automatically better. In the reported results, Hybrid-Review has very low precision, while CR-Agent underperforms, which the source attributes to coordination overhead and error propagation (Zeng et al., 1 Sep 2025).

7. Multi-Review aggregation, significance, and limitations

The paper’s proposed improvement strategy is Multi-Review, motivated by overlap analyses showing instability and complementary coverage across runs and models. The study reports that across different LLMs, only 36 successfully identified change-points overlapped, and across five runs of the same LLM, only 27 successfully identified change-points overlapped. This is interpreted as evidence that single-pass reviews are noisy and that different runs or models capture partly complementary issues (Zeng et al., 1 Sep 2025).

Multi-Review operates by running PR-Review or another ACR system multiple times on the same pull request, collecting $n$ review reports, and then feeding them to an LLM aggregator that synthesizes the reports, filters valid points, discards erroneous suggestions, and outputs a final unified review. The paper distinguishes Multi-Agg, which aggregates reports from multiple distinct LLMs, and Self-Agg, which aggregates multiple executions of the same LLM. In the reported experiments, Gemini-2.5-Flash is used as the representative model with $n \in \{0,1,3,5,10\}$, where $n=0$ corresponds to baseline PR-Review without aggregation (Zeng et al., 1 Sep 2025).

The strongest highlighted result is Gemini-2.5-Flash + Self-Agg ($n=10$), which reaches Overall F1 21.91%, a 43.67% increase, and Overall Recall 30.44%, a 118.83% increase. The paper also reports gains for smaller open-source models, including Qwen-Chat-7B Self-Agg ($n=10$), which improves overall F1 by 26.13% to 14.67%, and Qwen-Chat-32B Self-Agg ($n=10$), which improves overall F1 by 19.25% to 16.41%. The source states that the larger gain is in recall, while precision remains the next frontier for improvement (Zeng et al., 1 Sep 2025).

The paper identifies several threats to validity. Under internal validity, the stated threats are implementation bugs and inaccuracy in manual verification, mitigated by code review of the implementation and public release of code. Under external validity, the threats are the choice of LLMs, ACR tools, and selected projects, mitigated by choosing representative tools and 12 popular, well-maintained open-source GitHub projects. Under construct validity, the central threat is the reliability of the objective LLM-based evaluator, mitigated by the reported human validation study showing high consistency with human experts, especially for hit detection (Zeng et al., 1 Sep 2025).

Taken together, SWRBench defines a shift in automated code review evaluation from local comment generation toward repository-context-rich PR review. Its distinguishing features are the use of whole pull requests, full project context, manually verified change-points, and reference-based semantic evaluation. The reported experimental results indicate that current ACR systems remain weak on this task, particularly because of low precision, while the Multi-Review results suggest that aggregation can materially improve performance without changing the underlying benchmark definition (Zeng et al., 1 Sep 2025).