- The paper introduces agentic code review, a repository-grounded workflow where reviewer agents explore code, run tests, approve or reject pull requests, and generate actionable diagnoses for revision.
- Agentic review outperforms diff-based and retrieval-based single-turn methods, raising resolve rates from 44.1% to 52.6% on weakly generated patches and widening its advantage on harder issues.
- Review-guided revision improves test-time efficiency, reaching 38.4% resolution versus 32.3% for reviewer-gated resampling while using 2.44 samples on average, though subtle false approvals remain a major challenge.
SWE-Review addresses a structural gap in current AI-assisted software engineering: while coding agents can generate candidate pull requests (PRs) for real-world issues, the process is open-loop—there is no systematic mechanism for deciding whether a PR resolves its issue or for diagnosing how it should be revised when it does not. The paper proposes agentic code review as the closing mechanism: a reviewer agent explores the repository in an executable environment, emits an approve/request-changes decision, and produces a structured diagnosis that drives revision. The work contributes a benchmark (SWE-Review-Bench), a training corpus (SWE-Review-Traj), distilled reviewer models, and end-to-end evaluations across review, training transfer, and test-time scaling (2607.06065).
The review task is formalized as repository-grounded: given a repository checkout, a natural-language issue, and a candidate PR (diff, title, body), the reviewer may browse files, search code, and execute commands before producing two outputs—a binary decision and, when requesting changes, a diagnosis citing concrete defects and actionable fixes. The reviewer never sees the golden patch or hidden test results. Evaluation uses three metrics: Completion Rate (CR), whether a parseable review is produced; Decision Accuracy (DA), agreement between the decision and executable resolve status; and Resolve Rate after Revision (RRR), the final resolve rate after approved patches are kept and rejected patches are revised once from the feedback.
SWE-Review-Bench contains 1,384 candidate PRs derived from 500 SWE-bench Verified issues, generated by three models spanning distinct quality tiers under the OpenHands-SDK scaffold: GLM-5 (72.2% resolve rate, n=500), Qwen3-Coder-30B-A3B (50.9%, n=462), and Qwen3-30B-A3B (27.5%, n=422). This stratification matters because reviewer behavior differs sharply across candidate-quality regimes; the paper's error analysis shows false approvals dominate on high-quality generators while false rejections predominate on weak ones. A limitation worth noting at the outset: DA and RRR presuppose a SWE-style harness with hidden-test verification and a standardized revise-from-feedback procedure, so the metrics measure functional correctness only—not style, security, maintainability, or project conventions.
Agentic versus single-turn review
The central empirical claim is that interactive repository exploration outperforms fixed-context single-turn review. Holding the reviewer fixed at Claude Opus 4.6, agentic review beats both diff-only and diff-plus-retrieved-context single-turn baselines on all three generator splits, in both DA and RRR. The gains are largest for weaker generators: on Qwen3-30B-A3B PRs, agentic review raises RRR from 44.1% under the best single-turn setting to 52.6%—nearly double the 27.5% no-review baseline. Stratifying instances into difficulty tertiles via file-level divergence from the golden patch and fail-to-pass test count shows the agentic–single-turn gap widens monotonically from easy to hard patches.
A case study on sympy-13877 illustrates the mechanism. A candidate patch adds a NaN guard in exprtools.py that suppresses a crash but leaves det() returning incorrect nan values; the root cause is a discarded cancel(ret) result upstream in matrices.py. Single-turn reviewers approve the symptom fix; the agentic reviewer traces the call chain, runs the reproducer, observes f(5) = nan and f(6) = nan post-patch, and requests changes with a one-line upstream fix. The paper's interpretation—that the advantage lies not merely in more context but in adaptively gathering decision-relevant evidence—is supported by the fact that adding retrieved context improves single-turn review without closing the gap.
Trajectory curation and validation
To address data scarcity for open reviewers, SWE-Review-Traj is built from SWE-rebench issues (~6k after decontamination against the benchmark repositories), generating 14,156 candidate PRs with the same three generators. GLM-5 with thinking enabled serves as the teacher; notably, the prompt forces the teacher to reconstruct the issue's root cause independently before reading the candidate patch, mitigating anchoring bias toward the proposed fix. Decision-correctness filtering against executable outcomes yields 8,914 trajectories.
Because a correct rejection decision does not guarantee a correct diagnosis, the paper validates diagnoses along two axes. Semantic validation by two proprietary judges (Claude Opus 4.6 and GPT-5.4) yields mean overall scores of only ~3.1 on a 5-point scale, with suggestion correctness around 2.8—the authors concede the diagnoses contain meaningful but imperfect information; inter-judge agreement is substantial (κ=0.72). Stricter semantic filtering to 6,789 trajectories conferred no clear benefit over decision-correct filtering alone. Functional validation is more decisive: on 100 sampled rejections, revision RRR rises from 3% (no review) to 8% (decision only) to 21% (teacher diagnosis), recovering roughly two-thirds of the gap to a 32% oracle upper bound. This establishes that the structured diagnosis carries actionable signal beyond the binary verdict.
Distillation and cross-task transfer
Supervised fine-tuning on SWE-Review-Traj transforms small open models into usable reviewers. For Qwen3-8B, CR rises from ~4% to 71–84% and DA from near-random to 67–72%; RRR gains reach +7.6 points on the Qwen3-Coder split and +8.3 points for the distilled 30B-A3B model on the weakest split. Consistent with capability asymmetry, the distilled reviewers provide no RRR gain over no-review when judging GLM-5 patches—clear gains there emerge only with frontier-scale reviewers.
A stronger result concerns mixed training. Fine-tuning Qwen3-8B on issue-resolution trajectories paired with equal volumes of review trajectories improves direct resolve rate at every scale (27.6→28.4, 31.2→36.8, 34.0→37.8%), indicating review supervision transfers positively to patch generation rather than teaching an isolated judging skill. The mixed model also becomes a competent reviewer (CR 67.6–87.4%, DA 67.4–72.3%), enabling a self-contained generate-review-revise loop with final resolve-rate gains up to 10.6 points. In the appendix, this single checkpoint sustains iterative self-review from 34.8% to 44.0% over five samples.
Test-time scaling
Agentic review also functions as a test-time verifier with a richer interface than scalar scorers. Using Qwen3-30B-A3B as the resolver (baseline 22.9%), review-guided iterative revision reaches 38.4% within a five-sample budget while consuming only 2.44 samples on average, because approval gates early stopping. Reviewer-gated resampling without diagnostic feedback plateaus at 32.3% even at budget 16 (8.9 average samples), and verifier-selected best-of-N reaches only 25.6% at full cost. On a token-efficiency basis the gap is stark: 2.28 versus 0.35 percentage points per million tokens—a 6.5× advantage for directed feedback over independent resampling. Against size-matched published verifiers (SWE-Lego-Verifier-8B, OpenHands Critic) at budget 16, the trained reviewer dominates on both agents tested, and the Critic sometimes underperforms random selection.
Reviewer behavior and failure analysis
Appendix analyses yield several findings relevant to practitioners. Exploration efficiency, not volume, predicts quality: Opus 4.6 achieves the highest weighted DA (81.8%) using 24.1 steps and 148K tokens, whereas distilled 8B reviewers use 57 steps and 2.36M tokens for 69.1% DA. Reproducer execution is the strongest behavioral correlate of correct decisions (Opus runs reproducers in 99.9% of trials); the distilled models compensate with broad undirected search. Incorrect decisions consume significantly more compute across all models (p<0.001), suggesting excess exploration signals uncertainty rather than rescuing accuracy.
Error taxonomy for Opus 4.6 reveals a "test paradox": among false approvals, 82% of reports mention tests passing, and 62% of falsely approved patches include AI-written test files that validate the patch's own mental model; among false rejections, 42% stem from the reviewer's own failing tests being misattributed to the patch. False approvals concentrate in the hard tertile (16.7% FA rate versus 7.6% easy), driven by subtle logic bugs (37.7%), insufficient reproducer coverage (32.1%), and cross-file reasoning gaps (29.0%). These results imply that as PR generators improve, reviewer errors will concentrate increasingly on subtle false approvals—an open problem the paper identifies but does not solve.
Limitations and open questions
The paper scopes itself explicitly to AI-generated PRs for SWE-style bug resolution; feature implementation, refactoring, migrations, and architectural changes are unaddressed, as are non-functional review criteria. Relative comparisons are confined to selected repositories, model families, prompts, and the OpenHands-SDK scaffold. Several questions remain open: whether stronger test-adequacy analysis can reduce false approvals on near-correct patches; whether explicit functional-equivalence checks can reduce the misreading-driven false rejections; and whether distillation can close the reproducer-formulation competency gap that currently separates open reviewers from frontier models.
Conclusion
SWE-Review demonstrates that code review, formulated as an agentic, repository-grounded task with executable verification, is a practical pivot for closed-loop issue resolution. Its benchmark and trajectory corpus enable reproducible study of reviewer training; its experiments show consistent gains from agentic over single-turn review, positive transfer from review supervision to patch generation, and efficient test-time scaling through review-guided iteration. The remaining bottlenecks—subtle false approvals on high-quality patches and the cost-efficiency gap of distilled reviewers—define the immediate research agenda this work leaves open.
