- The paper introduces a trajectory-level evaluation of four terminal-based and five static code reviewers using 362 human-verified repository tasks, revealing that agentic systems achieve substantially better localization but lower human-alignment rates.
- The study finds that reviewer costs vary by roughly two orders of magnitude, with exploration and validation dominating spending while planning consumes less than 10% of effort across agents.
- The paper shows that successful reviews involve more upfront planning, whereas excessive validation often reflects weak initial grounding and fails to improve outcomes, motivating cost-sensitive and behavior-aware benchmarks.
Motivation and scope
Terminal-based agentic code review shifts review feedback to local development, before a pull request is opened. Unlike static LLM reviewers that generate comments from fixed inputs (diffs and PR metadata), agentic reviewers autonomously navigate the repository—invoking tools such as grep, find, and cat—to gather evidence before producing comments. The authors argue that existing benchmarks are mismatched to this setting: they evaluate only final outputs under static prompting, providing no visibility into how agents plan, explore, and validate. This paper addresses that gap through a trajectory-level empirical study of four production-grade terminal-based reviewers (Claude Code, Gemini CLI, Rovo Dev, CodeRabbit) against five static LLM reviewers (GPT-5.4, GPT-5.2, Claude Sonnet 4.6/4.5, Gemini 2.5 Pro), analyzing behavior, operational cost, and human alignment (2607.16740).
Dataset construction
The study builds AgenticCR-Verified, reconstructed from the ContextCRBench corpus of 68K code review discussions across 99 repositories. A review task is defined as T(repo,commit hash,diff hunks,issue,PR→comments). A three-stage filter reduces 30K unique task candidates to 1,078 tasks: excluding repositories larger than 1GB, retaining only tasks whose commits remain resolvable and whose diff hunks match via token matching, then stratifying by diff size (Small–Medium ≤100 additions, Large 101–500, Extra Large >500). Manual verification removes bot-generated or non-substantive comments and extreme commits (>2,000 LOC), yielding 362 human-verified tasks. This attrition—from 30K candidates to 362 verified tasks—is itself an implicit finding: most historical review data cannot be faithfully replayed at repository level.
Evaluation methodology
All agents run in Harbor, a sandboxed execution harness standardizing conditions across heterogeneous reviewers. Trajectories are recorded in ATIF (Agent Trajectory Interchange Format) and segmented into five phases—initialization, exploration, planning, generation, validation—via keyword matching against message text, reasoning content, and tool descriptions. Four phase-level signals are measured: steps, tool calls, tokens, and estimated cost.
Review quality uses two comment-level criteria: localization (a generated comment within ±5 lines of a human comment) and human alignment (identifying a relevant issue, judged by an LLM-as-a-Judge framework). Tasks are aggregated into accomplished (both criteria met by at least one comment), partially accomplished (one criterion), or unaccomplished (neither). Temperature is set to 0 for all reviewers.
Agentic reviewers outperform static LLMs on overall performance—the percentage of comments satisfying both criteria simultaneously:
| Reviewer |
# Comments |
% Localized |
% Human Aligned |
Overall |
| Rovo Dev |
860 |
20.93% |
15.10% |
5.81% |
| Claude Code |
947 |
22.39% |
14.37% |
5.60% |
| Gemini CLI |
392 |
12.24% |
13.28% |
2.30% |
| CodeRabbit |
803 |
7.22% |
8.54% |
1.37% |
| GPT-5.2 |
3180 |
7.86% |
21.44% |
2.61% |
| GPT-5.4 |
2305 |
9.15% |
17.39% |
2.43% |
| Gemini 2.5 Pro |
3166 |
7.96% |
18.71% |
2.37% |
| Claude Sonnet 4.5 |
9060 |
7.70% |
12.68% |
1.72% |
| Claude Sonnet 4.6 |
4823 |
7.26% |
14.65% |
1.87% |
The advantage of Rovo Dev and Claude Code is driven almost entirely by localization (>20% versus 7–9% for all others), achieved while generating far fewer comments than static baselines (e.g., 9060 from Sonnet 4.5). Notably, static models achieve higher human-alignment rates than agentic ones; repository grounding improves precision of placement rather than issue identification per se. The authors caution that absolute scores are low because both strict criteria must hold concurrently, and agents may surface valid issues absent from the benchmark ground truth, which register as failures.
Trajectory behavior and cost (RQ1–RQ3)
Effort allocation differs sharply across agents. Rovo Dev is validation-driven (41% of steps, 46% of tool calls and cost devoted to validation); Claude Code is exploration-centric (58% of steps, 73% of tool calls, 51% of cost on exploration, with only 11–14% validation); Gemini CLI is balanced and lightweight. Across all agents, planning consistently consumes less than 10% of effort despite the open-ended nature of the task.
Tool-use analysis shows heavy reliance on general-purpose shell commands (cd, git, grep) in Rovo Dev and Claude Code, suggesting agents default to low-level execution over structured tooling—a pattern the authors associate with higher exploration overhead. Gemini CLI concentrates on a small set of specialized file-read and git operations.
Cost asymmetries are pronounced. Rovo Dev averages 7.4 validation and 5.5 exploration steps per task, incurring $0.75 in validation and $0.44 in exploration costs with over ten validation tool calls per task. Claude Code matches the exploration step count (7.5) but costs only $0.10 there, indicating materially more efficient retrieval. Gemini CLI operates at ≤$0.007 per phase. Exploration and validation dominate expenditure; planning is near-zero cost. The implication drawn is direct: cost does not imply performance, motivating cost-normalized evaluation metrics.
What distinguishes successful trajectories (RQ4)
Comparing accomplished against unaccomplished reviews (excluding partial cases), successful trajectories allocate significantly more effort to planning for Rovo Dev and Gemini CLI (p<0.05, Mann–Whitney U test). Generation effort differences are not statistically significant. Conversely, unsuccessful trajectories show heavier validation effort—largely re-checking context (reopening diffs and files) without improving outcomes. The paper's central behavioral claim follows: success is associated with stronger upfront deliberation, whereas excessive validation signals weak initial grounding that re-checking cannot recover.
Limitations and open questions
The authors concede several constraints. Phase labels derive from keyword heuristics that may misattribute latent intent (the same text can serve different purposes), though labels were manually validated. Human alignment relies on an LLM-as-a-Judge, which may introduce noise despite using an evaluated judge. Comments identifying issues outside the ground truth are scored as failures, potentially understating agent capability. Findings may not generalize to future agent versions, and CodeRabbit's undisclosed backend model and non-exposed trajectories limit comparability. Open questions include whether explicit planning interventions causally improve review success, how to design cost-normalized metrics that account for the observed two-orders-of-magnitude cost spread between comparable performers, and whether improved repository navigation strategies can close the localization gap without inflating exploration cost.
Conclusion
This trajectory-level study establishes that repository-grounded exploration is what separates agentic from static code review—specifically through localization—and that planning, the least-invested phase (<10% of effort), is the one most strongly associated with review success, while excessive validation correlates with failure. The released AgenticCR-Verified dataset and replication package provide infrastructure for trajectory-aware, cost-sensitive evaluation of future agentic reviewers.