Benchmarking Agentic Review Systems

Abstract: A new class of agentic review systems are emerging as a remedy to the pressure placed on peer review systems by AI-assisted research, but it is unclear how they should be evaluated. We evaluate two open-source systems (OpenAIReview and coarse), one proprietary system (Reviewer3), and a zero-shot baseline, across six LLMs spanning frontier and efficient models. First, we study whether AI reviews on ICLR/NeurIPS papers track with papers' quality as approximated by external signals such as citations and acceptance decisions. Every system performs above chance in pairwise accuracy, and the best is OpenAIReview + GPT-5.5 at 83.0%. Second, to test whether systems can catch errors with known ground truth, we construct a perturbation benchmark that injects four categories of errors into papers across eight arXiv subject classes and measure detection recall. The strongest configuration (OpenAIReview + GPT-5.5) catches 71.6% of injected errors, leaving substantial room for improvement. The union of detections across six models reaches 83.3% recall, suggesting different models detect different errors and better harness design can potentially increase performance. Beyond these benchmarks, we study a public deployment of OpenAIReview with real users. Votes on its comments skew positive at 1.44 to 1, and the most common complaints are about false positives and minor nitpicks. Together, by evaluating full review systems backed by state-of-the-art models on real research papers, we show that while AI reviews still have room for improvement, they can already track human quality judgments well, catch important errors, and earn positive feedback from real users.

• The paper demonstrates that agentic review systems, such as OpenAIReview with GPT-5.5, achieve up to 0.83 pairwise accuracy in correlating review outputs with paper quality.
• It utilizes a perturbation benchmark to show that these systems detect 71.6% of injected errors overall, reaching 85% on experimental mistakes.
• The results highlight high complementarity among different review models while emphasizing precision calibration as a key challenge for future improvement.

Authoritative Summary of "Benchmarking Agentic Review Systems" (2606.19749)

Introduction and Context

The paper addresses the evaluation of agentic review systems—LLM-driven frameworks for automated peer review—amid increasing pressure on the conventional peer review pipeline imposed by AI-assisted and large-scale submissions, especially at top venues such as ICLR and NeurIPS. The significance of these systems is heightened by the risk of the so-called "review death spiral": as submissions balloon and human reviewer capacity saturates, review quality degrades and the system risks collapse. The central thesis is that improvements in review precision are necessary to stabilize peer review, and that agentic review systems—fully-pipelined, prompt-structured LLM frameworks for automatic review—offer a pragmatic intervention for this challenge.

System Selection and Benchmarking Paradigm

The paper empirically compares four systems: OpenAIReview, 'coarse, the proprietary Reviewer3, and a zero-shot baseline, each evaluated on six LLMs spanning GPT-5.5, Claude Opus 4.7, DeepSeek-V4-Flash, Qwen3.6-35B-A3B, Gemini-3.1-Flash-Lite, and Grok-4.1-Fast. The benchmarking is conducted on two axes:

1. Quality Signal Tracking: Assessment of whether review outputs (number and severity of comments) correlate with independent external proxies for paper quality (citations, acceptance decisions, review scores, composite metrics).
2. Perturbation-Based Error Recall: Quantitative evaluation using a large-scale perturbation set, where controlled errors—spanning surface mathematical errors, faulty claims, logical gaps, and experimental misinterpretations—are injected into papers from eight arXiv domains.

Additional ecological evaluation is performed using interaction logs and user feedback from a public deployment of OpenAIReview.

Experimental Design

Quality Proxy Correlation

The experimental design relies on SNOR, a dataset that links OpenReview records with Semantic Scholar metadata. For each of the four quality proxies, 30 high- and 30 low-quality papers are sampled per proxy. Systems process the first 10 pages of each paper, and pairwise accuracy is measured: the probability that a system surfaces more issues on a low-quality than a high-quality paper per sampled pair.

Perturbation Benchmark

The perturbation benchmark systematically injects errors, validated both automatically and via manual audit (confirming 82.5% validity), leveraging both LLM and deterministic rules. Detection of errors is measured as recall: fraction of injected perturbations surface-matched and explanation-matched by the reviewer model output.

Model Harnesses

OpenAIReview employs sequential passage-level review with a maintained running summary, which, along with local context, prompts the model to identify technical and logical issues. 'coarse uses a hierarchical, agentic multi-agent pipeline with both per-section and macro-overview agents, and Reviewer3 is assessed as a closed black-box.

Main Results

Quality Signal Tracking

All systems, across all LLMs, demonstrate statistically significant tracking of human-designated paper quality. The best pair, OpenAIReview + GPT-5.5, attains 0.83 pairwise accuracy (where chance is 0.5), with similar trends for Reviewer3 and the strongest zero-shot configuration. Importantly, model scale is critical—frontier models outperform efficient ones by large margins.

• Weaker papers attract more and more severe comments.
• This effect is consistent across models, proxies, and severity tiers, and does not require explicit training on acceptance or quality signals.

Perturbation-Based Recall

Error recall rates reinforce that system harness matters:

• OpenAIReview + GPT-5.5 detects 71.6% of injected errors overall, with up to 85% recall on experimental errors.
• The union of different model detections raises possible recall to 83.3%, indicating substantial model-level complementarity.
• 'coarse and Reviewer3 trail behind, reflecting their focus on editorial/prioritization rather than exhaustive enumeration.

Prose-level errors (claims, reasoning, experimental design) are detected at far higher rates relative to surface-level math perturbations, reflecting the importance of wide-context summarization and multi-passage reasoning.

Complementarity Analysis

Overlap analysis demonstrates high complementarity between systems and backends; human reviewers focus on paper-level novelty, scope, or methodological critiques, while agentic systems excel in enumerating formal, technical, and surface-level errors. Among AI models, different backends flag overlapping but still distinct sets of errors.

Public Deployment and Practical Value

From 27,587 comments generated for 1,100 papers, user feedback is predominantly positive (like/dislike ratio 1.44:1), with a significant share of comments being marked as resolved—indicating practical use. The main precision failures are false positives and trivial nitpicks, not recall. This aligns with the benchmarking outcomes and highlights calibration/precision as a key open problem.

Theoretical and Practical Implications

Implications for Peer Review

• Automated agentic reviewing can meaningfully augment human review workflows, providing both scale and coverage, and tracking human judgment signals robustly despite lack of explicit training (inductive emergence).
• Model choice and harness (system design) are critical levers; harnesses enable better recall (and detailed error localization), while aggregated multi-backend analysis suggests ensemble-based harnesses can further improve coverage.
• Precision remains a bottleneck: practical deployments reveal unhelpful or nitpicking output, emphasizing the value of refining prompting/instruction and perhaps adopting reviewer calibration.

Potential for Systemic Impact

Near-term, agentic reviewing offers scalability for overloaded conferences, and deployable systems (OpenAIReview, 'coarse) already perform robustly out-of-the-box. As LLMs progress, recall will likely approach human-level for most technical dimensions, with the humans retaining primacy in holistic, paper-level evaluation (novelty, methodological innovation, scientific positioning).

Long-term, the convergence of model complementarity, dynamic prompt design, and agentic workflows could yield review pipelines that reliably surface technical flaws, shifting human reviewer focus towards evaluation, synthesis, and higher-level critique. Integration of adaptive system-level ensembles (combining diverse LLMs and harnesses) is pointed out as a high-potential research direction.

Limitations and Prospects for Future Work

The perturbation suite, while extensive, is constructed LLM-driven, which may bias error typology toward LLM-salient mistakes. Precision is weakly addressed (though deployment signals are consistent with known practical limitations). Further, review usefulness extends beyond defect identification to synthesis, prioritization, and calibration, which are not directly benchmarked. Finally, the evaluated systems, although representative, do not exhaust the design space—future benchmarks must accommodate more varied output formats and use cases (author-facing tools, area chair interfaces, etc.).

Conclusion

This work constitutes the most thorough system-level evaluation of agentic review pipelines to date. The empirical evidence demonstrates that these systems, especially with advanced LLMs and suitable harness design, reliably track external signals of paper quality, surface a substantial proportion of injected errors, and provide actionable feedback in practical deployments. Precision, harness customization, and ensemble methods are the primary axes for future improvement. This establishes a technical foundation for integrating agentic reviewing at scale in both academic and industrial research settings.