OpenAIReview: Agentic Paper Review System

• OpenAIReview is an open-source agentic paper review system that processes academic papers passage by passage using persistent context.
• The system employs sequential reviews with running summaries to consolidate detailed, passage-grounded comments on technical and logical aspects.
• Benchmark evaluations reveal high pairwise accuracy (0.83) and robust error detection, underscoring its advanced review capabilities.

Searching arXiv for the OpenAIReview benchmark paper and closely related peer-review systems/benchmarks. arXiv query: OpenAIReview benchmarking agentic review systems (Nguyen et al., 18 Jun 2026) OpenAIReview is an open-source agentic academic paper review system that takes a full paper as input and returns an overall feedback paragraph together with a list of passage-grounded comments, where each comment includes a quoted passage and an explanation of the issue. In the benchmark study that made it a focal object of comparison, OpenAIReview is characterized as a sequential, context-maintaining harness rather than a raw single-prompt reviewer: it reviews a paper passage by passage, conditions each review on neighboring passages plus a running summary of the paper so far, and then deduplicates and consolidates comments into a final review artifact (Nguyen et al., 18 Jun 2026).

1. Emergence within AI-assisted peer review

OpenAIReview belongs to a newer class of agentic review systems developed in response to pressure on conference peer review from rising submission volumes and AI-assisted research. In the benchmark that evaluates it, it is compared directly against a zero-shot baseline, an open-source multi-agent reviewer called [coarse](https://www.emergentmind.com/topics/coarse), and a proprietary system called Reviewer3, with the stated goal of evaluating systems that authors can actually run today rather than only isolated prompting strategies (Nguyen et al., 18 Jun 2026).

Its significance is best understood against the broader trajectory of automated reviewing. A pilot study on GPT-4 found mean author-rated helpfulness of 3.0 for GPT reviews versus 3.1 for human reviews, but also greater variance in GPT review quality, suggesting early usefulness but limited consistency (Robertson, 2023). Subsequent work moved toward more structured evaluation: ReviewEval proposed dimensions such as alignment with expert reviews, factual accuracy, analytical depth, constructiveness, and adherence to reviewer guidelines, arguing that review quality cannot be reduced to stylistic resemblance (Garg et al., 17 Feb 2025). CoCoReviewBench pushed this further by emphasizing completeness and correctness, noting that human reviews are incomplete and sometimes wrong, and therefore poor gold references when used naively (Deng et al., 8 May 2026).

This broader literature also clarifies why OpenAIReview is evaluated as a full system rather than only as a text generator. A large-scale analysis of 1,683 papers and 6,495 expert reviews found that current LLMs are relatively strong on descriptive and affirmational content such as summaries and strengths, but consistently underperform in identifying weaknesses, raising substantive questions, and adjusting feedback based on paper quality (Li et al., 13 Sep 2025). OpenAIReview therefore occupies a specific position in the field: it is not merely another model asked to “review this paper,” but an attempt to improve review behavior through system design.

2. System architecture and operating procedure

OpenAIReview’s central design choice is sequential passage-level reviewing with persistent context. Instead of placing an entire paper into a single monolithic prompt, it splits the document on paragraph boundaries and greedily merges adjacent paragraphs into passages of up to roughly 8,000 characters. Each passage is reviewed with access not only to the passage itself, but also to five preceding passages and two following passages, an asymmetry intended to weight earlier technical context more heavily because definitions and notation often appear earlier in the manuscript (Nguyen et al., 18 Jun 2026).

A second key component is the running summary. After each passage-level review call, a separate model call updates a summary whose token budget is $\max(4{,}000,\, T/10)$, where $T$ is document length in tokens. This summary stores notation and definitions, key equations, theorems or propositions, assumptions, and key claims. The summary is then carried forward to later passage reviews, allowing the system to accumulate document-level state while keeping each review call local enough to remain manageable (Nguyen et al., 18 Jun 2026).

The review prompt is explicitly oriented toward substantive scientific issues. It asks the model to check for mathematical or formula errors, notation inconsistencies, inconsistency between prose and formal definitions, parameter or numerical inconsistencies, insufficient justification, questionable claims, ambiguity that could mislead, and underspecified methods. It also instructs the model to first try to understand the authors’ intent, check whether a concern is resolved by context before flagging it, be lenient with introductory or overview sections, avoid forward-reference false positives, and not flag formatting, typesetting, capitalization, trivial observations, or incomplete boundary text. Passage-level outputs are required to be a JSON array with fields "title", "quote", "explanation", and "type", where type is "technical" or "logical" (Nguyen et al., 18 Jun 2026).

After passage-level analysis, OpenAIReview serializes all comments and makes a consolidation call that removes duplicates, merges closely related issues, drops comments that merely restate standard conventions, and re-anchors surviving comments to source passages by quote matching. The final output consists of an overall feedback paragraph and a set of quoted comments. The system accepts papers in PDF, markdown, LaTeX, and other formats. For PDFs it uses a fallback parser chain of Mistral OCR, DeepSeek OCR, Marker, and pymupdf4llm, जबकि LaTeX, Word, and arXiv HTML bypass OCR (Nguyen et al., 18 Jun 2026).

Component	Mechanism	Function
Segmentation	Paragraph splitting with greedy merging to roughly 8,000 characters	Defines review passages
Local context	Five preceding and two following passages	Preserves nearby technical context
Global context	Running summary with budget $\max(4{,}000,\, T/10)$	Carries forward definitions, equations, assumptions, and claims
Consolidation	Post-hoc deduplication and merging	Produces final coherent comment set

This architecture suggests that OpenAIReview is best viewed as a review harness over frontier or efficient LLM backends rather than as a fixed reviewer model. The benchmark study confirms this interpretation by sweeping the same harness across six model backends (Nguyen et al., 18 Jun 2026).

3. Benchmarking methodology

OpenAIReview is evaluated on two main benchmarks, each targeting a different property of review systems. The first asks whether review behavior tracks external proxies of paper quality. The second asks whether the system can detect injected errors with known ground truth. Together they distinguish correlation with quality signals from direct error-finding ability (Nguyen et al., 18 Jun 2026).

For the quality-tracking benchmark, the data come from SNOR, which links OpenReview submissions for ICLR 2017–2025 and NeurIPS 2021–2025 to Semantic Scholar citation counts, decisions, and reviewer scores. The main evaluation restricts to ICLR and NeurIPS 2021–2022 papers with at least three official reviews and a non-null average score. Four proxy tasks are defined, each with 30 high-quality and 30 low-quality papers: community-level, conference-level, reviewer-level, and a composite proxy. On the main frontier subset this yields 74 papers, and each paper is reviewed on the first 10 pages by every relevant method-model pair (Nguyen et al., 18 Jun 2026).

The principal metric is pairwise accuracy: the percentage of low- and high-quality paper pairs where the lower-quality paper receives more comments, with ties counting as 0.5. Formally,

$$\mathrm{Acc} = \frac{1}{|L||H|} \sum_{\ell \in L}\sum_{h \in H} \left[ \mathbf{1}(c_\ell > c_h) + 0.5\,\mathbf{1}(c_\ell = c_h) \right],$$

where $L$ and $H$ are the low- and high-quality sets and $c_\ell, c_h$ are comment counts. The study also reports $\Delta = \bar{c}_{\rm low} - \bar{c}_{\rm high}$ and notes that pairwise accuracy is equivalent to the Mann–Whitney AUC. Confidence intervals are obtained by a nonparametric cluster bootstrap over papers with $B = 5000$ resamples (Nguyen et al., 18 Jun 2026).

The perturbation benchmark is more direct. It starts from 74 arXiv papers spanning eight subject classes—Computational Complexity, Machine Learning, Econometrics, Experimental High-Energy Physics, Mathematics, Atomic and Cluster Physics, Genomics, and Applied Statistics—and injects four categories of errors: surface, claim, logic or reasoning, and experimental. The perturbation pipeline extracts candidate spans from LaTeX, generates candidate perturbations with Gemini-3 Flash Preview, validates them structurally, verifies them with Claude Sonnet 4.6, and injects surviving edits into the source. The final benchmark contains 3,365 injected edits: 1,042 surface, 1,123 claim, 221 reasoning, and 979 experimental perturbations (Nguyen et al., 18 Jun 2026).

Detection uses a two-stage procedure. First, a quote-match stage requires approximate containment between the perturbed text and the review comment’s quote, with default threshold $\tau = 0.75$. Second, Gemini-3 Flash Preview judges whether the explanation identifies the same error on a 1–5 scale, and a score of at least 3 counts as a detection. Recall can therefore be written as

$T$0

where a perturbation counts as detected if any emitted comment passes both localization and semantic-match stages. Confidence intervals again come from a paper-cluster bootstrap with 5,000 resamples (Nguyen et al., 18 Jun 2026).

The open systems—zero-shot, OpenAIReview, and coarse—are tested across six backends: GPT-5.5, Claude Opus 4.7, DeepSeek-V4-Flash, Qwen3.6-35B-A3B, Gemini-3.1-Flash-Lite, and Grok-4.1-Fast, all accessed via OpenRouter without reasoning mode. Reviewer3 is treated as a fixed closed system rather than a backend-swept harness (Nguyen et al., 18 Jun 2026).

4. Empirical performance

OpenAIReview is the strongest system overall in the benchmark study when paired with GPT-5.5. On the quality-tracking benchmark, OpenAIReview + GPT-5.5 achieves pairwise accuracy of 0.83 with 95% confidence interval $T$1, with mean comments per paper $T$2. Its proxy-specific accuracies are 0.87 for community, 0.66 for conference, 0.94 for reviewer, and 0.84 for composite quality, each with the corresponding reported confidence intervals (Nguyen et al., 18 Jun 2026).

The same configuration also leads the perturbation benchmark. OpenAIReview + GPT-5.5 reaches 71.6% recall on the 24-paper frontier subset, with 95% confidence interval $T$3. By error type, it achieves 85.2% recall on experimental errors, 83.3% on claim errors, 70.0% on reasoning errors, and 47.3% on surface errors. The distribution of gains is important: OpenAIReview’s largest advantages over zero-shot appear on prose-level errors rather than local math edits, which is consistent with its running-summary design (Nguyen et al., 18 Jun 2026).

Evaluation	Best OpenAIReview configuration	Result
Quality tracking	OpenAIReview + GPT-5.5	0.83 pairwise accuracy, 95% CI $T$4
Perturbation benchmark	OpenAIReview + GPT-5.5	71.6% recall, 95% CI $T$5
Multi-model union under OpenAIReview	Union across six models	83.3% recall, 95% CI $T$6

Comparison against the other systems clarifies the size of the margin. On the frontier quality-tracking subset, the strongest alternative pairs are zero-shot + DeepSeek-V4-Flash at 0.80, Reviewer3 at 0.80, and coarse + Grok-4.1-Fast at 0.66. On the perturbation benchmark, zero-shot + GPT-5.5 reaches 59.8%, Reviewer3 26.5%, and the best coarse configuration 20.7%, all below OpenAIReview + GPT-5.5 (Nguyen et al., 18 Jun 2026).

OpenAIReview also outperforms zero-shot on every shared backend for perturbation recall: GPT-5.5 by +11.8 points, Claude Opus 4.7 by +13.8, Grok-4.1-Fast by +21.2, DeepSeek-V4-Flash by +24.8, Qwen3.6-35B-A3B by +14.0, and Gemini-3.1-Flash-Lite by +18.3. This suggests that the harness contributes nontrivially beyond backend quality alone (Nguyen et al., 18 Jun 2026).

The system is not uniformly best in every domain. On the 24-paper frontier subset it wins every domain except Mathematics, where zero-shot + GPT-5.5 reaches 79.3% recall while OpenAIReview + GPT-5.5 reaches 62.1%. Computational Complexity is the hardest domain overall, with OpenAIReview scoring 45.7% on the frontier subset and 32.7% on the full efficient-only benchmark (Nguyen et al., 18 Jun 2026).

A further finding is complementarity across backends. The union of detections across all six model backends under OpenAIReview reaches 83.3% recall, which is 11.7 points above the best single model, with bootstrap improvement confidence interval $T$7. The study also reports low paragraph-overlap across models, supporting the view that different backends detect different classes of issues (Nguyen et al., 18 Jun 2026).

5. Deployment behavior, failure modes, and limitations

Beyond offline benchmarking, OpenAIReview was deployed publicly using Claude Opus 4.6 as backend. The deployment dataset contains 1,360 completed reviews on 1,100 distinct papers. Across 27,587 comments shown, only 690 comments received a vote, corresponding to a 2.5% engagement rate. Of those, 407 were upvotes and 283 were downvotes, yielding the reported positive skew of 1.44 to 1. The system also recorded 1,348 resolved comments, or 4.9% of all comments shown. Among the 109 papers with at least one resolved comment, an average of 50% of comments were resolved (Nguyen et al., 18 Jun 2026).

The deployment feedback identifies the dominant error profile. Among 283 downvoted comments, 109 were categorized as false positives, 49 as trivial nitpicks, 43 as unreasonable asks for detail, 18 as parsing or OCR artifacts, 63 as correct but dismissed by the author, and 1 as unclear. The paper therefore characterizes the main deployment weakness as precision rather than recall. This is consistent with the benchmark structure: the perturbation benchmark measures recall, not precision, and the authors explicitly note that a system can emit many false positives while still scoring well on recall (Nguyen et al., 18 Jun 2026).

Several benchmark limitations matter for interpreting OpenAIReview’s status. First, the quality-tracking benchmark uses comment count as a coarse proxy; a concise review that identifies one fatal flaw could be undervalued by this metric. Second, the perturbation benchmark is generated and filtered with LLM assistance, which may bias the injected errors toward mistakes that are salient to LLMs. Third, even the best OpenAIReview configuration still misses a substantial fraction of injected errors: 71.6% recall implies that 28.4% remain uncaught on the frontier subset. Surface math errors are especially difficult, with only 47.3% recall for OpenAIReview + GPT-5.5 (Nguyen et al., 18 Jun 2026).

These failure modes align with concerns raised elsewhere in the AI-review literature. Work on presentation-only review gaming shows that AI reviewers can be manipulated by visible, policy-compliant reframing that leaves core evidence fixed, and it specifically identifies structured outputs such as strengths, weaknesses, and ratings as likely attack surfaces (Yang et al., 11 Jun 2026). This suggests that OpenAIReview-style feedback, while useful, can also become an optimization target if authors iteratively rewrite against it. Likewise, CoCoReviewBench reports that AI reviewers remain limited in correctness and are prone to hallucinations despite strong completeness, reinforcing that helpful-looking review structure is not equivalent to reliable scientific judgment (Deng et al., 8 May 2026).

6. Position within the broader review-system ecosystem

OpenAIReview is best situated as one design point within a rapidly diversifying ecosystem of AI-assisted scholarly review systems. OpenReviewer is an end-to-end review generator built around a specialized Llama-3.1-8B-Instruct model fine-tuned on about 79,000 high-confidence expert reviews from ICLR and NeurIPS, and is primarily positioned as a pre-submission feedback tool for authors rather than a replacement for human peer review (Idahl et al., 2024). By contrast, OpenAIReview is benchmarked as a general review harness that can be paired with multiple frontier and efficient backends and evaluated passage by passage rather than only as a monolithic paper-to-review generator (Nguyen et al., 18 Jun 2026).

A different branch of the literature shifts attention away from comment generation and toward scalar scoring. AIPR validates a prompt-only first-pass manuscript scorer against public ICLR outcomes, reporting AUROC 0.82 on a 300-paper cohort and emphasizing that the main deployment object may be a numeric triage signal rather than prose quality (Georgantas, 14 Jun 2026). ReviewGuard, in another sense, aligns LLM-assisted peer review with long-term scientific impact rather than contemporaneous reviewer preference, reaching Spearman correlation $T$8 with future citations on rejected-then-published papers and flagging 10.2% of high-impact rejected papers versus 1.8% for human reviewers (Rasool et al., 29 May 2026). These systems pursue adjacent but distinct objectives: OpenAIReview concentrates on issue-finding and review text; AIPR concentrates on first-pass scoring; impact-aligned ReviewGuard concentrates on rescue of overlooked papers.

There is also a governance-oriented branch that evaluates the reviews themselves rather than the papers. Another ReviewGuard introduces a four-stage framework for detecting deficient peer reviews using OpenReview data, GPT-4.1 annotation with human validation, synthetic augmentation, and fine-tuning, reporting that deficient reviews are shorter, simpler, more negative, lower-rated, and slightly more confident than sufficient reviews (Zhang et al., 18 Oct 2025). In that context, OpenAIReview can be seen as part of a broader transition from using LLMs merely to generate reviews toward embedding them in review-assistance, review-audit, and decision-support infrastructure.

The most plausible long-term role of OpenAIReview is therefore neither autonomous peer reviewer nor simple prompt template. It is better understood as modular reviewer-assistance infrastructure: a system that can track paper quality above chance, detect many injected errors, surface passage-grounded concerns, and receive net-positive user feedback, while still requiring human oversight for precision, prioritization, and final judgment (Nguyen et al., 18 Jun 2026). This suggests a broader lesson for AI-assisted peer review: system design choices such as sequential context maintenance, passage grounding, and comment consolidation can materially change behavior, but they do not remove the need for external evaluation regimes that test correctness, completeness, robustness, and susceptibility to gaming.