CoCoReviewBench: AI Reviewer Benchmark

• CoCoReviewBench is a benchmark that evaluates AI reviewers by measuring both the completeness and correctness of review outputs on curated scientific papers.
• It organizes review content into a taxonomy with 5 top-level categories and 23 subcategories for detailed, category-specific analysis.
• The benchmark filters atomic opinions via reviewer–author conflict analysis and meta-review adjudication to improve reliability and factual grounding.

CoCoReviewBench is a completeness- and correctness-oriented benchmark for evaluating AI reviewers on scientific papers. It was introduced to address a specific failure mode of prior review evaluation: human reviews are neither exhaustive nor uniformly reliable, so overlap with raw human reviews is a weak proxy for whether an AI critique is actually correct, well grounded, or comprehensive. The benchmark curates 3,900 papers from ICLR and NeurIPS, organizes review content into a shared taxonomy of 5 top-level categories and 23 subcategories, and filters reference opinions through reviewer–author–meta-review discussion traces to strengthen correctness (Deng et al., 8 May 2026).

1. Motivation and evaluation philosophy

The benchmark is built around two claims. First, peer reviews are incomplete: a reviewer often comments on only a subset of salient issues in a paper. Second, peer reviews can themselves be wrong, especially when reviewers disagree with one another or when authors rebut specific claims. CoCoReviewBench therefore treats Completeness and Correctness as separate benchmark design problems rather than assuming that all human review text is a gold reference (Deng et al., 8 May 2026).

The incompleteness evidence reported in the benchmark is explicit. A single reviewer covers only 3.03 / 5 top-level categories on average and 5.10 / 23 subcategories on average. Aggregating all reviewers improves this to 3.98 / 5 top-level categories and 9.23 / 23 subcategories, which the paper describes as an 81% improvement in subcategory coverage over a single reviewer. Even after aggregation, average subcategory coverage remains only about 40%. This is the core reason the benchmark does not score AI reviews against an undifferentiated union of raw human comments.

The correctness problem is quantified through disagreement analysis. More than 13% of submissions have an overall-score gap of 4 or more between the highest and lowest reviewer scores. Using inter-reviewer conflicts together with meta-review adjudication, the benchmark identifies incorrectness in 22.13% of papers and 7.63% of reviews. Using reviewer–author conflicts together with meta-review adjudication, it identifies incorrectness in 75.72% of papers and 36.76% of reviews. These numbers motivate the paper’s rejection of raw human-review overlap as a sufficient evaluation target.

A direct implication is the benchmark’s critique of conventional metrics such as BLEU, ROUGE-L, and BERTScore. In this setting, those metrics reward surface or semantic overlap with human reviews, not whether the AI comment is factually correct, well justified, or useful. This suggests that reviewer evaluation should be structured around issue coverage and filtered reference quality rather than imitation of reviewer phrasing.

2. Corpus, taxonomy, and subset construction

CoCoReviewBench is constructed from NeurIPS 2021–2024 and ICLR 2017–2025 review data. The benchmark samples 300 papers per year, yielding 3,900 papers in total. Each paper must have at least three independent reviewers, and at least 75% of reviews have author responses. The benchmark also balances scores by stratified sampling after linearly mapping scores to $[0,10]$ and partitioning with thresholds 3 / 5 / 7. For the paper text used at evaluation time, the authors verify that the PDF predates the reviews; when an earlier OpenReview PDF is unavailable, they retrieve an arXiv version from within nine months before review release (Deng et al., 8 May 2026).

The released benchmark statistics are:

• 3,900 papers
• 14.1k review comments
• 134.8k opinions
• 115.9k opinion clusters
• 108.6k correct opinions as references

The benchmark defines a shared two-level taxonomy with 5 top-level categories and 23 subcategories. The top-level categories are:

1. Quality
2. Clarity
3. Significance
4. Originality
5. Policy

The paper explicitly mentions subcategories such as Originality-Method, Originality-Analysis, Quality-Experiment, Quality-Comparisons, Clarity-Writing, and Significance-StateOfTheArt. The taxonomy is derived from the NeurIPS 2025 Reviewer Guidelines and the ACL Rolling Review Review Form.

The benchmark’s completeness mechanism is category-specific subset formation. For a given category or subcategory, a paper is included in that sub-benchmark only if the human reviews contain comments in that category. If the category is absent from the human references for that paper, evaluation is skipped for that category. This avoids penalizing an AI reviewer for raising a potentially valid issue in a category that the human panel simply did not discuss.

This design makes CoCoReviewBench a collection of category-conditioned sub-benchmarks rather than a single flat review-matching dataset. A plausible implication is that the benchmark is intended to evaluate review quality within supported issue types, while measuring cross-category breadth separately through Completeness.

3. Construction pipeline and correctness filtering

The construction pipeline has two major parts: completeness-aware classification and correctness-aware filtering. The starting point is not whole reviews as atomic units, but atomic opinions extracted from review and rebuttal text (Deng et al., 8 May 2026).

The initial segmentation stage is rule-based. It uses:

• newlines
• semantic transitions such as “However,” or “Furthermore”
• minimum and maximum token-length constraints

An LLM then assigns sentence-level labels indicating which sentences belong to the same atomic opinion and links the associated author response. Sentences with the same label are aggregated into atomic discussion records. Each atomic opinion is then classified into one of the 23 subcategories.

Because LLM segmentation can merge multiple issues, the pipeline adds a secondary segmentation stage. When an atomic opinion receives multiple categories, the system applies finer-grained rule-based splitting, a smaller maximum length, LLM reclassification, and splitting at category-change boundaries. This is intended to improve atomicity before reference construction.

Correctness filtering uses the discussion process itself as supervision. Within each subcategory, the pipeline groups discussions about the same topic, checks whether grouped opinions conflict, and uses the meta-review to determine which opinion is more reliable. Redundant opinions are removed, and if multiple correct opinions target the same topic, only the longest correct opinion is retained as the representative reference because it usually contains the fullest justification.

The same logic is applied to reviewer–author exchanges. A reviewer opinion and its corresponding author response are paired; if an explicit conflict exists, the meta-review is used as a coarse adjudication signal. Opinions judged incorrect are excluded from the reference set. The paper is careful to note that this does not produce a perfect oracle; it only aims to be more reliable than raw unfiltered review text or no-reference LLM judging.

The benchmark also reports several validation results for this pipeline. At each construction step, the authors compare six strong LLMs and use leave-one-out cross-validation. They then conduct human verification on 50 papers using ML PhD students. Reported accuracies are:

• Review classification: 85.45% have entirely correct categories
• Opinion aggregation: 93.41% correct
• Inter-reviewer conflict error detection: 81.40%
• Reviewer-author conflict error detection: 66.83%

Reviewer–author conflict adjudication is harder, especially for rejected papers. The reported accuracy is 50.03% on rejected papers and 79.76% on accepted papers. This suggests that meta-reviews provide only coarse point-level supervision, particularly when final decisions are negative.

4. Task formulation and evaluation protocol

In the benchmark task, an AI reviewer is given a paper and produces a review text. The generated review is then post-processed into atomic opinions and subcategory labels before scoring. To reduce evaluation cost, CoCoReviewBench trains two distilled models from Qwen3-8B: ReviewSplit for segmentation and ReviewClassify for subcategory classification (Deng et al., 8 May 2026).

ReviewSplit is trained with GRPO using the reward

$R=\max(0.5, \mathrm{OmegaIndex}+\mathbbm{1}(\mathrm{Correct\ Format})).$

The training uses 32 trajectories per instance. ReviewClassify is trained with instruction tuning rather than GRPO, and under human annotation it achieves entirely correct classifications in 87.09% of reviews.

Once AI reviews are segmented and classified, category-level evaluation is performed with an LLM judge. The judge is GPT-5-Mini with temperature 0. For each applicable category, it assigns 1–5 scores on five dimensions:

1. Correctness
2. Thoroughness
3. Grounding
4. Verifiability
5. Clarity

These dimensions separate whether a comment is right, whether it covers the category deeply, whether it points to a specific part of the paper, whether it can be checked from reasoning or references, and whether it is coherently written.

Category-level evaluation is performed only when both the human and AI provide comments in that category. This restriction keeps category scoring focused on comment quality rather than breadth. Breadth is measured separately by Completeness. The paper does not provide a closed-form equation for Completeness, but it defines the procedure precisely: for each paper, the union of categories covered by all human reviewers is treated as the full set with score 100; the AI score is the ratio between the average number of categories covered by the AI review and the number of categories covered by the full human panel, averaged over papers.

The benchmark also defines a paper-level score. Here, all categories’ AI comments and all human references are fed jointly to the same judging prompt, which outputs a single paper-level score. Human performance is estimated with leave-one-out validation: one human review is treated as the candidate, and the remaining reviews are treated as references.

For the experiments in the paper, the authors evaluate on one third of the 3,900-paper dataset, i.e. 100 papers per year, while preserving the original score distribution. They also report the old metrics—BLEU, ROUGE-L, and BERTScore with RoBERTa-large—by taking the maximum score against any reviewer, to show how differently those metrics rank systems.

5. Empirical findings

The benchmark’s human leave-one-out baseline is:

• Correctness: 3.55
• Thoroughness: 2.37
• Grounding: 3.75
• Verifiability: 2.38
• Clarity: 4.15
• Average: 3.26
• Paper-level: 3.61
• Completeness: 55.66

The strongest systems are the closed-source frontier models. GPT-5.2 achieves 3.91 Correctness, 3.01 Thoroughness, 4.67 Grounding, 3.16 Verifiability, 4.47 Clarity, 3.85 Average, 4.51 paper-level score, and 84.49 Completeness. GPT-5-Mini achieves 3.84, 2.95, 4.52, 2.91, 4.46, 3.74, 4.42, and 89.97, respectively. Gemini-3-Pro reaches 3.61 Average and 4.11 paper-level with 67.69 Completeness. The paper states that these frontier systems exceed the human leave-one-out baseline on most benchmark dimensions (Deng et al., 8 May 2026).

Among open-source reasoning models, Qwen3-32B reaches 3.40 Average, 3.82 paper-level, and 73.17 Completeness, while QwQ-32B reaches 3.45, 3.98, and 79.83. Among specialized reasoning reviewers, DeepReviewer-14B reaches 3.46 Average, 3.97 paper-level, and 81.98 Completeness. At the low end, SEA-E-7B records 3.02 Correctness, 1.93 Thoroughness, 2.13 Grounding, 1.27 Verifiability, 3.77 Clarity, 2.47 Average, 2.28 paper-level, and 51.54 Completeness.

A central conclusion is that many AI reviewers are clearer but less correct and less thorough than humans. Clarity is often above the human baseline of 4.15, whereas Correctness and Thoroughness are substantially harder. The benchmark also finds that reasoning models are better grounded and more verifiable. For example, GPT-5.2 and GPT-5-Mini substantially exceed the human baseline on Grounding and Verifiability, and reasoning-oriented open models narrow the gap more on these dimensions than on Correctness.

The Completeness metric yields a different picture from the category-quality metrics. Human leave-one-out Completeness is 55.66, but many AI systems exceed it: GPT-5.2 reaches 84.49, GPT-5-Mini 89.97, QwQ-32B 79.83, and DeepReviewer-14B 81.98. The paper interprets this as evidence that AI reviews are often broader but shallower: they cover more categories than a single human review, but they do not match the aggregate human panel and often lack depth within the categories they touch.

Category-wise analysis shows that AI reviewers are strongest on Quality. Originality and Significance are mixed but not dramatically worse. The main weaknesses are Clarity and especially Policy. The paper says that Quality is already close to, or slightly above, human performance on average, while Policy is the clearest lagging dimension in both Correctness and Thoroughness.

The benchmark is also designed to expose hallucinations. In the text-only review setting, models still generate figure-related opinions, even though they do not see images. The rate is low—fewer than 0.05 opinions per paper—but it occurs in every model. The paper treats this as a concrete hallucination signal. It also reports that if incorrect human opinions are treated as references, the human–AI correctness gap narrows substantially, which suggests that AI reviewers can learn to reproduce erroneous human reviewing patterns rather than independently correct them.

6. Relation to adjacent benchmarks, limitations, and significance

CoCoReviewBench occupies a distinct position within the emerging literature on AI review evaluation. Unlike “Beyond Rating” (Li et al., 21 Apr 2026), which evaluates generated reviews through Content Faithfulness, Argumentative Alignment, Focus Consistency/Alignment, Question Constructiveness, and AI-Likelihood, CoCoReviewBench centers its design on category-specific completeness and discussion-based correctness filtering. Unlike “Benchmarking Agentic Review Systems” (Nguyen et al., 18 Jun 2026), which evaluates full review systems through quality-proxy correlation and perturbation-based error detection, CoCoReviewBench evaluates generated review text against filtered human reference opinions across category-level dimensions such as Correctness, Thoroughness, Grounding, Verifiability, and Clarity. This suggests that CoCoReviewBench is primarily a benchmark for review-content quality under incomplete and noisy human supervision, rather than for score prediction alone or synthetic error-catching alone.

The benchmark’s main limitations are explicit. Conflict-based filtering can only detect errors that become visible through explicit disagreements. If there is no author response, if the response is too mild, or if an incorrect review point goes uncontested, the benchmark may miss that error. Meta-reviews are also only coarse adjudicators; they often lack point-by-point justifications, especially for rejected papers. This is consistent with the lower reviewer–author conflict validation accuracy.

There is also computational overhead. Evaluation requires segmentation and category classification of AI reviews before judging, although the distilled 8B models are intended to mitigate the cost. Coverage is limited to ICLR and NeurIPS, so the benchmark reflects OpenReview-style machine learning review norms rather than a universal model of peer review. The paper therefore does not claim generality across all academic disciplines.

Even with these caveats, CoCoReviewBench contributes a clear methodological shift. It reframes reviewer evaluation around two benchmark-design primitives: do not penalize valid critique for missing human coverage, and do not treat all human critique as equally trustworthy. Its practical implication is that AI reviewer research should optimize not merely for overlap with reviewer wording, but for category-aware correctness, depth, grounding, and verifiability. The benchmark and associated models are released at https://github.com/hexuandeng/CoCoReviewBench (Deng et al., 8 May 2026).