ReviewerToo: AI-Enhanced Peer Review

• ReviewerToo is a modular, AI-assisted framework that integrates multi-agent protocols with literature-backed evidence to enhance peer review.
• Its pipeline architecture employs distinct agents (e.g., LitLLM, Reviewer, Author, Metareviewer) to deliver structured, evidence-anchored evaluations.
• Empirical results on ICLR papers demonstrate that ReviewerToo nearly matches human review quality, promoting reproducible and fair outcomes.

ReviewerToo is a modular, AI-assisted peer-review framework designed to address the escalating demands and challenges of modern scientific publishing, particularly those related to review consistency, subjectivity, and scalability. It blends systematic LLM-based evaluation with structured multi-agent protocols to supplement human reviewers, supports granular experimentation, and offers a reproducible scaffold for hybrid, scalable, and fair peer review systems (Sahu et al., 9 Oct 2025).

1. Modular Pipeline Architecture

ReviewerToo conceptualizes peer review as a structured pipeline of composable “agents,” each corresponding to a critical review stage. This architecture enables fine-grained customization and controlled ablation:

• Literature Review Agent (LitLLM): Parses the manuscript, retrieves top-$k$ related works via Semantic Scholar APIs, applies a debate-inspired ranking protocol, and synthesizes a literature summary. This summary is consumed downstream by other agents as grounding evidence.
• Reviewer Agents (Persona Panel): Each agent receives (i) the manuscript, (ii) literature summary, and (iii) a role-specific prompt embedding official review rubrics and specialized “personas” (e.g., critical, permissive, empiricist, theorist). The agents issue structured evaluations comprising succinct summaries, explicit strengths/weaknesses, rubric-based scores, grounded justifications, and categorical recommendations (Oral, Spotlight, Poster, Reject, Desk Reject).
• Author Agent: Generates an evidence-driven rebuttal targeting all major critiques, referencing both reviewer comments and the literature summary. The rebuttal is archived independently for each review aggregation scenario.
• Metareviewer Agent: Aggregates reviewer opinions and author rebuttal. It performs stance summarization, consensus/bias detection, rebuttal impact quantification, automated fact-checking versus the literature, weighted evidence assignment, and produces a meta-review with a calibrated accept/reject recommendation.

This modularity enables integration or substitution of LLMs, humans, or additional logic in any subsystem, supporting both human-only, AI-only, or hybrid feedback loops. Each agent is stateless and operates on rich, structured input bundles (Sahu et al., 9 Oct 2025).

2. Formal Evaluation Metrics and Protocols

ReviewerToo rigorously quantifies both decision fidelity and review quality with metrics explicitly defined in LaTeX:

• Classification (macro-averaged by class):
  • Precision: $$\text{Precision} = \frac{1}{|C|}\sum_{c\in C}\frac{\text{TP}_c}{\text{TP}_c + \text{FP}_c}$$
  • Recall: $$\text{Recall} = \frac{1}{|C|}\sum_{c\in C} \frac{\text{TP}_c}{\text{TP}_c + \text{FN}_c}$$
  • F1: $$\frac{1}{|C|}\sum_{c\in C} \frac{2\,\text{Precision}_c\,\text{Recall}_c}{\text{Precision}_c + \text{Recall}_c}$$
  • Accuracy: $$\frac{\sum_c (\text{TP}_c + \text{TN}_c)}{\sum_c (\text{TP}_c + \text{TN}_c + \text{FP}_c + \text{FN}_c)}$$
• Consistency: Cohen’s $\kappa = \frac{p_o - p_e}{1 - p_e}$, where $p_o$ = observed, $p_e$ = expected (chance) agreement.
• LLM-As-Judge Quality Measures: ELO-based ratings, with per-pair outcomes $S_A \in \{0, 0.5, 1\}$ and updates via $R'_A = R_A + K (S_A - E_A)$, $$\text{Precision} = \frac{1}{|C|}\sum_{c\in C}\frac{\text{TP}_c}{\text{TP}_c + \text{FP}_c}$$0 and annealing $$\text{Precision} = \frac{1}{|C|}\sum_{c\in C}\frac{\text{TP}_c}{\text{TP}_c + \text{FP}_c}$$1.
• Error Rates: $$\text{Precision} = \frac{1}{|C|}\sum_{c\in C}\frac{\text{TP}_c}{\text{TP}_c + \text{FP}_c}$$2, $$\text{Precision} = \frac{1}{|C|}\sum_{c\in C}\frac{\text{TP}_c}{\text{TP}_c + \text{FP}_c}$$3.

Both five-way (review assignment) and binary (accept/reject) tasks are evaluated. Review “helpfulness” is operationalized via LLM-based pairwise ELO comparisons between agent and human reviews (Sahu et al., 9 Oct 2025).

3. Experimental Validation and Results

Experiments utilize a curated sample of 1,963 papers from ICLR 2025, balanced across all decision categories. The backbone LLM is gpt-oss-120b. Baselines include both classical supervised classifiers (XGBoost on TF–IDF/BERT embeddings, fine-tuned BERT), single-agent LLMs, ensemble/voting schemes, and human annotators.

Key Results:

Metric	Meta (All)	Human Avg	Human Top 1%
Binary Acc (accept/reject)	81.8%	83.9%	92.4%
Five-way F1	28.1	—	—
Five-way Accuracy	52.5%	—	—
Review-Quality (ELO, Meta)	1,657	540	1,316

A plausible implication is that ensemble AI reviewer configurations can closely approach, and sometimes exceed, the mean human reviewer both in labeling and review quality, though not the most expert humans. LLM reviewers show domain-specific strengths (fact-checking, literature coverage, clarity) but weaker performance in assessing methodological novelty and theoretical contributions. Sycophantic effects—higher false positive rates post-rebuttal—were observed (Sahu et al., 9 Oct 2025).

4. Strengths, Weaknesses, and Agent Design

Strengths:

• High inter-reviewer consistency and reproducibility due to systematic execution of rubrics.
• Large-scale parallelism enables reviewing tens of thousands of submissions with consistent standards.
• Conditioning reviewer agents with literature retrieval (“LitLLM”) leads to more actionable, evidence-based feedback than the average human, as measured by ELO.
• Ensemble meta-review and majority-vote schemes reduce bias and amplify reliability.

Weaknesses:

• Limited calibration on fine-grained decisions (e.g., oral vs spotlight presentations).
• Review quality degrades when exposed to author rebuttals without adversarial calibration (“sycophancy”).
• LLMs miss fine conceptual novelty and subtle theoretical merit—no single persona outperforms the ensemble.
• Persona disagreement mirrors variance among human reviewers, requiring aggregation protocols for stability (Sahu et al., 9 Oct 2025).

5. Best Practices and Guidelines for Deployment

Based on systematic empirical and qualitative assessment, ReviewerToo proposes guidelines for integration into conference pipelines:

1. Deploy AI as a complementary resource to “raise the floor” in review consistency rather than fully replacing human expert judgment, especially for borderline or high-impact papers.
2. Employ reviewer ensembles and a metareviewer protocol to stabilize aggregator outputs and mitigate persona- or rubric-specific biases.
3. Condition reviewers on both conference rubrics and external literature. Incorporate rebuttal handling with adversarial or calibration prompts to minimize sycophantic artifacts.
4. Optimize primarily for both label accuracy and feedback quality (helpfulness, technical depth).
5. Route complex or novelty-centric decisions to humans; exploit AI for coverage, consistency, and fact-checking.
6. Actively include extremal reviewer personas to diagnose and bound systematic biases; use metareviewer agents for fact-checking and claim verification (Sahu et al., 9 Oct 2025).

These procedural recommendations set a reproducible, modular blueprint for implementing hybrid peer-review systems and inform future design of AI–human collaborative workflows.

6. Comparative Developments and Related Systems

ReviewerToo occupies a distinct position among peer review automation systems by emphasizing modular, persona-driven, and ensemble review agents, systematic rubric adherence, and meta-review protocols. Related frameworks, such as DeepReview (Zhu et al., 11 Mar 2025), also structure the review process in multi-stage chains (novelty detection, multi-aspect feedback, reliability verification) and leverage fine-tuned LLMs for greater review quality. Systems like P2R (Pan et al., 7 Apr 2026) focus instead on reviewer assignment using explicit reviewer profiling and hybrid LLM scoring, while Reviewer2 (Gao et al., 2024) advances the diversity and specificity of LLM-generated feedback via prompt engineering.

The distinguishing characteristic of ReviewerToo is its composability: agents are self-contained and can be systematically swapped, combined, or ablated for controlled experimental study and real-world production deployment (Sahu et al., 9 Oct 2025).

---

References:

• "ReviewerToo: Should AI Join The Program Committee? A Look At The Future of Peer Review" (Sahu et al., 9 Oct 2025)
• "DeepReview: Improving LLM-based Paper Review with Human-like Deep Thinking Process" (Zhu et al., 11 Mar 2025)
• "Beyond Paper-to-Paper: Structured Profiling and Rubric Scoring for Paper-Reviewer Matching" (Pan et al., 7 Apr 2026)
• "Reviewer2: Optimizing Review Generation Through Prompt Generation" (Gao et al., 2024)