Can AI Be a Good Peer Reviewer? A Survey of Peer Review Process, Evaluation, and the Future

Abstract: Peer review is a multi-stage process involving reviews, rebuttals, meta-reviews, final decisions, and subsequent manuscript revisions. Recent advances in LLMs have motivated methods that assist or automate different stages of this pipeline. In this survey, we synthesize techniques for (i) peer review generation, including fine-tuning strategies, agent-based systems, RL-based methods, and emerging paradigms to enhance generation; (ii) after-review tasks including rebuttals, meta-review and revision aligned to reviews; and (iii) evaluation methods spanning human-centered, reference-based, LLM-based and aspect-oriented. We catalog datasets, compare modeling choices, and discuss limitations, ethical concerns, and future directions. The survey aims to provide practical guidance for building, evaluating, and integrating LLM systems across the full peer review workflow.

• The paper presents a taxonomy of five AI paradigms for generating peer reviews, each with distinct methodologies and measurable performance.
• It evaluates methods via human-centric, reference-based, LLM-based, and aspect-oriented frameworks, highlighting specific trade-offs and challenges.
• Findings indicate that fine-tuned and RL-based systems yield actionable, detailed feedback, while ethical and reliability concerns remain significant.

AI for Peer Review: Methodologies, Evaluation, and Future Directions

Introduction

The paper "Can AI Be a Good Peer Reviewer? A Survey of Peer Review Process, Evaluation, and the Future" (2604.27924) offers a comprehensive survey of AI-driven methods for automating and assisting the peer review process. The work identifies a dual focus: the evolution of peer review generation methodologies under the LLM revolution, and a systematic taxonomy of evaluation frameworks and datasets. The survey positions itself as a critical reference for rigorous engineering, evaluation, and deployment of LLM-based peer review systems, emphasizing nuanced critique generation, after-review workflows (rebuttal, meta-review, revision), and integrative benchmarks.

Peer Review Generation Paradigms

Peer review generation is classified into five primary paradigms: foundation approaches, fine-tuning methods, agent-based systems, reinforcement learning (RL) optimization, and generation enhancement.

• Foundation approaches include early multi-document summarization and analytic sub-tasks (acceptance prediction, score regression), which underlie modern dataset curation but lack end-to-end review generation capacity.
• Fine-tuning methods leverage domain-specific corpora and supervised adaptation (e.g., Llama-8B on expert reviews) to produce format-conforming, critical feedback with significantly improved specificity, actionable comments, and higher review/decision hit rates.
• Agent-based methods emulate human panel workflows; these frameworks use role decomposition (planner, investigator, reviewer, controller) or iterative refinement via multi-agent collaboration, aligning review output with targeted critique, validity diagnosis, and dynamic interactivity.
• Reinforcement learning introduces human-aligned reward signals and multi-objective optimization for deeper, actionable reviews. RL paradigms (e.g., Group Relative Policy Optimization) mitigate the tendency toward shallow, generic feedback and enable RLHF-driven cycles between manuscript and critique generation.
• Generation enhancement encompasses retrieval-augmented generation (RAG), self-corrective refinement loops, and structure/style control to ensure factual grounding, detailed rebuttal-responsive improvements, and alignment with actionable criteria.

  Figure 1: Five main paradigms for peer review generation are illustrated: foundation, fine-tuning, agent-based, RL-based, and enhancement methods.

After-Review Modeling: Rebuttal, Meta-Review, Revision

Beyond initial critique, the survey rigorously covers AI methods for after-review tasks:

• Rebuttal generation is increasingly treated as structured, multi-turn dialog, with explicit decomposition of reviewer criticism and evidence-based response planning. Author-in-the-loop approaches and minimal guidance strategies demonstrably improve factuality and targeted refutation.
• Meta-review generation synthesizes multiple reviews via extract-then-write, argument structure analysis, and checklist-guided introspection. Sentence-level functional labeling and facet-level judgment extraction improve consensus consolidation and decision support.
• Revision from reviews links granular critique to paper-edit alignment, with sentence- and span-level annotation facilitating edit intent understanding, evaluation, and the design of more content-aware revision systems.

Evaluation Frameworks: Taxonomy and Benchmarking

Evaluation remains a paramount challenge for reliability, scalability, and depth. The paper offers a taxonomy with four main categories:

• Human-centric evaluation via expert scoring and pairwise preference arenas yields the most direct and nuanced assessment, but is limited by scalability and subjectivity.
• Reference-based evaluation uses text similarity metrics (BLEU, ROUGE, BERTScore) and content overlap, providing reproducibility but failing to capture deep reasoning and validity.
• LLM-based evaluation automates rubric-based assessment via judge models, supporting scalable, nuanced evaluation but introducing systemic biases reliant on prompt design and model calibration.
• Aspect-oriented evaluation decomposes quality into multi-dimensional, fine-grained targets (e.g., novelty, clarity, evidence use), surfacing critical strengths, failure modes, and robustness vulnerabilities.

  Figure 2: Main evaluation methods: human-centric, reference-based, LLM-based, and aspect-oriented evaluation.

Dataset curation has evolved markedly post-2023, expanding scope, granularity, and annotation richness—though coverage outside NLP/ML remains sparse and structural alignment challenges persist. The survey recommends efforts for better role standardization, sentence-to-paper mapping, semi-supervised expansion, and cross-disciplinary benchmarks.

Numerical Results and Contradictory Claims

• Fine-tuned LLMs, such as OpenReviewer, achieve considerable diversity and critique specificity, outperforming zero-shot baselines by substantial margins in review and decision hit rates.
• RL-based frameworks like REMOR and CycleResearcher optimize multi-objective reward signals for actionable feedback and establish closed research-review-refinement loops, demonstrating superior alignment with nuanced human preferences.
• DeepReviewer-14B, trained in the DeepReview framework, achieves win rates of 88.21% and 80.20% against GPT-o1 and DeepSeek-R1, outperforming larger models while utilizing fewer tokens.
• Human studies find LLM feedback perceived as valuable by 57–82% of users, but caution against reliability, high variance, and compromised integrity in scientific validity assessment.
• Aspect-oriented evaluations consistently reveal LLM blind spots—overweighting technical correctness and underweighting novelty—and expose susceptibility to adversarial manipulation.

Practical and Theoretical Implications

The integration of LLMs into the peer review pipeline enables scalable assistance and automation but raises sharp ethical and technical concerns. Affiliation bias, reviewer anchoring, and the surge in AI-modified review text highlight risks to fair and reproducible scientific judgment. Theoretical advances suggest a shift from gatekeeping to improvement-oriented review, leveraging actionable signals for manuscript revision and iterative scientific progress. RL and agentic frameworks are poised to unlock deeper reasoning, but require holistic evaluation frameworks and rigorous content-level benchmarking for robust deployment.

The transition to human-in-the-loop workflows and multimodal review systems (incorporating visual, table, and code artifacts) is necessary for comprehensive, high-fidelity critique. Future directions include causal modeling of review points to decisions, venue-fit analysis via CfP integration, and extension beyond NLP/ML for domain-general review models.

Conclusion

AI-driven peer review methodologies—underpinned by LLMs, agentic simulation, RL optimization, and advanced bench-marking—have dramatically redefined the landscape of scholarly critique but remain fundamentally challenged by evaluation, bias mitigation, and domain generalization. The survey provides authoritative guidance for methodological engineering and evaluative rigor, advocating transparent and ethical deployment as the field advances toward deeper, improvement-oriented, and multidisciplinary automated peer review systems.