Generative Adversarial Reviews: When LLMs Become the Critic (2412.10415v1)

Abstract: The peer review process is fundamental to scientific progress, determining which papers meet the quality standards for publication. Yet, the rapid growth of scholarly production and increasing specialization in knowledge areas strain traditional scientific feedback mechanisms. In light of this, we introduce Generative Agent Reviewers (GAR), leveraging LLM-empowered agents to simulate faithful peer reviewers. To enable generative reviewers, we design an architecture that extends a LLM with memory capabilities and equips agents with reviewer personas derived from historical data. Central to this approach is a graph-based representation of manuscripts, condensing content and logically organizing information - linking ideas with evidence and technical details. GAR's review process leverages external knowledge to evaluate paper novelty, followed by detailed assessment using the graph representation and multi-round assessment. Finally, a meta-reviewer aggregates individual reviews to predict the acceptance decision. Our experiments demonstrate that GAR performs comparably to human reviewers in providing detailed feedback and predicting paper outcomes. Beyond mere performance comparison, we conduct insightful experiments, such as evaluating the impact of reviewer expertise and examining fairness in reviews. By offering early expert-level feedback, typically restricted to a limited group of researchers, GAR democratizes access to transparent and in-depth evaluation.

• The paper introduces a novel Generative Agent Reviewers (GAR) framework that automates peer review with graph-based representations and custom reviewer personas.
• The paper details a multi-round review process where iterative feedback is synthesized, achieving an f1 score of 0.66 that rivals human performance.
• The paper's findings imply that GAR can enhance scalability and fairness in academic peer review, reducing dependence on limited expert availability.

Generative Adversarial Reviews: When LLMs Become the Critic

The paper "Generative Adversarial Reviews: When LLMs Become the Critic" by Nicolas Bougie and Narimasa Watanabe addresses critical challenges in the academic peer review process and proposes a novel framework, Generative Agent Reviewers (GAR), which utilizes LLM based agents to act as automated reviewers. This approach is particularly relevant in light of the increasing complexity and volume of manuscripts, alongside the biases and inconsistencies prevalent in traditional peer review systems.

Overview of GAR's Architecture

The GAR framework is designed to mimic the traditional peer-review process by extending LLM capabilities with memory functions and reviewer personas derived from historical data. Central to this system is the use of graph-based manuscript representation, which allows for the condensation and logical organization of a paper's content by linking ideas, evidence, and technical details. The review process is multi-faceted and includes:

1. Graph-Based Representation: Manuscripts are condensed into graph structures that establish connections between ideas, claims, and results, enhancing the LLM's ability to process and evaluate papers efficiently.
2. Reviewer Personas: The framework simulates various reviewer characteristics such as strictness and focus areas, which are inferred from past review behaviors. This personalization aligns synthetic reviewers closer with their human counterparts.
3. Review Process: GAR employs a multi-round assessment process where reviewers provide iterative feedback, integrating insights from its memory module. A meta-reviewer then synthesizes these reviews to predict the likelihood of a paper's acceptance.

Experimental Validation and Results

The empirical analysis demonstrates that GAR performs comparably to human reviewers in terms of providing detailed feedback and predicting paper outcomes, with capabilities aligning closely in terms of scope and depth of reviews. The experiments are designed to measure GAR's effectiveness against traditional human reviewers and other LLM-powered systems like ReviewerGPT and AI-Review.

Quantitatively, GAR exhibits high consistency with human reviewer assessments, showcasing an f1 score of 0.66 across evaluated datasets, matching human performance in conference paper reviews such as ICLR and NeurIPS. Moreover, through an LLM evaluator's lens, GAR reviews were frequently chosen over human-generated reviews, highlighting the framework's ability to maintain consistency and depth in feedback, paramount for academic rigor.

Implications and Future Directions

The work has significant implications for the automation and scalability of the peer review process. By democratizing access to high-quality feedback, GAR can potentially alleviate the bottleneck of expert availability in niche domains and support researchers in improving the robustness of their submissions prior to peer evaluation. Theoretically, it opens avenues for further refining AI models to encapsulate more nuanced human-like judgment and fairness in reviews. However, challenges such as potential biases and the ability to evaluate truly novel contributions remain areas for further investigation.

Speculatively, future developments in AI could enhance GAR with advanced contextual understanding mechanisms, potentially integrating real-time updates from the latest research trends and citations to autonomously assess paper novelty. Furthermore, continuous improvement of the persona modeling aspect can lead to even more human-like feedback, maintaining the integrity and balance between automation and scholarly expertise.

In conclusion, this research presents a promising step towards improving efficiency, consistency, and accessibility in academic peer review systems using LLM-driven agents. As the technology progresses, it could significantly enhance the scalability and effectiveness of peer review processes while simultaneously providing valuable early-stage feedback to researchers globally.