Determine the role of LLMs in AI-assisted peer reviews

Determine whether large language models are primarily used for surface-level writing assistance (e.g., enhanced spell-checking and editing) or for formulating the core substantive arguments in AI-assisted peer reviews written for venues such as the International Conference on Learning Representations (ICLR), and clarify which usage mode predominates in practice.

Background

The paper investigates the prevalence and impact of AI-assisted peer reviews at ICLR 2024 using GPTZero for detection and quasi-experimental designs to estimate effects on scores and acceptance rates. Beyond detection, the authors highlight that even when AI assistance can be identified, the specific role LLMs play in composing reviews is unknown. They propose two distinct modalities: LLMs as writing aids (e.g., enhanced spell-checking) versus LLMs as generators of core arguments. The distinction carries major implications for the integrity and epistemic function of peer review, as the latter could undermine the review process while the former may improve clarity, particularly for non-native English speakers.

This uncertainty remains unresolved within the paper, as their methodology focuses on detection and outcomes rather than directly observing how reviewers employ LLMs. The authors thus explicitly flag this as an open question regarding real-world usage patterns of LLMs in peer reviewing.