Conjecture: Similar AI-assisted review dynamics in other top AI conferences

Ascertain whether other major AI conferences with reviewer pools overlapping those of ICLR—such as the Conference on Neural Information Processing Systems (NeurIPS)—exhibit similar prevalence and effects of AI-assisted peer reviews on submissions and decisions as observed at ICLR 2024.

Background

The authors focus on ICLR due to its open peer-review model, which provides public access to reviews and decisions needed for their analyses. They note that many top AI conferences share substantial overlap in reviewer pools, suggesting that the practices and impacts measured at ICLR may generalize. They quantify this overlap, reporting a 53.4% reviewer overlap between ICLR 2024 and NeurIPS 2023.

Based on these overlaps, the authors explicitly conjecture that related conferences could be subject to similar dynamics (e.g., widespread AI assistance and measurable impacts on scores and acceptance). Testing this conjecture requires analogous data access and methodologies applied to other venues.