GPT4 is Slightly Helpful for Peer-Review Assistance: A Pilot Study (2307.05492v1)
Abstract: In this pilot study, we investigate the use of GPT4 to assist in the peer-review process. Our key hypothesis was that GPT-generated reviews could achieve comparable helpfulness to human reviewers. By comparing reviews generated by both human reviewers and GPT models for academic papers submitted to a major machine learning conference, we provide initial evidence that artificial intelligence can contribute effectively to the peer-review process. We also perform robustness experiments with inserted errors to understand which parts of the paper the model tends to focus on. Our findings open new avenues for leveraging machine learning tools to address resource constraints in peer review. The results also shed light on potential enhancements to the review process and lay the groundwork for further research on scaling oversight in a domain where human-feedback is increasingly a scarce resource.
- John C Burnham. The evolution of editorial peer review. Jama, 263(10):1323–1329, 1990.
- Ray Spier. The history of the peer-review process. TRENDS in Biotechnology, 20(8):357–358, 2002.
- Inga Vesper. Peer reviewers unmasked: largest global survey reveals trends. Nature, pages 7–8, 2018.
- The evolving crisis of the peer-review process. Journal of Marketing Analytics, 10(3):185–186, 2022.
- Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020.
- URL https://nips.cc/.
- Is the future of peer review automated? BMC Research Notes, 15(1):1–5, 2022.
- Ai-assisted peer review. Humanities and Social Sciences Communications, 8(1):1–11, 2021.
- Reviewergpt? an exploratory study on using large language models for paper reviewing. arXiv preprint arXiv:2306.00622, 2023.
- Peerassist: leveraging on paper-review interactions to predict peer review decisions. In Towards Open and Trustworthy Digital Societies: 23rd International Conference on Asia-Pacific Digital Libraries, ICADL 2021, Virtual Event, December 1–3, 2021, Proceedings 23, pages 421–435. Springer, 2021.
- Predicting paper acceptance via interpretable decision sets. In Companion Proceedings of the Web Conference 2021, pages 461–467, 2021.
- Weijie Su. You are the best reviewer of your own papers: An owner-assisted scoring mechanism. Advances in Neural Information Processing Systems, 34:27929–27939, 2021.
- Eliciting informative feedback: The peer-prediction method. Management Science, 51(9):1359–1373, 2005.
- Bo Waggoner and Yiling Chen. Information elicitation sans verification. In Proceedings of the 3rd workshop on social computing and user generated content (SC13), 2013.
- Elicitability and knowledge-free elicitation with peer prediction. In Proceedings of the 2014 international conference on Autonomous agents and multi-agent systems, pages 245–252, 2014.
- An information theoretic framework for designing information elicitation mechanisms that reward truth-telling. ACM Transactions on Economics and Computation (TEAC), 7(1):1–33, 2019.
- Anoformer: Time series anomaly detection using transformer-based gan with two-step masking.
- Understanding black-box predictions via influence functions. In International conference on machine learning, pages 1885–1894. PMLR, 2017.
- Chimle: Conditional hierarchical imle for multimodal conditional image synthesis. Advances in Neural Information Processing Systems, 35:280–296, 2022.
- Zachary Robertson (6 papers)