Pruning LLM-generated questions

Develop an effective and inexpensive method for pruning LLM-generated questions so that the retained questions have a higher proportion of edit-inducing questions.

Background

The paper evaluates questions generated by GPT-4o and o3 as feedback for scientific manuscripts. Although the models produce substantially more edit-inducing questions than human peer reviewers in absolute terms, only approximately 7–10% of their generated questions are edit-inducing, indicating substantial over-generation of questions that do not lead to edits in the final manuscript.

The authors identify increasing the rate of useful, edit-inducing questions as a practical priority for authors and characterize the development of an effective pruning procedure as an unresolved problem. The desired method should reduce the number of generated questions while preserving or enriching those that provide actionable feedback, and should do so at low computational or financial cost.