Are large language models capable of refinement?

Determine whether contemporary large language models are capable of refining their own prior responses across successive interaction turns, i.e., producing improved outputs over multiple turns after an initial attempt, in the general setting addressed by RefineBench.

Background

The paper motivates the need to evaluate LLMs' ability to refine their prior responses, noting conflicting evidence: some early methods suggested refinement works while subsequent analyses concluded it does not. Prior evaluations have largely focused on verifiable tasks such as mathematics or code, leaving open-ended and free-form domains underexplored, and the role of feedback control insufficiently examined.

RefineBench is introduced to test this capability across 11 domains and with checklist-based evaluation under self-refinement and guided refinement. The authors emphasize that, despite progress, the fundamental question of whether LMs can refine at all in broad settings has not been settled.