RefineBench: Evaluating Refinement Capability of Language Models via Checklists (2511.22173v1)

Abstract: Can LMs self-refine their own responses? This question is increasingly relevant as a wide range of real-world user interactions involve refinement requests. However, prior studies have largely tested LMs' refinement abilities on verifiable tasks such as competition math or symbolic reasoning with simplified scaffolds, whereas users often pose open-ended queries and provide varying degrees of feedback on what they desire. The recent advent of reasoning models that exhibit self-reflection patterns in their chains-of-thought further motivates this question. To analyze this, we introduce RefineBench, a benchmark of 1,000 challenging problems across 11 domains paired with a checklist-based evaluation framework. We evaluate two refinement modes: (1) guided refinement, where an LM is provided natural language feedback, and (2) self-refinement, where LMs attempt to improve without guidance. In the self-refinement setting, even frontier LMs such as Gemini 2.5 Pro and GPT-5 achieve modest baseline scores of 31.3% and 29.1%, respectively, and most models fail to consistently improve across iterations (e.g., Gemini-2.5-Pro gains only +1.8%, while DeepSeek-R1 declines by -0.1%). By contrast, in guided refinement, both proprietary LMs and large open-weight LMs (>70B) can leverage targeted feedback to refine responses to near-perfect levels within five turns. These findings suggest that frontier LMs require breakthroughs to self-refine their incorrect responses, and that RefineBench provides a valuable testbed for tracking progress.

• The paper demonstrates that state-of-the-art LMs struggle with self-refinement even after multiple iterations, showing minimal improvement.
• It employs a comprehensive checklist protocol across 11 diverse domains to rigorously evaluate both self- and guided refinement.
• Guided refinement with explicit feedback greatly boosts performance, highlighting error localization as the main challenge in autonomous correction.

Evaluating LLM Refinement with RefineBench

Motivation and Problem Formulation

Iterative refinement—the multi-turn improvement of generated outputs in response to either explicit feedback or self-evaluation—is a core desideratum for interactive LMs, especially as user queries frequently demand post-hoc correction, clarification, or expansion. Prior work has primarily evaluated LM refinement on simple or verifiable tasks (e.g., mathematical problem-solving, symbolic reasoning), often with strong inductive or structural priors. However, such setups do not reflect the heterogeneity of real-world queries, nor do they differentiate between self-driven and externally-guided revision protocols. This work introduces RefineBench, a comprehensive multi-turn benchmark targeting the measurement of LM refinement capacity across self-refinement (no explicit feedback) and guided refinement (explicit checklist-based feedback). RefineBench is further distinguished by its coverage of both verifiable and open-ended domains, and its evaluation protocol based on rigorously designed checklists capturing fine-grained response requirements.

Figure 1: (Left) Compared to prior benchmarks, even state-of-the-art LMs attain minimal self-refinement gains on RefineBench; (Right) bottlenecks often arise from inability to localize errors without explicit guidance, motivating the examination of partial guidance and variable feedback granularity.

Dataset Composition and Evaluation Protocol

RefineBench consists of 1,000 challenging items spanning 11 domains—including STEM, humanities, law, and social science—each paired with a checklist (average 9.9 items per instance) specifying binary evaluation criteria. Problem sources encompass university entrance exams, bar exam questions, and established reasoning benchmarks; non-textual content (e.g., images, tables) is verbalized to maximize applicability to text-only LMs, and rigorous translation and verification pipelines ensure English accessibility for all domains.

Checklist construction combines institutional rubric extraction with multi-LM generation followed by human expert curation and validation. Checklists consistently obtain high approval rates ($>96\%$ expert validation; see Figure 2), ensuring reliability of the binary evaluation process.

Figure 3: Left—full example instance from RefineBench; Right—a schematic of the self-refinement and guided refinement protocols used for evaluation.

Evaluation proceeds via an LLM-as-judge setup, most commonly using GPT-4.1 for binary grading. In the self-refinement protocol, the LM revises its output without any explicit feedback; iteration continues up to 5 rounds or until the model terminates. In guided refinement, the reviser receives explicit feedback itemizing all failed checklist elements, and refinement proceeds identically. The framework supports partial guidance, enabling analysis of performance as a function of feedback granularity.

Figure 4: Basic statistics and the domain distribution of RefineBench instances; the benchmark ensures high coverage, with math, humanities, and law in particular strongly represented.

Empirical Results

A total of 34 frontier LMs—covering open-source and proprietary models, both reasoning- and instruction-tuned—were evaluated under both refinement regimes. Two principal findings emerge:

1. State-of-the-art LMs fail to self-refine on challenging, open-ended benchmarks. Even under five turns, top-tier models such as Gemini 2.5 Pro and GPT-5 achieve only 31.3% and 29.1% accuracy, respectively, with mean $\Delta$ (improvement) generally in [–2.5%, 2.6%] across all categories. Many models actually regress over refinement iterations. This failure is not domain-specific: math, law, engineering, humanities all exhibit similar trends.

2. With explicit, fine-grained feedback, even modest models rapidly approach near-perfect performance. Under guided refinement, models above 70B parameters, both proprietary and open-source, achieve $>$90% checklist completion within five turns (e.g., Claude-Opus-4.1 98.4% at $t=5$, +79.7% $\Delta$), while smaller models exhibit more gradual improvement. The gap between self-refinement and guided refinement is thus vast, as shown in Figure 1 and 2.

Figure 2: Human evaluation corroborates the high quality and appropriateness of the constructed checklists (96.1% average approval).

Failure Mode Analysis and Behavioral Trends

Error Localization vs. Correction Capacity. When provided with explicit feedback detailing which aspects are unsatisfactory (i.e., failed checklist items as guidance, but not the solutions), models show significant improvement (cf. LLaMA-3.1-70B-Instruct: +43.6% over self-refinement). Thus, the primary bottleneck is the inability to localize or identify what should be revised, not incorporating the correction once it is indicated.

Partial Guidance Regimes. When only a random subset of checklist feedback is revealed at each iteration, response improvement is restricted to the provided feedback; aspects not surfaced remain unimproved, mapping directly to real-world user behavior where partial guidance predominates.

Reasoning LMs vs. Instruction-Tuned LMs. Reasoning-specialized LMs (e.g., Qwen3-30B-A3B-Thinking, DeepSeek-R1) modestly outperform size-matched instruction-tuned baselines on self-refinement, yet absolute improvements remain limited. Interestingly, DeepSeek series models often regress over multiple rounds—their Chain-of-Thought patterns reveal that after initial self-correction, subsequent iterations mostly repeat prior reasoning without exploring new failure modes.

Figure 5: DeepSeek-R1 reasoning token usage and behavioral markers drop sharply after the initial refinement turn, indicating lack of new hypothesis exploration and insufficient error localization in subsequent turns.

Figure 6: Transition matrix for DeepSeek-R1 responses over refinement iterations—incorrect responses are strongly persistent, and the probability of transitioning from incorrect to correct is minimal without external feedback.

Domain Variation. Some domains (notably law) exhibit stronger self-refinement (e.g., Claude-Opus-4.1: +7.8%) than others. This may be explained by legal reasoning tasks offering more explicit satisfaction criteria, or by domain artifacts in the LMs’ pretraining.

Figure 7: The diversity of checklist items per instance, confirming that each item targets a distinct, non-overlapping quality criterion for robust multi-faceted evaluation.

Implications and Future Directions

Practically, RefineBench demonstrates that, outside of narrow, highly-structured tasks, LM self-refinement is not a solved problem—even with extensive scaling and advanced reasoning pretraining. Guidance (whether by humans, users, or via external critic LMs) remains essential for meaningful output improvement. Current LMs mostly fail to autonomously discover unsatisfied requirements or to generalize self-reflective behavior to real-world, multi-turn dialogue.

Reward functions incorporating fine-grained criteria (e.g., checklist-derived reward shaping), explicit error localization, and richer forms of intrinsic model feedback may be necessary to unlock robust self-correction. The sharp improvement under guided and partial-guided regimes motivates hybrid protocols in real-world deployments (e.g., checklist-driven user interfaces or explainable verdict enumeration).

Moreover, the discrepancy between reasoning LMs’ self-verification claims and their actual multi-turn performance on RefineBench calls into question prevailing evaluation practices, emphasizing the need for more challenging, free-form, and multi-domain protocols for future model selection and development.

Conclusion

RefineBench constitutes a rigorous multi-domain, multi-turn benchmark for analyzing both self-refinement and guided refinement in LMs, operationalized via highly-validated, fine-grained checklists. Experimental evidence demonstrates a pronounced incapacity for self-refinement in contemporary LMs outside of narrow scenarios, but rapid, near-perfect trajectory under explicit feedback—exposing the critical bottleneck in error localization. The benchmark can drive future research in reward design, training objectives, and adaptive guidance protocols for truly self-improving LLMs and highlights the need for a paradigm shift in the development and evaluation of interactive AI systems.