Refine-n-Judge: Iterative LLM Refinement

• Refine-n-Judge is an automated, iterative framework that curates high-quality preference data for LLM fine-tuning by coupling refinement and judgment roles.
• It employs a dual-stage loop where one LLM iteratively refines candidate outputs and validates improvements in-line without external reward models.
• Empirical results show up to 98.4% preference rates over baselines, highlighting its scalability, reduced verbosity, and effective bias mitigation.

Refine-n-Judge is an automated, iterative framework for curating high-quality preference data for fine-tuning LLMs. Distinct from earlier self-improvement and reward modeling paradigms, Refine-n-Judge operates by employing a single LLM in a dual capacity: as both refiner—responsible for improving candidate outputs—and judge—responsible for verifying whether each refinement constitutes a substantive improvement. This strategy is designed to generate sequences of strictly-increasing quality, ensuring that only meaningful enhancements are preserved in the training data without resorting to external reward models or human-in-the-loop annotation (Cayir et al., 3 Aug 2025).

1. Iterative Refinement and Judgment Pipeline

The Refine-n-Judge pipeline consists of a loop over two tightly coupled LLM-driven stages:

1. Refinement: Given a query and a candidate output (Ansₜ), the LLM synthesizes a refined output (Ansₜ₊₁), applying explicit feedback prompts to target axes of quality such as accuracy, completeness, clarity, conciseness, and relevance.
2. Judgment: The LLM then evaluates the relative quality of Ansₜ versus Ansₜ₊₁ using dedicated judgment prompts. If the refinement is preferred, the improved output becomes the new baseline; if not, the process terminates for that query, and the most recent accepted answer is retained as the “best.”

Formally, the iterative process is captured as: $$\text{Ans}_0 \rightarrow \text{Ans}_1 \rightarrow \text{Ans}_2 \rightarrow \cdots \rightarrow \text{Ans}_n$$ where each update is given by

$\text{Ans}_{t+1} = \text{Refine}(q, \text{Ans}_t)$

and accepted only if

$$\text{Judge}(q, \text{Ans}_t, \text{Ans}_{t+1}) = \text{“better”}$$

This continues until the LLM judge no longer prefers further refinements. The method is outlined algorithmically in the paper’s workflow diagrams.

2. Distinctions from Previous Iterative and Reward-based Methods

Unlike earlier iterative self-refinement frameworks such as SELF-REFINE, Refine-n-Judge incorporates an explicit quality control mechanism at each step. Traditional methods often lack a stopping criterion, leading to potential regressions such as verbosity inflation or unnecessary content changes. Refine-n-Judge’s core innovation is tying the refinement loop to an in-line, model-internal preference judgment, ensuring only agreed improvements are retained. This design removes the need for external comparison models, separate reward models, or burdensome human comparison data.

Empirical comparisons with multi-answer selection baselines indicate that Refine-n-Judge outputs are preferred 98.4% of the time over those produced by simple zero-shot or selection strategies, underscoring the importance of coupled refinement and judgment.

3. Empirical Performance and Preference Metrics

Refine-n-Judge delivers substantial and quantifiable improvements on established evaluation benchmarks for LLMs:

Model	AlpacaEval	AlpacaEval 2.0	MT-Bench
Llama 3.1-8B (Original TULU)	79.3%	34.8%	6.9
Llama 3.1-8B (Refine-n-Judge)	84.8%	37.5%	7.5
Llama 3.3-70B (Original)	88.2%	51.7%	8.4
Llama 3.3-70B (Refine-n-Judge)	90.5%	54.3%	8.6

Preference experiments report that, when assessed in head-to-head comparisons by GPT-4, outputs generated by the Refine-n-Judge process are chosen 74% of the time over those of a refinement-only baseline. These results demonstrate that the judgment mechanism is critical in preventing the acceptance of regressions and that fine-tuned models exhibit stronger performance across a range of tasks, including code, math, and general conversational QA.

4. Applicability Across Domains and Data Sources

Refine-n-Judge is domain-agnostic and has been evaluated on a variety of public datasets covering coding (e.g., Code Alpaca), mathematical reasoning (GSM8K), and dialogue-centric data sources (UltraChat, OpenAssistant). Its robustness is especially pronounced on noisy or challenging initial answers; the process converges even when the starting point is a suboptimal response.

A key application is the automatic construction of “preference chains” (ranked sequences of candidate responses with explicit pairwise preferences), which are ideal for driving preference-based fine-tuning pipelines. Such chains enable supervised learning schemes (including direct preference optimization and reward modeling) without the scale limitations of human annotation.

5. Design Considerations and Bias Mitigation

Refine-n-Judge incorporates structural design elements to counteract known biases:

• Answer Position Swapping: By varying the order in which candidate and refined outputs are presented to the judge during pairwise evaluation, the pipeline reduces position bias.
• Verbosity and Redundancy Control: The system penalizes unnecessary elaboration by encoding conciseness among the evaluation criteria, addressing the tendency of LLMs to favor verbose outputs during iterative self-improvement.
• Single-Model Integration: By using a single LLM for both refinement and judging, the risk of “gaming” or inter-model misalignment is minimized, and the approach remains computationally simple and scalable.

6. Implications and Future Research Trajectories

The Refine-n-Judge paradigm points to scalable, fully automated preference data curation, with the following forward-looking considerations:

• Multi-Judge Voting: As the decisiveness of the LLM judge diminishes for marginal improvements, incorporating consensus mechanisms or using diverse judgment models could maintain robust evaluation in later iterations.
• Adaptation to New Criteria: The pipeline’s prompts can be conditionally augmented to target new axes of quality, e.g., user engagement or friendliness, for data curation in expanded domains.
• Ethical Considerations: As with all automated dataset curation methods, ongoing work is needed to detect and correct for subtle biases or harmful stereotypes that might propagate during automated refinement.

Ongoing research efforts seek to extend Refine-n-Judge to further domains, investigate calibration and bias-mitigation strategies, and explore the integration of multi-agent or staged judgment modules for continued robustness and coverage.

---

Refine-n-Judge represents an evolution of LLM-driven refinement that tightly couples generation and evaluation stages within a unified, iterative system. Its demonstrated gains in benchmark preference rates and downstream model performance support its adoption as a scalable, annotation-free approach for high-quality preference data curation in large-scale LLM training pipelines (Cayir et al., 3 Aug 2025).