Preference Fine-Tuning Stage

• Preference fine-tuning stage is the post-training process that converts relative judgments from pairwise comparisons and critiques into targeted policy updates.
• It refines model behavior following supervised fine-tuning, enhancing outputs in multi-modal tasks like image editing, summarization, and continuous-control safety alignment.
• The stage leverages diverse preference signals to update parameters while mitigating issues such as forgetting and drift from base model capabilities.

Searching arXiv for papers on preference fine-tuning and the cited topic. arxiv_search(query="preference fine-tuning direct preference optimization supervised fine-tuning image editing vision-LLMs", max_results=10) Preference fine-tuning stage is the post-training phase in which a model is updated to increase its preference for desired outputs relative to counter-responses under a fixed data or annotation budget. In the recent literature, the stage appears in standard LLM pipelines after pretraining and supervised fine-tuning, but it also appears in VLM instruction synthesis, summarization, diffusion-model alignment, VLA policy adaptation, and continuous-control safety alignment. Across these settings, the stage is defined less by a single loss than by a common function: converting preference information—pairwise comparisons, ranked candidates, rubric-conditioned choices, critique-derived edits, or revision chains—into parameter updates that move the policy toward target behavior while controlling drift from a reference model or base capability set (Lin et al., 20 May 2026, Gallego, 13 Jun 2025).

1. Conceptual position in post-training pipelines

Modern post-training pipelines typically place supervised fine-tuning before preference optimization. In this ordering, SFT supplies a strong generative prior or instruction-following prior, while the preference stage refines behavior along axes such as correctness, helpfulness, safety, controllability, or executability. Several papers make this role explicit: post-training commonly proceeds through SFT, preference learning such as RLHF or DPO, and sometimes safety-oriented repair; in image editing, EditCaption uses Stage 1 SFT followed by Stage 2 DPO; and in robotic VLAs, the Imitation $\rightarrow$ Preference $\rightarrow$ Interaction pipeline uses SFT initialization before offline stage-aware preference optimization (Lin et al., 20 May 2026, Wang et al., 9 Apr 2026, Xu et al., 4 Dec 2025).

The stage is not limited to one modality. In VLM instruction synthesis, the model must reason over source–target image differences rather than describe a single frame; in diffusion models, the objective is to prefer one image over another under the same prompt; in continuous control, the preference is over trajectories rather than responses to a fixed prompt. This suggests that “preference fine-tuning stage” is best understood as a family of alignment procedures operating on relative judgments rather than absolute labels, with the exact data structure determined by the underlying modality (Wang et al., 9 Apr 2026, Zhao et al., 13 Mar 2025, Verma et al., 20 May 2026).

A recurrent design question is whether the stage should remain sequential or be integrated with SFT. Sequential SFT $\rightarrow$ preference learning is standard, but several papers argue that it is suboptimal because the second stage can forget what the first stage learned. PAFT instead trains SFT and preference alignment in parallel from the same base model and merges the resulting deltas, while joint post-training frameworks optimize both objectives concurrently to reduce oscillation between SFT and preference optima (Pentyala et al., 2024, Fernando et al., 2024).

2. Preference supervision and data construction

The core object of the stage is the preference example. In the canonical text setting, a preference tuple is $(x, y^+, y^-)$, where $y^+$ is preferred over $y^-$. EditCaption instantiates this in a multimodal image-pair setting with $x=(I_{\mathrm{src}}, I_{\mathrm{tgt}})$, where the rejected instruction is produced by the SFT model and the chosen instruction is written by a human annotator to correct orientation inconsistency, viewpoint ambiguity, or insufficient fine-grained attributes. Stage 2 uses 10K human preference pairs built directly from SFT outputs and human corrections, with no additional automated filtering reported (Wang et al., 9 Apr 2026).

Preference signals need not come from direct human A/B rankings. Configurable Preference Tuning constructs synthetic preference pairs conditioned on rubric-derived system prompts, teaching the same response pair to reverse preference under a different rubric score. Refine-n-Judge builds monotone preference chains $$[\mathrm{Ans}_0,\mathrm{Ans}_1,\ldots,\mathrm{Ans}_n]$$ by iteratively refining an answer and judging whether the refinement is better; the reported training stage then uses the best answer $\mathrm{Ans}_n$ for SFT, although the chain also supports pairwise preference training. Fine-grained text-span feedback goes further by marking liked and disliked spans, then generating left-to-right improvement chains whose adjacent steps become localized preference pairs (Gallego, 13 Jun 2025, Cayir et al., 3 Aug 2025, CH-Wang et al., 29 Dec 2025).

In vision and generative modeling, pair construction is often synthetic but structured. POVID uses ground-truth multimodal answers as preferred responses and generates dispreferred responses in two ways: GPT-4V injects plausible hallucinations into correct answers, and image distortion elicits the VLLM’s own hallucinations under noisy input. RPO uses a critique $\rightarrow$ instruction $\rightarrow$0 edit $\rightarrow$1 relabel pipeline for diffusion models, creating preference pairs from original and edited images. For visual program generation, dispreferred Ladder Diagram programs are created by graph editing, and the hardest negative is selected by minimum graph edit distance (Zhou et al., 2024, Zhao et al., 13 Mar 2025, Kang et al., 23 Feb 2025).

A distinct line of work emphasizes that not all preference alignment requires pairwise ranking as the final optimization target. AlignSum uses high-quality human-annotated summaries as the preference signal but implements alignment by staged supervised cross-entropy on a Data Pyramid, and Refine-n-Judge likewise improves data quality through preference chains while reporting SFT on the final revised answer rather than pairwise preference loss (Han et al., 2024, Cayir et al., 3 Aug 2025).

3. Objectives and reference anchoring

The canonical direct preference optimization objective compares the model’s log-probability gap between preferred and rejected outputs against a frozen reference model. In EditCaption’s multimodal form, with $\rightarrow$2,

$\rightarrow$3

Here $\rightarrow$4 is the frozen SFT checkpoint and $\rightarrow$5 controls how aggressively the policy deviates from the reference (Wang et al., 9 Apr 2026).

This reference-anchored margin has numerous adaptations. CPT conditions the policy on a rubric-derived system prompt $\rightarrow$6, yielding $\rightarrow$7 over $\rightarrow$8; PREFINE adapts DPO-style reasoning to continuous-control trajectories with per-state contrasts and an added SFT anchor for reward retention; and FlowPRO’s RPRO replaces trajectory-level contrast with per-state contrast for flow-matching VLA action heads, adding an explicit proximal regularizer to prevent the reward-hacking failure mode of plain Flow-DPO (Gallego, 13 Jun 2025, Verma et al., 20 May 2026, Wu et al., 3 Jun 2026).

Several papers alter the preference loss to better match data geometry. Cooling-Weighted DPO scales the negative term by a cooling weight derived from the model’s average token log-probability on each negative, suppressing uninformative gradients from easy or off-distribution losers. In fine-grained revision-based supervision, APO-down is selected because DPO can exhibit catastrophic degeneration on highly similar preference pairs. In trajectory settings, Stage-Aware TPO decomposes preference over action sequences into stage-level Bradley–Terry terms with stage costs, so that credit is assigned to Reach, Grasp, Transport, or Place rather than only to the full rollout (Zhang et al., 13 Oct 2025, CH-Wang et al., 29 Dec 2025, Xu et al., 4 Dec 2025).

A separate but related line treats preference as a direction for data selection rather than only as a training loss. PRISM constructs a preference-aware target representation

$\rightarrow$9

then scores candidate SFT examples by $\rightarrow$0, using current-model preference weights $\rightarrow$1 to select or filter data under a budget (Lin et al., 20 May 2026).

4. Relation to supervised fine-tuning

A consistent empirical pattern is that preference optimization rarely replaces SFT; it depends on it. EditCaption reports that DPO-only underperforms severely, whereas SFT-only is strong and SFT+DPO is best. The paper attributes this to the need for a strong generative prior before preference optimization can effectively correct residual errors such as left/right reversals or vague viewpoint descriptions (Wang et al., 9 Apr 2026).

Budget-allocation studies generalize this observation. Under fixed annotation budgets, SFT on the base model dominates in low-data regimes below 1,000 annotated examples, while larger budgets favor combinations of SFT and preference fine-tuning, often with increasing shares allocated toward preference data. Completely eliminating SFT and running DPO directly on the base model produces a cold start problem on tasks such as mathematics; allocating even a small portion of the budget to SFT first can substantially improve downstream analytical performance (Raghavendra et al., 16 Feb 2025).

At the same time, several papers argue that naive sequencing incurs forgetting or alignment tax. Joint post-training treats SFT and preference learning as a multi-objective optimization problem rather than two disconnected phases, while PAFT avoids sequential interference by training SFT and preference adapters in parallel and then merging them. AlignSum, though not pairwise-preference-based, embodies the same intuition in summarization: first fit domain-general signals, then specialize to scarce human-preferred data so that the high-entropy preference signal is not drowned out (Fernando et al., 2024, Pentyala et al., 2024, Han et al., 2024).

5. Modality-specific instantiations

In language modeling, preference fine-tuning spans sequence-level and token-level formulations. LoRA-LiteE frames the stage as supervised preference classification over “Model A wins,” “Model B wins,” or “tie,” using LoRA adapters and an ensemble rather than RLHF. Preference-grounded token-level guidance instead learns a bounded token-level reward $\rightarrow$2 from sequence-level rankings, then uses either REINFORCE-style dense rewards or reward-weighted MLE to fine-tune the LM (Yang et al., 2024, Yang et al., 2023).

In summarization, the stage may be implemented without explicit pairwise optimization. AlignSum uses Gaussian Resampling to match summary-length distributions and two-stage Hierarchical Fine-Tuning to adapt from extractive and abstractive synthetic summaries to scarce human-annotated summaries. The preference signal is present in the high-quality human rewrites, but the optimization remains supervised cross-entropy rather than DPO or RLHF (Han et al., 2024).

In VLMs, the stage is often tied to hallucination reduction or differential visual reasoning. POVID preference-tunes a VLLM against truthful vs. hallucinated answers generated through GPT-4V hallucination injection and noise-triggered hallucinations, explicitly targeting modality misalignment. EditCaption uses human-corrected instruction pairs to teach a VLM to describe transformations between images in precise, executable language. Learning-dynamics-aware VLM work adds another axis: the usefulness of a negative depends on its hardness, so preference optimization must model training dynamics rather than treat all rejected examples equivalently (Zhou et al., 2024, Wang et al., 9 Apr 2026, Zhang et al., 13 Oct 2025).

In diffusion modeling, preference fine-tuning has moved from single population-level rewards to more structured and personalized regimes. RPO curates richer synthetic preference pairs through critique-driven edits before applying Diffusion-DPO. PPD conditions the diffusion denoiser on user preference embeddings extracted from a VLM, then fine-tunes with a user-conditioned Diffusion-DPO objective so that one model can align to multiple users and interpolate between preferences at inference (Zhao et al., 13 Mar 2025, Dang et al., 11 Jan 2025).

In robotics and continuous control, preference fine-tuning becomes policy adaptation over trajectories. STARE-VLA performs stage-aware offline preference optimization on decomposed trajectory segments, PREFINE adapts DPO to safe continuous control with policy-sampled counterfactuals and an SFT anchor, and FlowPRO uses intervention-and-rollback data plus per-state RPRO losses to fine-tune flow-matching VLAs without explicit rewards (Xu et al., 4 Dec 2025, Verma et al., 20 May 2026, Wu et al., 3 Jun 2026).

6. Empirical regularities, limitations, and controversies

Three regularities recur. First, preference data quality matters at least as much as objective choice. EditCaption targets systematic failure modes directly; Refine-n-Judge stops iterative improvement when the judge prefers the previous answer; and revision-chain supervision uses structural checks to ensure one contiguous edit per step. These designs all attempt to prevent noisy or weak preference signals from reaching optimization (Wang et al., 9 Apr 2026, Cayir et al., 3 Aug 2025, CH-Wang et al., 29 Dec 2025).

Second, not all negatives are equally informative. Cooling-Weighted DPO argues that trivially wrong negatives inject uninformative gradients; PRISM shows that target examples should be weighted by current model preference because examples closer to the current model are more actionable; and the two-stage alignment assumption argues that diverse off-policy data are valuable in a preference injection stage, whereas high-quality on-policy pairs are preferable in a later preference fine-tuning stage (Zhang et al., 13 Oct 2025, Lin et al., 20 May 2026, Sun et al., 14 Aug 2025).

Third, preference alignment is constrained by representativeness and judge dependence. PRISM notes that noisy preference estimates and poor Hessian approximations can misguide data selection. CPT depends on teacher quality, rubric clarity, and the risk of bias propagation from synthetic data. POVID and RPO rely on AI-generated negatives or edits whose plausibility determines the strength of the resulting signal. More broadly, this suggests that the preference fine-tuning stage is not a single algorithmic primitive but an interface between model optimization and a chosen notion of preference, with all the instability, bias, and coverage limitations that such notions entail (Lin et al., 20 May 2026, Gallego, 13 Jun 2025, Zhou et al., 2024, Zhao et al., 13 Mar 2025).

Taken together, these works present preference fine-tuning stage as a technically heterogeneous but conceptually unified phase of post-training. Its defining operation is the conversion of relative judgments into policy updates that sharpen desired behavior while preserving a usable base model. The principal open problem is not merely how to optimize a preference loss, but how to construct preference signals, references, curricula, and negatives that remain informative as models and tasks scale (Wang et al., 9 Apr 2026, Pentyala et al., 2024, Sun et al., 14 Aug 2025).