Reference-Based Multimodal Preference Alignment

Updated 4 July 2026

The paper introduces reference-based multimodal preference alignment, showing that optimizing outputs against explicit exemplars enhances model grounding and reduces hallucinations.
It details mechanisms for constructing visual and textual references via pairwise and unary comparisons combined with adaptive rejection weighting strategies.
Empirical results demonstrate improved trustworthiness, personalization, and performance across systems like CLIP, Stable Diffusion, and various multimodal language models.

Searching arXiv for the cited work on reference-based multimodal preference alignment and closely related methods. Reference-based multimodal preference alignment denotes a class of alignment procedures in which outputs are optimized relative to explicit exemplars—such as preferred and non-preferred images, perturbed visual counterexamples, unary reference answers, reference images, or multimodal interaction histories—rather than solely through generic supervised targets or language-only chosen-versus-rejected pairs. Across recent work, the paradigm appears in few-shot adaptation of CLIP and Stable Diffusion with Bradley–Terry preference models, vision-aware preference optimization for multimodal LLMs (MLLMs), modality-fair image-text alignment for hallucination reduction, unary reference-answer alignment for LLMs, self-corrective multimodal recommendation, and training-free reference-image-guided diffusion control (Gallego, 2023, Lu et al., 22 Apr 2025, Jiang et al., 2024, Zhao et al., 14 Apr 2025, Guan et al., 13 Aug 2025, Li et al., 25 Aug 2025).

1. Problem formulation and conceptual scope

A recurrent motivation in this literature is that standard preference alignment pipelines are often insufficiently grounded in the non-text modalities. In the MLLM setting, standard Direct Preference Optimization (DPO) is typically constructed from pairs of the form $(Q, I, y_w)$ and $(Q, I, y_l)$ , where the image is fixed and only the response changes. This setup is effective for learning which answer sounds better, but it can overemphasize language preferences and underemphasize visual evidence, making hallucination difficult to eliminate (Lu et al., 22 Apr 2025). Closely related work argues that multimodal preference optimization becomes text-biased because the optimization signal is driven primarily by text reward differences, while the image is not directly rewarded in a fine-grained way; the consequence is weak visual grounding and visually unsupported fluent outputs (Jiang et al., 2024).

The term reference-based therefore covers several distinct but related constructions. In text-to-image adaptation, the reference may be a pairwise preference between candidate images or a set of preferred and non-preferred aesthetic exemplars (Gallego, 2023). In MLLM trustworthiness alignment, the reference is often the original image, contrasted with a minimally modified rejected image that differs only in semantically important regions (Lu et al., 22 Apr 2025, Jiang et al., 2024). In language-model alignment, the reference may be a high-quality unary answer, with similarity to that answer replacing binary human preference data (Zhao et al., 14 Apr 2025). In recommendation, the reference can be the user’s own multimodal tipping history, from which a structured preference representation is composed (Guan et al., 13 Aug 2025). In training-free diffusion steering, the reference is a single image from which an MLLM infers global and local preference signals for prompt enrichment and generation control (Li et al., 25 Aug 2025).

A common misconception is that multimodal preference alignment is necessarily equivalent to optimizing textual chosen-versus-rejected responses. The surveyed methods directly contest that view. They introduce image-side rejections, exemplar-conditioned embedding updates, similarity-to-reference rewards, or multimodal history-conditioned recommendation objectives, all of which expand alignment beyond purely textual preference comparison.

2. Reference construction mechanisms

The concrete meaning of “reference” varies by task, but in each case the reference is operationalized as a structured source of contrastive or similarity supervision.

Work	Reference object	Alignment target
(Gallego, 2023)	Pairwise preferred vs. rejected images; preferred aesthetic exemplars	CLIP embedding or Stable Diffusion text conditioning
(Lu et al., 22 Apr 2025)	Original image and vision-based rejected image	MLLM response grounding
(Jiang et al., 2024)	Original image and key-region perturbed image	Modality-fair trustworthiness alignment
(Zhao et al., 14 Apr 2025)	High-quality reference answer(s)	LLM helpfulness, safety, and confidence
(Guan et al., 13 Aug 2025)	User tipping history as multimodal references	Author recommendation and explanation
(Li et al., 25 Aug 2025)	Reference image	Training-free text-to-image preference steering

In the Bradley–Terry adaptation framework, pairwise human preference is the primitive supervision unit. For two images $y_1$ and $y_2$ compared under a text query embedding $x$ , the preferred image is the one that should receive higher latent strength under the CLIP similarity score. The same logic is reused for generation, where preferred and non-preferred images are sampled from SAC-style human aesthetic rankings and converted into pairwise preferences (Gallego, 2023).

AdaViP constructs vision-based rejected samples by perturbing the image rather than the response. Its pipeline uses RAM to identify categories, GroundingDINO to localize them with bounding boxes, SAM to generate segmentation masks, and LaMa to remove the segmented objects via inpainting while preserving scene structure. A candidate set of perturbed images is then scored against the preferred response with CLIP after splitting the response into self-contained sentences; the rejected image is the candidate with the lowest CLIP score (Lu et al., 22 Apr 2025). The resulting pair, $(Q, \mathcal{I}_w, y_w)$ versus $(Q, \mathcal{I}_l, y_w)$ , is a visually grounded contrast constructed relative to the original image.

MFPO also uses image-level references, but its rejected image differs from the original only in key regions selected from the text. Important keywords are extracted using a multipartite graph over keywords encoding positional relationships, semantic similarity, and contextual relevance; these keywords are mapped to regions with a lightweight Segment Anything Model, and diffusion noise is applied only to the selected regions. If reliable mapping fails, the method falls back to global noise perturbation over the full image (Jiang et al., 2024). The original image thus becomes the reference visual condition, and the perturbed image becomes a minimally altered negative example.

RefAlign departs from pairwise preference data entirely. It replaces chosen-versus-rejected labels with a unary high-quality reference answer $y^\star$ , and the reward is a similarity metric between a sampled generation and that reference. Its motivating claim is that binary preference labeling and reward modeling are resource-intensive, whereas selecting a single best response or obtaining a strong reference answer may be cheaper in scenarios such as AI preference distillation (Zhao et al., 14 Apr 2025).

MSPA defines the reference as the user’s historical tipping sequence

$S = \{s_1, s_2, \dots, s_n\},$

where each tipped author is described by textual profile, visual features, audio-derived text, and comment data. An MLLM converts this history into a behavioral preference text $P_{\text{text}}$ and embedding $(Q, I, y_l)$ 0, which are then aligned with candidate authors’ multimodal attributes (Guan et al., 13 Aug 2025). In this formulation, multimodal references are aggregated over time rather than presented as a single exemplar.

The training-free diffusion framework uses a reference image $(Q, I, y_l)$ 1 to infer preferences through an MLLM. The model extracts global preference keywords spanning Artistic Style, Emotional / Atmospheric Resonance, Thematic, Visual Elements, and Others, then enriches a base prompt into a complex prompt, object-level sub-prompts, and a background prompt (Li et al., 25 Aug 2025). Here the reference is neither a pairwise comparator nor a label set, but a visual specification of user intent.

3. Objective families and optimization strategies

The Bradley–Terry formulation provides one of the simplest objective families in this area. Given normalized embeddings and similarity

$(Q, I, y_l)$ 2

the probability that $(Q, I, y_l)$ 3 is preferred to $(Q, I, y_l)$ 4 under query embedding $(Q, I, y_l)$ 5 is

$(Q, I, y_l)$ 6

and the negative log-likelihood for a labeled pair $(Q, I, y_l)$ 7 is

$(Q, I, y_l)$ 8

Because the score is linear in the embedding, the gradient with respect to $(Q, I, y_l)$ 9 has the closed form

$y_1$ 0

leading to the single-step update

$y_1$ 1

This mechanism adapts the prompt embedding at inference time without modifying CLIP encoders or diffusion-model weights. When only positive examples are available, a baseline update moves the embedding toward the average of preferred images,

$y_1$ 2

but the paper reports that the Bradley–Terry version is better because it uses both positive and negative signals (Gallego, 2023).

AdaViP generalizes beyond binary response preference by jointly optimizing language-based and vision-based rejections. It denotes the preferred sample as $y_1$ 3, language-rejected samples as $y_1$ 4, and vision-rejected samples as $y_1$ 5, then uses a Plackett-Luce-style unified preference:

$y_1$ 6

with $y_1$ 7. The expected reward is replaced by the DPO implicit reward

$y_1$ 8

and the gradient includes adaptive rejected-sample weights

$y_1$ 9

The adaptive weighting is designed to avoid the rigid equal-weight assumption that can cause one rejection type to dominate optimization (Lu et al., 22 Apr 2025).

MFPO also remains within a DPO-style family, but explicitly decomposes optimization into text preference loss, image preference loss, and a margin loss. The image-side term compares the preferred response $y_2$ 0 under the clean image $y_2$ 1 and the perturbed image $y_2$ 2:

$y_2$ 3

The total loss is

$y_2$ 4

This directly rewards the model for assigning higher preference to the response under the original image than under the perturbed image (Jiang et al., 2024).

RefAlign uses a different objective family: a REINFORCE-style policy-gradient update over full generated responses, with no critic model and no reference model. Given sampled responses $y_2$ 5 and reference answer $y_2$ 6, the reward is

$y_2$ 7

instantiated with BERTScore in experiments. The advantage is computed with the sampled-group mean as baseline,

$y_2$ 8

and the update is

$y_2$ 9

The advantage is clipped to $x$ 0, and extensions to safety and confidence alignment are built by composing additional similarity-based reward terms (Zhao et al., 14 Apr 2025).

MSPA adopts Group Relative Policy Optimization (GRPO). For a query $x$ 1, the old policy samples a group of outputs $x$ 2, and the normalized advantage is

$x$ 3

Its final reward combines exact recommendation accuracy, structured output format, and semantic similarity between predicted and ground-truth authors:

$x$ 4

The similarity term is cosine similarity between multimodal features of the predicted and ground-truth authors (Guan et al., 13 Aug 2025).

The training-free diffusion framework does not optimize model parameters. Instead, it injects global preference information into prompt embeddings through orthogonal projection,

$x$ 5

and modulates local cross-attention by region-aware masks and bounding boxes predicted by the MLLM (Li et al., 25 Aug 2025). This is alignment by inference-time control rather than gradient-based adaptation of model weights.

4. Alignment pathways across tasks

Reference-based multimodal preference alignment is not confined to a single application class. In text-to-image systems, it appears in both embedding-space adaptation and explicit diffusion control. The Bradley–Terry method modifies the text embedding or prompt representation used by CLIP or Stable Diffusion, so that the embedding moves toward preferred image embeddings and away from rejected ones (Gallego, 2023). The training-free diffusion framework instead decouples preference understanding from preference-guided generation: the MLLM extracts preference keywords from the reference image, expands the prompt into complex, object-level, and background sub-prompts, infers bounding boxes, and then combines global keyword guidance with local region-aware cross-attention modulation (Li et al., 25 Aug 2025).

In MLLMs, the main pathway is visually grounded rejection construction. AdaViP keeps the response fixed and changes the image so that the rejected sample becomes visually inconsistent with the preferred answer, thereby forcing the model to distinguish missing or altered evidence (Lu et al., 22 Apr 2025). MFPO takes a similar but independently motivated route: it compares the same preferred answer under the original image and a key-region-perturbed image, and supplements this with a margin term and an easy-to-hard curriculum based on semantic entropy (Jiang et al., 2024). Both methods are responses to the same diagnosis: text-only preference optimization can appear aligned while still being weakly grounded in visual tokens.

In language-model alignment without an explicit image channel, the reference pathway shifts from multimodal contrast to reference-answer similarity. RefAlign treats a whole sampled response as the action and uses BERTScore against a high-quality reference answer as the reward. The same mechanism is extended to safety alignment by introducing helpful and harmless references, and to confidence alignment by adding a confidence-quality consistency reward (Zhao et al., 14 Apr 2025). Although this setting is not image-conditioned, it remains reference-based in the strong sense that alignment supervision is derived from similarity to explicit exemplars rather than from binary preference pairs.

In recommendation, MSPA uses MLLM-generated preference text as the intermediary representation between user history and candidate authors. The recommendation prompt requires fields such as “User Preference,” “Recommendation Reason,” and “Answer,” so selection and explanation are coupled within the same multimodal decoding process (Guan et al., 13 Aug 2025). A plausible implication is that reference-based alignment can be interpreted not only as preference scoring but also as structured preference externalization.

5. Empirical behavior and benchmark evidence

The empirical literature reports improvements in few-shot personalization, hallucination reduction, trustworthiness, recommendation quality, and training-free preference-guided generation.

System	Representative reported result	Domain
Bradley–Terry CLIP adaptation (Gallego, 2023)	With 50 training pairs: CLIP-L original 0.548, CLIP-L positive-only $x$ 6, CLIP-L BT $x$ 7	Text-to-image preference prediction
AdaViP-7B (Lu et al., 22 Apr 2025)	Object HalBench Non-Rsp 93.7, Non-Men 96.4; MMHal-Bench hallucination 28.0 vs DPO 42.0	MLLM trustworthiness
MFPO on LLaVA-v1.5-7B (Jiang et al., 2024)	MMHalBench score 2.69, HalRate 0.49; Object HalBench CHAIR $x$ 8 13.4, CHAIR $x$ 9 6.6	Modality-fair hallucination reduction
RefAlign (Zhao et al., 14 Apr 2025)	Llama-3-8B-Instruct: 38.9 LC win rate, 47.0 raw win rate on AlpacaEval 2; Llama-2-7B TruthfulQA ECE 0.018 from vanilla 0.633	General preference and confidence alignment
MSPA (Guan et al., 13 Aug 2025)	U2A $(Q, \mathcal{I}_w, y_w)$ 0 77.78%, Recall@5 0.250; A2A Alignment Rate 0.809	Multimodal recommendation
Training-free instant alignment (Li et al., 25 Aug 2025)	Style Loss 0.663, Emotion Accuracy 60.2%, CLIP Score 0.288, Human preference 53.2%	Reference-image-guided diffusion

Several more granular comparisons are notable. In the Bradley–Terry study, CLIP-H improves from 0.644 to 0.788 and CLIP-G from 0.624 to 0.772, while PickScore is reported at 0.738. In Stable Diffusion generation, human evaluation by two annotators on 20 prompts gives win rates of 0.267 for CLIP-L original, 0.273 for CLIP-L positive-only, and 0.460 for CLIP-L BT (Gallego, 2023).

AdaViP reports an ablation in which naive equal-weight mixing of language- and vision-based preferences degrades AMBER discriminative performance: DPO yields F1 = 78.8 and Acc = 64.8, “+ Vision” drops to F1 = 72.6 and Acc = 51.8, while “+ Adaptive” improves to F1 = 85.8 and Acc = 79.9 (Lu et al., 22 Apr 2025). This directly supports the claim that adaptive weighting is not merely a regularization detail but a central part of the optimization design.

MFPO reports that easy-to-hard training outperforms end-to-end optimization on MMHalBench and Object HalBench, with 2.69 / 0.49 / 13.4 / 6.6 versus 2.53 / 0.53 / 16.0 / 8.0. Its loss-composition ablation shows that text-only optimization, image-only optimization, and text + image all improve over weaker baselines, but full MFPO with text + image + margin performs best (Jiang et al., 2024).

RefAlign presents a different empirical argument: in Anthropic HH, the average BERTScore similarity between chosen and rejected responses is about 0.054, and in about 70% of cases BERTScore-based selection is no worse than reward-model-based selection (Zhao et al., 14 Apr 2025). This is the quantitative basis for replacing binary preference labels with unary reference answers.

MSPA reports consistent offline replay gains as well as benchmark gains. Click AUC increases from 0.8315 to 0.8321, Click UAUC from 0.6416 to 0.6428, Gift AUC from 0.9501 to 0.9513, and Gift UAUC from 0.7167 to 0.7189. Its ablation shows $(Q, \mathcal{I}_w, y_w)$ 1 improving from 66.93 for the baseline to 73.08 without $(Q, \mathcal{I}_w, y_w)$ 2 and to 77.78 for full MSPA (Guan et al., 13 Aug 2025).

The training-free diffusion framework reports superiority over ViPer on style, emotion, CLIP, ImageReward, thematic alignment, and visual element alignment, and states that user studies show nearly 2× the user preference of Viper in the main comparison (Li et al., 25 Aug 2025). This suggests that reference-based multimodal alignment can be effective even without any additional training, provided that preference understanding and preference-guided control are sufficiently expressive.

6. Assumptions, limitations, and research directions

The principal limitations are heterogeneous but structurally related to how references are defined. The Bradley–Terry approach assumes pairwise comparisons, no ties in the preference data, and only addresses pairwise ranking; richer ranking models such as Plackett–Luce are explicitly suggested for multi-level preferences like $(Q, \mathcal{I}_w, y_w)$ 3 (Gallego, 2023). Its efficiency depends on the linearity of CLIP similarity and on the availability of informative pairwise preference examples.

AdaViP is developed for images and text only, with future work explicitly mentioned on video and audio. Its rejected-sample construction depends on RAM, GroundingDINO, SAM, LaMa, and CLIP, and it assumes that removing key visual elements creates a valid and informative rejected sample (Lu et al., 22 Apr 2025). MFPO likewise depends on accurate mapping from keywords to regions; when that fails, it falls back to global perturbation, which the paper treats as less informative than fine-grained region-based corruption (Jiang et al., 2024).

RefAlign’s limitations center on reward specification. The paper highlights an over-length issue, attributes it to long reference answers and recall-oriented similarity, and notes that using precision or F1 can shorten outputs but hurts benchmark performance. It also depends on high-quality references, reports that results improve when references are stronger, and states that human-written references have not yet been tested even though they are considered the gold standard (Zhao et al., 14 Apr 2025).

The training-free diffusion framework notes that minor preferences may be missed because of limitations in MLLM training data, and that current diffusion backbones still fail on some unseen conditions (Li et al., 25 Aug 2025). MSPA, while framed as self-corrective and interpretable, reports averaging outputs over three generations due to instability, which indicates that reinforcement-based multimodal recommendation still inherits decoding variance from the underlying MLLM (Guan et al., 13 Aug 2025).

Taken together, these limitations suggest several common research directions. One is richer preference structure: moving from pairwise to multi-level or groupwise ranking. Another is broader modality coverage: extending image-text alignment to video and audio. A third is improving the quality of references themselves, whether through stronger exemplar selection, human-written references, or more reliable construction of image-side negatives. A final direction is preserving fine-grained and minority preference signals without sacrificing semantic fidelity or stability. Across the surveyed work, reference-based multimodal preference alignment is best understood not as a single algorithm but as a design principle: alignment supervision is anchored to explicit exemplars, and the central technical problem is how to convert those exemplars into stable, modality-aware optimization or control signals.