Multi-faceted Cross-modal DPO

Updated 14 July 2026

MCM-DPO is a cross-modal preference optimization framework that integrates multifaceted signals from text, images, and context to generate concise, context-aware alt-text.
It extends standard DPO by jointly optimizing single-, pairwise-, and multi-preference dimensions, addressing issues like noisy annotations and modality insensitivity.
Empirical results show significant improvements in metrics like ROUGE-L and CIDEr, indicating better visual grounding and reduced hallucination in alt-text outputs.

Multi-faceted Cross-modal Direct Preference Optimization (MCM-DPO) is a preference-alignment framework for alt-text generation that learns from multifaceted, multimodal preference signals spanning images, post-text context, and alt-text responses. It extends standard Direct Preference Optimization (DPO) beyond text-only preferences and single-dimension comparisons by jointly optimizing single-, pairwise-, and multi-preference dimensions across textual, visual, and cross-modal factors. In the alt-text setting, this design addresses noisy user annotations, inconsistent annotation standards, and multimodal LLMs’ insensitivity to contextual information, while preserving the reference-policy anchoring and logistic preference training structure of DPO (Fu et al., 1 Oct 2025).

1. Task setting and conceptual scope

MCM-DPO is defined around the alt-text generation task, where the model takes a prompt $x$ , an image $m$ , post-text context $c$ when available, and outputs an alt-text response $y$ . In this formulation, alt-text is not treated as ordinary image captioning. The target output is a concise, context-relevant description for accessibility, especially for blind and low-vision users, and the objective emphasizes brevity, high information value, and contextual relevance rather than exhaustive visual detail. The framework explicitly analyzes cases where information such as “Elsa” comes from post-text; the resulting alt-text is concise and context-dependent, whereas captions are detailed (Fu et al., 1 Oct 2025).

The motivation for MCM-DPO rests on three empirical difficulties. First, user-written alt-text on social platforms contains grammatical errors, inconsistent quality, and varied stylistic standards. Second, annotation standards differ in length, specificity, and context reliance; only a tiny fraction of social images are annotated, reported as approximately $2\%$ on Twitter and $0.68\%$ on Pinterest in the crawl. Third, many multimodal models are optimized for captioning and tend to produce verbose descriptions, while conventional multimodal preference tuning usually applies preferences only at the response level and neglects alignment with images and context (Fu et al., 1 Oct 2025).

This leads to a central distinction from supervised fine-tuning (SFT). SFT depends on accurate target annotations and large-scale labeled datasets, but social alt-text is noisy and inconsistent. DPO instead learns from relative preferences between chosen and rejected outputs. MCM-DPO preserves that relative-learning advantage and augments it so that preference signals can be defined not only over responses, but also over images, contexts, and their combinations. A common misconception is that MCM-DPO is merely multimodal DPO with images appended to a textual prompt; the defining feature is the explicit optimization of cross-modal agreement and contextual grounding through multiple preference facets rather than response-only comparison (Fu et al., 1 Oct 2025).

The starting point is a multimodal DPO objective with a reference policy $\pi_{\mathrm{ref}}$ and inverse temperature $\beta$ . For multimodal samples with prompt $x$ , image $m_w$ , chosen response $m$ 0, and rejected response $m$ 1, the base multimodal DPO objective is

$m$ 2

In MCM-DPO, context $m$ 3 enters the preferences, and the reward $m$ 4 is instantiated by $m$ 5 times a log probability ratio with respect to $m$ 6. The framework organizes seven preference dimensions into three modules.

Module	Facet	Compared elements
Single Preference Optimization	RPO, VPO, CPO	response-only, image-only, context-only
Pairwise Preference Optimization	VRPO, CRPO, VCPO	image+response, context+response, image+context
Multi-Preference Optimization	MTPO	image+context+response

The three single-facet losses are

$m$ 7

$m$ 8

$m$ 9

The three pairwise losses are

$c$ 0

$c$ 1

$c$ 2

The multi-preference loss is

$c$ 3

These are combined into the aggregate target

$c$ 4

The weights $c$ 5, $c$ 6, and $c$ 7 control single-, paired-, and multi-facet contributions; the best observed setting is $c$ 8, $c$ 9, and $y$ 0. No extra margin terms are introduced; stability relies on facet weighting and temperature scaling, while the reference policy provides implicit KL anchoring through the log-likelihood ratio construction (Fu et al., 1 Oct 2025).

The significance of this decomposition is methodological rather than merely notational. Standard DPO focuses on text-only chosen versus rejected outputs. MCM-DPO substitutes or combines $y$ 1, $y$ 2, and $y$ 3 pairs, so the optimization can distinguish textual quality, visual grounding, and contextual relevance within one preference-training framework. This suggests a more structured form of credit assignment than vanilla multimodal DPO, especially in domains where the preferred response depends on external context rather than image content alone (Fu et al., 1 Oct 2025).

3. Data resources and preference construction

MCM-DPO is coupled to two alt-text datasets, TAlt and PAlt, sourced from Twitter and Pinterest. Together they provide 202k supervised triplets of context, image, and alt-text, along with 18k preference pairs across facets. The test sets have roughly 1.7k samples each (Fu et al., 1 Oct 2025).

Dataset	Training	Preference
PAlt	101,715	8,212
TAlt	101,078	9,938

Pinterest data were scraped from 22M posts using 100k common words for diversity, filtered to approximately 150k rough samples, with only $y$ 4 containing alt-text. Twitter uses the public dataset from Srivatsan et al. with 371k image–tweet–alt-text pairs. Filtering removes non-static images such as GIFs, non-English content, hashtags, and alt-text shorter than five words. Deduplication uses CLIP-based embedding similarity and visual match clusters via pixel overlap. Named entity recognition is used for Person Name Replacement, substituting names with “[person]”, and grammar is corrected with geimini-1.5-flash for both context and alt-text (Fu et al., 1 Oct 2025).

Preference construction is central to the method. Chosen responses are human-written alt-text after grammar correction. Rejected responses are generated by Gemini, specifically gemini-1.5-flash, using a “suboptimal alt-text” prompt with temperature sampling and manual filtering to ensure inferiority in accuracy and coherence while avoiding stylistic bias. Quality verification is also performed by Gemini, using position-swapping checks: a pair is kept only if the evaluator consistently selects the chosen item before and after swap, and the rejected sample can be regenerated up to three times (Fu et al., 1 Oct 2025).

The framework also constructs context and image rejections. Context rejections $y$ 5 are sampled from other training samples to encourage non-redundancy and context sensitivity. For image rejections $y$ 6, several strategies are explored: Diffusion noise, Blackness, Crop, Rotation, and Randomness. The best results are obtained with Diffusion noise at $y$ 7, while excessive dissimilarity is reported to hurt learning. This design encodes a substantive assumption about cross-modal preferences: negative samples should remain difficult enough to preserve semantic proximity, rather than collapsing into trivial mismatches (Fu et al., 1 Oct 2025).

4. Backbone, optimization protocol, and training regimes

The base multimodal LLM is LLaVA-1.6 (7B), pairing CLIP’s ViT as the vision encoder with Vicuna-7B as the language backbone. Stage 1 is supervised fine-tuning on the 202k mixed TAlt and PAlt triplets for one epoch at image resolution 336, learning rate $y$ 8, and batch size 128. Stage 2 performs preference optimization with DPO, mDPO, or MCM-DPO on TAlt and PAlt preference sets for three epochs with learning rate $y$ 9 and batch size 64. The reference policy $2\%$ 0 is the SFT checkpoint, following the standard DPO setup (Fu et al., 1 Oct 2025).

The training pipeline evaluates four vision-encoder paradigms. Paradigm-1 freezes the vision encoder in SFT and alignment; Paradigm-2 freezes it in SFT and trains it in alignment; Paradigm-3 trains it in SFT and freezes it in alignment; Paradigm-4 trains it in both stages. Paradigm-4 yields the best overall performance. For example, on PAlt, DPO with Paradigm-4 obtains ROUGE-L 32.71 and CIDEr 157.73, while MCM-DPO with Paradigm-4 reaches ROUGE-L 39.54 and CIDEr 207.98 (Fu et al., 1 Oct 2025).

The stagewise optimization loop is explicit. During preference optimization, facet batches are sampled for single facets, paired facets, and the joint multi-facet case. For each facet, the reward is computed as

$2\%$ 1

and the corresponding loss $2\%$ 2 is applied. The aggregated loss uses the weighted objective above, with $2\%$ 3, $2\%$ 4, and $2\%$ 5 (Fu et al., 1 Oct 2025).

The reported systems are trained on $2\%$ 6 A100 80GB GPUs. Standard DPO requires approximately 4.6 hours, and MCM-DPO requires approximately 5.0 hours, corresponding to a reported $2\%$ 7 overhead, while inference latency is unchanged. Code, data, and models are released for research use only; the dataset license is CC BY-NC 4.0, and usage follows LLaMA/Vicuna terms (Fu et al., 1 Oct 2025).

5. Empirical performance, ablations, and observed behavior

Evaluation uses ROUGE-L, BLEU4, METEOR, and CIDEr against user-written alt-text references. On PAlt, LLaVA-1.6 (7B)+SFT obtains ROUGE-L 24.63, BLEU4 7.63, METEOR 41.13, and CIDEr 99.27. Adding DPO raises these to 32.71, 11.11, 40.84, and 157.73; mDPO gives 36.63, 12.90, 41.37, and 183.89; MCM-DPO reaches 39.54, 14.48, 40.60, and 207.98. On TAlt, the corresponding progression is SFT 33.83/14.90/33.29/144.64, DPO 35.32/15.87/36.32/146.32, mDPO 35.74/15.78/35.77/153.15, and MCM-DPO 36.32/16.25/36.62/158.74 (Fu et al., 1 Oct 2025).

Human evaluation reports MCM-DPO greater than DPO by 29%, specifically 39% versus 10% preference, and Paradigm-4 greater than Paradigm-1 by 21%, specifically 32% versus 11%. Qualitative analysis states that MCM-DPO generates alt-text that is better grounded, concise, and context-aware, avoids repetition of post-text, and limits hallucinations in comparison to SFT, DPO, and unadapted caption models. A representation-space visualization further shows generated alt-text representations moving closer to ground truth under MCM-DPO than under SFT, DPO, or baseline LLaVA (Fu et al., 1 Oct 2025).

Ablation studies isolate the contribution of individual facets. Removing the Multi facet ( $2\%$ 8) sharply drops performance to ROUGE-L 33.60 and CIDEr 165.18. The contextual facet is reported as crucial, with large declines when it is removed through pair ablations. A sweep over $2\%$ 9 finds the best value at $0.68\%$ 0, again producing ROUGE-L 39.54 and CIDEr 207.98. For rejected-image construction, Diffusion with $0.68\%$ 1 performs best, and noise-step sensitivity peaks at that setting (Fu et al., 1 Oct 2025).

The method also transfers beyond alt-text evaluation. On ObjHal and AMBER, MCM-DPO reduces hallucination rates; the appendix reports, for example, AMBER Hal 27.9% for MCM-DPO versus 48.6% for LLaVA, a relative reduction of 42.6%, while maintaining object coverage. This does not convert the model into a general hallucination-free vision-language system, but it indicates that the multifaceted preference construction captures cross-modal grounding signals useful outside the immediate training task (Fu et al., 1 Oct 2025).

6. Position within the broader MCM-DPO research landscape

Within the preference-optimization literature, the distinctive design choice of MCM-DPO is to increase the dimensionality and modality of preferences rather than modify reward models or eliminate reference policies. The alt-text formulation retains $0.68\%$ 2 and $0.68\%$ 3 scaling from DPO, but enriches the preference space with single, paired, and multi-preference comparisons over response, image, and context (Fu et al., 1 Oct 2025).

Related work shows that this design pattern has broader manifestations. CHiP introduces visual preference optimization and hierarchical textual preference optimization across response, segment, and token levels for multimodal LLM hallucination reduction (Fu et al., 28 Jan 2025). CcDPO uses context-level and needle-level preference optimization for multi-image MLLMs, targeting omission, conflation, and detail misinterpretation through a context-to-cue hierarchy (Li et al., 28 May 2025). In diffusion models, MCDPO replaces scalar aggregation with a disentangled Bradley–Terry objective conditioned on a preference outcome vector, explicitly addressing reward conflict among axes such as Human, PickScore, Aesthetic, HPSv2, and CLIP (Jang et al., 11 Dec 2025). Semi-DPO provides a complementary diagnosis, theoretically showing that collapsing multi-dimensional visual preferences into binary labels creates conflicting gradient signals in diffusion DPO, and then treating conflicting pairs as noisy unlabeled data in a semi-supervised loop (Liu et al., 27 Apr 2026). CaPO addresses multi-reward inconsistency through reward calibration, Pareto-frontier pair selection, and a regression objective for diffusion alignment (Lee et al., 4 Feb 2025). DPOFusion applies instance-level DPO to infrared–visible image fusion, aligning human perceptual quality, VLM semantic congruence, and task-driven metrics (Su et al., 7 May 2026).

A more explicitly generalized route appears in Mix- and MoE-DPO, which formulates DPO under a latent expert-assignment model with mixture policies, a Mixture-of-Bradley–Terry likelihood, and an ELBO objective. That framework is summarized as a concrete path toward a Multi-faceted Cross-modal DPO in which a shared multimodal backbone or independent multimodal experts are combined with facet-specific rewards and input-dependent gating (Bohne et al., 9 Oct 2025). This suggests a broader interpretation of MCM-DPO as a family of cross-modal, multi-axis preference-alignment methods rather than a single loss design, although the named alt-text system remains the canonical instance in which the term is explicitly introduced (Fu et al., 1 Oct 2025).

The main limitations of the alt-text formulation are also clear. The pipeline depends on preference quality, since rejected generation and verification rely on Gemini-assisted procedures that may introduce bias. Domain coverage is constrained by the Pinterest crawl, where only $0.68\%$ 4 of posts had alt-text, and the investigation of cultural and gender biases is left for future work. Multifaceted preference construction increases training complexity, although the measured overhead is modest. Ethical concerns remain around privacy, bias, and residual hallucination despite person-name replacement and improved grounding (Fu et al., 1 Oct 2025).

Taken together, MCM-DPO denotes a transition from response-level alignment to structured multimodal preference alignment. In its original alt-text form, the method operationalizes seven losses over response, image, and context; in the broader literature, related systems apply the same principle to hallucination reduction, multi-image reasoning, diffusion alignment, and image fusion. The consistent theme is that a single holistic binary preference is often too coarse for multimodal generation, whereas decomposed, cross-modal preference signals provide a more faithful alignment target (Fu et al., 1 Oct 2025).