M3PO: Multimodal Model-Guided Preference Optimization

Updated 8 July 2026

The paper introduces M3PO, which optimizes LVLMs by selecting ‘learning-valuable’ response pairs using multimodal alignment scores and model confidence.
It employs a dual-signal ranking process that combines external multimodal assessment with the model’s internal confidence to mine informative hard negatives.
Experiments on LLaVA models show that M3PO improves visual instruction-following outcomes with consistent performance gains over traditional tuning methods.

Multimodal-Model-Guided Preference Optimization (M3PO) is a data-efficient alignment method for Large Vision-LLMs (LVLMs) in visual instruction-following that constructs preference pairs entirely from model-generated candidates and a multimodal scoring model, rather than from large amounts of human preference labels, and then optimizes the resulting pairs with Direct Preference Optimization (DPO) (Gao et al., 17 Aug 2025). In the formulation presented in "M3PO: Multimodal-Model-Guided Preference Optimization for Visual Instruction Following" (Gao et al., 17 Aug 2025), an LVLM $p_\theta(\cdot)$ receives an image $I$ and an instruction $Q$ , generates multiple candidate responses, scores them by external multimodal quality and internal model confidence, selects the most "learning-valuable" preferred–dispreferred pair, and fine-tunes base models such as LLaVA-1.5-7B and LLaVA-1.5-13B with LoRA-based DPO (Gao et al., 17 Aug 2025).

1. Problem formulation and motivation

M3PO is formulated for visual instruction-following, where an LVLM $p_\theta(\cdot)$ takes multimodal input $x=(I,Q)$ and generates a textual response $R$ (Gao et al., 17 Aug 2025). The alignment target is not merely fluent generation, but responses that are visually grounded in $I$ , correctly follow the instruction $Q$ , and satisfy human-like preferences such as helpfulness, correctness, and non-hallucination (Gao et al., 17 Aug 2025).

The method is motivated by limitations of supervised fine-tuning (SFT), RLHF, and vanilla DPO in multimodal settings. In the description accompanying M3PO, SFT is characterized as depending on expensive, slow, and inconsistent human annotation, while also failing to expose the model to hard negatives from its own generation space (Gao et al., 17 Aug 2025). RLHF is described as requiring a multimodal reward model, suffering from possible reward-model noise or bias over visual inputs, and incurring substantial computational overhead (Gao et al., 17 Aug 2025). Vanilla DPO, although simpler, still relies on preference pairs from human annotation or reward models and does not systematically mine the model’s own confidently generated but externally incorrect responses (Gao et al., 17 Aug 2025).

The central motivation is therefore that multimodal preference data is costly because annotators must jointly understand the image and instruction, visual grounding is harder to judge than text-only correctness, and labels show variability and bias on complex scenes (Gao et al., 17 Aug 2025). M3PO addresses this by generating multiple responses per image–instruction pair, evaluating them automatically for multimodal alignment, and combining that external assessment with the model’s own confidence to identify "confidently wrong" outputs that are especially informative for alignment (Gao et al., 17 Aug 2025).

This places M3PO within a broader line of multimodal preference optimization methods that seek finer control over hallucination, modality balance, and hard-negative selection. Related work has emphasized modality bias and noise-robust objectives (Zhang et al., 23 Mar 2025), mixed preference signals for multimodal reasoning (Wang et al., 2024), adversarial negative mining and hybrid offline–online optimization (Liu et al., 20 May 2025), cross-modal and hierarchical DPO objectives (Fu et al., 28 Jan 2025), sentence-level adaptive weighting (Wang et al., 25 May 2025), token-level uncertainty-aware exploration (Zhang et al., 6 May 2026), clinically weighted multimodal preference data (Zhu et al., 2024), and theory-consistent symmetric preference structures (Liu et al., 13 Jun 2025). M3PO’s distinct contribution is to make hard-negative selection depend jointly on external multimodal quality and internal model confidence (Gao et al., 17 Aug 2025).

2. Candidate generation and preference-pair construction

For each image–instruction pair $(I,Q)$ , M3PO first performs candidate generation. The base LVLM, such as LLaVA-1.5-7B or LLaVA-1.5-13B, generates a set of $N=32$ diverse responses,

$I$ 0

using stochastic decoding such as top- $I$ 1 or temperature sampling (Gao et al., 17 Aug 2025).

Each candidate is then scored by two signals. The first is an external Multimodal Alignment Score,

$I$ 2

produced by a separate multimodal assessment system (Gao et al., 17 Aug 2025). The second is the model’s internal confidence, represented by the log-probability

$I$ 3

typically computed as the sum of token-level log-probabilities under the reference LVLM (Gao et al., 17 Aug 2025).

M3PO constructs a single preferred–dispreferred pair per input. The preferred response $I$ 4 is the candidate with highest multimodal alignment score,

$I$ 5

The dispreferred response $I$ 6 is not simply the lowest-scoring candidate. Instead, it is chosen to be a hard negative: a response that is externally worse but not much lower in model confidence, so that it reflects a mistake the model is actually inclined to make (Gao et al., 17 Aug 2025).

This construction yields a dataset

$I$ 7

which is then used for DPO fine-tuning (Gao et al., 17 Aug 2025). The guiding principle is to find, for each visual instruction, the most learning-valuable pair rather than a random or trivially separable comparison (Gao et al., 17 Aug 2025).

A plausible implication is that M3PO shifts the bottleneck of preference alignment away from human pairwise labeling and toward the quality of candidate generation and multimodal assessment. That implication is consistent with the method’s reliance on generated candidate pools and external multimodal scoring, but it remains an interpretation rather than an explicitly stated claim.

3. Multimodal Alignment Score, self-consistency, and the M3P-Score

The Multimodal Alignment Score (MAS) is defined as a scalar score assigned by an external multimodal assessment model for a response $I$ 8 given image $I$ 9 and instruction $Q$ 0 (Gao et al., 17 Aug 2025). The description states that MAS is not derived from the LVLM being fine-tuned and that the implementation mentions using CLIP/BLIP-2 combined with GPT-4V (Gao et al., 17 Aug 2025). MAS is intended to reflect visual relevance, semantic correctness, and instruction adherence, but it is used comparatively within each candidate set rather than as an absolute calibrated reward (Gao et al., 17 Aug 2025).

The internal confidence signal is the response log-probability,

$Q$ 1

which functions as a measure of self-consistency or generative confidence (Gao et al., 17 Aug 2025). High confidence does not imply correctness; M3PO explicitly targets cases where confidence remains high even when external multimodal quality is low (Gao et al., 17 Aug 2025).

The core ranking mechanism is the M3P-Score. After choosing $Q$ 2 by MAS, M3PO defines, for each other candidate $Q$ 3,

$Q$ 4

The dispreferred response is then

$Q$ 5

The default experimental hyperparameters are $Q$ 6 and $Q$ 7 (Gao et al., 17 Aug 2025).

The first term increases when $Q$ 8 is clearly worse than the best response according to MAS. The second term penalizes candidates that are much lower-confidence than $Q$ 9, thereby downweighting easy negatives that the model would not generate anyway (Gao et al., 17 Aug 2025). The overall effect is to privilege responses that are externally inferior yet internally plausible.

This hard-negative logic distinguishes M3PO from several nearby approaches. RM-DPO is described as using a reward model to produce or filter preference pairs but not integrating internal confidence into dispreferred-sample selection (Gao et al., 17 Aug 2025). ASPO, by contrast, refines preference optimization at sentence level using CLIP similarity and perplexity-derived adaptive rewards (Wang et al., 25 May 2025). CHiP adds visual preference optimization and hierarchical textual preference optimization at response, segment, and token levels (Fu et al., 28 Jan 2025). UE-DPO reallocates token-level learning pressure using epistemic uncertainty about visual grounding (Zhang et al., 6 May 2026). SymMPO enforces symmetric response preference across similar images while maintaining DPO-theoretic consistency (Liu et al., 13 Jun 2025). M3PO remains response-level, but its distinctive operation is to select the negative by combining MAS differences with self-confidence (Gao et al., 17 Aug 2025).

4. DPO objective and implementation details

Once $p_\theta(\cdot)$ 0 has been constructed, M3PO applies a standard DPO loss over preferred and dispreferred responses: $p_\theta(\cdot)$ 1 Here $p_\theta(\cdot)$ 2 is the fine-tuned policy, $p_\theta(\cdot)$ 3 is the reference LVLM, $p_\theta(\cdot)$ 4 controls preference strength, and $p_\theta(\cdot)$ 5 is the sigmoid (Gao et al., 17 Aug 2025).

In implementation, M3PO is instantiated on LLaVA-1.5-7B and LLaVA-1.5-13B (Gao et al., 17 Aug 2025). Fine-tuning is LoRA-based, with only low-rank adapter parameters trainable while base weights remain frozen, which is described as lowering memory use, speeding training, and reducing catastrophic forgetting (Gao et al., 17 Aug 2025). The reported training settings are batch size 8, learning rate $p_\theta(\cdot)$ 6, 5% warm-up, and typically 1 epoch of DPO training over the M3PO preference dataset (Gao et al., 17 Aug 2025).

Training pairs are curated from LLaVA-Instruct-150K and ShareGPT4V-80K (Gao et al., 17 Aug 2025). For each pair, the system generates 32 candidates with the base LLaVA model, computes MAS via CLIP/BLIP-2 + GPT-4V, computes log-probabilities via base LLaVA, and selects $p_\theta(\cdot)$ 7 using the M3P-Score (Gao et al., 17 Aug 2025).

The method is explicitly characterized as "DPO + multimodal model-guided data construction" (Gao et al., 17 Aug 2025). That formulation is useful because it identifies M3PO less as a new preference objective than as a preference-data construction strategy specialized for multimodal hard-negative mining (Gao et al., 17 Aug 2025).

5. Experimental results and efficiency

M3PO is evaluated on MME-Bench, POPE, IFT, and Human Pref. Score, with an evaluation suite of approximately 15k image–instruction pairs (Gao et al., 17 Aug 2025). It is compared against Base LLaVA-1.5-7B/13B, SFT, RLHF (Simulated), DPO (Vanilla), and RM-DPO (Gao et al., 17 Aug 2025).

For LLaVA-1.5-7B, M3PO reports 1402.3 on MME-Bench, 87.35 on POPE Acc., 71.80 on IFT, and 3.38 on Human Pref., exceeding the corresponding scores of SFT, simulated RLHF, vanilla DPO, and RM-DPO (Gao et al., 17 Aug 2025). For LLaVA-1.5-13B, M3PO reports 1537.8 on MME-Bench, 89.30 on POPE Acc., 74.70 on IFT, and 3.65 on Human Pref., again outperforming the listed baselines (Gao et al., 17 Aug 2025).

Model	MME-Bench	POPE Acc.	IFT	Human Pref.
LLaVA-1.5-7B + M3PO	1402.3	87.35	71.80	3.38
LLaVA-1.5-13B + M3PO	1537.8	89.30	74.70	3.65

The reported interpretation is that the gains are modest but consistent, indicating that model-guided hard-negative mining provides incremental but reliable improvements in multimodal instruction following (Gao et al., 17 Aug 2025).

The ablation on the confidence term is central to the method’s justification. Full M3PO with $p_\theta(\cdot)$ 8 reaches 1402.3 on MME-Bench, 87.35 on POPE, 71.80 on IFT, and 3.38 on Human Pref., whereas the variant without the confidence term ( $p_\theta(\cdot)$ 9) reaches 1395.0, 87.00, 71.20, and 3.25 respectively (Gao et al., 17 Aug 2025). This is presented as evidence that using model confidence is crucial because it avoids easy, low-confidence negatives and improves learning from negatives the model would actually produce (Gao et al., 17 Aug 2025).

Hyperparameter sensitivity analysis reports best performance at $x=(I,Q)$ 0 when $x=(I,Q)$ 1, and best performance at $x=(I,Q)$ 2 when $x=(I,Q)$ 3, for LLaVA-1.5-7B (Gao et al., 17 Aug 2025). This is described as indicating robust behavior while also showing that balancing MAS difference and confidence matters (Gao et al., 17 Aug 2025).

In efficiency terms, for LLaVA-7B the reported train times and peak memory are 18 GPU hours and 16 GB for SFT, 30 GPU hours and 24 GB for simulated RLHF, 10 GPU hours and 14 GB for vanilla DPO, 12 GPU hours and 14 GB for RM-DPO, 10 GPU hours and 14 GB for M3PO fine-tuning only, and 20 GPU hours total for M3PO including data generation plus fine-tuning, with peak memory listed as 16/14 GB (Gao et al., 17 Aug 2025). The stated conclusion is that DPO-based methods, including M3PO, are more efficient than RLHF, and that M3PO adds a one-time cost for candidate generation and MAS scoring while keeping fine-tuning cost comparable to vanilla DPO and RM-DPO (Gao et al., 17 Aug 2025).

6. Relation to adjacent multimodal preference-optimization methods

M3PO belongs to a rapidly expanding family of multimodal preference-optimization approaches, but it occupies a specific methodological niche. It differs from CHiP, which introduces a visual preference optimization module and a hierarchical textual preference optimization module spanning response, segment, and token levels (Fu et al., 28 Jan 2025). It differs from ASPO, which adds sentence-level adaptive rewards based on CLIP similarity and perplexity without introducing extra models or parameters (Wang et al., 25 May 2025). It differs from UE-DPO, which uses token-level epistemic uncertainty under blurred images to reweight preferred and dispreferred tokens (Zhang et al., 6 May 2026). It also differs from SymMPO, which constructs symmetric response pairs across similar images and adds a preference margin consistency loss while preserving DPO-theoretic cancellation properties (Liu et al., 13 Jun 2025).

A second nearby cluster concerns modality bias and hard-negative design. NaPO constructs language-biased and vision-biased negatives by masking modalities and uses a noise-aware preference objective that interpolates between BCE-like and MAE-like behavior via a negative Box–Cox transformation (Zhang et al., 23 Mar 2025). MBPO constructs adversarial hard negatives by perturbing images, measures modality usage through Image Information Gain, and combines offline preference learning with online GRPO on verifiable tasks (Liu et al., 20 May 2025). MMedPO creates plausible hallucinations and lesion-region neglect examples in medical LVLMs, then weights preference pairs by clinically derived relevance scores (Zhu et al., 2024). These methods emphasize that multimodal preference optimization can be guided by modality balance, clinical salience, adversarial perturbation, or noise robustness. M3PO instead emphasizes the conjunction of external multimodal alignment and internal self-confidence as a selector for informative hard negatives (Gao et al., 17 Aug 2025).

A third line focuses on reasoning-oriented or transfer-oriented preference learning. MPO combines DPO, BCO, and SFT losses to improve multimodal chain-of-thought reasoning using the MMPR dataset (Wang et al., 2024). MINT uses an upstream multimodal model to generate chosen and rejected label lists and aligns downstream unimodal or vision-language decoders primarily with ORPO (Wu et al., 9 May 2025). These are related in that they also rely on model-guided preference construction, but their aim is not specifically the M3PO criterion of finding the most learning-valuable preferred–dispreferred pair from a candidate pool according to MAS and log-probability (Gao et al., 17 Aug 2025).

This suggests that M3PO is best understood not as a universal replacement for other multimodal preference methods, but as a targeted recipe for response-level hard-negative mining in visual instruction-following. Where CHiP and ASPO inject finer-grained supervision, and where MBPO or NaPO directly target modality imbalance and noisy synthetic labels, M3PO optimizes the informativeness of pair selection itself (Gao et al., 17 Aug 2025).

7. Limitations, misconceptions, and extensions

The method’s stated strengths are data efficiency, reduced dependence on human preference labels, targeted hard-negative mining, multimodal robustness through MAS, and compatibility with simple DPO fine-tuning using LoRA (Gao et al., 17 Aug 2025). Its stated limitations are equally important.

First, M3PO relies on external MAS models such as CLIP/BLIP-2 + GPT-4V or similar systems, which may be proprietary, expensive to query, biased, or weak in specialized domains such as medical or fine-grained scenes (Gao et al., 17 Aug 2025). Second, the candidate pool size is fixed at $x=(I,Q)$ 4, which may under-sample difficult examples and oversample easy ones depending on task complexity (Gao et al., 17 Aug 2025). Third, the formulation targets single-turn visual instruction-following; extending it to multi-turn dialogue would require rethinking MAS, confidence, and preference construction over histories rather than isolated responses (Gao et al., 17 Aug 2025). Fourth, although LoRA reduces cost, scaling to larger LVLMs or continuous-training pipelines presents engineering challenges (Gao et al., 17 Aug 2025).

A common misconception would be to treat M3PO as merely vanilla DPO with synthetic labels. The method’s distinguishing mechanism is the M3P-Score, which explicitly seeks low-MAS but high-confidence negatives rather than arbitrary rejected responses (Gao et al., 17 Aug 2025). Another misconception would be to interpret MAS as a mathematically specified analytic reward. In the description, MAS is a scalar from an external multimodal assessment model, not a closed-form expression (Gao et al., 17 Aug 2025).

The paper itself lists several possible extensions: better alignment scorers using ensembles or task-specific MAS models, adaptive candidate generation with variable $x=(I,Q)$ 5, multi-turn extensions, and transfer of the M3PO paradigm to other modalities such as audio–text or video–text models (Gao et al., 17 Aug 2025). A plausible implication is that M3PO could be combined with techniques from adjacent work—for example, noise-aware loss design from NaPO (Zhang et al., 23 Mar 2025), sentence-level weighting from ASPO (Wang et al., 25 May 2025), or token-level uncertainty guidance from UE-DPO (Zhang et al., 6 May 2026)—to form hybrid systems that improve both pair selection and intra-response credit assignment. That implication goes beyond the explicit claims of the M3PO paper, but it is consistent with the surrounding research landscape.

In conceptual terms, M3PO defines a specific answer to a broader alignment question in multimodal modeling: if a model can already generate diverse responses, which pair should be used to teach it most effectively? Its answer is to prefer the externally best response and oppose it with the externally worse response the model is still most liable to believe (Gao et al., 17 Aug 2025).