MISP-DPO: Multi-Negative Preference Optimization

• MISP-DPO is a framework for aligning vision-language models by leveraging multiple semantically diverse negatives to enhance preference learning.
• It employs semantic-aware negative mining in CLIP space with a sparse autoencoder to uncover interpretable deviation factors.
• The method uses a Plackett–Luce ranking objective and importance sampling to reduce training cost and mitigate hallucinations.

MISP-DPO, short for Importance Sampling for Multi-Negative Multimodal Direct Preference Optimization, is a framework for aligning vision-LLMs through preference learning with multiple, semantically diverse negative images rather than a single negative image. It was introduced to address a limitation of existing multimodal DPO methods, which typically rely on oversimplified pairwise comparisons generated by perturbation or similarity-based retrieval and therefore can induce optimization bias and hallucinations. The method combines semantic-aware negative mining in CLIP space, a sparse autoencoder for interpretable deviation factors, a Plackett–Luce objective for multi-negative ranking, and an importance sampling estimator that reduces training cost while retaining informative supervision (Li et al., 30 Sep 2025).

1. Problem setting and terminological scope

MISP-DPO is situated in the broader literature on multimodal Direct Preference Optimization for vision-LLMs. In the formulation described in the paper, a multimodal prompt $x$ is paired with a preferred image $m_p$ and aligned response $y_p$, and training encourages the policy $\pi_\theta$ to prefer $(m_p, y_p)$ over one or more negative image alternatives. The central claim is that pairwise multimodal DPO uses a single negative image per comparison, and that such a setup fails to capture the multi-faceted nature of visual errors such as object identity, attributes, layout, and scene context (Li et al., 30 Sep 2025).

The paper motivates multi-negative comparisons by arguing that ranking a positive image above a diverse set of negatives forces the policy to resolve multiple constraints simultaneously. This suggests a richer preference signal than one-dimensional deviation modeling. The method is explicitly framed as the first framework to incorporate multiple, semantically diverse negative images in multimodal DPO via the Plackett–Luce model (Li et al., 30 Sep 2025).

The acronym is potentially ambiguous in adjacent literatures. In the threat-intelligence literature on the MISP sharing platform, a 2019 paper states that it does not use the term “MISP-DPO”, though it notes that communities may refer to a “decaying policy” for indicators in MISP and that “MISP-DPO” can be instantiated as that policy object (Mokaddem et al., 2019). That usage is distinct from the multimodal alignment method. Likewise, the acronym DPO also denotes “Dynamic-Programming Optimization on Hybrid Constraints” in a separate line of work on Boolean MPE and project-join trees, which is unrelated to direct preference optimization (Phan et al., 2022).

2. Architectural pipeline

The MISP-DPO pipeline has two principal components: semantic-aware negative mining and multi-negative preference optimization. The mining stage begins by embedding prompts and candidate images in CLIP space. For a preferred pair $(m_p, x)$, the paper computes CLIP image and text embeddings $h_v = f_v(m_p)$ and $h_t = f_t(x)$ and fuses them by outer product and vectorization:

$$e = \operatorname{vec}(h_v \times h_t^\top) \in \mathbb{R}^{d^2}.$$

For each candidate negative image $m_n^i$, it then forms a difference vector

$m_p$0

These difference vectors define the semantic deviations that the model will use for negative selection (Li et al., 30 Sep 2025).

A sparse autoencoder (SAE) is trained over these difference vectors to uncover interpretable factors such as object, color, and layout. Negative candidates are then scored using three criteria given in the paper: reconstruction difficulty, semantic deviation, and mutual diversity. Reconstruction difficulty is measured by the error $m_p$1; semantic deviation is measured by the latent magnitude $m_p$2; and diversity is imposed through cosine dissimilarity among selected latent vectors (Li et al., 30 Sep 2025).

The optimization stage adopts a Plackett–Luce-style objective to rank the positive image above a set of $m_p$3 negatives. Because evaluating a large candidate set is computationally expensive, the method constructs a proposal distribution $m_p$4 from SAE-derived scores and diversity bonuses, then selects a smaller importance-sampled set $m_p$5. The high-level training loop described in the paper samples preferred triplets from RLHF-V, retrieves a large negative pool such as COCO, scores candidates in CLIP/SAE space, constructs $m_p$6, computes the image and text preference losses, applies the importance-sampled gradient estimator, and updates $m_p$7 via LoRA fine-tuning or full-parameter optimization (Li et al., 30 Sep 2025).

3. Mathematical formulation

The starting point is canonical pairwise DPO with a reference policy $m_p$8:

$m_p$9

where

$y_p$0

In the multimodal pairwise setting, the preferred response $y_p$1 is evaluated under the preferred image $y_p$2 and a negative image $y_p$3 for the same prompt $y_p$4 (Li et al., 30 Sep 2025).

MISP-DPO extends this pairwise structure through a Plackett–Luce ranking model. For a set containing the positive and $y_p$5 negatives, if the utilities are $y_p$6, the probability that the positive ranks first is

$y_p$7

The general Plackett–Luce likelihood over a full ranking $y_p$8 with active set $y_p$9 at stage $\pi_\theta$0 is

$\pi_\theta$1

This ranking formulation is the core departure from single-negative DPO (Li et al., 30 Sep 2025).

For each negative image $\pi_\theta$2, the paper defines the preference advantage

$\pi_\theta$3

These advantages are aggregated via a softmax over negatives, yielding the image-side loss

$\pi_\theta$4

The paper further gives the gradient decomposition

$\pi_\theta$5

$\pi_\theta$6

and

$\pi_\theta$7

This places the multi-negative update in a softmax-weighted preference-gradient form (Li et al., 30 Sep 2025).

The method also includes a text-side preference loss

$\pi_\theta$8

and combines modalities through

$\pi_\theta$9

The scalar $(m_p, y_p)$0 balances image and text terms (Li et al., 30 Sep 2025).

4. Semantic-aware multi-negative selection

The negative selection mechanism is the most distinctive component of MISP-DPO. Rather than drawing negatives from perturbations alone, the method scores a large open-domain candidate set using the latent structure exposed by the SAE. The training objective for the SAE is presented in both generic and instantiated form. The paper gives the instantiated loss as

$(m_p, y_p)$1

where $(m_p, y_p)$2 is the target average activation, $(m_p, y_p)$3 is the empirical activation of hidden unit $(m_p, y_p)$4, and $(m_p, y_p)$5 controls sparsity strength. In experiments, the latent dimension is 128 and $(m_p, y_p)$6 (Li et al., 30 Sep 2025).

Candidate scoring combines normalized reconstruction difficulty and latent magnitude:

$(m_p, y_p)$7

where $(m_p, y_p)$8 and $(m_p, y_p)$9 are the maximum reconstruction error and latent magnitude across candidates. Selection then uses a greedy diversity-promoting rule:

$(m_p, x)$0

The stated objective is to promote broad semantic coverage while emphasizing hard negatives (Li et al., 30 Sep 2025).

This design is presented as a response to a concrete failure mode of single-negative multimodal DPO: a model can learn a narrow rejection heuristic, such as rejecting a salient but incidental attribute, instead of learning robust cross-modal grounding. By spreading supervision across object, color, layout, and context deviations, MISP-DPO seeks to reduce spurious correlations and hallucinations. A plausible implication is that the method uses the SAE not merely as a compression module but as a structured proposal mechanism that shapes the geometry of the negative distribution seen during optimization (Li et al., 30 Sep 2025).

5. Importance sampling and training efficiency

The paper emphasizes that evaluating all negatives is expensive. If a prompt has $(m_p, x)$1 candidate negatives, exhaustive evaluation is $(m_p, x)$2 per prompt, whereas importance sampling reduces the cost to $(m_p, x)$3 with $(m_p, x)$4, yielding an approximate speedup of $(m_p, x)$5 in forward and backward passes for the image-side loss (Li et al., 30 Sep 2025).

The importance-sampled gradient estimator is obtained by rewriting the gradient under the proposal distribution $(m_p, x)$6. With importance weight

$(m_p, x)$7

the paper gives

$(m_p, x)$8

The proposal $(m_p, x)$9 is derived from SAE-based scores and diversity bonuses, favoring informative and diverse negatives (Li et al., 30 Sep 2025).

The reported training configuration uses LoRA fine-tuning with rank=64, $h_v = f_v(m_p)$0, learning rate $h_v = f_v(m_p)$1, $h_v = f_v(m_p)$2, $h_v = f_v(m_p)$3, 2×A100, per-device batch size 2, and gradient accumulation 8, for an effective batch size 32. The empirical ablation identifies $h_v = f_v(m_p)$4 negatives per prompt as optimal; the paper states that $h_v = f_v(m_p)$5 is insufficient, while $h_v = f_v(m_p)$6 can degrade performance due to redundancy and noise. Ablations also place the best $h_v = f_v(m_p)$7 range at 0.45–0.75, with 0.5 used in the main experiments (Li et al., 30 Sep 2025).

The paper notes practical caveats around the sampling scheme. Importance weights $h_v = f_v(m_p)$8 can be heavy-tailed if the proposal assigns low mass to high-advantage negatives; weight normalization or mild clipping may therefore be desirable in practice. This is explicitly presented as a limitation rather than as part of the core formalism (Li et al., 30 Sep 2025).

6. Empirical results, prior work, and limitations

MISP-DPO is evaluated on five benchmarks: MMHal-Bench, HallusionBench, POPE, WildVision, and MMVP. The models are LLaVA-1.5-7B-HF, Qwen2.5-VL-7B, and Qwen2.5-VL-3B; training data are drawn from RLHF-V (≈5k samples) and negatives are mined from COCO (Li et al., 30 Sep 2025).

For LLaVA-1.5-7B-HF, the paper reports that MMHal-Bench hallucination rate is reduced from 51.04% (Base) to 32.29% (MISP-DPO). On HallusionBench, MISP-DPO achieves aA 57.52, fA 25.43, and qA 24.83; on WildVision, reward −46.4 and win rate 20.6, compared with Base −55.7 and 17.0; on MMVP, accuracy 63.00; and the average improvement over Base is +30.09% (Li et al., 30 Sep 2025).

For Qwen2.5-VL-7B, the paper reports MMHal-Bench hallucination rate 11.46% (MISP-DPO) versus 18.09% (Base) and 13.83–16.67% for single- and multi-negative variants. It also reports HallusionBench aA 71.24, fA 43.77, qA 45.61, POPE 88.66 accuracy, and average improvement over Base +5.35%. For Qwen2.5-VL-3B, the reported numbers are MMHal-Bench hallucination rate 13.54% versus 22.34% (Base), HallusionBench aA 65.51, fA 38.44, qA 38.02, WildVision reward 8.6, win rate 52.4, and average improvement over Base +19.89% (Li et al., 30 Sep 2025).

The comparison set includes prior multimodal DPO variants such as mDPO, CHiP, S-VCO, and retrieval-augmented DPO, which the paper characterizes as typically pairwise and frequently dependent on perturbation or similarity retrieval. MISP-DPO is positioned relative to ranking-based extensions such as Softmax-DPO and DMPO, which generalize DPO to multi-negative textual settings using Plackett–Luce or soft ranking objectives. The novelty claim is threefold: semantic-aware multi-negative selection via SAE in CLIP space, a Plackett–Luce objective for multimodal comparisons, and an importance sampling strategy guided by the SAE-derived proposal distribution (Li et al., 30 Sep 2025).

The paper also identifies clear limitations. Performance depends on CLIP embedding quality; SAE factors may be entangled or hard to map to attributes in some domains; very large $h_v = f_v(m_p)$9 can increase gradient variance while offering diminishing returns; and the effectiveness of the proposal distribution depends on maintaining sufficient mass on high-advantage negatives. These caveats place the method in a broader pattern familiar from preference-based alignment: richer supervision improves robustness, but only if the representation geometry and sampling distribution remain well calibrated (Li et al., 30 Sep 2025).