UniME-V2: Universal Multimodal Embedding

• UniME-V2 is a universal multimodal embedding framework that unifies text, image, audio, video, and other modalities using soft semantic matching and advanced hard negative mining.
• It employs an MLLM-guided mechanism to compute soft semantic scores, enhancing compositional discrimination and retrieval precision with joint pairwise and listwise optimization.
• Experiments across 36 datasets show state-of-the-art performance, with significant gains in tasks requiring nuanced, fine-grained semantic discrimination.

Universal Multimodal Embedding (UniME-V2) denotes a family of models, algorithms, and learning frameworks that produce a unified vector space for diverse modalities—encompassing images, text, audio, video, graphs, fused modalities, and more—such that their embeddings enable cross-modal retrieval, grounding, and generalization across a wide spectrum of downstream tasks. The UniME-V2 paradigm is characterized by advanced mechanisms for discriminative representation learning, hard negative mining guided by Multimodal LLMs (MLLMs), and explicit modeling of soft semantic similarities to achieve state-of-the-art performance in universal embedding benchmarks and compositional reasoning tasks (Gu et al., 15 Oct 2025).

1. Conceptual Foundations and Objectives

UniME-V2 is motivated by critical limitations in classic dual-encoder approaches—such as contrastive image–text pre-training (CLIP)—which suffer from token truncation, modality-isolated encoding, weak compositionality, and simplistic negative sampling (Gu et al., 24 Apr 2025). The essential UniME-V2 goal is to establish a truly universal, discriminative embedding space across any combination of modalities and instruction types. The central innovations can be summarized as:

• Leveraging the advanced comprehension and generative abilities of MLLMs to assess and guide embedding space learning.
• Introducing soft semantic matching scores computed by MLLM-based judges to capture fine-grained distinctions between candidates.
• Mining diverse, high-quality hard negatives via global retrieval and semantic assessment, rather than relying solely on in-batch negatives.
• Employing soft supervision (with soft labels from the MLLM judge) in the optimization objective, relaxing rigid one-to-one mapping and improving semantic alignment.
• Integrating listwise and pairwise optimization objectives in reranking modules to enhance downstream retrieval precision and discrimination.

This holistic methodology realizes a universal embedding that is robust, compositionally expressive, and generalizes to both seen and unseen (out-of-distribution) modalities and tasks.

2. MLLM-as-a-Judge: Semantic Soft Labeling and Hard Negative Mining

A core contribution of UniME-V2 is the MLLM-as-a-Judge mechanism (Gu et al., 15 Oct 2025). The model deploys a pre-trained MLLM to semantically assess the alignment between a query (q) and a candidate (c) using an instruction such as:

"I will provide you with a query and a candidate. Evaluate whether the candidate meets the requirements of the query..."

The MLLM outputs logits for “Yes” (e_y) and “No” (e_n) classes, allowing computation of the soft semantic matching score:

$s_i = \frac{e_{y,i}}{e_{y,i} + e_{n,i}}$

These scores offer a scalar, continuous assessment of semantic alignment between each query–candidate pair, providing:

• Filters to exclude false negatives from the hard negative pool.
• Soft supervision signals (as opposed to strict binary indicators) for aligning predicted embedding similarities with genuine semantic relations.

For large datasets, potential hard negative candidates Ωₚ for query $q$ are first selected by global retrieval using a strong baseline (e.g., VLM2Vec (Jiang et al., 7 Oct 2024)), thresholded by backbone similarity. The MLLM further evaluates each, retaining only those candidates which pose challenging semantic ambiguity.

3. Learning Objectives and Supervisory Signal

UniME-V2 aligns the model-predicted similarity distribution with the semantic matching scores by minimizing a symmetric Kullback–Leibler divergence (effectively, the Jensen–Shannon divergence) between the probabilistic similarity vector $P$ and the MLLM-guided soft label vector $Q$, where: $$P(e_q, E_c) = \frac{\exp\left(\cos(e_q, e_{c^+})/\tau\right)}{\exp\left(\cos(e_q, e_{c^+})/\tau\right) + \sum_i\exp\left(\cos(e_q, e_{c^-_i})/\tau\right)}$$

$$Q(S_c) = \frac{\exp\left(s_{q,c_t}/\tau\right)}{\exp\left(s_{q,c_t}/\tau\right) + \sum_i\exp\left(s_{q,c_i}/\tau\right)}$$

where $e_q$ is the encoded query, $e_{c^+}$ the label-positive encoding, $e_{c^-_i}$ the hard negative encodings, $s_{q,c_t}$ the semantic score for the target, and $s_{q,c_i}$ for each selected negative.

The loss is: $$L = \frac{1}{2}\left[ \mathrm{KL}(P \| Q) + \mathrm{KL}(Q \| P) \right]$$

Soft labels permit the model to express degrees of semantic similarity, better matching the nuanced nature of complex, compositional multimodal tasks.

4. UniME-V2-Reranker: Joint Pairwise and Listwise Optimization

In addition to primary embedding training, UniME-V2 introduces a reranking module that further refines retrieval results by leveraging joint pairwise and listwise objectives. The reranker is trained on scenarios where the model is given a set of candidates (including ground-truth and mined hard negatives) and is tasked with:

• Predicting a "Yes"/"No" outcome in pairwise comparisons, operationalized by cross-entropy loss (e.g., $$L_{pair} = CE(\mathrm{YES}, \eta(q, c_t)) + CE(\mathrm{NO}, \eta(q, c_h))$$).
• Identifying the correct index of a target candidate among a list, also using cross-entropy loss ($L_{list}$).

The full reranker loss is $L_{pair} + L_{list}$. This dual-level objective ensures the model not only differentiates between the best and hardest incorrect candidate but also ranks the entire candidate set more accurately, leading to improved recall and robustness—especially in challenging, fine-grained discrimination scenarios (Gu et al., 15 Oct 2025).

5. Benchmarking, Results, and Empirical Findings

Comprehensive experiments on the MMEB benchmark (spanning 36 datasets and tasks: classification, retrieval, VQA, grounding, etc.) consistently show that UniME-V2 outperforms state-of-the-art baselines (including VLM2Vec, GME, and original UniME) by margins of 2–4 points in average precision and recall metrics. The model achieves significant gains particularly on:

• Tasks requiring compositional reasoning and nuanced discrimination among fine-grained semantics.
• Retrieval under both in-distribution and out-of-distribution settings.
• Scenarios with abundant hard negatives, where the semantic soft labeling mitigates the impact of noisy or borderline cases.

A cyclical candidate selection ensures that at least ten diverse hard negatives are used per query, with duplication or fallback to random candidates for rare shortfalls.

6. Contextual Significance and Future Directions

UniME-V2 addresses key limitations in prior art by:

• Replacing rigid, binary supervision with a framework where similarity judgments are informed by the semantic capacity of MLLMs.
• Enhancing the diversity and difficulty of hard negatives used during training, which is central to producing embeddings capable of supporting zero-shot transfer, open-set retrieval, and semantic clustering.
• Integrating listwise (global ranking) and pairwise (fine discrimination) reranking signals, effectively bridging retrieval and re-ranking model strengths.

Envisioned directions for UniME-V2 include leveraging even more advanced MLLMs as judges, expanding to novel modalities (e.g., extending to video, audio, biomedical signals beyond text and images), improving scalable deployment, and further automating the mining and alignment of high-fidelity hard negatives.

7. Technical Comparisons with Related Approaches

UniME-V2 contrasts with previous universal embedding work on several axes:

Method	Negative Mining	Supervision Type	Semantic Alignment
CLIP	In-batch (random)	Hard (binary targets)	Cosine similarity only
VLM2Vec	In-batch/global	Hard (binary targets)	Contrastive InfoNCE
UniME-V2	Global + MLLM-guided	Soft (semantic scores)	MLLM-as-Judge guidance

This soft-label paradigm avoids forced one-to-one mappings and enables the embedding space to reflect graded semantic similarity—an essential property for fine-grained, compositional, and retrieval-augmented downstream tasks.

---

UniME-V2 represents a substantial evolution in the design and training of universal multimodal embedding models. By integrating MLLMs as a source of fine-grained semantic supervision and employing joint optimization techniques, the framework achieves state-of-the-art discriminative and compositional performance, setting a standard for future advances in the area of universal embedding learning (Gu et al., 15 Oct 2025).