MM-Embed: Universal Multimodal Embeddings

• MM-Embed is a family of universal multimodal embedding models that encode text, images, and combinations into a shared embedding space for direct similarity computation.
• It uses a bi-encoder architecture with modality-specific encoders, lightweight projectors, and contrastive loss with hard negative mining to optimize retrieval performance.
• The model mitigates modality bias through explicit instruction prompts and prompt-and-rerank strategies, achieving state-of-the-art results on benchmarks like M-BEIR and MTEB.

MM-Embed

MM-Embed refers both to a family of universal multimodal embedding models leveraging multimodal LLMs (MLLMs) and to several related technical innovations for robust, general-purpose multimodal retrieval. MM-Embed models encode data from multiple modalities (e.g., text, images, and combinations thereof) into a shared embedding space, enabling direct similarity computations for information retrieval, semantic matching, and downstream reasoning tasks. Recent developments center on overcoming the intrinsic modality bias and architectural constraints of MLLMs, advancing contrastive learning techniques, and exploring fine-grained alignment and efficiency in large-scale scenarios (Lin et al., 2024).

1. Model Architecture and Universal Embedding Space

MM-Embed adopts a bi-encoder paradigm grounded in MLLMs. The architecture consists of modality-specific encoders (e.g., CLIP vision encoders, text encoders) and a language-centric backbone, often augmented by lightweight projectors and conditioning mechanisms.

For the canonical instantiation on LLaVa-Next, each input (query or candidate) is processed via a combination of:

• Vision encoder: A frozen CLIP-large image encoder generating embedding vectors for image tokens.
• Projector: An MLP that adapts visual features into the language embedding space.
• LLM: A 7B-parameter LLM (Mistral 7B) augmented with LoRA adapters.
• Input instruction: Task-specific prompts prepended to queries and candidates to encode contextual semantics and ensure instruction-aware embedding.
• Embedding extraction: The representation is typically taken from the final hidden state corresponding to a summary token (e.g., <eos>), with l₂ normalization ensuring cross-modal comparability. Encoders process both unimodal (text, image) and multimodal (text+image) inputs.

Bi-encoder retrieval operates by encoding the query and candidate separately:

$$f_q(q) = \text{normalize}\left(h(\text{inst}, q)\right), \quad f_c(c) = \text{normalize}\left(h(\text{inst}, c)\right)$$

where $h$ encompasses vision pre-processing, tokenization, input concatenation, and final aggregation (Lin et al., 2024).

2. Training Objectives: Contrastive Loss and Hard Negative Mining

Training uses the InfoNCE contrastive loss:

$$L_{\text{InfoNCE}} = - \frac{1}{N} \sum_{i = 1}^N \log \frac{\exp(f_q(q_i)^\top f_c(c_i) / \tau)}{\sum_{j=1}^N \exp(f_q(q_i)^\top f_c(c_j) / \tau)}$$

where temperature $\tau$ controls the sharpness of similarity distributions.

Standard in-batch negatives are augmented by modality-aware hard negative mining to address the tendency of MLLMs to prefer certain modalities, e.g., returning text in text–>image retrieval. Hard negative mining operates by:

• Retrieving top-50 candidates with a pre-fine-tuned “random negative” retriever.
• Partitioning negatives into: C₁ (high-ranking but undesired modality) and C₂ (lower ranking but correct modality).
• Sampling negatives equally from C₁ and C₂ for robust optimization.

Continual fine-tuning further integrates text-only retrieval data (e.g., MS MARCO, NQ), mining hard negatives using a high-performing text retriever and optimizing over joint multi-modal/text batches (Lin et al., 2024).

3. Modality Bias and Mitigation Strategies

Empirically, MLLM-based retrievers suffer from modality bias: retrieving candidates in the “wrong” modality even when the query specifies otherwise (e.g., returning text for a text–>image query). Modality accuracy at top-1 (M.A.@1) quantifies this effect, with MLLM retrievers underperforming text-specialized models (e.g., CLIP) (Lin et al., 2024).

Mitigation is achieved via:

• Modality-aware hard negative mining (as detailed above), which explicitly teaches the model to avoid affordable but modality-incorrect solutions.
• Bi-encoder instruction prompts that make the target modality explicit.
• Mixing batches from single- and multi-modal retrieval tasks, aligning the model’s semantic space across tasks.

4. Benchmark Evaluation and Results

MM-Embed is evaluated on the M-BEIR benchmark—a collection of 10 sources spanning 16 retrieval tasks and four retrieval domains (text, image, multi-modal). M-BEIR includes a diversity of retrieval problems, such as:

• text–>image: MSCOCO, Fashion200K, VisualNews
• text–>text: WebQA, EDIS
• multi-modal: InfoSeek, OVEN, CIRR, FashionIQ

Key metrics include Recall@K and nDCG@K, assessing retrieval on relevance.

Quantitative results:

• On M-BEIR overall R@5 (or R@10 on Fashion): MM-Embed achieves 51.9 (Mrand: 47.4, Mhard: 48.3).
• For single-modal queries: MM-Embed improves to 56.4 versus 51.0–53.5 for baselines.
• On MTEB (text-only): MM-Embed nDCG@10 = 60.3 versus 59.4 for NV-Embed-v1 (top-5 leaderboard).
• For multi-modal queries: MM-Embed boosts to 47.0 from 39.5–40.1 (Lin et al., 2024).

Prompt-and-rerank with off-the-shelf LLaVa-Next as a zero-shot reranker is shown to provide significant additional gains (up to +8.9 mAP@5 on CIRCO for composed queries).

5. Prompt-and-Rerank Strategies for Complex Queries

Beyond primary retrieval, MM-Embed explores zero-shot reranking by leveraging an MLLM as a pointwise True/False classifier. For each of the top-K retrieved candidates, a relevance prompt is constructed (e.g., “Does the above image match the caption? True or False”), and the LLM’s probability of ‘True’ is used for reranking.

This method delivered consistent improvements:

• On InfoSeek (img+txt → txt), R@5 improved from 35.8 to 37.1.
• On CIRCO (composed image retrieval), mAP@5 improved from 32.3 to 39.9 for Mrand and 29.0 to 37.9 for Mhard (Lin et al., 2024).
• Notably, such gains are largest on composed/multimodal and challenging query types.

6. Limitations and Future Directions

While MM-Embed advances universal multimodal retrieval, several limitations remain:

• Residual cross-modal bias, especially if hard negative mining is incomplete or suboptimal.
• Computational expense of large MLLM inference and reranking (not yet suitable for extreme-scale, millisecond-latency deployments).
• Aggregation bottlenecks due to single-token summarization—potentially omitting finer details in cross-modal relations.

Proposed directions include:

• Knowledge distillation into smaller bi-encoders (e.g., CLIP variants) for efficient deployment.
• Integrated, one-stage architectures where retriever and reranker functions are fused.
• Extending to additional modalities (video, audio, document, etc.) and highly compositional, multi-turn tasks (Lin et al., 2024).

7. Context in Multimodal Representation Learning

MM-Embed builds on a lineage of advances in multimodal and universal embeddings, integrating architectural and training innovations:

• Parallel Decoupling Frameworks for rich, diverse embedding spaces using mutual information minimization (Wang et al., 3 Nov 2025).
• Hard negative gradient amplifying (Xue et al., 28 May 2025) and explicit curriculum design (Chen et al., 7 Jan 2026) for more robust cross-batch learning.
• Reasoning-enhanced and pair-aware adaptive reasoning selection (Wang et al., 7 Apr 2026), demonstrating that generation and retrieval objectives can be synergistically unified.
• Strong performance and broad coverage on MMEB, MTEB, and emerging large-scale benchmarks have positioned MM-Embed paradigms at the current state of the art for universal retrieval.

Universal multimodal retrieval with MLLMs via MM-Embed represents a transition from modality-specific to truly cross-modal representation learning, with empirically validated recipes for task and modality integration, and a blueprint for scalable, instruction-aware, and extensible embedding models (Lin et al., 2024).