Universal Multimodal Retrieval

• Universal multimodal retrieval is the task of mapping diverse modalities like text, image, video, and audio into a shared semantic space for any-to-any search.
• It employs unified embedding strategies, generalized contrastive losses, and reasoning-augmented frameworks to overcome modality gaps and enhance retrieval accuracy.
• UMR systems improve scalability, cross-modal alignment, and multilingual support, achieving state-of-the-art metrics on extensive benchmarks.

Universal multimodal retrieval (UMR) is the task of retrieving the most semantically relevant items from a large, heterogeneous corpus where both queries and candidate documents may consist of any combination of modalities, such as text, images, video, and audio. UMR systems are designed to support “any-to-any” search scenarios, for example, retrieving an image with a text query, retrieving a full document with a composed (image+text) query, or supporting text-to-video search. Modern UMR aims to abolish modality-specific silos, instead unifying all modalities within a single embedding space or retrieval framework, while robustly handling real-world and complex search demands—such as reasoning over compositional queries, supporting fine-grained constraints, and enabling cross-lingual generalization. Achieving universal retrieval requires innovations in model architecture, embedding learning, training data curation, curriculum learning, reasoning-augmented representations, cross-modal alignment, and evaluation methodology.

1. Theoretical Foundations and Design Principles

UMR formalizes the retrieval problem over a corpus $\mathcal{D} = \{d_i\}_i$ where each $d_i$ consists of one or more modalities: text $t_i$, image $v_i$, audio $a_i$, and/or video $u_i$. Queries $q$ may similarly span any or all present modalities. The solution seeks an embedding model $f$ or a bi-encoder pair $(f_q, f_c)$ such that $z_q = f_q(q)$ and $d_i$0 reside in a shared metric space—typically $d_i$1—and a similarity function $d_i$2 supports effective nearest-neighbor retrieval for any combination of query/candidate modalities (Zhang et al., 2024, Lin et al., 2024, Zhang et al., 6 Feb 2026).

A central challenge is closing the “modality gap”: naive alignment leads to clustering by modality rather than semantics and causes severe degradation when queries and keys are of mismatched or fused types (Lee et al., 30 Sep 2025). UMR frameworks therefore employ generalized contrastive losses and other alignment objectives that both minimize intra-class distances (across modalities) and maximize intra-modal separation.

Further, universal retrieval must handle:

• Heterogeneity: arbitrary query–candidate combinations (text/image/audio/video/fused).
• Semantic alignment: mapping disparate modalities to a shared conceptual space.
• Latent reasoning: disambiguating underspecified queries, handling composition, and entity linking.
• Scalability: supporting millions of items and high-throughput inference (low $d_i$3 cost).
• Adaptivity: dynamic reasoning and processing depth conditioned on per-query complexity (e.g., TRACE’s adaptive routing (Hao et al., 3 Mar 2026)).
• Multilinguality: language-agnostic embedding and support for intent-rich, conversational queries (Zhang et al., 21 Jan 2026, Madasu et al., 2022).

2. Model Architectures and Embedding Strategies

Unified Embedding and Bi-/Dual-Encoders

Most state-of-the-art UMR systems rely on a shared bi-encoder architecture—distinct query and document encoders with shared or aligned weights—extracting embeddings via mean pooling, [EOS]/[CLS] token pooling, or via prompt-induced compression (Lin et al., 2024, Zhang et al., 2024). Multi-modal LLMs (MLLMs) such as Qwen2-VL and LLaVa are increasingly used as transversal backbone models (Li et al., 20 Jul 2025, Zhang et al., 2024).

Fusion Mechanisms:

• Score-level fusion: Weighted sums of unimodal encoder outputs; often used for CLIP-based systems (Wei et al., 2023, Lin et al., 2024).
• Feature-level fusion: Early or late fusion via small Transformer heads to enable inter-modality cross-attention (Wei et al., 2023, Zhou et al., 2024).
• Token-level interleaving: Used for text–image pairs; jointly processed by the LLM for unified embeddings (Lin et al., 2024).

Unified modalities: Systems such as GME, UNITE, Omni-Embed-Nemotron, and OmniRet support text, image, video, and audio with unified or late fusion, using separate modality-specific encoders funneled into a shared transformer bottleneck or a joint projection/pooling layer (Zhang et al., 2024, Xu et al., 3 Oct 2025, Huynh et al., 2 Mar 2026).

Fine-grained pooling: Advanced pooling schemes such as Attention Sliced Wasserstein Pooling preserve more local detail during corpus compression for audio and video (Huynh et al., 2 Mar 2026).

Reasoning-Augmented and Agentic Retrieval

Several recent frameworks extend bi-encoder models with generative or agentic abilities:

• Chain-of-Thought (CoT) reasoning: TRACE and V-Retrver use generative LLMs to decompose complex queries into explicit reasoning traces, either compressing the trace to a single embedding (TRACE) or using stepwise, agentic reasoning to rerank and verify candidates (V-Retrver) (Hao et al., 3 Mar 2026, Chen et al., 5 Feb 2026).
• Agentic interleaved reasoning: V-Retrver alternates between hypothesis generation and targeted visual inspection via external tools (e.g., SELECT-IMAGE, ZOOM-IN), reducing speculative errors in visually ambiguous scenarios (Chen et al., 5 Feb 2026).
• Curriculum and adaptive routing: Models like TRACE learn to dynamically allocate reasoning depth, enabling high throughput on reflexive queries and deeper CoT for complex, compositional queries (Hao et al., 3 Mar 2026).

Multilingual and Multimodal Alignment

Models such as MuMUR and LaBSE-ViT-L/14-based systems integrate multilingual text encoders with unified vision and audio backbones, supporting retrieval across a hundred languages and any input structure (Zhang et al., 21 Jan 2026, Madasu et al., 2022). Pseudo-label translation and multi-task NLU integration further boost cross-lingual generalization.

3. Data Curation, Training Methodologies, and Loss Functions

Proper data curation, curriculum design, and objective selection are pivotal for universal retrieval:

Data Curation

• Diverse and balanced modality coverage: Curated mixtures of text–text, text–image, and text–video pairs are essential. UNITE uses a carefully controlled composition (e.g., 21.6% TT, 39.2% TI, 36.1% TV) to close the modal gap (Kong et al., 26 May 2025).
• Synthetic fused-modal data: GME synthesizes millions of high-quality text–image combined pairs through automatic doc2query, entity extraction, and image retrieval/generation (Zhang et al., 2024). MegaPairs generates over 26 million synthetic image–text–instruction triplets using LLM-prompted annotation and open-domain image mining (Zhou et al., 2024).
• Fine-grained and semantically explicit annotation: Reasoning-augmented retrieval augments both queries and corpus entries via dense, VLM-generated captions and query rewriting (Zhang et al., 6 Feb 2026).

Losses and Optimization

• Generalized contrastive learning (GCL): Extends the standard InfoNCE loss to cover all cross-modality pairs within each batch (image, text, fused), ensuring generalization to novel and unseen query–candidate modality combinations (Lee et al., 30 Sep 2025).
• Modality-aware hard negative mining: To mitigate bias and prevent the model collapsing to the dominant modality, negatives are explicitly balanced or mined to match the modal distribution of positives (Lin et al., 2024, Li et al., 20 Jul 2025).
• Masked contrastive learning (MAMCL): UNITE masks out negatives with different modality tags in the contrastive loss, improving cross-modal calibration (Kong et al., 26 May 2025).
• Reinforcement learning and curriculum: V-Retrver and Retrv-R1 deploy multi-stage training sequences—SFT, rejection-based filtering, and evidence-aligned RL (e.g., Group Relative Policy Optimization)—to incentivize efficient, low-hallucination stepwise reasoning (Chen et al., 5 Feb 2026, Zhu et al., 3 Oct 2025).

Reasoning-augmented retrieval and pretext tasks

• Captioning and reasoning augmentation: Explicitly generating semantic descriptions for both queries and candidates (e.g., region-based CoT for fine-grained vision, or structured VLM-based captions for images) supports robust matching for underspecified or compositional queries (Zhang et al., 6 Feb 2026, Guo et al., 6 Aug 2025).
• Integration with NLU: NLU heads for intent classification and slot-filling are incorporated for robust retrieval with noisy, natural-language, or multilingual queries (Zhang et al., 21 Jan 2026).

4. Evaluation Protocols, Benchmarks, and Empirical Results

Benchmarks

• M-BEIR: Standardized, multi-dataset universal multimodal benchmark; 16 retrieval tasks, 5.6M candidates. Many UMR models, including U-MARVEL, GME, MM-Embed, TRACE, V-Retrver, and UniIR report on this suite (Li et al., 20 Jul 2025, Zhang et al., 2024, Lin et al., 2024, Hao et al., 3 Mar 2026, Chen et al., 5 Feb 2026, Wei et al., 2023).
• UMRB: Comprehensive benchmark including text, image, visual document, and fused-modal retrieval, with over 47 subtasks (Zhang et al., 2024).
• MMEB, MMEBv2, CaReBench, ACM, FineVideo: Multi-modal, audio-centric, and video-centric tasks for evaluating broader modality coverage (Huynh et al., 2 Mar 2026, Kong et al., 26 May 2025, Xu et al., 3 Oct 2025).

Metrics

Common metrics include Recall@K (R@1,5,10,50), nDCG@K, mAP@5, median/mean rank, and task-specific VQA accuracy. Modality accuracy and retrieval efficiency (queries/sec, memory, compute) are also increasingly reported.

Empirical Results

• SOTA performance: GME, U-MARVEL, TRACE, Retrv-R1, and V-Retrver all report consistently leading Recall@K or nDCG@K on M-BEIR and UMRB. For example, V-Retrver-7B achieves R@K of 69.7% vs. 64.8% for U-MARVEL-7B, and +4.9% gain over previous best (Chen et al., 5 Feb 2026).
• Fine-grained/compositional tasks: V-Retrver, TRACE, and reasoning-augmented approaches outperform prior embedding-only models on FashionIQ, CIRR, and compositional image retrieval (Chen et al., 5 Feb 2026, Hao et al., 3 Mar 2026).
• Data scale and ablation: Synthetic fusion and more contrastive negative mining yield substantial improvements. MegaPairs models trained on 26M synthetic pairs markedly outperform previous baselines trained on 70× less data (Zhou et al., 2024). GCL boosts local/global retrieval accuracy by up to +11 points depending on the backbone (Lee et al., 30 Sep 2025).
• Multilingual and cross-modal generality: Models such as MuMUR, LaBSE-based encoders, and UNITE demonstrate high multilingual R@1 and R@10 on image/text/video retrieval over 12+ languages (Madasu et al., 2022, Zhang et al., 21 Jan 2026, Kong et al., 26 May 2025).

5. Specializations, Extensions, and Applications

Reasoning-Augmented and Agentic Reasoning

Explicit chain-of-thought and external tool interaction (e.g., multimodal inspection, region zoom) enable grounding and robust matching under visual ambiguity and underspecified text queries (Chen et al., 5 Feb 2026, Hao et al., 3 Mar 2026). These approaches significantly reduce hallucination and speculative ranking, and improve zero-shot generalization to new modalities and instructions.

Few-Shot Fine-Grained Visual Classification

By casting FGVC as a multimodal retrieval task over structured attribute captions (CDV-Captioner), universal, training-free few-shot classification is enabled, outperforming both CLIP and fully-supervised MLLM baselines on challenging datasets (Guo et al., 6 Aug 2025).

Video and Audio-Centric Retrieval

Unified architectures (OmniRet, Omni-Embed-Nemotron, UNITE) now support seamless retrieval across text, vision, audio, and video, using attention-based resampling, late fusion, and large-scale curriculum training. These models achieve new records on video, document, and audio-centric tasks, while inefficient early fusion and naive poolings degrade performance (Huynh et al., 2 Mar 2026, Xu et al., 3 Oct 2025, Kong et al., 26 May 2025).

Multilingual and Intent-enriched Scenarios

Pseudo-labeled multilingual datasets, NLU integration, slot/value-attentive query representation, and multi-task curriculum yield state-of-the-art R@10 on XTD10, Multi30K, and other language-rich retrieval challenges (Zhang et al., 21 Jan 2026, Madasu et al., 2022).

6. Current Limitations and Open Problems

Universal retrieval models, despite their advances, are subject to several challenges:

• Information bottleneck: Jointly compressing rich multi-modal or long-sequence inputs into a single vector may discard salient fine-grained details (Huynh et al., 2 Mar 2026).
• Scaling Law: UMR model performance increases linearly with more data/training but remains limited by modality balance, efficiency, and sparse fused-modal corpora (Zhang et al., 2024).
• Modality and data imbalance: High-quality, balanced fused-modal data remains a bottleneck; models are sensitive to distribution drift and mix proportions (Kong et al., 26 May 2025).
• Inference efficiency: Increasing reasoning depth, agentic inspection (e.g., V-Retrver, Retrv-R1), and explicit CoT can reduce query throughput. Adaptive routing helps recover some speed on simple queries (Hao et al., 3 Mar 2026, Zhu et al., 3 Oct 2025).
• Fusion architecture: Late fusion is more scalable for video/audio, but richer cross-modal attention remains an area for improvement (Xu et al., 3 Oct 2025).
• Zero-shot compositionality: Robust generalization to unseen combinations of complex instructions, temporal/event graphs, or even new modalities (audio, 3D) is still open (Zhang et al., 6 Feb 2026, Huynh et al., 2 Mar 2026).

7. Outlook and Future Directions

Emerging directions include:

• Richer modality mix: Extending UMR to multi-image, multi-page, 3D, depth, and conversation-driven modalities (Zhang et al., 2024, Huynh et al., 2 Mar 2026).
• Dynamic curriculum and adaptive selection: Query-conditioned reason-selection, efficiency–effectiveness trade-off, and dynamically sampled data scheduling (Hao et al., 3 Mar 2026, Kong et al., 26 May 2025).
• Integration with Retrieval-Augmented Generation (RAG): Tight coupling of UMR systems with LLMs for open-domain QA, knowledge grounding, and compositional generation (Caffagni et al., 10 Sep 2025).
• All-in-one benchmarks: A major open need is for standard, multi-modality, dynamic evaluation suites covering retrieval, recommendation, summarization, dialog, and document QA (Kong et al., 26 May 2025).
• Multilingual and multi-intent scaling: Scaling UMR to >100 languages, supporting complex, noisy user queries, and dynamic intent detection/interpretation (Zhang et al., 21 Jan 2026, Madasu et al., 2022).
• Model efficiency: Compressing, pruning, and quantizing UMR systems for on-device and low-resource deployment (Guo et al., 6 Aug 2025, Li et al., 20 Jul 2025).
• Automated data synthesis: Expanding the use of large-scale synthetic data via LLM pipelines and instructive pair mining to drive further advances (Zhou et al., 2024, Zhang et al., 2024).

Ongoing progress in these dimensions suggests universal multimodal retrieval will become an increasingly practical paradigm, providing a foundation for broad, robust, and interpretable search across all digital media.