Omni-Embed-Nemotron: Unified Multimodal Retrieval

Updated 14 July 2026

Omni-Embed-Nemotron is a unified multimodal retrieval model that encodes text, image, audio, and video into a single shared embedding space.
It uses a bi-encoder architecture with modality-specific encoders and a late fusion strategy to preserve contextual and temporal details.
The model enhances retrieval quality on complex, multimodal documents and often outperforms text-only baselines in varied retrieval tasks.

Omni-Embed-Nemotron is a unified multimodal retrieval embedding model for text, image, audio, and video, designed to support both cross-modal retrieval such as text–video and joint-modal retrieval such as text–video+audio within a single shared embedding space. It is positioned as a retrieval-oriented response to the limitations of text-only retrievers in Retrieval-Augmented Generation pipelines and in corpora containing visually and semantically rich objects such as PDFs, slides, and videos, where layout, speech, and temporal context can carry task-relevant information (Xu et al., 3 Oct 2025).

1. Problem setting and retrieval scope

Omni-Embed-Nemotron addresses a retrieval regime in which queries and indexed items are not restricted to clean text. The motivating setting is one in which standard text-based retrievers, although effective on structured corpora, struggle with real-world documents whose evidence is distributed across text, page layout, imagery, speech, and video. The model extends retrieval beyond text and images to audio and video, while retaining the ability to execute both cross-modal and joint-modal matching through one model rather than a collection of modality-specific retrievers (Xu et al., 3 Oct 2025).

A central antecedent is the observation, associated in the paper with ColPali, that preserving document layout using image-based representations can improve retrieval quality. Omni-Embed-Nemotron generalizes that intuition from text-and-image retrieval to a four-modality setting. In practical terms, this means that a query can target evidence located in transcript text, in a slide image, in audio content, in a video stream, or in fused audio-video evidence, without changing retrieval infrastructure.

2. Bi-encoder design and modality fusion

The model uses a bi-encoder architecture in which queries and corpus items are encoded into a shared, unified embedding space, enabling scalable retrieval through similarity computation. The architecture includes modality-specific encoders for text, image, audio, and video, while all modalities, including combinations such as video+audio, are mapped to the same embedding space. This shared-space design is what permits both cross-modal and joint-modal retrieval with a single index and a single scoring interface (Xu et al., 3 Oct 2025).

Its backbone is the Qwen-Omni / Qwen2.5-Omni-3B Thinker component, with the Talker module removed. A defining architectural decision is that audio and video are processed through separate encoding streams and are only merged at the embedding stage. The paper contrasts this with Qwen-style interleaved timestamped fusion. For retrieval rather than generation, keeping audio and video streams independent, with no cross-modal token interleaving, is reported to be superior because it better preserves contextual integrity and temporal structure.

This fusion choice is one of the model’s most consequential retrieval-specific claims. It implies that the architectural optimum for omni-modal generation need not coincide with the architectural optimum for omni-modal retrieval. A plausible implication is that retrieval objectives reward modality-preserving representation geometry more than early token-level entanglement.

3. Optimization, adaptation, and implementation

Training is based on contrastive learning with the InfoNCE objective,

$\mathcal{L}(q, d^+, D_N) = -\log \frac{\exp(\text{sim}(q, d^+)/\tau)}{\sum_{d_i \in \{d^+\} \cup D_N} \exp(\text{sim}(q, d_i)/\tau)}.$

Here, $q$ is the query embedding, $d^+$ is the positive document embedding, $D_N$ is the set of negatives, $\text{sim}(\cdot)$ is the similarity function, and $\tau$ is the temperature. Hard negatives are adopted from NV-Retriever using the top-k-with-percentage-to-positive threshold: negatives are chosen if their similarity to the query is less than $0.95$ of the query–positive similarity, with $K=2$ (Xu et al., 3 Oct 2025).

The adaptation strategy is parameter-efficient. The visual and audio encoders are frozen, only the LLM is updated, LoRA is applied with rank $r=16$ and scaling factor $\alpha=32$ , and bidirectional attention replaces causal attention. Initial training uses text-text and text-image pairs, while text-video and text-audio pairs are introduced only later experimentally. The reported training corpora include ColPali train set, Wiki-SS-NQ, VDR, Docmatix, NQ, StackExchange, SQuAD, HotpotQA, and FineVideo.

Sequence length is a nontrivial systems constraint in this design. The paper reports the following modality-dependent sequence lengths.

Modality setting	Sequence length
Text (Transcript + OCR)	3497
Audio-Only	3222
Video-Only	20758
Audio + Video (Fusion)	23960
Audio + Video (Separately)	23960

These figures clarify why processor configuration and fusion strategy matter operationally: video and audio-video inputs push token budgets far beyond those of transcript-based retrieval.

4. Empirical retrieval behavior

Evaluation spans video, image, and text retrieval. On video retrieval, the model is assessed on LPM, an educational lecture dataset with slides, video, audio, and ASR transcripts, and on FineVideo, a 10,000-video, 46,553-query open-domain benchmark. On FineVideo, Omni-Embed-Nemotron outperforms the text-only baselines evaluated on transcripts, while on LPM the strongest text-centric baseline remains slightly ahead. Averaged across the two datasets, however, Omni-Embed-Nemotron is the highest overall model in the reported comparison (Xu et al., 3 Oct 2025).

Retrieval setting	Metric	Omni-Embed-Nemotron result
LPM	NDCG@10 / NDCG@5	0.8465 / 0.8355
FineVideo	NDCG@10 / NDCG@5	0.5662 / 0.5486
ViDoRe V1	Avg. nDCG@5	85.7
Text retrieval benchmarks	Avg. nDCG@10	0.6059

The baseline comparisons in the paper are informative. On FineVideo, the best text-only baseline, Qwen/Qwen3-Embedding-4B, reaches NDCG@10 $q$ 0, while Omni-Embed-Nemotron reaches $q$ 1. On LPM, Qwen/Qwen3-Embedding-4B reaches NDCG@10 $q$ 2, compared with $q$ 3 for Omni-Embed-Nemotron. The paper also reports that domain-specific fine-tuning on FineVideo raises performance from NDCG@10 $q$ 4 and NDCG@5 $q$ 5.

Ablation findings further sharpen the retrieval picture. In lecture settings, text-only performs best; audio-only is better than video-only; audio+video fusion recovers some ground relative to the individual modalities. On FineVideo, the text-only variant surpasses all text baselines, and when text is missing, audio yields the best non-text results. The paper explicitly states that naive early fusion underperforms late fusion.

5. Diagnostic benchmarks, modality bias, and external critiques

Independent full-modality evaluation complicates the headline retrieval results. MMEB-V3 introduces a 190-task benchmark covering text, image, video, audio, visual document, and agent tasks, together with OmniSET, where tuples $q$ 6 share the same semantics across modalities. In that diagnostic setting, Omni-Embed-Nemotron is reported to often fail to retrieve the instructed target modality. The cited OmniSET entries are stark: $q$ 7, $q$ 8, $q$ 9, $d^+$ 0, and $d^+$ 1 in Hit@1. MMEB-V3 interprets such results as evidence that current multimodal embeddings exhibit query-modality bias, strong cross-modal asymmetry, and instruction-induced shifts that are insufficient or misaligned with the target modality; for OmniEmbed-Nemotron, the reported mean instruction shift is $d^+$ 2, but not necessarily toward the correct modality cluster (Huang et al., 25 Apr 2026).

A separate line of critique appears in e5-omni. That work argues that many recent omni-modal embeddings, including Omni-Embed-Nemotron, rely heavily on implicit alignment inherited from pretrained VLM backbones. In its comparative summary, Omni-Embed-Nemotron is described as using a single global temperature and naive hard-negative mining rather than explicit mechanisms for modality-aware temperature calibration, debiased curricula, and covariance alignment. Quantitatively, e5-omni-7B reports an MMEB-V2 average of $d^+$ 3 against $d^+$ 4 for Omni-Embed-Nemotron, and an AudioCaps Recall@1 of $d^+$ 5 against $d^+$ 6 for Omni-Embed-Nemotron (Chen et al., 7 Jan 2026).

Taken together, these external analyses indicate that Omni-Embed-Nemotron’s strengths are most visible in retrieval tasks where modality evidence improves matching over transcript-only baselines, while its weaknesses emerge in controlled settings that require reliable modality-aware retrieval behavior under explicit target-modality constraints.

Omni-Embed-Nemotron belongs to a rapidly expanding family of omni-modal systems, but its emphasis is specifically retrieval-centric. OmniEmbed, developed in the Tevatron 2.0 ecosystem, also maps text, images, audio, and video into a shared vector space and, after MultiVENT 2.0 fine-tuning, reports nDCG@10 $d^+$ 7 on the MAGMaR shared task. Conan-embedding-v3 takes a different route, using a decouple--fuse--recover pipeline and reporting $d^+$ 8 on MMEB and $d^+$ 9 on the 30-task MAEB audio suite (Zhan et al., 11 Jun 2025, Li et al., 8 Jun 2026).

By contrast, HyperCLOVA X 8B Omni and Nemotron 3 Nano Omni are unified multimodal understanding-and-generation systems rather than retrieval-specialized bi-encoders. HyperCLOVA X 8B Omni uses a shared next-token prediction interface over an interleaved multimodal sequence with text, audio, and vision as both inputs and outputs, while Nemotron 3 Nano Omni is an efficient open multimodal model with native support for text, images, video, and audio and a 256K-token context window (Team, 5 Jan 2026, NVIDIA et al., 27 Apr 2026).

Within that broader landscape, Omni-Embed-Nemotron is significant because it argues for a retrieval-specialized adaptation path from general omni-modal backbones: a bi-encoder objective, bidirectional attention, frozen modality encoders, and modality-preserving late fusion. This suggests that omni-modal retrieval is not merely a by-product of omni-modal generation. Instead, it may require its own representational geometry, negative sampling regime, and fusion design to remain competitive on text and image retrieval while extending meaningfully to audio and video.