Llama-Nemoretriever-Colembed: Cross-Modal Retrieval

• Llama-Nemoretriever-Colembed is a unified cross-modal retrieval model that combines bidirectional attention with ColBERT-style late interaction for detailed token-level matching.
• The model offers 1B and 3B parameter variants, with the 3B variant achieving state-of-the-art results on ViDoRe benchmarks through fine-grained similarity comparisons.
• Its two-stage training—from contrastive text pretraining to multimodal fine-tuning—demonstrates robust performance in digital libraries, enterprise search, and other retrieval applications.

Llama-Nemoretriever-Colembed is a unified text-image retrieval model developed to address the challenges of cross-modal retrieval tasks, delivering state-of-the-art performance across established benchmarks as of June 2025. The architecture is based on modifications to the NVIDIA Eagle2 Vision-LLM (VLM) and is characterized by the combination of bidirectional attention and a ColBERT-style late interaction mechanism. Two model variants have been released: a 1B-parameter model and a 3B-parameter model, with the 3B variant achieving leading results on the ViDoRe leaderboards.

1. Architectural Foundations

Llama-Nemoretriever-Colembed is built upon the NVIDIA Eagle2 VLM, introducing two central innovations. First, the architecture transitions from causal (uni-directional) attention—where each token attends only to its predecessors—to bidirectional attention, enabling every token in an input sequence to attend to all other tokens, regardless of their position. This shift significantly enhances the model's capacity to form fully-contextualized token representations, which is crucial for the retrieval setting where both queries and documents may contain complex, multimodal information.

Second, the model incorporates a ColBERT-style late interaction mechanism. Standard bi-encoder paradigms collapse document and query representations into single pooled vectors, sacrificing token-level detail. In contrast, the late-interaction approach preserves distinct token embeddings for both queries and documents. During inference, the model implements the MaxSim operator for token-level matching:

$$R(q, d) = \sum_{i=1}^{|q|} \max_{j=1}^{|d|} \text{sim}(q_i, d_j)$$

where $q = \{q_1, \ldots, q_n\}$ and $d = \{d_1, \ldots, d_m\}$ represent the respective token embeddings of the query and document, and $\text{sim}(\cdot, \cdot)$ is a similarity function such as cosine similarity or dot product. This mechanism allows for fine-grained interactions that underlie the model's high retrieval accuracy.

2. Retrieval Performance and Benchmarking

The 3B-parameter model variant of Llama-Nemoretriever-Colembed demonstrates high effectiveness on visual document retrieval tasks. On the ViDoRe V1 benchmark, it achieves an nDCG@5 of 91.0, and on ViDoRe V2, an nDCG@5 of 63.5, outperforming previous benchmarks (best baselines being 89.9 and 60.7, respectively). These gains underscore the importance of fine-grained, token-level matching in ranking relevant results. In practical terms, this improved capability translates to users consistently receiving more accurate and relevant documents or images in response to their queries compared to previous retrieval systems.

3. Storage and Computational Trade-offs

The integration of the late interaction mechanism introduces substantial trade-offs with respect to storage and computational efficiency. By retaining token-level representations for every document, the storage requirements scale with the number of tokens rather than the number of documents alone. For example, the complete output dimension (3072) requires approximately 10,000 GB to store embeddings for one million images—a storage cost orders of magnitude higher than that of pooled bi-encoder models.

There is likewise a computational cost: the MaxSim operation over all token pairs between a query and a set of candidate documents increases inference latency, especially as the number of tokens or documents scales. To mitigate these overheads, the authors propose the use of linear projection to reduce the embedding dimensionality—for instance, from 3072 down to 512—which yields reduced storage demands while incurring only a modest reduction in retrieval accuracy. Alternative strategies such as binary quantization are also discussed as potential means to further economize resources.

A summary of the primary trade-offs is as follows:

Mechanism	Storage Requirement	Retrieval Performance	Inference Latency
Late Interaction	High (per-token)	Superior	Increased
Traditional Bi-Encoder	Low (per-document)	Lower	Faster
Linear Proj. Variant	Intermediate	Slightly Reduced	Reduced (vs full)

The choice between late-interaction models and bi-encoders may thus depend on application-specific requirements for accuracy versus resource constraints.

4. Two-Stage Training Methodology

Training proceeds in two distinct stages:

• Stage 1: The model, equipped with bidirectional attention, is pretrained on large-scale text-only retrieval corpora using contrastive learning. The InfoNCE loss is employed:

$$\mathcal{L}(q, d^+, D_n) = -\log \left[ \frac{\exp(\text{sim}(q, d^+)/\tau)}{\sum_{d \in \{d^+\} \cup D_n} \exp(\text{sim}(q, d)/\tau)} \right]$$

where $d^+$ is the positive document and $D_n$ the set of negatives. This stage builds a robust language-based retrieval backbone and exhibits strong transferability to the multimodal retrieval setting.

• Stage 2: Fine-tuning is performed on aligned text-image pair datasets, encouraging the model to bridge modalities by aligning text and vision feature spaces. Throughout both stages, hard-negative mining is utilized: negatives are chosen where similarity scores are within 95% of the positive instance, driving the model to discriminate between subtle relevancy cues.

5. Applications and Use Cases

Llama-Nemoretriever-Colembed is intended for deployments where retrieval across both textual and visual modalities is essential. Typical applications include:

• Digital Libraries: Facilitating retrieval of relevant documents or images in multimedia corpora comprising text, scanned pages, figures, and diagrams.
• Enterprise and Legal Search: Handling search within scanned reports, slide decks, or heterogeneous business document collections.
• Retrieval-Augmented Generation Systems: Enabling downstream models to access both text and imagery as context for generation or analysis.

Its scalability and precision make it particularly advantageous in settings that require exact identification of multimodal information from extensive and diverse datasets.

6. Broader Impact and Research Implications

The innovations introduced by Llama-Nemoretriever-Colembed highlight new directions for research in multimodal retrieval. The employment of bidirectional attention for richer contextualization, paired with fine-grained token-level late interaction, provides a robust paradigm for bridging diverse data modalities. However, the marked increase in storage and computational cost emphasizes the need for further advances in efficient model architectures, approximate search mechanisms, or cascaded reranking modules that can preserve high retrieval performance while meeting practical deployment requirements. This line of work signals the growing importance of multimodal retrieval for information access and the ongoing tension between performance gains and operational costs.