Nemotron ColEmbed V2: Late Interaction Retrieval

• Nemotron ColEmbed V2 is a family of visual document retrieval models that integrate vision-language backbones with ColBERT-style late interaction for precise multi-modal matching.
• The architecture leverages bidirectional attention, dynamic image tiling, and advanced data sampling with hard-negative mining to boost retrieval effectiveness and state-of-the-art performance.
• Empirical results on ViDoRe benchmarks demonstrate superior NDCG scores, with each variant balancing trade-offs between storage, latency, and retrieval quality for RAG systems.

Nemotron ColEmbed V2 refers to a family of late interaction embedding models explicitly architected for high-capacity visual document retrieval in the context of Retrieval-Augmented Generation (RAG) pipelines. These models integrate vision-LLM (VLM) backbones with ColBERT-style late interaction mechanisms and advanced data processing techniques to deliver state-of-the-art performance on the ViDoRe benchmark suite, which targets retrieval tasks involving complex, multi-modal (text, layout, visual) documents such as PDFs and presentation slides. The system is designed to maximize retrieval effectiveness for downstream generative tasks that require precise grounding in large, visually encoded enterprise document repositories (Moreira et al., 3 Feb 2026).

1. Model Family and Architectural Principles

Nemotron ColEmbed V2 comprises three principal model variants:

Model Name	Parameters	Embedding Dim.	Backbone
llama-nemotron-colembed-vl-3b-v2	3.99B	3072	NVIDIA Eagle 2 + Llama 3.2 3B
nemotron-colembed-vl-4b-v2	4.43B	2560	Qwen3-VL-4B-Instruct
nemotron-colembed-vl-8b-v2	8.14B	4096	Qwen3-VL-8B-Instruct

Key architectural features include:

• Vision-LLM Backbone Selection: The 3B variant initializes from the NVIDIA Eagle 2 VLM, which combines a SigLip 2 vision encoder with a Llama 3.2 3B language decoder. The 4B and 8B models use Qwen3-VL (4B and 8B), each augmenting a SigLip 2 encoder with a vision-language merger and Qwen3 LLM core.
• Bidirectional Attention: All variants replace the traditional causal (autoregressive) self-attention with fully bidirectional self-attention in every Transformer layer, enabling richer context propagation suitable for retrieval tasks as opposed to generation (Moreira et al., 3 Feb 2026, Xu et al., 7 Jul 2025).
• DeepStack Visual Token Injection: For the Qwen3-VL-based models, DeepStack is used to inject intermediate visual tokens at multiple LLM layers, facilitating deep multimodal fusion.
• Dynamic Image Tiling: Dynamic tiling segments document images into tiles during training (max_input_tiles=2) and inference (up to 8), feeding these into the vision encoder to balance context window size and efficiency.
• Late-Interaction Mechanism: Inspired by ColBERT, both query and document (page image) are encoded as token embedding sequences. At retrieval time, the scoring function computes, for query tokens $\{q_i\}_{i=1}^{n_q}$ and document tokens $\{d_j\}_{j=1}^{n_d}$:

$$\mathrm{score}(q, d) = \sum_{i=1}^{n_q} \max_{j=1}^{n_d} \mathrm{sim}(q_i, d_j)$$

using either dot-product or cosine similarity.

• Model Merging (Model Soups): Final checkpoints are weighted averages of several independently trained instances: 8 (3B) or 4 (4B, 8B), resulting in empirical gains of +0.8% to +1.5% NDCG@10.

2. Data Processing, Sampling, and Training Regimen

To guarantee both domain generalization and fine-grained discrimination, Nemotron ColEmbed V2 employs advanced data stratification and hard negative mining:

• Cluster-Based Sampling: Using a 3B VLM, positive document pages are first embedded into 3072-dimensional vectors. PCA reduces the dimensions to 50, and K-means (with $k$ from the gap statistic, typically 14) partitions the space. Positives are then uniformly sampled per cluster to avoid overfitting to dominant domains.
• Hard-Negative Mining: For every query, candidates are retrieved using an internal Llama-Eagle 3B embedding model. Negatives whose similarity falls below $0.95 \times \text{sim}(q, \text{pos})$ are selected (NV-Retriever strategy), yielding sentences or passages that are semantically close but non-matching—a key factor in improving contrastive learning.
• Contrastive Objective: The InfoNCE loss drives both positive alignment and negative separation. For inputs $q$, $d^+$ (positive), and $D_N$ (negatives):

$$\mathcal{L}(q, d^+, D_N) = -\log \left[\frac{\exp(\mathrm{sim}(q, d^+)/\tau)}{\sum_{d_i \in \{d^+\} \cup D_N} \exp(\mathrm{sim}(q, d_i)/\tau)}\right]$$

where $\tau=0.05$ (temperature), and $\mathrm{sim}(\cdot, \cdot)$ is the late-interaction MaxSim score.

• Training Stages: The 3B variant undergoes two-stage training—initial text-only triplet training followed by cross-modal (image-retrieval) fine-tuning—while 4B/8B rely on single-stage image-only contrastive learning due to stronger VLM pretraining.

3. Empirical Performance and Benchmarking

Nemotron ColEmbed V2 achieves state-of-the-art ranking on several leading retrieval leaderboards:

Model Name	ViDoRe V3 NDCG@10	ViDoRe V1+V2 NDCG@5	MIRACL-Vision Avg NDCG@10
nemotron-colembed-vl-8b-v2	63.42 (1st)	84.80 (2nd)	0.6860 (best)
nemotron-colembed-vl-4b-v2	61.54 (3rd)	83.87 (3rd)	0.6272
llama-nemotron-colembed-vl-3b-v2	59.79 (6th)	83.64 (4th)	0.6127

Ablation studies reveal:

• Model Merging: +0.8–1.5% NDCG@10 (largest for 8B).
• Hard Negatives: +0.7% NDCG@10.
• Cluster Sampling: +0.5% NDCG@10 through data domain balance.
• Late Interaction vs. Single-Vector Pooling: Up to +5% NDCG@10.

These results reflect highly effective multi-vector token-level modeling, augmented by sophisticated sampling and merging strategies (Moreira et al., 3 Feb 2026).

4. Storage, Latency, and Engineering Considerations

The late interaction paradigm necessitates storage of all per-token document embeddings, yielding substantial storage and compute demands:

Model	Embedding Dim.	Avg Tokens/Page	Storage per 1M Pages (FP16)
3B	3072	2304	13.2 TB
4B	2560	--	3.7 TB
8B	4096	773	5.9 TB
Bi-encoder	2048	1	3.8 GB

Mitigations include:

• Embedding Dimension Projection: Linear reduction of latent dimension. For example, 8B projected to 512D yields storage at 13% of full size and retains 96% NDCG@10; to 128D achieves 3% storage and 95.4% retention.
• Latency: Token-level MaxSim requires specialized vector database structures and imposes higher per-query costs. Trade-offs appear when comparing with bi-encoder plus cross-encoder reranker pipelines, which can approach ColBERT's accuracy at a small latency overhead (e.g., ~2.4 s for reranking) and dramatically reduced storage footprint (Moreira et al., 3 Feb 2026, Xu et al., 7 Jul 2025).

5. Integration in RAG Systems and Deployment Choices

Nemotron ColEmbed V2 is tailored for RAG pipelines comprising document-level batch encoding (storing per-page multi-vector embeddings) and query-time MaxSim retrieval, with optional cross-encoder reranking to fill LLM prompts.

• Throughput-Sensitive Scenarios: Embedding-size projection or hybrid pipelines are recommended.
• Model Selection:
  • 3B: Compact model size minimizes encoding cost, but dynamic tiling leads to the largest storage requirement per page.
  • 4B: Optimal storage–accuracy compromise (3.7 TB/1M pages, NDCG@10 = 61.54).
  • 8B: Maximizes retrieval quality (NDCG@10 = 63.42), preferred when quality trumps storage and compute constraints (Moreira et al., 3 Feb 2026).

Practical implications are that late interaction's overhead can be managed via projection, influencing the selection of models based on specific storage and latency budgets.

6. Methodological Lineage and Comparative Landscape

Nemotron ColEmbed V2 builds upon prior models such as llama-nemoretriever-colembed (Xu et al., 7 Jul 2025), which established the efficacy of bidirectional attention and ColBERT-style token-level retrieval in vision-language contexts. Early models leveraged two-stage training (text-to-image), now standard in the 3B variant, and systematically replaced causal attention with bidirectional mechanisms throughout. Comparative evaluations against bi-encoder architectures and reranking-based approaches confirm the superiority of token-level MaxSim for fine-grained cross-modal matching, albeit with marked engineering trade-offs.

7. Limitations and Future Directions

Current bottlenecks for Nemotron ColEmbed V2 involve:

• Scalability: Large storage costs (>10 TB per million documents in full-dim ColEmbed) limit applicability to truly web-scale repositories.
• Latency: MaxSim operation's computational requirements restrict real-time, user-facing retrieval unless mitigated by dimensionality reduction or hybrid retrieval–reranking methods.
• Domain Adaptation: While cluster-based sampling mitigates overfitting, continual domain shift remains a challenge, suggesting ongoing work on adaptive sampling and negative generation.

A plausible implication is that further advances will hinge on improved embedding compression, hardware-aware indexing structures for MaxSim, and more sample-efficient cross-modal alignment methods.

---

For detailed methodology, benchmarking, and architectural insights, see (Moreira et al., 3 Feb 2026) and foundational prior work (Xu et al., 7 Jul 2025).