---
title: 'Llama-Embed-Nemotron-8B: Multilingual Embedding Model'
url: https://www.emergentmind.com/topics/llama-embed-nemotron-8b
type: topic
---

# Llama-Embed-Nemotron-8B: Multilingual Embedding Model

Llama-Embed-Nemotron-8B is an open-weights, instruction-aware multilingual text embedding model based on the Llama-3.1-8B architecture. Designed as a universal solution for retrieval, classification, and semantic textual similarity (STS), it demonstrates state-of-the-art (SOTA) performance across multilingual and cross-lingual settings, as validated by its leading position on the Multilingual Massive Text Embedding Benchmark (MMTEB) as of October 2025. The training process leverages a novel combination of public and synthetically generated data, extensive ablation studies, and model merging strategies to maximize both accuracy and generalizability across over 250 languages.

## 1. Model Architecture and Embedding Mechanism

The model initializes from Llama-3.1-8B, comprising 32 decoder-only Transformer layers with a hidden size $d_{\text{model}}=4096$, 32 attention heads, and feed-forward dimension of 16,384. Departing from its original causal mask, every layer adopts full bi-directional attention to facilitate optimal information propagation for embedding tasks.

Input sequences $S$ are tokenized and passed through the network, yielding final hidden states $H \in \mathbb{R}^{L \times d_{\text{model}}}$. A global average pooling scheme is applied:
$$
v = \frac{1}{L} \sum_{i=1}^L H_i \in \mathbb{R}^{d_{\text{model}}}
$$
where $v$ serves as the fixed-dimensional embedding vector for all downstream tasks.  

The model is instruction-aware: each input is explicitly prefixed:
```
Instruct: {task_instruction}
Query: {T}
```
allowing user-defined task specialization at inference. A shared encoder is employed, with bi-encoder usage (query/document) for retrieval tasks and uni-encoder operation for STS/classification.

## 2. Training Data Curation and Preprocessing

The training corpus consists of 16.1 million query-document (Q–D) pairs, partitioned as follows:

| Subset       | Source/Use-case            | Volume (M Q–D) |
|--------------|----------------------------|:--------------:|
| Non-synthetic | Nemotron-CC-v2 Diverse-QA (pretrain), public corpora (fine-tune: MIRACL, HotpotQA, MS MARCO, NQ, SQuAD, etc.) | 7.7           |
| Synthetic    | Open-weight LLMs SDG (pretrain, fine-tune for retrieval/class/STS/bitext) | 8.4           |

Synthetic data generation uses LLMs such as gpt-oss-20b, gpt-oss-120b, Mixtral-8x22B-Instruct, Llama-3.3-70B-Instruct, Llama-4-Scout-17B, and Llama-4-Maverick-17B. Strategies include end-to-end triplet creation, seed-corpus query generation with hard negative retrieval, back-translation/multilingual dataset augmentation, and retrieval-augmented generation for bitext.

Preprocessing steps standardize the data with Llama’s SentencePiece BPE tokenizer, sequence truncation (max 512 tokens), Unicode normalization, deduplication, and explicit balancing across 250+ languages via per-language quotas.

## 3. Contrastive Objective, Negative Sampling, and Loss Ablation

Llama-Embed-Nemotron-8B employs the InfoNCE contrastive objective:
$$
\mathcal{L}(q,d^+,D_N) = -\log \frac{\exp(\operatorname{sim}(q,d^+)/\tau)}{\sum_{d_i \in \{d^+\} \cup D_N} \exp(\operatorname{sim}(q,d_i)/\tau)}
$$
with $\operatorname{sim}(\cdot,\cdot)$ as cosine similarity and $\tau=0.02$.

Negative sampling distinguishes pretraining (one hard negative per Q–D) from fine-tuning (four hard negatives per Q–D). Hard negatives are mined using an ensemble (e5-mistral-7B, Qwen3-Embedding-8B) and a "top-k with threshold" rule ($\operatorname{sim}(q,d_{-}) < 0.95 \cdot \operatorname{sim}(q,d^+)$).

Comparative ablation with alternatives (Gecko: hard+in-batch+same-tower, Qwen3-Embedding: same-tower, Gemini: hard+in-batch positive) demonstrates that the hard-negative configuration is effective at both scale and performance.

**InfoNCE Loss Variant Ablation**

| Variant                | Borda Votes | Mean(Task) | Mean(Type) |
|------------------------|:-----------:|:----------:|:----------:|
| Gecko                  |   37,903    |   63.45    |   55.86    |
| Qwen3-Embedding        |   36,835    |   62.14    |   55.49    |
| Gemini                 |   38,135    |   63.83    |   55.90    |
| Ours (HNs only)        |   38,225    |   64.03    |   56.04    |

## 4. Synthetic Data Generation Approaches and Multilingual Strategy

Synthetic data generation is central to model generalization, notably for low-resource languages. The primary strategies encompass:

1. End-to-end triplet generation from scratch.
2. Seed-document $\rightarrow$ query $\rightarrow$ hard negatives mining.
3. Back-translation for enforcing cross-lingual and multilingual diversity.
4. Retrieval-augmented generation tailored for bitext use-cases.

High-resource datasets are machine-translated by LLMs into target languages; per-language quotas are enforced in the synthetic mix for balanced representation.

Ablation shows that mixing synthetic data sources yields superior MMTEB metrics over exclusive reliance on a single source. Qualitative analysis via human inspection appraises factuality and diversity.

| SDG Source        | Borda Votes | Mean(Task) | Classification | Clustering |
|-------------------|:-----------:|:----------:|:--------------:|:----------:|
| No synthetic      |   37,348    |   61.95    |     62.16      |   49.94    |
| gpt-oss-20b       |   37,732    |   62.54    |     63.71      |   50.45    |
| Mix of all models |   37,812    |   62.89    |     64.39      |   50.95    |

Empirical results on five classification benchmarks demonstrate that combining 1M synthetic samples with in-domain training data significantly boosts accuracy, especially in non-English and low-resource settings.

## 5. Model Merging and Instruction Prompting

Six checkpoints, each corresponding to different data regimes or hyperparameter configurations ($\{W_i\}_{i=1}^6$), are merged with equal weights:
$$
W_{\text{merge}} = \frac{1}{6} \sum_{i=1}^6 W_i
$$
This approach, termed "model soups", leads to a +119 Borda vote improvement over the best single checkpoint (39,573 vs 39,454; Mean(Task): 69.46 vs 68.62 on MMTEB).

Instruction encoding is applied purely via runtime prompt manipulation—user-supplied task-instruction strings are prepended—requiring no further weight updates. Embeddings dynamically adapt to instruction context via prompt-tuned attention on the shared encoder.

## 6. Benchmark Results and Empirical Evaluation

On MMTEB as of October 21, 2025, Llama-Embed-Nemotron-8B ranks first on Borda score:

| Model                          | Borda Rank | Borda Votes | Mean(Task) | Mean(Type) |
|---------------------------------|:----------:|:-----------:|:----------:|:----------:|
| llama-embed-nemotron-8b        |     1      |   39,573    |   69.46    |   61.09    |
| gemini-embedding-001           |     2      |   39,368    |   68.37    |   59.59    |
| Qwen3-Embedding-8B             |     3      |   39,364    |   70.58    |   61.69    |

Per-type metric evaluations employ Cosine MRR and Recall@10 for retrieval, accuracy for classification, and Pearson/Spearman correlation for STS. The model maintains robust performance in both high- and low-resource language tasks, and in cross-lingual transfer setups.

Consistent advantages are observed over both open- and closed-weight baselines, with stability across diverse language tasks attributed to the data composition, effective negative sampling, and model merging strategies.

## 7. Practical Deployment and Usage

Inference proceeds at approximately 1 ms per 512-token sequence on an A100 80 GB GPU, with peak memory consumption near 20 GB for batch size 128. A single A100 or V100 is sufficient for typical inference; multi-GPU scaling is applicable for large-scale retrieval scenarios.

Typical usage for embedding extraction employs the HuggingFace Transformers interface:

```python
from transformers import AutoTokenizer, AutoModel
import torch

tok = AutoTokenizer.from_pretrained("nvidia/llama-embed-nemotron-8b")
model = AutoModel.from_pretrained("nvidia/llama-embed-nemotron-8b", trust_remote_code=True)

def embed(texts, instr="Retrieve relevant documents"):
    inputs = tok([f"Instruct: {instr}\nQuery: {t}" for t in texts],
                 return_tensors="pt", padding=True, truncation=True, max_length=512)
    H = model(**inputs).last_hidden_state
    v = H.mean(dim=1)     # global average pooling
    return torch.nn.functional.normalize(v, dim=1)

embs = embed(["What is RAG?"], instr="Retrieve semantically similar text")
```

Recommended best practices include input truncation/chunking for sequences longer than 512 tokens, use of mean-pooling, embedding normalization, and indexing with vector databases such as FAISS. Downstream adaptation can be performed by attaching learned MLP heads to pooled embeddings for task-specific refinement.

---

Llama-Embed-Nemotron-8B's open methodology, comprehensive ablation, and instruction-aware interface establish it as a high-performance, flexible backbone for state-of-the-art multilingual retrieval, classification, and semantic similarity applications [2511.07025].

Source: https://www.emergentmind.com/topics/llama-embed-nemotron-8b