BGE-M3: Unified Multilingual Retrieval Model

• BGE-M3 is a multilingual encoder that unifies dense, sparse, and multi-vector retrieval modes to handle texts from short queries to long documents.
• It builds on XLM-RoBERTa-large with extended sequence length and employs self-knowledge distillation to align its multiple retrieval heads.
• The model demonstrates versatility in various tasks, powering multilingual retrieval, reranking, and prompt classification across diverse languages.

BGE-M3, introduced as “M3-Embedding,” is a multilingual text-encoding model designed to unify three retrieval functionalities—dense retrieval, sparse retrieval, and multi-vector retrieval—within a single encoder, while also handling inputs from short sentences to documents of up to 8,192 tokens (Chen et al., 2024). It is built on XLM-RoBERTa-large, further pre-trained via RetroMAE on multilingual unlabeled corpora, and trained with a self-knowledge-distillation scheme intended to align the model’s parallel retrieval heads rather than optimize them as separate, potentially conflicting objectives (Chen et al., 2024). In subsequent work, BGE-M3 has been used not only as a retriever in multilingual and cross-lingual information retrieval, but also as a backbone for Arabic and Khmer RAG pipelines, a Vietnamese reranker, and a multilingual safe/harmful prompt classifier (Alsubhi et al., 1 Jun 2025).

1. Design scope and model identity

BGE-M3 is explicitly defined around three axes: multi-linguality, multi-functionality, and multi-granularity (Chen et al., 2024). In the original formulation, the model supports over 100 languages, exposes three retrieval heads in parallel, and extends maximum input length from 512 to 8,192 tokens so that short queries and long documents can be encoded by the same model family without separate architectures (Chen et al., 2024).

The encoder backbone is XLM-RoBERTa-large with 550M parameters and a vocabulary of 250,000 subword tokens covering 105+ languages (Chen et al., 2024). A downstream Khmer RAG study reports BGE-M3 as a 567 million parameter model; that study also describes it as a Transformer-based encoder trained to produce fixed-length embeddings for arbitrary text spans (Ros et al., 21 May 2026). The coexistence of these two reported parameter counts reflects differences in how downstream papers describe the model rather than a resolved architectural contradiction.

The model’s stated goal is not merely multilingual sentence embedding. Its defining claim is that one encoder can serve approximate nearest-neighbor lookup, inverted-index retrieval, and fine-grained late interaction by combining dense, sparse, and multi-vector scoring heads in one system (Chen et al., 2024). This distinguishes it from multilingual dual encoders that only expose a single dense representation. A plausible implication is that BGE-M3 is best understood as a retrieval platform model rather than only an embedding checkpoint.

2. Encoder architecture and retrieval functions

BGE-M3 is built on top of XLM-RoBERTa-large and is extended in two ways: it is further pre-trained via RetroMAE on 184M unlabeled samples drawn from Pile, mC4, and Wudao, and its maximum position embedding is extended to 8,192 (Chen et al., 2024). The model then attaches three retrieval heads to the shared encoder.

The dense head uses the final hidden state at the \texttt{[CLS]} position, followed by $\ell_2$ normalization:

$$e_q = \mathrm{norm}(H_q[0]), \qquad e_p = \mathrm{norm}(H_p[0]),$$

with similarity

$s_{\mathrm{dense}}(q,p)= e_q \cdot e_p.$

The sparse head computes a per-token lexical weight through a linear projection:

$$w_{q_i}=\mathrm{ReLU}(W_{\mathrm{lex}}^\top H_q[i]),$$

and scores overlapping tokens by

$$s_{\mathrm{lex}}(q,p)=\sum_{t\in q\cap p} w_{q_t}\cdot w_{p_t}.$$

The multi-vector head projects every hidden state and uses ColBERT-style late interaction:

$$E_q[i]=\mathrm{norm}(W_{\mathrm{mul}}^\top H_q[i]),$$

$$s_{\mathrm{mul}}(q,p)=\frac1N\sum_{i=1}^N \max_{j=1}^M E_q[i]\cdot E_p[j].$$

For highest-accuracy hybrid reranking, the original paper sums the three scores:

$$s_{\mathrm{rank}} = s_{\mathrm{dense}} + s_{\mathrm{lex}} + s_{\mathrm{mul}}.$$

This architecture is the technical basis for the model’s “multi-functionality” claim (Chen et al., 2024). It also explains why later work can repurpose BGE-M3 beyond first-stage retrieval. In ViRanker, for example, the BGE-M3 backbone is used in cross-encoder mode on concatenated query-document input up to 1,024 tokens, with standard self-attention replaced by Blockwise Parallel Transformer modules and a two-layer MLP head producing a scalar relevance score (Dang et al., 11 Sep 2025).

3. Training procedure, self-distillation, and efficiency mechanisms

The original paper argues that naïvely assigning one contrastive loss to each retrieval head leads to conflicting gradients (Chen et al., 2024). Its remedy is self-knowledge distillation: the sum of dense, sparse, and multi-vector scores is treated as a teacher distribution, and each head is trained to align with that ensemble in addition to its own InfoNCE objective.

For each head $*\in\{\mathrm{dense},\mathrm{lex},\mathrm{mul}\}$, the standard contrastive loss is

$$\mathcal{L}_* = - \log \frac{\exp(s_*(q,p^+)/\tau)}{\sum_{p\in\{p^+,P^-\}} \exp(s_*(q,p)/\tau)}.$$

The ensemble score is

$$e_q = \mathrm{norm}(H_q[0]), \qquad e_p = \mathrm{norm}(H_p[0]),$$0

which is converted into a softmax teacher distribution. The final objective combines the three head-specific losses with a distillation term averaged across heads (Chen et al., 2024).

The training workflow is multi-stage. Stage 0 further pre-trains the encoder with RetroMAE on unsupervised data using dense loss only. Stage 1 fine-tunes on labeled and synthetic data with all three heads, hard negatives from ANCE sampling, and self-distillation (Chen et al., 2024). The same paper also reports a batching strategy intended to preserve large in-batch negative pools even for 8K-token inputs: length-based grouping, split-batch training with gradient checkpointing, and cross-GPU embedding broadcast. In its reported example, per-device batch size at length 8,192 increases from 6 without split-batch to 130 with split-batch (Chen et al., 2024).

At inference time, the paper proposes several deployment paths. Dense or multi-vector heads can back FAISS or ColBERT indexes, sparse weights can populate a SPLADE-style inverted index, and query-time retrieval can combine ANN and BM25 before reranking by $$e_q = \mathrm{norm}(H_q[0]), \qquad e_p = \mathrm{norm}(H_p[0]),$$1 (Chen et al., 2024). When GPU memory is insufficient and no document fine-tuning is possible, the model uses the “MCLS” trick: inserting a \texttt{[CLS]} token every 256 tokens and averaging the resulting CLS representations (Chen et al., 2024).

4. Empirical retrieval performance

The original BGE-M3 paper reports multilingual, cross-lingual, and long-document retrieval results using MIRACL, MKQA, MLDR, and NarrativeQA (Chen et al., 2024). The strongest results are typically obtained by hybrid use of all three heads rather than by a single head alone.

Benchmark	Setting	Reported BGE-M3 result
MIRACL dev, 18 languages	nDCG@10 avg	M3 All = 70.0
MKQA, 25 lang$$e_q = \mathrm{norm}(H_q[0]), \qquad e_p = \mathrm{norm}(H_p[0]),$$2en	Recall@100 avg	M3 All = 75.5
MLDR long docs	nDCG@10	M3 All = 65.0
NarrativeQA	nDCG@10	M3 All = 61.7

These numbers sit alongside more granular comparisons in the paper. On MIRACL dev, M3 Dense reaches 67.8, M3 Sparse 53.9, M3 Multi-vec 69.0, and M3 Dense+Sparse 68.9, with M3 All at 70.0 (Chen et al., 2024). On MKQA, M3 Dense reaches 75.1 and M3 All 75.5 (Chen et al., 2024). On MLDR long-document retrieval, M3 Dense is 52.5, M3 Sparse 62.2, M3 Multi-vec 57.6, M3 Dense+Sparse 64.8, and M3 All 65.0 (Chen et al., 2024).

The ablations are central to the model’s interpretation. Disabling self-knowledge distillation and supervising each head separately hurts the sparse head by approximately 17 points in nDCG@10 on MIRACL (Chen et al., 2024). The reported multi-stage training path improves MIRACL dense retrieval from 59.3 to 64.8 to 67.8 (Chen et al., 2024). This suggests that the model’s performance is not reducible to XLM-RoBERTa-large plus longer context; the training recipe is part of the claimed contribution.

5. Use in RAG, reranking, and prompt filtering

BGE-M3 has been used in several downstream systems that treat it either as a dense retriever or as a transferable multilingual backbone.

In Arabic RAG, a systematic analysis of chunking strategies, embedding models, rerankers, and generators reports that sentence-aware chunking outperforms fixed-size, recursive, and semantic chunking, and that BGE-M3 and Multilingual-E5-large emerge as the most effective embedding models (Alsubhi et al., 1 Jun 2025). The study uses cosine similarity over query and chunk embeddings,

$$e_q = \mathrm{norm}(H_q[0]), \qquad e_p = \mathrm{norm}(H_p[0]),$$3

retrieves the top-$$e_q = \mathrm{norm}(H_q[0]), \qquad e_p = \mathrm{norm}(H_p[0]),$$4 chunks, and optionally reranks them with bge-reranker-v2-m3 (Alsubhi et al., 1 Jun 2025). The paper reports only aggregated RAGAS composite scores for BGE-M3: ARCD 80.29, ArSQUAD 51.97, SaudiWiki 88.31, QA4MRE 48.97, Quran Tafseer 82.72, Hindawi Books 73.70, with an average of 70.99 (Alsubhi et al., 1 Jun 2025). It further reports that BGE-M3 leads Multilingual-E5-large by 0.68 points on average, with the largest gains on Hindawi Books and ARCD, while E5-large leads on ArSQUAD and SaudiWiki (Alsubhi et al., 1 Jun 2025). The same study explicitly notes that it does not provide per-metric breakdowns, confidence intervals, or significance testing.

In Khmer retrieval-augmented question answering, BGE-M3 is benchmarked against Jina-Embeddings-v3 and Qwen3-Embedding on a 200-question golden set over telecom-domain documents (Ros et al., 21 May 2026). At $$e_q = \mathrm{norm}(H_q[0]), \qquad e_p = \mathrm{norm}(H_p[0]),$$5, BGE-M3 achieves Hit Rate@3 = 0.285, File Hit Rate@3 = 0.700, MRR@3 = 0.221, and Precision@3 = 0.112 (Ros et al., 21 May 2026). The paper states that this more than doubles Hit Rate@3 relative to Jina-Embeddings-v3 and yields a 63% gain over Qwen3-Embedding (Ros et al., 21 May 2026). At the same time, it emphasizes that absolute recall remains modest and identifies retrieval as the primary bottleneck in end-to-end Khmer RAG (Ros et al., 21 May 2026).

In Vietnamese reranking, ViRanker repurposes BGE-M3 from an embedder into a cross-encoder reranker, replacing standard self-attention with Blockwise Parallel Transformer modules and fine-tuning with triplet loss and hybrid hard-negative mining (Dang et al., 11 Sep 2025). On MMARCO-VI, the reported scores are NDCG@3 = 0.6815 and MRR@3 = 0.6641, compared with 0.6625 and 0.6458 for PhoRanker and 0.6088 and 0.5841 for BGE-Reranker-V2-M3 (Dang et al., 11 Sep 2025). This demonstrates one concrete route by which the BGE-M3 backbone has been adapted from first-stage retrieval to early-rank reranking.

In SafeGen, BGE-M3 serves a different role: a fine-tuned multilingual text classifier that filters prompts as “safe” or “harmful” before image generation (Nam et al., 14 Dec 2025). The paper describes a Transformer-family encoder with subword tokenization, a CLS representation passed through dropout and a linear classification head, balanced-class batching, and a Class-Balanced Focal Loss term added to cross-entropy (Nam et al., 14 Dec 2025). On the held-out test set, the reported results are Accuracy = 0.8215 and F1-Score = 0.8145 for the fine-tuned BGE-M3, versus 0.1865 and 0.1840 for the pretrained base (Nam et al., 14 Dec 2025). This use does not redefine BGE-M3’s original purpose, but it shows that the backbone has been treated as a multilingual encoder that can be specialized well beyond retrieval.

6. Limitations, failure modes, and nomenclature

The limitations reported across papers are consistent with a pragmatic rather than universal interpretation of BGE-M3. In Arabic RAG, BGE-M3 shows slight underperformance relative to Multilingual-E5-large on highly structured extractive tasks such as ArSQUAD and SaudiWiki, and the study notes noisy recall on multi-paragraph inference in QA4MRE (Alsubhi et al., 1 Jun 2025). It also reports that semantic chunking under-performs on Arabic data and suggests that BGE-M3’s sentence-level representations are stronger than its document-level similarities for Arabic in literary and religious text (Alsubhi et al., 1 Jun 2025). Because that study omits confidence intervals and statistical significance tests, the robustness of small absolute differences remains unresolved.

In Khmer RAG, the main limitation is not comparative weakness but insufficient absolute recall: Hit Rate@3 remains 0.285, no Khmer-specific fine-tuning is applied, and the multilingual tokenizer may under-represent Khmer subwords (Ros et al., 21 May 2026). The paper therefore recommends hybrid indexing, script normalization, domain-adaptive fine-tuning, iterative retrieval, and continuous monitoring of Hit Rate@k and MRR (Ros et al., 21 May 2026). These are deployment recommendations rather than guarantees of improvement.

SafeGen exposes a different class of limitation: fine-tuning details such as optimizer choice, learning-rate schedule, and number of epochs are not specified, and no confusion matrix is reported (Nam et al., 14 Dec 2025). That omission constrains deeper error analysis. ViRanker similarly notes that full architectural hyperparameters of BGE-M3 itself should be taken from the original reference rather than inferred from the reranker paper (Dang et al., 11 Sep 2025).

There is also a nomenclature issue. An unrelated paper on mixtures of Gaussian Bayesian networks uses the near-identical label “BGe-M3” for a tied-covariance marginal-likelihood scoring metric (Grzegorczyk, 10 Nov 2025). That object is analytically and conceptually distinct from BGE-M3 the multilingual embedding model. Confusion between the two arises from string similarity alone, not from shared methodology.

Taken together, the literature presents BGE-M3 as a single multilingual encoder whose practical importance lies in its combination of three retrieval modes, long-context capacity, and a self-distillation training scheme, with downstream evidence that the model can be transferred into dense retrieval, reranking, and classification settings across Arabic, Khmer, Vietnamese, English, and Vietnamese-English workflows (Chen et al., 2024).