---
title: 'Multilingual-E5-Large: Open-Source Text Embeddings'
url: https://www.emergentmind.com/topics/multilingual-e5-large-me5-l
type: topic
---

# Multilingual-E5-Large: Open-Source Text Embeddings

Searching arXiv for relevant papers on Multilingual-E5-large to ground the article in current literature.
arxiv_search query="Multilingual E5 Text Embeddings mE5 large technical report" max_results=5
Multilingual-E5-large (mE5-L) is an open-source multilingual text embedding model in the E5 family, released in mid-2023 and trained to map text into dense vectors for retrieval, semantic similarity, classification, bitext mining, and related embedding tasks. The model follows the English E5 training recipe, with contrastive pre-training on 1 billion multilingual text pairs and subsequent fine-tuning on labeled datasets, and it has since served both as a benchmarked multilingual retriever and as a backbone for task-specific adaptation, including Arabic machine-generated text detection and explicit post hoc cross-lingual alignment [2402.05672][2603.10007][2602.21543][2605.23618].

## 1. Model identity and reported architecture

The most detailed architectural account describes mE5-large as initialized from XLM-RoBERTa-large (Conneau et al. 2020). In that description, the backbone has 24 Transformer layers, hidden size 1024, feed-forward inner size 4096, 16 attention heads, and approximately 560 million parameters. The Transformer stack is unchanged relative to the initializer, and a single pooling/prefix-tuned projection head maps the final CLS or mean-pooled representation to a 1024-dimensional embedding [2402.05672].

In retrieval-oriented use, mE5-L is also described as a single shared 24-layer Transformer with a 512-token maximum length, using lightweight role prefixes such as `query:` and `passage:` rather than separate encoder towers for queries and documents [2605.23618]. A downstream Arabic classification report independently summarizes the encoder as having 24 layers, hidden size $d=1024$, feed-forward inner dimension $d_{ff}\approx 4096$, and total parameters $\approx 770$ M, while a later alignment paper explicitly states that exact large-variant layer counts, hidden sizes, attention-head counts, and parameter totals are not specified there [2603.10007][2602.21543]. This suggests that secondary reports are not fully uniform in how they restate the large variant’s configuration.

The model is multilingual by construction. One technical summary characterizes its pretraining coverage as spanning approximately 50 languages, including Arabic, via mC4/CCNet-derived web crawls and mined translation bitexts [2603.10007]. A later dense-retrieval benchmark instead emphasizes zero-shot support across the major Latin, Cyrillic, Indic, and East-Asian scripts tested in that study [2605.23618]. Taken together, these descriptions place mE5-L as a general-purpose multilingual encoder rather than an English-centric embedding model extended only marginally to other languages.

## 2. Training recipe and contrastive objective

The training procedure reported for mE5-large has two stages. Stage 1 is weakly supervised contrastive pre-training on roughly 1 billion multilingual text pairs drawn from Wikipedia section title ↔ passage (150 M), mC4 title ↔ page (160 M), CCNews title ↔ article (160 M), NLLB parallel corpus (160 M), Reddit comment ↔ response (160 M), S2ORC title ↔ abstract and citation pairs (50 M), StackExchange question ↔ answer (50 M), xP3 prompt ↔ response (80 M), and miscellaneous SBERT collections (10 M). Pre-training runs for 30 k steps, with batch size 32,000, in-batch negatives only, AdamW, linear warmup plus decay, learning rate $1\times10^{-4}$ for the large model, and temperature $\tau=0.05$ [2402.05672].

The pre-training loss is standard InfoNCE over normalized embeddings. With positive pair $(x_i, x_i^+)$, embeddings $z_i=f_\theta(x_i)$ and $z_i^+=f_\theta(x_i^+)$, and cosine similarity $\operatorname{sim}(u,v)=u^\top v$, the per-example loss is

$$
\ell_i = -\log
\frac{\exp(\operatorname{sim}(z_i,z_i^+)/\tau)}
{\sum_{j=1}^{N}\exp(\operatorname{sim}(z_i,z_j^+)/\tau)},
$$

and the batch loss is

$$
L=\frac{1}{N}\sum_{i=1}^{N}\ell_i.
$$

In Stage 2, the model is fine-tuned on approximately 1.6 M labeled examples from mixed retrieval and similarity datasets: MS-MARCO passage (500 K) and document (70 K), Natural Questions, TriviaQA, and SQuAD (220 K total), NLI (275 K pairs), ELI5 long-form QA (100 K), NLLB parallel sentences (100 K), DuReader retrieval (86 K), FEVER (70 K), HotpotQA (70 K), Quora Duplicate Questions (15 K), Mr. TyDi (50 K), and MIRACL (40 K). This stage uses in-batch negatives, mined hard negatives, and knowledge distillation from a stronger cross-encoder on retrieval tasks, with batch size 512, learning rate $1\times10^{-5}$ for large, and 2 epochs [2402.05672].

A later downstream description restates the same training philosophy in broader operational terms: contrastive embedding learning over weakly supervised pairs such as search queries and clicked documents, translation pairs, and MS MARCO question–passage examples, with supervision signals drawn from user clicks, bilingual dictionaries, and retrieval logs [2603.10007]. The instruction-tuned derivative, mE5-large-instruct, further adds 500 K GPT-3.5/4-generated instruction–response pairs across 93 languages [2402.05672].

## 3. Embedding function and empirical profile

In dense retrieval use, mE5-L embeds queries $q$ and passages $d$ through a shared encoder $\phi(\cdot)\in\mathbb{R}^{1024}$ and scores relevance by cosine similarity,

$$
\operatorname{sim}(q,d)=
\frac{\phi(q)^\top \phi(d)}
{\|\phi(q)\|_2 \, \|\phi(d)\|_2}.
$$

Benchmarking work around mE5-L typically reports $nDCG@10$, Recall@k, and MRR as the main ranking metrics [2605.23618].

| Evaluation setting | Metric | mE5-L result |
|---|---|---|
| English MTEB [2402.05672] | Average score | 61.5 |
| MIRACL dev, avg over 16 languages [2402.05672] | nDCG@10 / R@100 | 66.5 / 94.3 |
| BUCC / Tatoeba [2402.05672] | F1 / accuracy | 98.6 / 75.7 |
| MMTEB multilingual, 132 tasks [2502.13595] | Average score | 58.6 |
| BEIR average over four subsets [2605.23618] | nDCG@10 | 0.546 |
| IT-RAG-Bench, 640 queries [2605.23618] | nDCG@10 / Recall@10 / MRR | 0.279 / 48.9% / 0.243 |
| AbjadGenEval Arabic detection [2603.10007] | Test F1 | 0.75 |

These results show that mE5-L is not confined to a single task family. In the original technical report it is strong on retrieval, bitext mining, and semantic similarity, with multilingual retrieval on MIRACL substantially exceeding BM25 and mDPR in the reported comparison [2402.05672]. In the broader MMTEB benchmark it remains competitive across a much larger multilingual task inventory, though not the overall leader [2502.13595].

The MMTEB evaluation context also matters. That benchmark reduces task count by inter-task correlation downsampling and accelerates retrieval evaluation via hard-negative sampling: for each query it pools the top 250 documents returned by BM25, e5-multilingual-large, and e5-Mistral-Instruct-7B, then evaluates on that reduced collection [2502.13595]. A plausible implication is that mE5-L’s published benchmark profile reflects both model quality and the increasingly standardized efficiency constraints of large multilingual embedding evaluations.

## 4. Comparative standing in multilingual retrieval and evaluation

On the original English MTEB benchmark of 56 datasets, mE5-large scores 61.5, improving over mE5-base at 59.5 and mE5-small at 57.9; the instruction-tuned variant reaches 64.4 [2402.05672]. On MIRACL dev averaged over 16 languages, mE5-large reports $nDCG@10=66.5$ and $R@100=94.3$, again exceeding the smaller E5 variants [2402.05672]. On BUCC and Tatoeba, the model reaches 98.6 BUCC F1 and 75.7 Tatoeba accuracy, with the instruct variant improving to 99.0 and 83.8 respectively [2402.05672].

In MMTEB’s fully multilingual benchmark, comprising 132 tasks after reduction, mE5-L attains an average score of 58.6 and ranks fourth. The reported category breakdown is 71.7 for bitext mining, 79.0 for pair classification, 59.9 for classification, 73.5 for STS, 54.1 for retrieval, 21.3 for multilabel classification, 42.9 for clustering, and 62.8 for reranking. On the Europe-focused subset it scores 58.5 and ranks fourth, while on the Indic-focused subset it scores 66.4 and ranks second [2502.13595]. The same benchmark reports that multilingual-e5-large-instruct, with only 560 million parameters, is the best-performing publicly available model among those evaluated, and it outperforms mE5-L across all categories [2502.13595].

A later retrieval benchmark places mE5-L against Google Embeddings 2, BGE-M3, E5-large, LaBSE, and Paraphrase-Multilingual-MPNet. On four zero-shot BEIR subsets, mE5-L obtains a BEIR average $nDCG@10$ of 0.546, compared with 0.638 for Google Embeddings 2. The per-dataset scores reported for mE5-L are 0.438 on FiQA-2018, 0.341 on NFCorpus, 0.704 on SciFact, and 0.702 on TREC-COVID [2605.23618]. On the synthetic Italian IT-RAG-Bench, however, the gap almost vanishes: mE5-L reaches $nDCG@10=0.279$ versus 0.282 for Google Embeddings 2, with Recall@10 of 48.9% versus 47.6% and MRR of 0.243 versus 0.259 [2605.23618]. This pattern positions mE5-L as a model whose competitiveness depends strongly on latency budgets, corpus characteristics, and context-window requirements rather than on a uniform ranking across all multilingual retrieval settings.

## 5. Fine-tuning for Arabic machine-generated text classification

In the AbjadGenEval shared task, GATech fine-tuned mE5-L for binary classification of Arabic text as human-written or machine-generated. The dataset contained 5,298 Arabic samples, balanced between 2,649 human-written and 2,649 machine-generated texts. The reported average length was 632 words per human-written text and 303 words per machine-generated text [2603.10007].

The classification head consumed a pooled vector $h_{\text{pool}}\in\mathbb{R}^d$ and applied LayerNorm, Dropout, a Dense layer with GELU, a second Dropout, and a final linear layer followed by softmax over two classes. The system also used Multi-Sample Dropout with five simultaneous dropout masks at rates $\{0.10,0.15,0.20,0.25,0.30\}$, averaging the resulting logits before softmax. Training used maximum sequence length 512 tokens with dynamic padding, batch size 16, gradient accumulation steps 4 for an effective batch size of 64, AdamW with weight decay 0.01, learning rate $2\times10^{-5}$ with cosine scheduler and 10% linear warmup, layer-wise learning rate decay factor 0.95, and 2 epochs [2603.10007].

Several pooling strategies were explored. Mean pooling used

$$
h_{\text{mean}}=\frac{1}{T}\sum_{t=1}^{T} h_t^L.
$$

Weighted layer pooling first mean-pooled each layer, learned normalized layer weights by softmax, and aggregated them. Multi-head attention pooling used $K=8$ learnable query vectors, computed token-level attention weights, and projected the concatenated context vectors back to dimension $d$. Gated fusion combined pooled vectors such as $h_{\text{mean}}$ and $h_{\text{attn}}$ through a sigmoid gate $g$ to produce $h_{\text{gate}}=g\odot h_{\text{mean}}+(1-g)\odot h_{\text{attn}}$ [2603.10007].

The final test F1 was 0.75 with simple mean pooling, while the more complex pooling variants yielded approximately 0.70–0.71. The paper attributes mean pooling’s advantage to the limited data size, the additional parameters introduced by learned pooling mechanisms, the strength of the pretrained language-agnostic token embeddings, and the possibility that the relevant human-versus-machine signal is distributed across the text rather than concentrated in a small set of salient tokens [2603.10007]. The same study argues that text length is a strong but brittle cue, and it notes that 512-token truncation may remove discriminative content from longer human-written examples.

## 6. Post hoc cross-lingual alignment and multi-way parallel supervision

Subsequent work has examined whether mE5-style encoders can be improved further through explicit alignment beyond their original multilingual contrastive training. One such study reuses the off-the-shelf mE5 encoder from Wang et al. (2024) and adds a multi-way parallel contrastive objective instead of introducing additional masked language modeling or translation language modeling. In that setup, for each sentence embedding $z_i$, the positive set $P(i)$ consists of all translations of the same sentence in the other languages in the batch, the negative set $A(i)$ consists of all remaining non-positives, and the fine-tuning objective is

$$
L = L_{\text{sup}} + \lambda R,
$$

where $L_{\text{sup}}$ is the supervised multi-positive contrastive loss and

$$
R = E_{i\in B}\|z_{\text{orig},i}-z_i\|^2
$$

penalizes drift from the model’s original pretrained embeddings [2602.21543].

The multi-way corpus in that study is built from English source sentences sampled equally from OPUS-Wikipedia and OPUS-OpenSubtitles, with minimum length 10 words and fixed seed 42. These sentences are translated with the NLLB-200 3.3 B model into Chinese, Japanese, French, German, Hindi, and Spanish; for each English row, three target languages are sampled uniformly at random to form a four-column row. The resulting dataset has 75,822 rows across train and validation [2602.21543].

For mE5, the reported fine-tuning setup uses batch size 128, 5 epochs, in-batch negatives, tuned temperature and regularization weight, and a positive-pair boosting factor $\exp(-z_i\cdot z_p)$ for positives with lower cosine scores. The paper reports gains of +0.7–0.8 points F1 or accuracy on BUCC and Tatoeba across language pairs, and it concludes that even high-quality sentence embedding models such as mE5 still benefit from small-scale multi-way parallel supervision [2602.21543]. Its broader recommendations are to include typologically varied languages, allow every language rather than only English to serve as an anchor, and scale the parallel corpus beyond approximately 75 K rows where possible.

## 7. Efficiency, chunking, and deployment trade-offs

mE5-large is the most resource-intensive of the standard multilingual E5 releases. The technical report lists approximately 560 M parameters and about 2.2 GB memory in fp16, compared with approximately 270 M and 1.1 GB for mE5-base and approximately 94 M and 0.35 GB for mE5-small. Approximate throughput on a V100 GPU at batch size 1 is reported as about 30 queries per second for large, 60 for base, and 150 for small [2402.05672]. The model’s principal systems trade-off is therefore straightforward: higher embedding quality at lower throughput and higher memory cost.

CPU-side measurements in a later retrieval benchmark report end-to-end latency, including embedding and FAISS flat search, on a single Apple M-series CPU. Under that setup, mE5-L achieves 31.0 ms median latency, 3.5 ms standard deviation, and 31.8 ms p95 over 50 warm-reused runs. E5-large and BGE-M3 are nearly identical at 30.9 ms, while LaBSE and mMPNet are faster at 16.6 ms and Google Embeddings 2 is much slower at 231.6 ms median with substantial variance [2605.23618]. That benchmark concludes that mE5-L is the preferred option when sub-100 ms SLAs matter.

Chunking sensitivity is also reported explicitly. On IT-RAG-Bench, fixed, sliding-window with 50% overlap, and semantic TextTiling chunking were evaluated at token lengths $L\in\{8,16,32,64,128\}$. For mE5-L, $nDCG@10$ is approximately 0.040, 0.080, and 0.075 at 8 tokens for fixed, sliding, and semantic chunking respectively; approximately 0.200, 0.255, and 0.275 at 16 tokens; and 0.276, 0.278, and 0.279 at 32 tokens, with the same values persisting at 64 and 128 tokens [2605.23618]. The reported interpretation is that all methods saturate by 32 tokens on that benchmark because the passages are short, and that semantic chunking provides measurable gains only in the 16-token regime. A practical consequence is that 32-token fixed chunking is a reasonable default when semantic segmentation cost is not justified.

Taken together, these operational results define mE5-L’s practical niche. It is not the smallest multilingual encoder, nor the absolute top scorer on every benchmark, but the published evidence consistently places it near the quality frontier while remaining deployable under local inference constraints that are materially tighter than those tolerated by higher-latency hosted alternatives [2402.05672][2502.13595][2605.23618].

Source: https://www.emergentmind.com/topics/multilingual-e5-large-me5-l