Multilingual-E5-Large: Open-Source Text Embeddings

• Multilingual-E5-Large is an open-source multilingual text embedding model that uses contrastive pre-training on 1B text pairs for retrieval and semantic tasks.
• It builds upon a 24-layer Transformer architecture initialized from XLM-RoBERTa-large with 560M parameters and employs a shared encoder for both queries and documents.
• The model achieves competitive benchmarks across multilingual tasks, supporting applications like cross-lingual retrieval, bitext mining, and Arabic machine-generated text detection.

Searching arXiv for relevant papers on Multilingual-E5-large to ground the article in current literature. arxiv_search query="Multilingual E5 Text Embeddings mE5 large technical report" max_results=5 Multilingual-E5-large (mE5-L) is an open-source multilingual text embedding model in the E5 family, released in mid-2023 and trained to map text into dense vectors for retrieval, semantic similarity, classification, bitext mining, and related embedding tasks. The model follows the English E5 training recipe, with contrastive pre-training on 1 billion multilingual text pairs and subsequent fine-tuning on labeled datasets, and it has since served both as a benchmarked multilingual retriever and as a backbone for task-specific adaptation, including Arabic machine-generated text detection and explicit post hoc cross-lingual alignment (Wang et al., 2024, Khamis, 17 Feb 2026, Fazili et al., 25 Feb 2026, Cirillo et al., 22 May 2026).

1. Model identity and reported architecture

The most detailed architectural account describes mE5-large as initialized from XLM-RoBERTa-large (Conneau et al. 2020). In that description, the backbone has 24 Transformer layers, hidden size 1024, feed-forward inner size 4096, 16 attention heads, and approximately 560 million parameters. The Transformer stack is unchanged relative to the initializer, and a single pooling/prefix-tuned projection head maps the final CLS or mean-pooled representation to a 1024-dimensional embedding (Wang et al., 2024).

In retrieval-oriented use, mE5-L is also described as a single shared 24-layer Transformer with a 512-token maximum length, using lightweight role prefixes such as query: and passage: rather than separate encoder towers for queries and documents (Cirillo et al., 22 May 2026). A downstream Arabic classification report independently summarizes the encoder as having 24 layers, hidden size $d=1024$, feed-forward inner dimension $d_{ff}\approx 4096$, and total parameters $\approx 770$ M, while a later alignment paper explicitly states that exact large-variant layer counts, hidden sizes, attention-head counts, and parameter totals are not specified there (Khamis, 17 Feb 2026, Fazili et al., 25 Feb 2026). This suggests that secondary reports are not fully uniform in how they restate the large variant’s configuration.

The model is multilingual by construction. One technical summary characterizes its pretraining coverage as spanning approximately 50 languages, including Arabic, via mC4/CCNet-derived web crawls and mined translation bitexts (Khamis, 17 Feb 2026). A later dense-retrieval benchmark instead emphasizes zero-shot support across the major Latin, Cyrillic, Indic, and East-Asian scripts tested in that study (Cirillo et al., 22 May 2026). Taken together, these descriptions place mE5-L as a general-purpose multilingual encoder rather than an English-centric embedding model extended only marginally to other languages.

2. Training recipe and contrastive objective

The training procedure reported for mE5-large has two stages. Stage 1 is weakly supervised contrastive pre-training on roughly 1 billion multilingual text pairs drawn from Wikipedia section title ↔ passage (150 M), mC4 title ↔ page (160 M), CCNews title ↔ article (160 M), NLLB parallel corpus (160 M), Reddit comment ↔ response (160 M), S2ORC title ↔ abstract and citation pairs (50 M), StackExchange question ↔ answer (50 M), xP3 prompt ↔ response (80 M), and miscellaneous SBERT collections (10 M). Pre-training runs for 30 k steps, with batch size 32,000, in-batch negatives only, AdamW, linear warmup plus decay, learning rate $1\times10^{-4}$ for the large model, and temperature $\tau=0.05$ (Wang et al., 2024).

The pre-training loss is standard InfoNCE over normalized embeddings. With positive pair $(x_i, x_i^+)$, embeddings $z_i=f_\theta(x_i)$ and $z_i^+=f_\theta(x_i^+)$, and cosine similarity $\operatorname{sim}(u,v)=u^\top v$, the per-example loss is

$$\ell_i = -\log \frac{\exp(\operatorname{sim}(z_i,z_i^+)/\tau)} {\sum_{j=1}^{N}\exp(\operatorname{sim}(z_i,z_j^+)/\tau)},$$

and the batch loss is

$d_{ff}\approx 4096$0

In Stage 2, the model is fine-tuned on approximately 1.6 M labeled examples from mixed retrieval and similarity datasets: MS-MARCO passage (500 K) and document (70 K), Natural Questions, TriviaQA, and SQuAD (220 K total), NLI (275 K pairs), ELI5 long-form QA (100 K), NLLB parallel sentences (100 K), DuReader retrieval (86 K), FEVER (70 K), HotpotQA (70 K), Quora Duplicate Questions (15 K), Mr. TyDi (50 K), and MIRACL (40 K). This stage uses in-batch negatives, mined hard negatives, and knowledge distillation from a stronger cross-encoder on retrieval tasks, with batch size 512, learning rate $d_{ff}\approx 4096$1 for large, and 2 epochs (Wang et al., 2024).

A later downstream description restates the same training philosophy in broader operational terms: contrastive embedding learning over weakly supervised pairs such as search queries and clicked documents, translation pairs, and MS MARCO question–passage examples, with supervision signals drawn from user clicks, bilingual dictionaries, and retrieval logs (Khamis, 17 Feb 2026). The instruction-tuned derivative, mE5-large-instruct, further adds 500 K GPT-3.5/4-generated instruction–response pairs across 93 languages (Wang et al., 2024).

3. Embedding function and empirical profile

In dense retrieval use, mE5-L embeds queries $d_{ff}\approx 4096$2 and passages $d_{ff}\approx 4096$3 through a shared encoder $d_{ff}\approx 4096$4 and scores relevance by cosine similarity,

$d_{ff}\approx 4096$5

Benchmarking work around mE5-L typically reports $d_{ff}\approx 4096$6, Recall@k, and MRR as the main ranking metrics (Cirillo et al., 22 May 2026).

Evaluation setting	Metric	mE5-L result
English MTEB (Wang et al., 2024)	Average score	61.5
MIRACL dev, avg over 16 languages (Wang et al., 2024)	nDCG@10 / R@100	66.5 / 94.3
BUCC / Tatoeba (Wang et al., 2024)	F1 / accuracy	98.6 / 75.7
MMTEB multilingual, 132 tasks (Enevoldsen et al., 19 Feb 2025)	Average score	58.6
BEIR average over four subsets (Cirillo et al., 22 May 2026)	nDCG@10	0.546
IT-RAG-Bench, 640 queries (Cirillo et al., 22 May 2026)	nDCG@10 / Recall@10 / MRR	0.279 / 48.9% / 0.243
AbjadGenEval Arabic detection (Khamis, 17 Feb 2026)	Test F1	0.75

These results show that mE5-L is not confined to a single task family. In the original technical report it is strong on retrieval, bitext mining, and semantic similarity, with multilingual retrieval on MIRACL substantially exceeding BM25 and mDPR in the reported comparison (Wang et al., 2024). In the broader MMTEB benchmark it remains competitive across a much larger multilingual task inventory, though not the overall leader (Enevoldsen et al., 19 Feb 2025).

The MMTEB evaluation context also matters. That benchmark reduces task count by inter-task correlation downsampling and accelerates retrieval evaluation via hard-negative sampling: for each query it pools the top 250 documents returned by BM25, e5-multilingual-large, and e5-Mistral-Instruct-7B, then evaluates on that reduced collection (Enevoldsen et al., 19 Feb 2025). A plausible implication is that mE5-L’s published benchmark profile reflects both model quality and the increasingly standardized efficiency constraints of large multilingual embedding evaluations.

4. Comparative standing in multilingual retrieval and evaluation

On the original English MTEB benchmark of 56 datasets, mE5-large scores 61.5, improving over mE5-base at 59.5 and mE5-small at 57.9; the instruction-tuned variant reaches 64.4 (Wang et al., 2024). On MIRACL dev averaged over 16 languages, mE5-large reports $d_{ff}\approx 4096$7 and $d_{ff}\approx 4096$8, again exceeding the smaller E5 variants (Wang et al., 2024). On BUCC and Tatoeba, the model reaches 98.6 BUCC F1 and 75.7 Tatoeba accuracy, with the instruct variant improving to 99.0 and 83.8 respectively (Wang et al., 2024).

In MMTEB’s fully multilingual benchmark, comprising 132 tasks after reduction, mE5-L attains an average score of 58.6 and ranks fourth. The reported category breakdown is 71.7 for bitext mining, 79.0 for pair classification, 59.9 for classification, 73.5 for STS, 54.1 for retrieval, 21.3 for multilabel classification, 42.9 for clustering, and 62.8 for reranking. On the Europe-focused subset it scores 58.5 and ranks fourth, while on the Indic-focused subset it scores 66.4 and ranks second (Enevoldsen et al., 19 Feb 2025). The same benchmark reports that multilingual-e5-large-instruct, with only 560 million parameters, is the best-performing publicly available model among those evaluated, and it outperforms mE5-L across all categories (Enevoldsen et al., 19 Feb 2025).

A later retrieval benchmark places mE5-L against Google Embeddings 2, BGE-M3, E5-large, LaBSE, and Paraphrase-Multilingual-MPNet. On four zero-shot BEIR subsets, mE5-L obtains a BEIR average $d_{ff}\approx 4096$9 of 0.546, compared with 0.638 for Google Embeddings 2. The per-dataset scores reported for mE5-L are 0.438 on FiQA-2018, 0.341 on NFCorpus, 0.704 on SciFact, and 0.702 on TREC-COVID (Cirillo et al., 22 May 2026). On the synthetic Italian IT-RAG-Bench, however, the gap almost vanishes: mE5-L reaches $\approx 770$0 versus 0.282 for Google Embeddings 2, with Recall@10 of 48.9% versus 47.6% and MRR of 0.243 versus 0.259 (Cirillo et al., 22 May 2026). This pattern positions mE5-L as a model whose competitiveness depends strongly on latency budgets, corpus characteristics, and context-window requirements rather than on a uniform ranking across all multilingual retrieval settings.

5. Fine-tuning for Arabic machine-generated text classification

In the AbjadGenEval shared task, GATech fine-tuned mE5-L for binary classification of Arabic text as human-written or machine-generated. The dataset contained 5,298 Arabic samples, balanced between 2,649 human-written and 2,649 machine-generated texts. The reported average length was 632 words per human-written text and 303 words per machine-generated text (Khamis, 17 Feb 2026).

The classification head consumed a pooled vector $\approx 770$1 and applied LayerNorm, Dropout, a Dense layer with GELU, a second Dropout, and a final linear layer followed by softmax over two classes. The system also used Multi-Sample Dropout with five simultaneous dropout masks at rates $\approx 770$2, averaging the resulting logits before softmax. Training used maximum sequence length 512 tokens with dynamic padding, batch size 16, gradient accumulation steps 4 for an effective batch size of 64, AdamW with weight decay 0.01, learning rate $\approx 770$3 with cosine scheduler and 10% linear warmup, layer-wise learning rate decay factor 0.95, and 2 epochs (Khamis, 17 Feb 2026).

Several pooling strategies were explored. Mean pooling used

$\approx 770$4

Weighted layer pooling first mean-pooled each layer, learned normalized layer weights by softmax, and aggregated them. Multi-head attention pooling used $\approx 770$5 learnable query vectors, computed token-level attention weights, and projected the concatenated context vectors back to dimension $\approx 770$6. Gated fusion combined pooled vectors such as $\approx 770$7 and $\approx 770$8 through a sigmoid gate $\approx 770$9 to produce $1\times10^{-4}$0 (Khamis, 17 Feb 2026).

The final test F1 was 0.75 with simple mean pooling, while the more complex pooling variants yielded approximately 0.70–0.71. The paper attributes mean pooling’s advantage to the limited data size, the additional parameters introduced by learned pooling mechanisms, the strength of the pretrained language-agnostic token embeddings, and the possibility that the relevant human-versus-machine signal is distributed across the text rather than concentrated in a small set of salient tokens (Khamis, 17 Feb 2026). The same study argues that text length is a strong but brittle cue, and it notes that 512-token truncation may remove discriminative content from longer human-written examples.

6. Post hoc cross-lingual alignment and multi-way parallel supervision

Subsequent work has examined whether mE5-style encoders can be improved further through explicit alignment beyond their original multilingual contrastive training. One such study reuses the off-the-shelf mE5 encoder from Wang et al. (2024) and adds a multi-way parallel contrastive objective instead of introducing additional masked language modeling or translation language modeling. In that setup, for each sentence embedding $1\times10^{-4}$1, the positive set $1\times10^{-4}$2 consists of all translations of the same sentence in the other languages in the batch, the negative set $1\times10^{-4}$3 consists of all remaining non-positives, and the fine-tuning objective is

$1\times10^{-4}$4

where $1\times10^{-4}$5 is the supervised multi-positive contrastive loss and

$1\times10^{-4}$6

penalizes drift from the model’s original pretrained embeddings (Fazili et al., 25 Feb 2026).

The multi-way corpus in that study is built from English source sentences sampled equally from OPUS-Wikipedia and OPUS-OpenSubtitles, with minimum length 10 words and fixed seed 42. These sentences are translated with the NLLB-200 3.3 B model into Chinese, Japanese, French, German, Hindi, and Spanish; for each English row, three target languages are sampled uniformly at random to form a four-column row. The resulting dataset has 75,822 rows across train and validation (Fazili et al., 25 Feb 2026).

For mE5, the reported fine-tuning setup uses batch size 128, 5 epochs, in-batch negatives, tuned temperature and regularization weight, and a positive-pair boosting factor $1\times10^{-4}$7 for positives with lower cosine scores. The paper reports gains of +0.7–0.8 points F1 or accuracy on BUCC and Tatoeba across language pairs, and it concludes that even high-quality sentence embedding models such as mE5 still benefit from small-scale multi-way parallel supervision (Fazili et al., 25 Feb 2026). Its broader recommendations are to include typologically varied languages, allow every language rather than only English to serve as an anchor, and scale the parallel corpus beyond approximately 75 K rows where possible.

7. Efficiency, chunking, and deployment trade-offs

mE5-large is the most resource-intensive of the standard multilingual E5 releases. The technical report lists approximately 560 M parameters and about 2.2 GB memory in fp16, compared with approximately 270 M and 1.1 GB for mE5-base and approximately 94 M and 0.35 GB for mE5-small. Approximate throughput on a V100 GPU at batch size 1 is reported as about 30 queries per second for large, 60 for base, and 150 for small (Wang et al., 2024). The model’s principal systems trade-off is therefore straightforward: higher embedding quality at lower throughput and higher memory cost.

CPU-side measurements in a later retrieval benchmark report end-to-end latency, including embedding and FAISS flat search, on a single Apple M-series CPU. Under that setup, mE5-L achieves 31.0 ms median latency, 3.5 ms standard deviation, and 31.8 ms p95 over 50 warm-reused runs. E5-large and BGE-M3 are nearly identical at 30.9 ms, while LaBSE and mMPNet are faster at 16.6 ms and Google Embeddings 2 is much slower at 231.6 ms median with substantial variance (Cirillo et al., 22 May 2026). That benchmark concludes that mE5-L is the preferred option when sub-100 ms SLAs matter.

Chunking sensitivity is also reported explicitly. On IT-RAG-Bench, fixed, sliding-window with 50% overlap, and semantic TextTiling chunking were evaluated at token lengths $1\times10^{-4}$8. For mE5-L, $1\times10^{-4}$9 is approximately 0.040, 0.080, and 0.075 at 8 tokens for fixed, sliding, and semantic chunking respectively; approximately 0.200, 0.255, and 0.275 at 16 tokens; and 0.276, 0.278, and 0.279 at 32 tokens, with the same values persisting at 64 and 128 tokens (Cirillo et al., 22 May 2026). The reported interpretation is that all methods saturate by 32 tokens on that benchmark because the passages are short, and that semantic chunking provides measurable gains only in the 16-token regime. A practical consequence is that 32-token fixed chunking is a reasonable default when semantic segmentation cost is not justified.

Taken together, these operational results define mE5-L’s practical niche. It is not the smallest multilingual encoder, nor the absolute top scorer on every benchmark, but the published evidence consistently places it near the quality frontier while remaining deployable under local inference constraints that are materially tighter than those tolerated by higher-latency hosted alternatives (Wang et al., 2024, Enevoldsen et al., 19 Feb 2025, Cirillo et al., 22 May 2026).