---
title: Multilingual Bi-Encoders
url: https://www.emergentmind.com/topics/multilingual-bi-encoders-c10b778f-945f-49f1-a5a3-89ccc73f528c
type: topic
---

# Multilingual Bi-Encoders

A multilingual bi-encoder is a neural architecture consisting of two independently operating encoders, each preprocesses its input sequence (e.g., query and candidate text, image and caption, or speech input) into a fixed-dimensional vector in a shared semantic space. Unlike traditional cross-encoder designs, bi-encoders precompute candidate representations offline and compare them to queries at runtime using a similarity metric such as dot product or cosine similarity. In the multilingual setting, a single bi-encoder is tasked with embedding inputs from multiple languages (and sometimes multiple modalities) so that semantically corresponding items—across languages and possibly modalities—occupy proximate locations in the latent space, enabling rapid and robust cross-lingual retrieval and matching.

## 1. Core Multilingual Bi-Encoder Architectures

The canonical architecture employs two towers (“Siamese structure”), each typically instantiated by a Transformer variant pretrained on multilingual corpora (e.g., mBERT, XLM-RoBERTa). Parameter sharing between towers is customary, though some approaches use independent encoders for source and target sides. The input is tokenized by a shared subword vocabulary and embedded, usually with positional and segment embeddings for textual inputs [2109.07157][2109.12870][1907.04307][2010.07972].

Pooling strategies for final embeddings include using the [CLS] token, average pooling over token-level outputs, or more sophisticated projections via linear or multilayer heads. For multimodal retrieval (e.g., image-text, speech-text), domain-specific encoder architectures are employed—such as Vision Transformers for images or Whisper/MMS for audio—followed by adapter layers and task-specific fusion mechanisms [2409.11214][2401.15896].

A representative text-only multilingual bi-encoder pipeline:

| Stage               | Example Component         | Dimensions          |
|---------------------|--------------------------|---------------------|
| Input               | mBERT/XLM-R, WordPiece   | ℝⁿ (tokens)         |
| Encoding            | 12-Layer Transformer     | ℝⁿ×768              |
| Pooling             | [CLS], avg/mean          | ℝ⁷⁶⁸ or ℝ⁵⁰⁰        |
| Projection          | Linear/MLP (optional)    | ℝᵈ                  |
| Similarity Scoring  | Dot/cosine               | Scalar              |

In multimodal cases, pools of encoder outputs for different input types are gated and fused according to language- or modality-specific selectors [2409.11214][2401.15896].

## 2. Training Objectives and Contrastive Losses

Bi-encoders are nearly always trained via contrastive loss functions that drive the embeddings for positive pairs (translation pairs, FAQ pairs, CV-vacancy matches, etc.) to be more similar than those for negative pairs. The dominant formulation is the in-batch softmax or InfoNCE loss [2109.07157][2109.12870][1902.08564][2010.02537][2010.07972]:

$$
\mathcal{L} = -\frac{1}{N}\sum_{i=1}^N
\log \frac{\exp(\text{sim}(e_i^A, e_i^B)/\tau)}{\sum_{j=1}^N \exp(\text{sim}(e_i^A, e_j^B)/\tau)}
$$

where $e_i^A$, $e_i^B$ are the encoded vectors of pair $i$, $\tau$ is a temperature, and negatives are drawn from all other batch candidates. In addition, additive margin softmax [1902.08564] and explicit alignment losses at both sentence and token granularity (AMBER: $\mathcal{L}_{\text{sent}}$, $\mathcal{L}_{\text{tok}}$) have improved stability and discriminative power in cross-lingual settings [2010.07972].

Negative sampling strategies range from in-batch negatives (computationally efficient), hard negatives sampled from top BM25/FAISS candidates, or batching semantically similar items (e.g., all FAQ pairs from one page) to stress fine distinction [2109.12870][2204.02363]. In multimodal settings, grouped InfoNCE with distributed aggregation is used to scale training to billions of examples [2401.15896].

## 3. Multilingual Alignment, Data Handling, and Cross-Lingual Transfer

Multilingual bi-encoders derive cross-lingual generalization primarily from Transformer backbones pretrained on joint multilingual corpora with shared subword vocabularies [2109.07157][2204.02363][1907.04307]. This produces anchor points for entity names and structural tokens, facilitating transfer even for low-resource languages or differing scripts. Fine-tuning on multilingual parallel data further aligns embeddings across languages; explicit alignment losses (e.g., AMBER—sentence-level and token-level) result in additional gains for sequence tagging and retrieval, most pronounced in low-resource languages [2010.07972][2205.00267].

Cross-lingual transfer is empirically robust: fine-tuning on any language improves target-language retrieval, and joint multilingual fine-tuning reliably outperforms monolingual approaches except in certain high-resource cases [2109.12870][2204.02363]. Multilingual bi-encoders also permit efficient code-switching and handling entity-rich retrieval where the same representation must serve many scripts or domains.

Tables below compare performance for monolingual and multilingual bi-encoder models on FAQ retrieval [2109.12870]:

| Language    | Monolingual MRR | Multilingual MRR |
|-------------|-----------------|------------------|
| English     | 82.9            | 82.5             |
| German      | 81.1            | 81.3             |
| Spanish     | 78.0            | 81.7             |
| French      | 71.0            | 80.7             |
| Dutch       | 70.4            | 81.2             |
| Russian     | 71.6            | 82.1             |

## 4. Scalability, Retrieval, and Practical Deployment

The bi-encoder design decouples inference from candidate encoding: all candidates (documents, answers, jobs, images) are embedded and indexed offline. At runtime only the query is encoded, and nearest-neighbor search over vector databases (e.g., FAISS, HNSW, IVF-PQ) retrieves top-k matches [2109.07157][2401.15896][1907.04307]. This pipeline achieves sub-millisecond per-query retrieval at scale (10⁶–10⁷ candidates), with batchable encoding and CPU-querying.

Grouped aggregation and gradient accumulation allow training on up to trillions of pairs by limiting GPU memory and communication [2401.15896]. For document-level retrieval, averaging sentence embeddings is highly effective and matches prior heavily engineered solutions [1902.08564].

Multilingual bi-encoder pipelines are maintainable: a single model supports all supported languages, avoids per-language forks, and can be periodically retrained on fresh matched pairs or new language data [2109.07157][2204.02363].

## 5. Empirical Results and Cross-Task Performance

Multilingual bi-encoders surpass lexical and monolingual dense baselines across retrieval, classification, and mining tasks. Notable empirical results:

- CV–vacancy matching: recall@50 improved from ~25% (TF-IDF baseline) to ~65%; cross-language matching saw relative lift of +40% compared to monolingual models [2109.07157].
- FAQ retrieval on MFAQ: multilingual XLM-RoBERTa outperformed monolingual per-language baselines by 5–15 MRR points for most languages, with ~6M FAQ pairs [2109.12870].
- UN parallel corpus: bi-directional dual-encoder with additive margin softmax yielded P@1 ≥ 86% for sentence retrieval, ~97% document retrieval [1902.08564].
- Image-text retrieval: M²-Encoder-10B reached 88.5% top-1 accuracy on ImageNet (English), 80.7% on ImageNet-CN (Chinese), and ≥91.2% recall@1 across major retrieval benchmarks [2401.15896].
- Speech-to-text: Ideal-LLM reduced ASR WER by 32.6% over baseline, BLEU scores for AST up to 36.78, attributing gains to dual-encoder fusion and language-adapted gating [2409.11214].

Bi-encoder models also minimize nationality-based disparity by reducing language as a proxy feature in hiring applications [2109.07157].

## 6. Limitations, Alignment Controversies, and Analysis

Explicit alignment objectives (e.g., word-level L2 distance, linear mapping) yield only marginal or statistically insignificant improvements over strong multilingual pretraining and larger model capacity [2010.02537]. Newly proposed contrastive alignment objectives are more robust to alignment noise but their observed gains fall within standard deviation of no alignment, especially for XLM-R-large models. Empirical analysis recommends prioritizing larger multilingual language models and robust multi-task evaluation over alignment engineering.

A residual issue is model brittleness—multilingual bi-encoders are sometimes overly surface-keyword dependent, exhibiting suboptimal robustness to paraphrase and adversarial rewording [2109.12870]. Extensions such as additive margin softmax and bidirectionality improve discrimination and recall, but further research is warranted on phrase-level and context-adaptive retrieval.

## 7. Recommended Practices and Future Directions

Recommended practices for multilingual dense retrieval include starting from large multilingual pre-trained encoders (mBERT, XLM-R), leveraging multi-stage fine-tuning (large English collection followed by in-language or cross-language finetuning), using shared subword vocabularies, and relying on in-batch negative sampling for scalable training [2204.02363]. Hybrid approaches with sparse retrieval (BM25) consistently outperform either method alone.

For extension to multimodal and low-resource scenarios, domain-specific encoder fusion (as in Ideal-LLM or M²-Encoder), balanced data distribution, and grouping/aggregation methods facilitate large-scale and multi-language coverage. Explicit alignment regularization may be most beneficial for smaller models or low-resource settings, but scaling data and model size remain empirically superior strategies [2401.15896][2010.07972][2205.00267].

Future work involves expanding to more languages, incorporating phrase- and entity-level alignment objectives, enhancing robustness, and extending grouped bi-encoder architectures to audio, video, and cross-modal retrieval [2401.15896][2409.11214][2205.00267].

Source: https://www.emergentmind.com/topics/multilingual-bi-encoders-c10b778f-945f-49f1-a5a3-89ccc73f528c