Language-Agnostic Embeddings

Updated 10 June 2026

Language-agnostic embeddings are vector representations that remove language-specific cues to preserve cross-lingual semantic content.
They utilize methods like subspace projection, normalization, and adversarial training to debias embeddings for robust multilingual transfer.
Empirical evaluations demonstrate significant improvements in tasks such as cross-lingual retrieval and zero-shot question answering.

Language-agnostic embeddings are vector representations that intentionally abstract away language- or modality-specific information, preserving only cross-linguistically shared, semantic or structural factors relevant for multilingual transfer, retrieval, or generalization. Unlike conventional multilingual representations that often encode both semantic content and language identity, language-agnostic embeddings aim to eliminate language-specific biases, clustering semantically equivalent content regardless of script, phonological inventory, or surface order. This property is critical for robust cross-lingual transfer in multilingual NLP, cross-script retrieval, and cross-modal semantic applications.

1. Motivation and Foundational Concepts

Large-scale multilingual LLMs (e.g., mBERT, XLM-R, LaBSE) demonstrate strong cross-lingual transfer yet their underlying embedding spaces encode not only semantics but also substantial language-specific factors such as syntax, script, and word-order biases. Embeddings from these models tend to cluster by language rather than meaning, which impairs zero-shot transfer for tasks such as cross-lingual retrieval or QA over a multilingual candidate pool. The goal of language-agnostic embeddings is to “erase” these spurious language cues, leaving only semantic, language-neutral components that enable strong alignment of equivalent content across languages (Xie et al., 2024).

Formally, a language-agnostic embedding $z$ from an original embedding $x$ is constructed by projecting $x$ into the orthogonal complement of a language-specific subspace $L$ . This decomposition is generalizable: for any representation $h\in\mathbb{R}^d$ ,

$h = h_{\text{lang}} + h_{\text{sem}},$

where $h_{\text{lang}} \in L$ (language-specific) and $h_{\text{sem}} \in S$ (language-neutral, semantic).

2. Empirical Characterization of Language-Specific Subspaces

Systematic probing of multilingual encoders reveals that language-specific information is not isolated in a single dimension or neuron but scattered throughout an $O(n)$ -dimensional subspace, with $n$ close to the number of languages. This subspace can be identified by linear projections such as singular value decomposition (SVD), Linear Discriminant Analysis (LDA), or centering (Liang et al., 2021, Utpala et al., 2023). Probing tasks (language identification, linguistic typology, clustering) demonstrate that removing the top $x$ 0 directions corresponding to language identity results in near-random language classification accuracy, but retains nearly all performance on structural or semantic downstream tasks. Notably, for mBERT 104-way probing, $x$ 1 suffices to nearly eliminate language information (Liang et al., 2021).

3. Methodologies for Inducing Language-Agnostic Embeddings

3.1. Subspace Projection and Linear Debiasing

A core class of methods uses unsupervised SVD to identify and remove the dominant directions capturing language identity:

SVD-based Null Space Projection (LSAR): Stack monolingual embeddings from each language, compute SVD, and form a projection $x$ 2, where $x$ 3 spans the top- $x$ 4 language-specific subspace. For any embedding $x$ 5, obtain the language-agnostic version via $x$ 6 (Xie et al., 2024).
PCA/Language Information Removal (LIR): Collect SVD or eigen-decomposition of the covariance of monolingual embeddings, select a small rank $x$ 7, and project out $x$ 8 main directions. This framing is model-agnostic and requires no fine-tuning (Yang et al., 2021).

Practical variations include per-language PCA (for visual or code embeddings), common subspace SVD (for code), and supervised DensRay/LDA projections (Utpala et al., 2023, Liang et al., 2021). The optimal subspace rank is typically selected by explained variance ( $x$ 9– $x$ 0) or validation on retrieval tasks; for $x$ 1 languages, $x$ 2 is often effective (Xie et al., 2024).

3.2. Post-hoc Normalization and Alignment

Alternative approaches focus on normalizing or re-aligning representation spaces:

BatchNorm/Vector Space NORM: Removing language-specific means and variances from each batch, e.g., via BatchNorm, sharpens separation by semantics and decreases language identity signals (Zhao et al., 2020).
Vector Space Joint-Alignment: Use small parallel corpora to re-align language spaces to a pivot (usually English) using a loss that pulls word-level representations together, optionally regularized to prevent distortion (Zhao et al., 2020).
Text-level Normalization: Syntactic or morphological normalization in preprocessing (e.g., de-contraction, word-order harmonization) increases cross-lingual alignment, with additive improvements observed on classification tasks (Zhao et al., 2020).

3.3. Adversarial Constraints and Universal Bottlenecking

Universal Grammar-inspired architectures constrain intermediate representations so that they are indistinguishable across languages by adversarial training—using the Wasserstein-1 distance between representations of different languages as a regularizer. This enforces a tight “universal” bottleneck in the representation, decoupling language parameters and semantics (Aghajanyan et al., 2018).

Language-agnosticity extends to speech, code, and vision. Phoneme embeddings derived from articulatory features generalize across languages and facilitate rapid adaptation in low-resource TTS (Lux et al., 2022). Cross-lingual visual embeddings for handwriting retrieval use asymmetric dual encoders anchored to language-agnostic semantic prototypes, achieving script-invariant retrieval (Chen et al., 16 Jan 2026). Multilingual code models benefit from syntax/semantic subspace separation, significantly raising semantic retrieval accuracy across programming languages (Utpala et al., 2023).

4. Large-Scale Benchmarks and Empirical Evaluations

Key benchmarks distinguish between “weak” alignment (cross-lingual transfer with no distractors in the same language) and “strong” alignment (retrieval from a multilingual pool with competing same-language distractors) (Roy et al., 2020).

LAReQA: Defines strong alignment as requiring cross-lingual semantic pairs to rank ahead of even same-language non-relevant pairs (Roy et al., 2020):

$x$ 3

On LAReQA (XQuAD-R), projecting out language-specific subspaces from mBERT embeddings nearly doubles mean average precision (mAP $x$ 4) (Xie et al., 2024).

Tatoeba, UN, BUCC: Bitext retrieval tasks measure nearest-neighbor accuracy across up to 112 languages. Removing language-specific signals with LSAR or LIR increases Tatoeba accuracy from $x$ 5 for mBERT, confirming improved agnosticism (Xie et al., 2024).
XNLI/RFEval: Combining normalization and alignment reduces cross-lingual transfer gaps by $x$ 6 (m-BERT) and $x$ 7 (XLM-R) points (Zhao et al., 2020).
Code XLCoST: Mean reciprocal rank boosts up to $x$ 8 via subspace removal for cross-language code retrieval (Utpala et al., 2023).
Handwriting OOD retrieval: Language-agnostic visual embeddings deliver $x$ 9 Acc@1 in cross-script retrieval with $L$ 0 the parameters of vision-language behemoths (Chen et al., 16 Jan 2026).

5. Applications Across Modalities

Language-agnostic embeddings now underpin cross-lingual sentence retrieval, QA, document alignment, code search, speech intent classification, handwriting retrieval, and sign language translation. Applications include:

Sentence and Document Retrieval: LAWDR applies the subspace-debiasing recipe to sentence-level document representations, achieving Recall@1 of $L$ 1 on WMT-19 document alignment (Gong et al., 2021).
Multimodal Supervision: SONAR multimodal embeddings, jointly trained on text and speech, enable language-agnostic sign language translation and cross-lingual abstractive summarization with improved factual consistency (Hamidullah et al., 22 Oct 2025, Chellaf et al., 9 Mar 2026).
Code Search: Language-agnostic code subspaces enable retrieval of semantically equivalent programs independent of programming language, with MRR increases up to $L$ 2 absolute (Utpala et al., 2023).
Speech and SLU: Universal phoneme and intent embeddings based on shared phonetic spaces or pre-trained universal phone recognizers (Allosaurus) outperform language-specific baselines in intent classification for low-resource languages (Lux et al., 2022, Yadav et al., 2021).
Speaker Disentanglement: LASPA leverages prefix-tuned cross-attention to explicitly disentangle speaker and language factors, improving EER for both seen and unseen languages (Menon et al., 2 Jun 2025).

6. Language-Agnostic Embedding Models

A diverse set of architectures deliver language-agnostic sentence or document embeddings:

Model	Core Method	Embedding Dim	Language Coverage	Key Performance
LaBSE	Dual-encoder+contrastive	768	109	Tatoeba Recall@1 83.7%
SONAR	Encoder–decoder+contrastive/gen	1024	200 (text & speech)	Tatoeba Recall@1 >95%
BGE-M3	Single-tower, multi-task+KD	1024	100+	Tatoeba Recall@1 ~97%
LEALLA	Thin-deep+k-distillation	128–256	109	Near-LaBSE performance, 7× smaller

All employ large-scale cross-lingual contrastive training, $L$ 3 normalization, and maximize semantic proximity while minimizing language or modality cues (Feng et al., 2020, Mao et al., 2023, Chellaf et al., 9 Mar 2026).

7. Limitations and Future Directions

Language-agnostic projection methods are primarily linear, removing syntax and script signals but potentially harming tasks sensitive to fine-grained syntactic or script differences if the subspace rank $L$ 4 is set too large (Xie et al., 2024). Nonlinear or kernel-based removals, adaptive rank selection, or adversarial domain generalization approaches are recognized as promising extensions. For code (Utpala et al., 2023), centering can over-subtract, and models already contrastively aligned may see diminished marginal returns. Universal-bottleneck and adversarial approaches (Aghajanyan et al., 2018) are computationally intensive and their absolute cross-lingual performance still lags bilingual systems. Training data for leading models such as LaBSE or SONAR remain English-centric, and guarantees for low-resource or typologically diverse languages require further empirical study (Mao et al., 2023, Chellaf et al., 9 Mar 2026).

A plausible implication is that as embedding models scale and coverage broadens to new modalities, robust language-agnostic subspaces will underpin large-scale multilingual, multimodal, and cross-domain retrieval or understanding systems. Adaptive or fine-grained disentanglement methods are likely to drive the next generation of universal semantic representations.

Key References:

Xie et al., “Discovering Low-rank Subspaces for Language-agnostic Multilingual Representations” (Xie et al., 2024)
Georgi et al., “Locating Language-Specific Information in Contextualized Embeddings” (Liang et al., 2021)
Saha et al., “Inducing Language-Agnostic Multilingual Representations” (Zhao et al., 2020)
Wang et al., “A Simple and Effective Method To Eliminate the Self Language Bias in Multilingual Representations” (Yang et al., 2021)
Feng et al., “Language-Agnostic BERT Sentence Embedding” (Feng et al., 2020)
Wu et al., “LAWDR: Language-Agnostic Weighted Document Representations from Pre-trained Models” (Gong et al., 2021)
Imani et al., “Language Agnostic Code Embeddings” (Utpala et al., 2023)
Lux & Vu, “Language-Agnostic Meta-Learning for Low-Resource Text-to-Speech with Articulatory Features” (Lux et al., 2022)
Chellaf et al., “Using Multimodal and Language-Agnostic Sentence Embeddings for Abstractive Summarization” (Chellaf et al., 9 Mar 2026)