RealAlign: Targeted Cross-Lingual Realignment

• RealAlign is a targeted realignment strategy that selectively improves cross-lingual transfer, particularly for low-resource and typologically distant languages.
• It employs a sentence-level averaging method with contrastive loss to efficiently align translated sentence embeddings before English fine-tuning.
• Strategically selected language subsets based on linguistic diversity often outperform exhaustive coverage, yielding notable gains in weak-transfer regimes.

Searching arXiv for the specified paper and closely related multilingual realignment work. RealAlign, in the sense sharpened by recent multilingual encoder research, denotes an explicit realignment stage applied before downstream English fine-tuning in order to improve zero-shot cross-lingual transfer, especially for low-resource and typologically distant languages (Nguyen et al., 9 Nov 2025). The term is given a more restricted and operational meaning than earlier usage: realignment is not uniformly helpful across languages, and it does not require aligning on every available language. Instead, the strongest gains arise where baseline transfer is weakest—particularly for low-resource languages (LRLs) and languages unseen during pretraining—and these gains can often be obtained with a much smaller, linguistically informed subset of realignment languages (Nguyen et al., 9 Nov 2025).

1. Conceptual scope and problem formulation

RealAlign addresses the instability of prior multilingual realignment results for encoder-only multilingual LLMs such as mBERT and XLM-R (Nguyen et al., 9 Nov 2025). These models already support zero-shot transfer from English, but performance drops sharply for languages that are typologically distant from English and especially for LRLs. Realignment is intended to correct this by explicitly pulling translated content closer in representation space (Nguyen et al., 9 Nov 2025).

A central claim of the recent formulation is that earlier mixed findings are structurally unsurprising. Multilingual encoders are already reasonably aligned for high-resource, related languages, leaving limited headroom for further gains there; classic realignment pipelines rely on word aligners such as FastAlign or AwesomeAlign and on good parallel corpora, both of which are much less reliable for LRLs; and prior work often assumed that adding more realignment languages would necessarily help, without testing whether language choice mattered more than coverage (Nguyen et al., 9 Nov 2025). This suggests that RealAlign is better understood as targeted representation shaping than as exhaustive multilingual data accumulation.

The problem is formalized as selecting a subset $S \subseteq L$ from a full language pool $L$, training a realigned model $\mathcal{M}_S$ on English-parallel data $\mathcal{D}_S$, then fine-tuning on downstream English supervision and evaluating average downstream performance. The ideal subset is written as

$S^* = \mathop{\arg\max}_{S \subseteq L} \dfrac{1}{|\mathcal{T}|} \sum_{T_i \in \mathcal{T} \text{Score}_{T_i}\left(\mathcal{M}^{T_i}_{S}\right)$

but this objective is non-differentiable and combinatorial, so the paper evaluates heuristic subset construction strategies instead (Nguyen et al., 9 Nov 2025).

This framing also clarifies who is expected to benefit. The main research questions concern whether all available languages are necessary, which types of subsets are most effective, whether benefits concentrate in high-resource, medium-resource, or low-resource settings, and whether those benefits generalize to languages absent from the realignment set itself (Nguyen et al., 9 Nov 2025). The answer advanced by current evidence is that RealAlign is particularly useful exactly where baseline cross-lingual transfer is weakest.

2. Realignment objective and training procedure

The recent formulation realigns two encoder-only multilingual models, mBERT and XLM-R, by inserting a separate realignment phase before downstream English fine-tuning (Nguyen et al., 9 Nov 2025). The main methodological simplification is to replace word-level alignment extraction with a sentence-level averaging trick: for each parallel sentence pair, token representations are averaged within each sentence, and a contrastive loss is applied directly between the two averaged sentence embeddings (Nguyen et al., 9 Nov 2025).

The loss is

$$\mathcal{L}(\theta) = \frac{1}{2B} \sum_{h \in H} \log \frac{ \exp\left( \text{sim}(h, \text{aligned}(h)) / T \right) }{ \sum\limits_{h' \in H,\; h' \neq h} \exp\left( \text{sim}(h, h') / T \right) }$$

where $H = \{(h_i, \tilde{h}_i)\}_{i=1}^B$ is a batch of aligned sentence representations, $h_i$ is the averaged embedding of a source sentence, $\tilde{h}_i$ the averaged embedding of its translation, $\text{sim}$ is cosine similarity, and $L$0 is the temperature (Nguyen et al., 9 Nov 2025). Operationally, the objective pulls each sentence toward its translation while pushing it away from other batch elements.

Parallel data come from OPUS-100, with fallback to NLLB when OPUS-100 does not cover a language (Nguyen et al., 9 Nov 2025). This is practically important because the averaging method requires only sentence pairs, not word alignments extracted from those corpora. A plausible implication is that RealAlign becomes feasible in settings where word-alignment quality would otherwise be a bottleneck.

This sentence-level approximation is not claimed to be superior to classic word-alignment-based realignment. FastAlign remains slightly better on most tasks, especially POS: for mBERT POS, 66.85 with averaging versus 70.31 with FastAlign; for XLM-R POS, 69.1 with averaging versus 71.29 with FastAlign (Nguyen et al., 9 Nov 2025). Its significance is computational: average realignment time is 0.35 h, whereas the FastAlign pipeline time is 1.93 h, making the averaging method about 5.5x faster for the realignment step alone (Nguyen et al., 9 Nov 2025). This supports large-scale subset studies that would otherwise be cumbersome.

3. Language subset selection and linguistic diversity

A major contribution of the current RealAlign formulation is to treat language selection as a first-class design variable rather than a byproduct of data availability (Nguyen et al., 9 Nov 2025). The candidate pool $L$1 contains 65 languages: 47 from XTREME-R plus 21 additional African languages, with some overlap, spanning high-, medium-, and low-resource settings (Nguyen et al., 9 Nov 2025). Subsets of size 5, 10, 20, or 40 are built using several heuristics.

The most prominent heuristic is featural diversity (URIEL+). Languages are represented with typological, geographic, phonological, syntactic, and phonetic-inventory features from URIEL+, and pairwise angular distance is used to measure diversity: $L$2 The most diverse subset of size $L$3 is

$L$4

while the least diverse subset is defined analogously with $L$5 (Nguyen et al., 9 Nov 2025). English is included in the diversity calculation but is not itself realigned (Nguyen et al., 9 Nov 2025).

Two additional subset families are evaluated. Language family diversity contrasts subsets drawn from distinct non-Indo-European families against subsets restricted to a single family, Indo-European. Script diversity contrasts distinct non-Latin scripts, Latin-script languages with maximal featural diversity, and Latin-script languages with minimal diversity (Nguyen et al., 9 Nov 2025). These are manually designed heuristics rather than learned selectors.

The empirical pattern is that diversity matters more than count. URIEL featural diversity and URIEL-style diversity within Latin-script languages are reported as the most reliable subset heuristics; these subsets consistently match the full multilingual setup and sometimes surpass it, especially on low-resource and unseen languages (Nguyen et al., 9 Nov 2025). Least-diverse subsets consistently underperform their diverse counterparts, family diversity helps more than restricting to one family, and pure non-Latin script diversity performs worst, with the suggested explanation that excluding Latin may weaken useful anchoring to English as the source fine-tuning language (Nguyen et al., 9 Nov 2025).

This evidence directly challenges the assumption that more realignment languages automatically yield better universal transfer. A plausible implication is that the relevant object for RealAlign is not coverage of the language universe, but coverage of informative linguistic variation.

4. Experimental regime and empirical results

The study covers 65 realignment languages, 29 LRLs, 3 downstream tasks, 4 random seeds, and 2 pretrained multilingual encoders (Nguyen et al., 9 Nov 2025). Realignment is followed by English-only fine-tuning, then zero-shot evaluation on target languages. The downstream tasks are POS tagging, NER, and NLI; for out-of-distribution generalization, the study also evaluates AmericasNLI, covering 10 typologically diverse languages absent from both pretraining and realignment language sets (Nguyen et al., 9 Nov 2025).

Resource stratification follows Joshi et al.’s classes: HRLs = class 5, MRLs = classes 3 and 4, and LRLs = classes 0, 1, 2 (Nguyen et al., 9 Nov 2025). The paper further distinguishes LRLs seen during pretraining from LRLs unseen during pretraining, and this distinction becomes central to the conclusions.

For aggregate in-distribution performance, full multilingual realignment is clearly beneficial overall. For XLM-R, fine-tuning only yields NLI 58.61, POS 65.98, NER 51.09; full 65-language realignment yields NLI 59.43, POS 69.14, NER 57.07, corresponding to gains of about +0.82 NLI, +3.16 POS, and +5.98 NER (Nguyen et al., 9 Nov 2025). For mBERT, fine-tuning only yields NLI 53.12, POS 62.20, NER 52.25; full 65-language realignment yields NLI 55.43, POS 66.87, NER 54.75, corresponding to about +2.31 NLI, +4.67 POS, and +2.50 NER (Nguyen et al., 9 Nov 2025).

The more novel result is that full coverage is not necessary. For XLM-R, a 41-language Latin-script diverse subset slightly exceeds the 65-language baseline on the average of the three task scores:

• 65 languages: $L$6
• 41-language Latin-diverse: $L$7

For mBERT, a 40-language URIEL-most-diverse subset is essentially tied with the 65-language baseline:

• 65 languages: $L$8
• 40-language URIEL-diverse: $L$9 (Nguyen et al., 9 Nov 2025).

The strongest gains appear for LRLs, especially unseen LRLs. The best realignment configuration improves performance on LRLs unseen during pretraining by up to 10 points over standard fine-tuning (Nguyen et al., 9 Nov 2025). By contrast, for HRLs and MRLs, fine-tuning alone remains competitive and realignment can even slightly hurt (Nguyen et al., 9 Nov 2025). This resolves a recurrent misunderstanding in the earlier literature: aggregate averages over easier, better represented languages can obscure the fact that realignment’s payoff is concentrated in genuinely underrepresented languages.

Out-of-distribution transfer further strengthens this interpretation. On AmericasNLI, realignment again beats fine-tuning only, and URIEL-diverse subsets outperform realignment on the full set of 65 languages (Nguyen et al., 9 Nov 2025). This suggests that for unseen LRLs, the type of languages used in the realignment set matters more than the number of languages.

5. Efficiency, scaling, and resource-constrained practice

The scaling analysis gives RealAlign an explicitly efficiency-oriented interpretation. Even with only 5 realignment languages, all strategies outperform simple fine-tuning (Nguyen et al., 9 Nov 2025). Performance rises with the number of languages, but not indefinitely: for XLM-R, most strategies plateau around 20 languages; for mBERT, scaling is more gradual and nearly linear, and some intermediate subsets outperform the 65-language baseline (Nguyen et al., 9 Nov 2025). This suggests different saturation dynamics across models, with XLM-R absorbing transferable alignment signal more quickly than mBERT.

A particularly practical result comes from the hard 10-language budget setting. Random subsets are drawn from pools such as only high-resource languages, only medium-resource languages, mixed pools, only pretraining-seen languages, only unseen languages, and only Joshi class-2 LRLs. The conclusion is that including LRLs and pretraining-unseen languages tends to work best (Nguyen et al., 9 Nov 2025). For XLM-R with 10 random languages, Joshi class 2 yields POS 68.8, NLI 59.9, NER 54.5, Avg 61.1; unseen languages yield POS 69.1, NLI 59.8, NER 53.5, Avg 60.8; HRLs only give Avg 60.2; fine-tuning only gives Avg 58.6; and the full 65-language baseline gives Avg 61.9 (Nguyen et al., 9 Nov 2025). For mBERT, Joshi class 2 yields Avg 57.5, unseen languages Avg 57.1, HRLs only Avg 55.9, fine-tuning only Avg 55.8, and the full 65-language baseline Avg 59.0 (Nguyen et al., 9 Nov 2025).

The gap between 10 languages and 65 languages is reported as only 0.8% to 1.5% despite a more than sixfold reduction in number of languages (Nguyen et al., 9 Nov 2025). This is one of the clearest operational lessons associated with RealAlign: if parallel data cannot be collected for all languages, a small, diverse subset—preferably containing LRLs or pretraining-unseen languages—captures most of the value. If LRL parallel data is unavailable, HRLs and MRLs can still serve as substitutes, with less than 1% degradation in some settings (Nguyen et al., 9 Nov 2025).

In this respect, the newer large-scale study also complements earlier evidence from layer-sensitive realignment work. "AlignFreeze" showed that realignment can affect all layers but can be most detrimental to the lower ones, and that freezing the lower half during realignment can prevent degradation in some multilingual encoder settings (Bakos et al., 18 Feb 2025). This suggests that RealAlign is not only about selecting the right languages, but also about limiting destructive interference with pretrained multilingual structure when alignment is applied.

6. Interpretation, limitations, and adjacent usages

The current formulation advances a more selective principle: if evaluation focuses on HRLs or already well-covered languages, realignment may add little and sometimes hurt; if the focus is LRLs, especially unseen ones, realignment is highly beneficial; among realignment strategies, linguistic diversity is more important than raw language count; and for robustness to unseen languages, a carefully selected diverse subset can outperform exhaustive coverage (Nguyen et al., 9 Nov 2025). This suggests that RealAlign should be understood as a targeted intervention for weak-transfer regimes rather than as a universal preprocessing step.

Several limitations remain. The study is restricted to encoder-only models rather than modern decoder-only LLMs; appendix experiments on Llama 3.1 8B with LoRA show mixed results, with realignment improving POS from 38.8 to 41.9 and NLI from 56.5 to 57.8, but hurting NER from 31.6 to 30.2 (Nguyen et al., 9 Nov 2025). The sentence-averaging method sacrifices some performance relative to FastAlign (Nguyen et al., 9 Nov 2025). Language selection is heuristic rather than learned. The 65-language pool, although broad, does not cover all linguistic diversity globally. And because the study relies on available OPUS/NLLB bitext, it cannot fully disentangle subset quality from parallel-data quality (Nguyen et al., 9 Nov 2025).

The term itself also has broader usage outside multilingual encoder realignment. "Flexible Realignment of LLMs" introduces RealAlign as a framework for controllable realignment of already aligned LMs via Training-time Realignment (TrRa) and Inference-time Realignment (InRa), with quantitative control through a realignment coefficient $\mathcal{M}_S$0 (Zhu et al., 15 Jun 2025). That usage concerns post-training control of reasoning or preference strength rather than cross-lingual encoder alignment. A plausible implication is that "RealAlign" is becoming a polysemous label spanning different post-hoc alignment adjustments, but in multilingual NLP the most technically specific contemporary meaning is the one established by the large-scale low-resource study: realign strategically, not exhaustively, and expect the largest gains on the languages that current multilingual transfer serves worst (Nguyen et al., 9 Nov 2025).