---
title: 'ALIGN-MLM: Alignment-Guided Pre-training'
url: https://www.emergentmind.com/topics/align-mlm
type: topic
---

# ALIGN-MLM: Alignment-Guided Pre-training

ALIGN-MLM denotes a masked-language-modeling-based alignment paradigm whose most explicit formulation is a multilingual pre-training objective that adds an auxiliary word embedding alignment loss to standard MLM, with the goal of improving zero-shot cross-lingual transfer [2211.08547]. In the literature provided here, the term also has a secondary, broader use: the earlier "align, mask, and select" (AMS) commonsense pre-training method was described as a form of "ALIGN-MLM" because the masked position is aligned with a knowledge-graph relation before prediction [1908.06725]. The shared idea across these usages is not a single benchmark or architecture, but the deliberate replacement of unguided masking with alignment-constrained masking or alignment-constrained representation learning.

## 1. Terminology and conceptual scope

In the multilingual setting, ALIGN-MLM is the name of a pre-training objective proposed to highlight "the importance of word embedding alignment" for multilingual pre-training [2211.08547]. Its central claim is that multilingual transfer depends critically on whether semantically equivalent lexical items across languages are brought close in embedding space. This use of the term is precise and refers to a specific auxiliary loss added to MLM.

A second usage appears in the discussion of the AMS method for commonsense knowledge injection into BERT-like encoders. There, ALIGN-MLM is presented as an interpretive label for a masking strategy in which the masked concept is explicitly tied to a ConceptNet relation through an align-mask-select pipeline [1908.06725]. This suggests that ALIGN-MLM is best understood as a family resemblance across methods rather than a single invariant algorithm: in one case the alignment object is a bilingual lexicon, and in the other it is a knowledge-graph triple aligned to natural-language sentences.

This terminological overlap is important because the two lines of work target different failure modes. The multilingual formulation addresses weak cross-lingual transfer caused by poor lexical alignment, especially under script and word-order divergence. The AMS-related formulation addresses the limited explicit incorporation of commonsense knowledge in pre-trained language representation models.

## 2. Multilingual ALIGN-MLM objective

The 2022 formulation of ALIGN-MLM uses a Transformer-based encoder, such as RoBERTa, and augments standard masked language modeling with an auxiliary alignment loss defined over a bilingual dictionary [2211.08547]. Let \(L_1\) and \(L_2\) denote two languages, \(\mathcal{B}\) a bilingual dictionary of word pairs \((B_{i1}, B_{i2})\), and \(\mathbf{E}[w]\) the embedding of token \(w\). The auxiliary loss maximizes cosine similarity between embeddings of translation pairs:

\[
L_{\text{ALIGN}} = \sum_{i=1}^{|\mathcal{B}|} \left[ 1 - \cos\_sim\left( \mathbf{E}[B_{i1}],\, \mathbf{E}[B_{i2}] \right) \right]
\]

The total objective is

\[
L_{\text{ALIGN-MLM}} = L_{\text{MLM}} + \alpha \cdot L_{\text{ALIGN}}
\]

where \(L_{\text{MLM}}\) is the standard masked language modeling loss applied to both language corpora, and \(\alpha\) is a hyperparameter; the supplied summary states that grid search shows \(10\) works well [2211.08547].

A defining property of this objective is that it requires only a small bilingual dictionary rather than full parallel corpora. The alignment signal is therefore word-level and explicit, rather than emerging indirectly from shared subword vocabularies, code-switching, or sentence-level parallel supervision. The paper’s formulation is consequently narrower than generic multilingual MLM, but more targeted: it directly optimizes the lexical-semantic correspondence that the authors argue is crucial for transfer.

## 3. Relation to MLM, XLM, and DICT-MLM

The paper positions ALIGN-MLM against three widely adopted objectives: MLM, XLM, and DICT-MLM [2211.08547]. Standard MLM pre-trains independently on each language and has no explicit cross-lingual alignment mechanism. Cross-lingual alignment may emerge, but the supplied summary characterizes this emergence as unreliable, especially when languages differ in script or syntax.

XLM with Translation Language Modeling uses parallel sentences and applies MLM on concatenated translation pairs. This supplies sentence-level cross-lingual supervision, but does not directly optimize word embedding alignment. It also requires large amounts of parallel data. DICT-MLM uses a bilingual dictionary to generate code-switched mixed training data by replacing masked tokens with translation equivalents, thereby encouraging cross-lingual similarity indirectly through contextualized training examples.

ALIGN-MLM differs in making the word embedding alignment objective first-class rather than incidental. The comparison advanced in the paper is therefore methodological as much as empirical: MLM relies on unsupervised emergence, XLM relies on expensive parallel data, and DICT-MLM relies on stochastic exposure to mixed examples, whereas ALIGN-MLM directly optimizes cosine similarity over translation pairs [2211.08547]. A plausible implication is that ALIGN-MLM is designed to isolate one variable—lexical embedding alignment—that earlier objectives confounded with many other cross-lingual factors.

## 4. Empirical results and alignment–transfer correlation

The experiments evaluate transfer between pairs of natural languages and systematically modified counterparts differing in script, word order, or syntax, with tasks drawn from XTREME: XNLI, NER, and POS-tagging [2211.08547]. The strongest reported gains occur when languages differ in both script and word order, denoted \(T_{trans} \circ T_{inv}\). In that setting, ALIGN-MLM outperforms XLM and MLM by 35 and 30 F1 points on POS-tagging; the supplied example gives 92.0 for ALIGN-MLM versus 61.8 for XLM.

The same summary reports that ALIGN-MLM is also superior for NER and XNLI under these difficult transformations, with gains greater than 3 points over DICT-MLM and greater than 1.5 over XLM. For script-only differences, all methods perform comparably. For syntax changes, ALIGN-MLM still outperforms MLM by more than 6 points on NER. It is also reported to beat XLM even when XLM is allowed up to 100% parallel data, and to use bilingual dictionaries more efficiently than DICT-MLM across all dictionary coverage ratios [2211.08547].

A central empirical claim is the strong correlation between alignment and transfer. The reported Spearman correlations are \(\rho = 0.727\) for XNLI with \(p < .01\), \(\rho = 0.781\) for NER, and \(\rho = 0.734\) for POS [2211.08547]. In the paper’s interpretation, better word-level alignment predicts better zero-shot transfer across objectives and language transformations. This does not merely support the usefulness of the auxiliary loss; it is presented as evidence that explicit embedding alignment should be optimized rather than left to emerge from generic multilingual pre-training.

## 5. The AMS interpretation: ALIGN-MLM as knowledge-guided masking

The earlier AMS work proposes a pre-training approach for incorporating commonsense knowledge into language representation models by constructing a large multi-choice question answering dataset from ConceptNet triples aligned to Wikipedia sentences [1908.06725]. The pipeline is "align, mask, and select": align a ConceptNet triple \((c_1, r, c_2)\) to a sentence containing both concepts, mask one concept to form a question, and select distractors from other concepts sharing the same relation with the unmasked concept. The resulting dataset, \(\mathcal{D}_{AMS}\), contains 16+ million samples, each with one correct answer and four distractors.

Within the supplied description, this procedure "can be seen as a form of 'ALIGN-MLM'" because whole concepts are masked in contextually relevant sentences and the masking decision is explicitly grounded in a knowledge relation [1908.06725]. The training objective becomes multi-class classification over five candidates rather than vocabulary-wide recovery:

\[
L = -\log p(c_i|s)
\]

with

\[
p(c_i | s) = \frac{\exp(\mathbf{w}^T \mathbf{c}_i)}{\sum_{k=1}^N \exp(\mathbf{w}^T \mathbf{c}_k)}
\]

where \(N=5\), \(c_i\) is the correct answer, \(\mathbf{w}\) softmax weights, and \(\mathbf{c}_i\) the \([CLS]\) embedding for candidate \(i\). For Winograd Schema Challenge, the paper uses a hinge-style loss.

Empirically, the AMS-pre-trained model improves commonsense benchmarks while maintaining comparable performance on general NLP tasks. On CommonsenseQA, BERT\_CS (large) achieves 62.2% versus 56.7% for BERT-large and 58.2% for CoS-E; on Winograd Schema Challenge, BERT\_CS (large) reaches 75.5% [1908.06725]. On GLUE, the supplied examples include BERT-large versus BERT\_CS-large scores of MNLI 86.7/85.9 versus 86.7/85.8, SST-2 94.9 versus 94.1, and CoLA 60.5 versus 60.7. The paper further states that pre-training directly on knowledge graph triples degrades general language performance, whereas the AMS procedure avoids this pitfall by using natural-language sentences.

## 6. Significance, limitations, and interpretive boundaries

Taken together, the supplied papers support a general view of ALIGN-MLM as alignment-guided masked pre-training: either align lexical items across languages before or during MLM, or align masked concepts to structured commonsense relations before prediction [2211.08547; 1908.06725]. In both cases, the method replaces undifferentiated masking with an external structure that constrains what should become close, what should be predicted, or which alternatives should be discriminated.

The multilingual version has a sharply defined empirical thesis: explicit word embedding alignment is crucial for multilingual pre-training and robust zero-shot transfer, particularly when languages differ in script and word order [2211.08547]. The commonsense version has a different thesis: knowledge-aligned masking in natural-language sentences can inject commonsense knowledge without degrading general language representation capabilities [1908.06725]. These are compatible but not identical claims.

Several limitations are also explicit in the supplied material. For AMS, construction relies on exact matching for entity alignment between ConceptNet and Wikipedia, is designed for English, and may overlap with CommonsenseQA because that benchmark is itself constructed using ConceptNet [1908.06725]. For multilingual ALIGN-MLM, the method depends on bilingual dictionaries and is evaluated through transfer scenarios that systematically modify properties such as script and word order [2211.08547]. This suggests that the term ALIGN-MLM should not be treated as a monolithic standard, but as a technically specific design principle whose concrete implementation depends on the alignment resource: dictionary entries in multilingual pre-training, or knowledge-graph triples in commonsense-oriented masking.

Source: https://www.emergentmind.com/topics/align-mlm