---
title: 'WSPAlign: Weakly Supervised Word Alignment'
url: https://www.emergentmind.com/topics/wspalign
type: topic
---

# WSPAlign: Weakly Supervised Word Alignment

WSPAlign is a word-alignment framework based on cross-language span prediction and, in its explicit titled form, stands for **Weakly Supervised span Prediction pre-training for word AlignmenT**. It was introduced to reduce dependence on **manual alignment annotation** and **parallel corpora** by pre-training on **noisy, partially aligned, and non-parallel paragraphs** mined at large scale, then fine-tuning on standard gold benchmarks [2306.05644]. The method preserves the span-prediction view associated with earlier multilingual BERT alignment work, in which a source token or span is treated as a query and the aligned target span is predicted, but it relocates the main source of supervision from manually created gold alignments to large-scale weak supervision [2004.14516].

## 1. Problem setting and conceptual basis

Word alignment is the task of identifying which words or token spans in one language correspond to which in another. It remains important for **annotation projection**, **terminology enforcement**, **under-translation detection**, and **cross-lingual representation learning** [2306.05644]. Earlier work had already shown that alignment can be cast as **cross-language span prediction**: a source token functions as the “question,” the target sentence is the “context,” and the aligned target span is the “answer,” in the style of **SQuAD v2.0** question answering [2004.14516].

WSPAlign is motivated by a specific limitation in prior aligners. Statistical systems such as **Giza++** and **FastAlign** depend on parallel corpora; unsupervised embedding-based methods such as **SimAlign** avoid parallel training data but are often weaker on benchmark alignment quality; and supervised systems such as **AWESoMe** and **SpanAlign** are strong but rely on manual alignment labels or parallel data [2306.05644]. WSPAlign keeps the strong span-prediction formulation but broadens the supervision source by relaxing the requirement for **correct**, **fully aligned**, and **parallel** training instances. The result is a pre-training scheme rather than merely a new architecture. This suggests that its main contribution is a data-scaling strategy for word alignment.

## 2. Weak supervision from Wikipedia

The weakly supervised corpus is constructed in two stages from Wikipedia. First, paragraphs are paired through **co-mentioned entities** using Wikipedia hyperlinks and Wikidata entity IDs. The construction is language-agnostic at the entity level: every hyperlink points to an entity ID, an inverted index is built from entity ID to all paragraphs mentioning that entity, and paragraph pairs are formed whenever two paragraphs mention the same entity [2306.05644].

Second, alignments are automatically annotated in two forms. **Wiki words** are entity-linked spans: if two paragraphs mention the same entity, the corresponding spans are aligned directly. **Common words** are obtained with a multilingual PLM and **bidirectional nearest-neighbor agreement** in the style of SimAlign: a word pair is kept only when the source word is the most similar target word and the target word is the most similar source word, with an additional filter retaining alignments where at least one aligned token is a common word [2306.05644].

The scale is central. Raw Wikipedia paragraph pairs from co-mentions reach **tens of millions**; the pre-training set samples **2,000,000** paragraph pairs; **wiki items** are annotated for all **2,000,000** pairs; and **common-word alignments** are annotated for **200,000** pairs. Filtering removes very short and very long pairs and retains only pairs whose sentence-embedding cosine similarity exceeds **0.75**, using **LaBSE** for multilingual data and **pcl-bert-base-uncased** for monolingual English data [2306.05644].

| Weak supervision type | Construction | Role |
|---|---|---|
| Wiki words | Entity-linked spans from co-mentioned Wikipedia entities | Direct span alignment |
| Common words | Multilingual PLM embeddings with bidirectional nearest-neighbor agreement | Supplementary lexical alignment |

The data are explicitly described as **noisy**, **partially aligned**, and **non-parallel**. The noise arises because the annotations are automatic rather than human-verified; the alignment is partial because only entity spans and selected common words are labeled; and the paragraph pairs need not be translations of each other, only related through co-mentioned entities [2306.05644].

## 3. Span-prediction model and inference

WSPAlign uses the same general formulation as SpanAlign. Given a source sentence \(X = x_1,\dots,x_{|X|}\), a source span \((i,j)\), and a target sentence \(Y = y_1,\dots,y_{|Y|}\), the model predicts the aligned target span \((k,l)\) [2306.05644]. The architecture consists of a backbone encoder \(f_{\theta^b}\), a start-position predictor \(g_{\theta^s}\), and an end-position predictor \(g_{\theta^e}\). The initialization is language-pair dependent: **mBERT** is used for **Zh-En** and **Ja-En**, while **XLM-R** is used for **De-En**, **Ro-En**, and **En-Fr** [2306.05644].

The span score is defined as
\[
w^{X \rightarrow Y}_{ijkl} = p_{start}(k|X,Y,i,j) \times p_{end}(l|X,Y,i,j),
\]
and the predicted target span is
\[
(\hat{k}, \hat{l}) = \arg\max_{(k,l):1<k<l<|Y|} w^{X \rightarrow Y}_{ijkl}.
\]
Training minimizes the negative log-likelihood of the predicted span, and inference is symmetrized by running the model in both directions, averaging token-pair probabilities, and keeping pairs above the threshold **0.4**, following SpanAlign [2306.05644].

This formulation inherits the principal advantages of the earlier supervised span-prediction method: it handles null alignments naturally through a QA-style setup and can exploit source-side context. In the predecessor system, using the whole source sentence as question context produced the best results, showing that contextual disambiguation is central to span-level alignment [2004.14516]. WSPAlign retains that modeling view but changes the training regime from fully supervised to large-scale weakly supervised pre-training.

## 4. Training protocol, benchmarks, and reported performance

Pre-training is conducted for **100,000 steps** with **2,000 warmup steps**, **max sequence length 384**, **batch size 96**, **hidden size 768**, a **12-layer Transformer encoder**, and **learning rate 1e-6**. Fine-tuning uses **5 epochs** in the supervised setting and **250 epochs** in the few-shot setting, where the few-shot regime uses **32 labeled examples** [2306.05644].

Evaluation is performed on five gold alignment datasets: **Zh-En**, **Ja-En**, **De-En**, **Ro-En**, and **En-Fr**. The compared baselines include **Giza++**, **FastAlign**, **SimAlign**, **AWESoMe**, **DiscAlign**, and **SpanAlign**. Reported metrics are **Precision**, **Recall**, **F1**, and **Alignment Error Rate (AER)**; the paper treats AER as secondary because it can overweight precision [2306.05644].

The main result is that WSPAlign sets a new state of the art when fine-tuned on standard benchmarks, improving upon the best supervised baseline by **3.3–6.1 points in F1** and **1.5–6.1 points in AER** [2306.05644].

| Language pair | SpanAlign | WSPAlign |
|---|---|---|
| Zh-En | **86.7 F1 / 13.3 AER** | **91.5 F1 / 8.5 AER** |
| Ja-En | **77.6 F1 / 22.4 AER** | **83.7 F1 / 16.3 AER** |
| De-En | **85.6 F1 / 14.4 AER** | **88.9 F1 / 11.1 AER** |
| Ro-En | **86.7 F1 / 12.2 AER** | **91.4 F1 / 8.6 AER** |
| En-Fr | **AER 4.0** | **AER 2.5** |

These gains are notable because SpanAlign was already a strong supervised baseline. In the earlier supervised span-prediction paper, for example, multilingual BERT with gold alignments achieved **F1 = 86.7** on **Zh-En** and **F1 = 77.6** on **Ja-En** without bitext pretraining [2004.14516]. WSPAlign improves those benchmark points through pre-training rather than by replacing the span-prediction architecture itself.

## 5. Few-shot behavior, transfer, and limitations

WSPAlign is also evaluated in **zero-shot**, **few-shot**, and **cross-lingual** settings. Without any manual fine-tuning, it already beats unsupervised baselines on all test sets. On **Ro-En**, the zero-shot model nearly matches supervised performance; on **Ro-En** and **De-En**, it can even slightly exceed the fully supervised baseline. With only **32 labeled examples**, the few-shot model substantially outperforms unsupervised baselines and approaches supervised performance [2306.05644].

Ablations identify two important ingredients in the weak supervision. Removing either **common-word** or **wiki-word** supervision hurts performance by about **3 points F1/AER** on **Zh-En**, although both ablated variants still outperform SpanAlign. The choice of base multilingual PLM also matters: **mBERT** works better for **Zh-En** and **Ja-En**, while **XLM-R** works better for **De-En** and **Ro-En** [2306.05644]. The paper also reports **monolingual pre-training** on English-only weak supervision. Pure zero-shot transfer is weaker than multilingual pre-training, but few-shot fine-tuning with **32 examples** makes it competitive, and with full supervised fine-tuning it can outperform the supervised baseline on all test sets [2306.05644]. This suggests that span-prediction pre-training alone captures some transferable structure even without explicit bilingual equivalence during pre-training.

The limitations are explicit. WSPAlign still depends on **multilingual POS tagging**, **multilingual PLMs**, and **Wikipedia hyperlinks / Wikidata**. These resources may be unavailable for very small or exceptional minority languages, and the paper does **not** evaluate on true low-resource benchmark datasets because such test sets do not exist yet. It also concludes that monolingual pre-training is promising, but not yet sufficient to fully replace multilingual resources [2306.05644].

## 6. Related formulations and terminological scope

The explicit title form **“WSPAlign”** appears in **“WSPAlign: Word Alignment Pre-training via Large-Scale Weakly Supervised Span Prediction”** [2306.05644]. Closely related work includes **“A Supervised Word Alignment Method based on Cross-Language Span Prediction using Multilingual BERT”** [2004.14516], which established the cross-language span-prediction formulation with **multilingual BERT** and **SQuAD v2.0** style supervision; **“Third-Party Aligner for Neural Word Alignments”** [2211.04198], which uses the output of an existing third-party word aligner as supervision for fine-tuning a pre-trained cross-lingual language model; and several distinct **Wasserstein alignment** formulations in optimal transport and cross-lingual embedding alignment, including **“On the Wasserstein alignment problem”** [2503.06838], **“Quantized Wasserstein Procrustes Alignment of Word Embedding Spaces”** [2212.02468], and **“A Data-dependent Approach for High Dimensional (Robust) Wasserstein Alignment”** [2209.02905].

| Work | Core formulation | Supervision or setting |
|---|---|---|
| “WSPAlign: Word Alignment Pre-training via Large-Scale Weakly Supervised Span Prediction” [2306.05644] | Word alignment pre-training via span prediction | Large-scale weak supervision from Wikipedia |
| “A Supervised Word Alignment Method based on Cross-Language Span Prediction using Multilingual BERT” [2004.14516] | Cross-language span prediction as QA | Manually created gold word alignment data |
| “Third-Party Aligner for Neural Word Alignments” [2211.04198] | Bidirectional similarity-based neural word alignment | Alignments from a third-party aligner |
| Wasserstein alignment works [2503.06838], [2212.02468], [2209.02905] | Optimal-transport or geometric alignment | Probability measures, embedding spaces, or compressed point sets |

This comparison matters because the label can be read too broadly. The third-party-supervised neural alignment paper explicitly notes that it does **not** define the name *WSPAlign* [2211.04198]. The Wasserstein papers address a different notion of alignment altogether, centered on optimal transport rather than span prediction. A plausible implication is that, in arXiv-style usage, **WSPAlign** is most precisely understood as the weakly supervised span-prediction pre-training framework for word alignment introduced in 2023, with earlier supervised span-prediction work as its immediate methodological precursor.

Source: https://www.emergentmind.com/topics/wspalign