WSPAlign: Weakly Supervised Word Alignment

• WSPAlign is a novel framework that applies weak supervision through cross-language span prediction to pre-train word alignment models.
• It leverages Wikipedia co-mentions and multilingual PLMs to automatically generate entity-linked and common-word alignments from noisy, non-parallel data.
• The approach achieves state-of-the-art F1 improvements and reduced AER across multiple language pairs, demonstrating efficacy in zero-shot and few-shot settings.

WSPAlign is a word-alignment framework based on cross-language span prediction and, in its explicit titled form, stands for Weakly Supervised span Prediction pre-training for word AlignmenT. It was introduced to reduce dependence on manual alignment annotation and parallel corpora by pre-training on noisy, partially aligned, and non-parallel paragraphs mined at large scale, then fine-tuning on standard gold benchmarks (Wu et al., 2023). The method preserves the span-prediction view associated with earlier multilingual BERT alignment work, in which a source token or span is treated as a query and the aligned target span is predicted, but it relocates the main source of supervision from manually created gold alignments to large-scale weak supervision (Nagata et al., 2020).

1. Problem setting and conceptual basis

Word alignment is the task of identifying which words or token spans in one language correspond to which in another. It remains important for annotation projection, terminology enforcement, under-translation detection, and cross-lingual representation learning (Wu et al., 2023). Earlier work had already shown that alignment can be cast as cross-language span prediction: a source token functions as the “question,” the target sentence is the “context,” and the aligned target span is the “answer,” in the style of SQuAD v2.0 question answering (Nagata et al., 2020).

WSPAlign is motivated by a specific limitation in prior aligners. Statistical systems such as Giza++ and FastAlign depend on parallel corpora; unsupervised embedding-based methods such as SimAlign avoid parallel training data but are often weaker on benchmark alignment quality; and supervised systems such as AWESoMe and SpanAlign are strong but rely on manual alignment labels or parallel data (Wu et al., 2023). WSPAlign keeps the strong span-prediction formulation but broadens the supervision source by relaxing the requirement for correct, fully aligned, and parallel training instances. The result is a pre-training scheme rather than merely a new architecture. This suggests that its main contribution is a data-scaling strategy for word alignment.

2. Weak supervision from Wikipedia

The weakly supervised corpus is constructed in two stages from Wikipedia. First, paragraphs are paired through co-mentioned entities using Wikipedia hyperlinks and Wikidata entity IDs. The construction is language-agnostic at the entity level: every hyperlink points to an entity ID, an inverted index is built from entity ID to all paragraphs mentioning that entity, and paragraph pairs are formed whenever two paragraphs mention the same entity (Wu et al., 2023).

Second, alignments are automatically annotated in two forms. Wiki words are entity-linked spans: if two paragraphs mention the same entity, the corresponding spans are aligned directly. Common words are obtained with a multilingual PLM and bidirectional nearest-neighbor agreement in the style of SimAlign: a word pair is kept only when the source word is the most similar target word and the target word is the most similar source word, with an additional filter retaining alignments where at least one aligned token is a common word (Wu et al., 2023).

The scale is central. Raw Wikipedia paragraph pairs from co-mentions reach tens of millions; the pre-training set samples 2,000,000 paragraph pairs; wiki items are annotated for all 2,000,000 pairs; and common-word alignments are annotated for 200,000 pairs. Filtering removes very short and very long pairs and retains only pairs whose sentence-embedding cosine similarity exceeds 0.75, using LaBSE for multilingual data and pcl-bert-base-uncased for monolingual English data (Wu et al., 2023).

Weak supervision type	Construction	Role
Wiki words	Entity-linked spans from co-mentioned Wikipedia entities	Direct span alignment
Common words	Multilingual PLM embeddings with bidirectional nearest-neighbor agreement	Supplementary lexical alignment

The data are explicitly described as noisy, partially aligned, and non-parallel. The noise arises because the annotations are automatic rather than human-verified; the alignment is partial because only entity spans and selected common words are labeled; and the paragraph pairs need not be translations of each other, only related through co-mentioned entities (Wu et al., 2023).

3. Span-prediction model and inference

WSPAlign uses the same general formulation as SpanAlign. Given a source sentence $X = x_1,\dots,x_{|X|}$, a source span $(i,j)$, and a target sentence $Y = y_1,\dots,y_{|Y|}$, the model predicts the aligned target span $(k,l)$ (Wu et al., 2023). The architecture consists of a backbone encoder $f_{\theta^b}$, a start-position predictor $g_{\theta^s}$, and an end-position predictor $g_{\theta^e}$. The initialization is language-pair dependent: mBERT is used for Zh-En and Ja-En, while XLM-R is used for De-En, Ro-En, and En-Fr (Wu et al., 2023).

The span score is defined as

$$w^{X \rightarrow Y}_{ijkl} = p_{start}(k|X,Y,i,j) \times p_{end}(l|X,Y,i,j),$$

and the predicted target span is

$$(\hat{k}, \hat{l}) = \arg\max_{(k,l):1<k<l<|Y|} w^{X \rightarrow Y}_{ijkl}.$$

Training minimizes the negative log-likelihood of the predicted span, and inference is symmetrized by running the model in both directions, averaging token-pair probabilities, and keeping pairs above the threshold 0.4, following SpanAlign (Wu et al., 2023).

This formulation inherits the principal advantages of the earlier supervised span-prediction method: it handles null alignments naturally through a QA-style setup and can exploit source-side context. In the predecessor system, using the whole source sentence as question context produced the best results, showing that contextual disambiguation is central to span-level alignment (Nagata et al., 2020). WSPAlign retains that modeling view but changes the training regime from fully supervised to large-scale weakly supervised pre-training.

4. Training protocol, benchmarks, and reported performance

Pre-training is conducted for 100,000 steps with 2,000 warmup steps, max sequence length 384, batch size 96, hidden size 768, a 12-layer Transformer encoder, and learning rate 1e-6. Fine-tuning uses 5 epochs in the supervised setting and 250 epochs in the few-shot setting, where the few-shot regime uses 32 labeled examples (Wu et al., 2023).

Evaluation is performed on five gold alignment datasets: Zh-En, Ja-En, De-En, Ro-En, and En-Fr. The compared baselines include Giza++, FastAlign, SimAlign, AWESoMe, DiscAlign, and SpanAlign. Reported metrics are Precision, Recall, F1, and Alignment Error Rate (AER); the paper treats AER as secondary because it can overweight precision (Wu et al., 2023).

The main result is that WSPAlign sets a new state of the art when fine-tuned on standard benchmarks, improving upon the best supervised baseline by 3.3–6.1 points in F1 and 1.5–6.1 points in AER (Wu et al., 2023).

Language pair	SpanAlign	WSPAlign
Zh-En	86.7 F1 / 13.3 AER	91.5 F1 / 8.5 AER
Ja-En	77.6 F1 / 22.4 AER	83.7 F1 / 16.3 AER
De-En	85.6 F1 / 14.4 AER	88.9 F1 / 11.1 AER
Ro-En	86.7 F1 / 12.2 AER	91.4 F1 / 8.6 AER
En-Fr	AER 4.0	AER 2.5

These gains are notable because SpanAlign was already a strong supervised baseline. In the earlier supervised span-prediction paper, for example, multilingual BERT with gold alignments achieved F1 = 86.7 on Zh-En and F1 = 77.6 on Ja-En without bitext pretraining (Nagata et al., 2020). WSPAlign improves those benchmark points through pre-training rather than by replacing the span-prediction architecture itself.

5. Few-shot behavior, transfer, and limitations

WSPAlign is also evaluated in zero-shot, few-shot, and cross-lingual settings. Without any manual fine-tuning, it already beats unsupervised baselines on all test sets. On Ro-En, the zero-shot model nearly matches supervised performance; on Ro-En and De-En, it can even slightly exceed the fully supervised baseline. With only 32 labeled examples, the few-shot model substantially outperforms unsupervised baselines and approaches supervised performance (Wu et al., 2023).

Ablations identify two important ingredients in the weak supervision. Removing either common-word or wiki-word supervision hurts performance by about 3 points F1/AER on Zh-En, although both ablated variants still outperform SpanAlign. The choice of base multilingual PLM also matters: mBERT works better for Zh-En and Ja-En, while XLM-R works better for De-En and Ro-En (Wu et al., 2023). The paper also reports monolingual pre-training on English-only weak supervision. Pure zero-shot transfer is weaker than multilingual pre-training, but few-shot fine-tuning with 32 examples makes it competitive, and with full supervised fine-tuning it can outperform the supervised baseline on all test sets (Wu et al., 2023). This suggests that span-prediction pre-training alone captures some transferable structure even without explicit bilingual equivalence during pre-training.

The limitations are explicit. WSPAlign still depends on multilingual POS tagging, multilingual PLMs, and Wikipedia hyperlinks / Wikidata. These resources may be unavailable for very small or exceptional minority languages, and the paper does not evaluate on true low-resource benchmark datasets because such test sets do not exist yet. It also concludes that monolingual pre-training is promising, but not yet sufficient to fully replace multilingual resources (Wu et al., 2023).

6. Related formulations and terminological scope

The explicit title form “WSPAlign” appears in “WSPAlign: Word Alignment Pre-training via Large-Scale Weakly Supervised Span Prediction” (Wu et al., 2023). Closely related work includes “A Supervised Word Alignment Method based on Cross-Language Span Prediction using Multilingual BERT” (Nagata et al., 2020), which established the cross-language span-prediction formulation with multilingual BERT and SQuAD v2.0 style supervision; “Third-Party Aligner for Neural Word Alignments” (Zhang et al., 2022), which uses the output of an existing third-party word aligner as supervision for fine-tuning a pre-trained cross-lingual LLM; and several distinct Wasserstein alignment formulations in optimal transport and cross-lingual embedding alignment, including “On the Wasserstein alignment problem” (Pal et al., 10 Mar 2025), “Quantized Wasserstein Procrustes Alignment of Word Embedding Spaces” (Aboagye et al., 2022), and “A Data-dependent Approach for High Dimensional (Robust) Wasserstein Alignment” (Ding et al., 2022).

Work	Core formulation	Supervision or setting
“WSPAlign: Word Alignment Pre-training via Large-Scale Weakly Supervised Span Prediction” (Wu et al., 2023)	Word alignment pre-training via span prediction	Large-scale weak supervision from Wikipedia
“A Supervised Word Alignment Method based on Cross-Language Span Prediction using Multilingual BERT” (Nagata et al., 2020)	Cross-language span prediction as QA	Manually created gold word alignment data
“Third-Party Aligner for Neural Word Alignments” (Zhang et al., 2022)	Bidirectional similarity-based neural word alignment	Alignments from a third-party aligner
Wasserstein alignment works (Pal et al., 10 Mar 2025, Aboagye et al., 2022, Ding et al., 2022)	Optimal-transport or geometric alignment	Probability measures, embedding spaces, or compressed point sets

This comparison matters because the label can be read too broadly. The third-party-supervised neural alignment paper explicitly notes that it does not define the name WSPAlign (Zhang et al., 2022). The Wasserstein papers address a different notion of alignment altogether, centered on optimal transport rather than span prediction. A plausible implication is that, in arXiv-style usage, WSPAlign is most precisely understood as the weakly supervised span-prediction pre-training framework for word alignment introduced in 2023, with earlier supervised span-prediction work as its immediate methodological precursor.