XL-DURel: Ordinal WiC Transformer

• XL-DURel is a fine-tuned multilingual Sentence Transformer that models graded semantic proximity by mapping contextual word usages onto a four-level DURel scale.
• It employs a siamese bi-encoder architecture with XLM-RoBERTa-large and explicit target marking to derive cosine similarity scores which are thresholded into ordinal labels.
• The model leverages mixed binary and ordinal datasets with ranking losses such as AnglE Loss, achieving state-of-the-art performance on both OGWiC and binary WiC benchmarks across multiple languages.

XL-DURel is a finetuned, multilingual Sentence Transformer model optimized for ordinal Word-in-Context classification. It models semantic proximity between two usages of the same target word on the four-level DURel scale—Unrelated, Distantly related, Closely related, and Identical—rather than reducing the task to a binary same-sense versus different-sense decision. The model is built on XLM-RoBERTa-large in an SBERT bi-encoder configuration, is trained with ranking objectives centered on AnglE Loss, and is designed to place binary and ordinal WiC tasks within a single similarity-based framework (Yadav et al., 19 Jul 2025). Its conceptual lineage derives from DURel (“Diachronic Usage Relatedness”), a sense-inventory-free framework that operationalizes lexical semantic change through intuitive relatedness judgments between contextualized uses of a word (Schlechtweg et al., 2018).

1. DURel as the conceptual substrate

DURel was introduced as a framework for annotating lexical semantic change without relying on predefined sense inventories. Its basic unit is a use of a target word $w$, defined as the sentence in which $w$ occurs together with one preceding and one following sentence. A use pair is a pair of such occurrences of the same word, and annotators rate how related the two uses are in meaning on a graded scale rather than assigning sense labels (Schlechtweg et al., 2018).

The DURel relatedness scale is a 5-option ordinal scale with four semantic levels and a meta-option: $4$ = Identical, $3$ = Closely Related, $2$ = Distantly Related, $1$ = Unrelated, and $0$ = Cannot decide. The scale is grounded in Blank’s continuum of semantic proximity, moving from Identity through Context Variance and Polysemy to Homonymy. This formulation makes the framework explicitly sense-inventory free: annotators are instructed not to enumerate senses, but to rely on intuitive judgments of semantic similarity between contextualized uses.

DURel extends earlier synchronic graded polysemy annotation to a diachronic setting by sampling three groups of use pairs for each target word: earlier pairs from $t_1$, later pairs from $t_2$, and compare pairs spanning $t_1$ and $w$0. In the original German instantiation, the corpus was DTA (Deutsches Textarchiv), the periods were 1750–1800 and 1850–1900, and the dataset comprised 22 target word instances and 1,320 use pairs, with ratings from five annotators. Agreement was measured with pairwise Spearman’s $w$1, yielding values in the range 0.57–0.68, all with $w$2, which established that graded relatedness judgments over historical data were robust enough to support evaluation benchmarks.

DURel also defined quantitative summaries over these annotations. For a target word $w$3, the framework computes $w$4, $w$5, and $w$6 for earlier, later, and cross-period pairs. It then derives

$w$7

and

$w$8

Within that framework, negative $w$9 values are associated with innovative change, while positive values are associated with reductive change. A plausible implication for XL-DURel is that its ordinal WiC formulation inherits the DURel view that semantic variation is best modeled as graded relatedness between contextualized uses rather than as categorical sense assignment.

2. From binary WiC to ordinal Word-in-Context

The classical Word-in-Context (WiC) task asks whether the same target word has the same sense in two different contexts, producing a binary label: $4$0 or TRUE for the same sense, and $4$1 or FALSE for a different sense. XL-DURel instead focuses on OGWiC (Ordinal Graded Word-in-Context), where the target is the degree of semantic proximity between the two usages on the DURel four-point scale (Yadav et al., 19 Jul 2025).

This reformulation changes both the linguistic target and the evaluation regime. In OGWiC, the labels are ordered, so a prediction that misses the gold label by two points is more severe than one that misses by one point. The task is therefore treated as ordinal classification, evaluated with Krippendorff’s $4$2 with ordinal distance. The paper contrasts this with GWiC, which uses 0–10 real-valued scores; OGWiC instead uses discrete labels with explicit linguistic interpretation and requires prediction of exact ordinal levels.

A central claim of XL-DURel is that binary WiC can be treated as a special case of ordinal WiC. The proposed mappings are explicit. For evaluation, ordinal labels 1 and 2 map to binary 0, while ordinal labels 3 and 4 map to binary 1. For unified training, binary 0 is mapped to ordinal 2, and binary 1 is mapped to ordinal 4. The rationale given is that many “different sense” WiC pairs are better modeled as polysemy than as pure homonymy, so they align more naturally with level 2 than with level 1. This yields a single four-level semantic space in which standard binary WiC becomes a coarse partition at the boundary between $4$3 and $4$4.

That unification is also methodological. XL-DURel argues that optimizing models for the general ordinal task improves performance on the more specific binary task. The model thus does not treat binary and ordinal WiC as unrelated benchmarks, but as different projections of the same underlying graded notion of usage relatedness. In that respect, the system can be read as transporting DURel’s annotation logic from diachronic lexical semantics into multilingual representation learning for contextual semantic comparison.

3. Architecture, representation, and prediction

XL-DURel is a multilingual Sentence Transformer (SBERT) model with a siamese bi-encoder architecture and XLM-RoBERTa-large as the sole encoder backbone. The base encoder has 561M parameters, and both input sentences are processed independently by the same shared encoder, after which a pooling operation produces a fixed-size embedding for each sentence (Yadav et al., 19 Jul 2025).

The input representation follows the WiC convention of pairing two usages of the same target lemma, but XL-DURel introduces explicit target marking. If $4$5 is the target span, the sentence is represented with special markers as

$4$6

Sentences are truncated to a maximum length of 128 tokens, and [CLS] and [SEP] are added. This adaptation allows a sentence-level SBERT architecture to encode contextual meaning with emphasis on the target span.

At inference time, the central score is cosine similarity between the two sentence embeddings: $4$7 XL-DURel does not use a dedicated ordinal classifier head. Instead, it predicts a continuous similarity score and then maps that score to ordinal labels 1–4 through three thresholds $4$8: $4$9 These thresholds are optimized on development data per language using Nelder–Mead simplex to maximize ordinal Krippendorff’s $3$0.

A frequent misconception is that XL-DURel is simply a multilingual four-way classifier. In fact, its operational core is a similarity space: the model learns embeddings whose pairwise geometry reflects semantic relatedness, and ordinal labels are induced afterward by thresholding. Another notable detail is that, although the model is trained with AnglE Loss, which replaces cosine similarity with angle difference in complex space inside a CoSENT-style ranking objective, the paper reports that cosine similarity at inference worked better than directly using the angle difference.

4. Training objectives and multilingual supervision

The training study compares four loss functions implemented in Sentence-Transformers: Contrastive Loss, Cosine Similarity Loss, CoSENT Loss, and AnglE Loss (Yadav et al., 19 Jul 2025). Contrastive Loss is binary and depends on a margin $3$1, set to 0.5. Cosine Similarity Loss treats the task as regression against a continuous target in $3$2. CoSENT is a pairwise ranking loss that enforces higher similarity for pairs with higher labels, with scaling hyperparameter $3$3. AnglE adopts the same ranking framework but replaces cosine similarity with angular distance in complex space.

For the regression and ranking objectives, labels are mapped into $3$4. The ordinal mapping is $3$5, $3$6, $3$7, $3$8. Binary WiC labels are mapped as binary 1 $3$9 and binary 0 $2$0, aligning them with ordinal levels 4 and 2, respectively. This is the mechanism that enables joint training over binary and ordinal data.

The multilingual supervision is assembled from two sources. The ordinal side is CoMeDi, an OGWiC dataset formed from a union of DURel-style resources across Chinese, English, German, Norwegian, Russian, Spanish, and Swedish, including ChiWUG, DWUG variants, DiscoWUG, RefWUG, DURel, SURel, NorDiaChange, RuSemShift, RuShiftEval, and RuDSI. Instances with <2 annotations, any “Cannot decide”, or heavy annotator disagreement (>1 point difference) are removed; the median label is computed, and only instances with integer medians are retained. The resulting split is 70/20/10 into train, test, and dev, with no target word overlap between splits. Final CoMeDi counts are 47,833 train, 8,287 dev, and 15,332 test.

The binary side merges XL-WiC, MCL-WiC, and AM2iCo into “WiC (train/dev)”, using the official training splits plus 75% of each dev split for training, leaving 25% dev for validation. These resources cover a substantially broader multilingual space. XL-WiC involves 12 languages; MCL-WiC covers 5 languages; and AM2iCo covers 15 languages with both monolingual and cross-lingual pairs. The unified configuration “WiC+CoMeDi” concatenates WiC and CoMeDi training material. No language-specific adapters are used: XLM-R-large alone is expected to learn a shared multilingual space for usage similarity.

Optimization settings are fully specified. The optimizer is AdamW with learning rate $2$1, batch size 32, weight decay 0.0, and 10 epochs of training. Warm-up is linear over 10% of training steps, followed by linear decay. Evaluation occurs every 25% of an epoch; Average Precision is used for Contrastive Loss, while Spearman correlation is used for the other objectives. Early stopping uses patience 10 evaluations after warm-up and a minimum improvement threshold of 0.02. Each run uses a single NVIDIA RTX A6000 (48GB) and requires approximately 40–50 GPU hours per model.

5. Empirical performance

The top-performing configuration is AnglE-WiC+CoMeDi, identified as XL-DURel. On the OGWiC (CoMeDi test) benchmark, it achieves $2$2 on CoMeDi-Avg, which is the best reported overall result in the paper (Yadav et al., 19 Jul 2025). The next-best systems are CoSENT-WiC+CoMeDi at $2$3 and Cosine Loss (COS-WiC+CoMeDi) at $2$4.

The baselines establish the significance of that gain. Published XL-LEXEME achieves $2$5 on the same benchmark. A re-trained XL-LEXEME with CL-WiC yields $2$6, and CL-WiC+CoMeDi yields $2$7. The shared-task winner DeepMistake reaches $2$8. Accordingly, XL-DURel clearly outperforms XL-LEXEME and slightly outperforms DeepMistake. The paper further states that AnglE-WiC+CoMeDi outperforms baselines on every single language and does so by a large margin on average.

On binary WiC evaluation, the differences are narrower but still informative. CoSENT-WiC+CoMeDi is best with $2$9 on WiC-Avg, while AnglE-WiC+CoMeDi and XL-LEXEME are both around 0.67. Across finetuned systems, performance falls between 0.62 and 0.68 on WiC-Avg. The broader conclusion is that optimizing for the general ordinal problem does not degrade binary WiC performance and can slightly improve it, particularly when ordinal data is added.

The comparative loss analysis is consistent across experiments. Ranking losses—specifically CoSENT and AnglE—outperform both Cosine Similarity Loss and binary-only Contrastive Loss on OGWiC. AnglE also improves over CoSENT under the same data configuration, with 0.67 versus 0.64 on CoMeDi-Avg. The interpretation advanced in the paper is that AnglE better exploits the ordinal signal, maintains stronger gradients near high similarity, and more effectively distinguishes fine-grained relatedness levels.

6. Interpretation, limitations, and practical significance

XL-DURel’s broader significance lies in its proposal of a unified treatment of WiC modeling across different task formulations. Traditional WiC benchmarks are usually approached as binary classification, while graded or ordinal tasks have often been treated separately. XL-DURel instead trains a single multilingual embedding model on mixed binary and ordinal data, using an ordinal-aware ranking objective and a common similarity space (Yadav et al., 19 Jul 2025).

This has immediate practical consequences. The released checkpoint is sachinn1/xl-durel on Hugging Face. In use, the target word span is marked with <t> and </t>, both sentences are encoded with the Sentence Transformer, cosine similarity is computed, and the score is then mapped either to ordinal labels 1–4 through language-specific thresholds or to binary labels via the $1$0 versus $1$1 partition. The paper also notes that the model can serve as a general-purpose contextualized embedder for word meaning in tasks such as lexical semantic change, sense clustering, and semantic similarity in context. This suggests continuity with DURel’s original role as an evaluation resource for lexical semantic change models (Schlechtweg et al., 2018).

The paper also identifies several limitations. Experiments are restricted to XLM-R-large, the SBERT architecture, and the specific OGWiC and WiC datasets used in training and evaluation. Future work is explicitly directed toward other encoders, including mBERT, multilingual T5, and LLaMA-based encoders, as well as broader testing of whether AnglE’s empirical advantage generalizes. Another unresolved question is why a model trained with AnglE Loss performs best when evaluated with cosine similarity rather than with the angle-based similarity used during training. The authors additionally suggest exploring direct ordinal classification objectives, including cumulative link models, to exploit ordinal structure more directly.

Within the history of lexical semantic modeling, XL-DURel can therefore be situated at the intersection of three lines of work: DURel’s sense-inventory-free relatedness annotation, multilingual Word-in-Context representation learning, and ordinal/ranking-aware optimization. The model’s central claim is not merely that multilingual WiC can be solved with a larger encoder, but that WiC tasks are better treated as instances of a single graded semantic relatedness problem whose structure is already implicit in the DURel scale.