---
title: 'XL-DURel: Ordinal WiC Transformer'
url: https://www.emergentmind.com/topics/xl-durel
type: topic
---

# XL-DURel: Ordinal WiC Transformer

XL-DURel is a finetuned, multilingual Sentence Transformer model optimized for ordinal Word-in-Context classification. It models semantic proximity between two usages of the same target word on the four-level DURel scale—**Unrelated**, **Distantly related**, **Closely related**, and **Identical**—rather than reducing the task to a binary same-sense versus different-sense decision. The model is built on **XLM-RoBERTa-large** in an **SBERT** bi-encoder configuration, is trained with ranking objectives centered on **AnglE Loss**, and is designed to place binary and ordinal WiC tasks within a single similarity-based framework [2507.14578]. Its conceptual lineage derives from **DURel** (“Diachronic Usage Relatedness”), a sense-inventory-free framework that operationalizes lexical semantic change through intuitive relatedness judgments between contextualized uses of a word [1804.06517].

## 1. DURel as the conceptual substrate

DURel was introduced as a framework for annotating **lexical semantic change** without relying on predefined sense inventories. Its basic unit is a **use** of a target word \(w\), defined as the sentence in which \(w\) occurs together with one preceding and one following sentence. A **use pair** is a pair of such occurrences of the same word, and annotators rate how related the two uses are in meaning on a graded scale rather than assigning sense labels [1804.06517].

The DURel relatedness scale is a **5-option ordinal scale** with four semantic levels and a meta-option: \(4\) = **Identical**, \(3\) = **Closely Related**, \(2\) = **Distantly Related**, \(1\) = **Unrelated**, and \(0\) = **Cannot decide**. The scale is grounded in Blank’s continuum of semantic proximity, moving from **Identity** through **Context Variance** and **Polysemy** to **Homonymy**. This formulation makes the framework explicitly **sense-inventory free**: annotators are instructed not to enumerate senses, but to rely on intuitive judgments of semantic similarity between contextualized uses.

DURel extends earlier synchronic graded polysemy annotation to a diachronic setting by sampling three groups of use pairs for each target word: **earlier** pairs from \(t_1\), **later** pairs from \(t_2\), and **compare** pairs spanning \(t_1\) and \(t_2\). In the original German instantiation, the corpus was **DTA (Deutsches Textarchiv)**, the periods were **1750–1800** and **1850–1900**, and the dataset comprised **22 target word instances** and **1,320 use pairs**, with ratings from **five annotators**. Agreement was measured with pairwise **Spearman’s \(\rho\)**, yielding values in the range **0.57–0.68**, all with **\(p < 0.01\)**, which established that graded relatedness judgments over historical data were robust enough to support evaluation benchmarks.

DURel also defined quantitative summaries over these annotations. For a target word \(w\), the framework computes \(Mean_e(w)\), \(Mean_l(w)\), and \(Mean_c(w)\) for earlier, later, and cross-period pairs. It then derives
\[
\Delta\text{later}(w)=Mean_l(w)-Mean_e(w)
\]
and
\[
compare(w)=Mean_c(w).
\]
Within that framework, negative \(\Delta\text{later}\) values are associated with **innovative change**, while positive values are associated with **reductive change**. A plausible implication for XL-DURel is that its ordinal WiC formulation inherits the DURel view that semantic variation is best modeled as graded relatedness between contextualized uses rather than as categorical sense assignment.

## 2. From binary WiC to ordinal Word-in-Context

The classical **Word-in-Context (WiC)** task asks whether the same target word has the same sense in two different contexts, producing a **binary label**: \(1\) or **TRUE** for the same sense, and \(0\) or **FALSE** for a different sense. XL-DURel instead focuses on **OGWiC (Ordinal Graded Word-in-Context)**, where the target is the **degree of semantic proximity** between the two usages on the DURel four-point scale [2507.14578].

This reformulation changes both the linguistic target and the evaluation regime. In OGWiC, the labels are ordered, so a prediction that misses the gold label by two points is more severe than one that misses by one point. The task is therefore treated as **ordinal classification**, evaluated with **Krippendorff’s \(\alpha\) with ordinal distance**. The paper contrasts this with **GWiC**, which uses **0–10 real-valued scores**; OGWiC instead uses discrete labels with explicit linguistic interpretation and requires prediction of exact ordinal levels.

A central claim of XL-DURel is that **binary WiC can be treated as a special case of ordinal WiC**. The proposed mappings are explicit. For evaluation, ordinal labels **1** and **2** map to binary **0**, while ordinal labels **3** and **4** map to binary **1**. For unified training, binary **0** is mapped to ordinal **2**, and binary **1** is mapped to ordinal **4**. The rationale given is that many “different sense” WiC pairs are better modeled as **polysemy** than as pure **homonymy**, so they align more naturally with level **2** than with level **1**. This yields a single four-level semantic space in which standard binary WiC becomes a coarse partition at the boundary between \(\{1,2\}\) and \(\{3,4\}\).

That unification is also methodological. XL-DURel argues that optimizing models for the **general ordinal task** improves performance on the more specific binary task. The model thus does not treat binary and ordinal WiC as unrelated benchmarks, but as different projections of the same underlying graded notion of usage relatedness. In that respect, the system can be read as transporting DURel’s annotation logic from diachronic lexical semantics into multilingual representation learning for contextual semantic comparison.

## 3. Architecture, representation, and prediction

XL-DURel is a **multilingual Sentence Transformer (SBERT) model** with a **siamese** bi-encoder architecture and **XLM-RoBERTa-large** as the sole encoder backbone. The base encoder has **561M parameters**, and both input sentences are processed independently by the same shared encoder, after which a pooling operation produces a fixed-size embedding for each sentence [2507.14578].

The input representation follows the WiC convention of pairing two usages of the same target lemma, but XL-DURel introduces **explicit target marking**. If \(w_{t_i}, \ldots, w_{t_i+k}\) is the target span, the sentence is represented with special markers as
\[
u = w_1, \ldots, \langle t \rangle, w_{t_i}, \ldots, w_{t_i+k}, \langle /t \rangle, \ldots, w_N.
\]
Sentences are truncated to a maximum length of **128 tokens**, and **[CLS]** and **[SEP]** are added. This adaptation allows a sentence-level SBERT architecture to encode contextual meaning with emphasis on the target span.

At inference time, the central score is **cosine similarity** between the two sentence embeddings:
\[
\text{sim}(u,v)=\cos(u,v)\in[-1,1].
\]
XL-DURel does not use a dedicated ordinal classifier head. Instead, it predicts a continuous similarity score and then maps that score to ordinal labels **1–4** through three thresholds \(\theta_1,\theta_2,\theta_3\):
\[
\hat{y}_\theta(s)=
\begin{cases}
1 & s < \theta_1 \\
2 & \theta_1 \le s < \theta_2 \\
3 & \theta_2 \le s < \theta_3 \\
4 & s \ge \theta_3.
\end{cases}
\]
These thresholds are optimized on development data **per language** using **Nelder–Mead simplex** to maximize **ordinal Krippendorff’s \(\alpha\)**.

A frequent misconception is that XL-DURel is simply a multilingual four-way classifier. In fact, its operational core is a **similarity space**: the model learns embeddings whose pairwise geometry reflects semantic relatedness, and ordinal labels are induced afterward by thresholding. Another notable detail is that, although the model is trained with **AnglE Loss**, which replaces cosine similarity with **angle difference in complex space** inside a CoSENT-style ranking objective, the paper reports that **cosine similarity at inference** worked better than directly using the angle difference.

## 4. Training objectives and multilingual supervision

The training study compares four loss functions implemented in **Sentence-Transformers**: **Contrastive Loss**, **Cosine Similarity Loss**, **CoSENT Loss**, and **AnglE Loss** [2507.14578]. Contrastive Loss is binary and depends on a margin \(m\), set to **0.5**. Cosine Similarity Loss treats the task as regression against a continuous target in \([0,1]\). CoSENT is a pairwise ranking loss that enforces higher similarity for pairs with higher labels, with scaling hyperparameter \(\lambda = 20\). AnglE adopts the same ranking framework but replaces cosine similarity with **angular distance in complex space**.

For the regression and ranking objectives, labels are mapped into \([0,1]\). The ordinal mapping is **\(1 \rightarrow 0.0\), \(2 \rightarrow \frac{1}{3}\), \(3 \rightarrow \frac{2}{3}\), \(4 \rightarrow 1.0\)**. Binary WiC labels are mapped as **binary 1 \(\rightarrow 1.0\)** and **binary 0 \(\rightarrow \frac{1}{3}\)**, aligning them with ordinal levels **4** and **2**, respectively. This is the mechanism that enables **joint training** over binary and ordinal data.

The multilingual supervision is assembled from two sources. The ordinal side is **CoMeDi**, an OGWiC dataset formed from a union of DURel-style resources across **Chinese, English, German, Norwegian, Russian, Spanish, and Swedish**, including **ChiWUG, DWUG variants, DiscoWUG, RefWUG, DURel, SURel, NorDiaChange, RuSemShift, RuShiftEval, and RuDSI**. Instances with **<2 annotations**, any **“Cannot decide”**, or **heavy annotator disagreement (>1 point difference)** are removed; the **median label** is computed, and only instances with **integer medians** are retained. The resulting split is **70/20/10** into train, test, and dev, with **no target word overlap** between splits. Final CoMeDi counts are **47,833** train, **8,287** dev, and **15,332** test.

The binary side merges **XL-WiC**, **MCL-WiC**, and **AM2iCo** into **“WiC (train/dev)”**, using the official training splits plus **75% of each dev split** for training, leaving **25% dev** for validation. These resources cover a substantially broader multilingual space. XL-WiC involves **12 languages**; MCL-WiC covers **5 languages**; and AM2iCo covers **15 languages** with both monolingual and cross-lingual pairs. The unified configuration **“WiC+CoMeDi”** concatenates WiC and CoMeDi training material. No language-specific adapters are used: **XLM-R-large** alone is expected to learn a shared multilingual space for usage similarity.

Optimization settings are fully specified. The optimizer is **AdamW** with learning rate **\(1 \times 10^{-5}\)**, batch size **32**, weight decay **0.0**, and **10 epochs** of training. Warm-up is linear over **10%** of training steps, followed by linear decay. Evaluation occurs every **25% of an epoch**; **Average Precision** is used for Contrastive Loss, while **Spearman correlation** is used for the other objectives. Early stopping uses **patience 10 evaluations after warm-up** and a **minimum improvement threshold of 0.02**. Each run uses a **single NVIDIA RTX A6000 (48GB)** and requires approximately **40–50 GPU hours per model**.

## 5. Empirical performance

The top-performing configuration is **AnglE-WiC+CoMeDi**, identified as **XL-DURel**. On the **OGWiC (CoMeDi test)** benchmark, it achieves **\(\alpha = 0.67\)** on **CoMeDi-Avg**, which is the best reported overall result in the paper [2507.14578]. The next-best systems are **CoSENT-WiC+CoMeDi** at **\(\alpha = 0.64\)** and **Cosine Loss (COS-WiC+CoMeDi)** at **\(\alpha = 0.62\)**.

The baselines establish the significance of that gain. Published **XL-LEXEME** achieves **\(\alpha = 0.58\)** on the same benchmark. A re-trained **XL-LEXEME** with **CL-WiC** yields **\(\alpha = 0.56\)**, and **CL-WiC+CoMeDi** yields **\(\alpha = 0.57\)**. The shared-task winner **DeepMistake** reaches **\(\alpha = 0.66\)**. Accordingly, XL-DURel **clearly outperforms XL-LEXEME** and **slightly outperforms DeepMistake**. The paper further states that **AnglE-WiC+CoMeDi outperforms baselines on every single language** and does so by a large margin on average.

On **binary WiC** evaluation, the differences are narrower but still informative. **CoSENT-WiC+CoMeDi** is best with **\(\alpha \approx 0.68\)** on **WiC-Avg**, while **AnglE-WiC+CoMeDi** and **XL-LEXEME** are both around **0.67**. Across finetuned systems, performance falls between **0.62 and 0.68** on WiC-Avg. The broader conclusion is that optimizing for the **general ordinal problem** does not degrade binary WiC performance and can slightly improve it, particularly when ordinal data is added.

The comparative loss analysis is consistent across experiments. **Ranking losses**—specifically **CoSENT** and **AnglE**—outperform both **Cosine Similarity Loss** and binary-only **Contrastive Loss** on OGWiC. AnglE also improves over CoSENT under the same data configuration, with **0.67** versus **0.64** on CoMeDi-Avg. The interpretation advanced in the paper is that AnglE better exploits the ordinal signal, maintains stronger gradients near high similarity, and more effectively distinguishes fine-grained relatedness levels.

## 6. Interpretation, limitations, and practical significance

XL-DURel’s broader significance lies in its proposal of a **unified treatment of WiC modeling across different task formulations**. Traditional WiC benchmarks are usually approached as binary classification, while graded or ordinal tasks have often been treated separately. XL-DURel instead trains a single multilingual embedding model on mixed binary and ordinal data, using an ordinal-aware ranking objective and a common similarity space [2507.14578].

This has immediate practical consequences. The released checkpoint is **`sachinn1/xl-durel`** on Hugging Face. In use, the target word span is marked with **`<t>`** and **`</t>`**, both sentences are encoded with the Sentence Transformer, cosine similarity is computed, and the score is then mapped either to **ordinal labels 1–4** through language-specific thresholds or to **binary labels** via the \(\{1,2\}\) versus \(\{3,4\}\) partition. The paper also notes that the model can serve as a **general-purpose contextualized embedder for word meaning** in tasks such as **lexical semantic change**, **sense clustering**, and **semantic similarity in context**. This suggests continuity with DURel’s original role as an evaluation resource for lexical semantic change models [1804.06517].

The paper also identifies several limitations. Experiments are restricted to **XLM-R-large**, the **SBERT** architecture, and the specific **OGWiC** and **WiC** datasets used in training and evaluation. Future work is explicitly directed toward other encoders, including **mBERT**, **multilingual T5**, and **LLaMA-based encoders**, as well as broader testing of whether AnglE’s empirical advantage generalizes. Another unresolved question is why a model trained with **AnglE Loss** performs best when evaluated with **cosine similarity** rather than with the angle-based similarity used during training. The authors additionally suggest exploring **direct ordinal classification objectives**, including **cumulative link models**, to exploit ordinal structure more directly.

Within the history of lexical semantic modeling, XL-DURel can therefore be situated at the intersection of three lines of work: DURel’s **sense-inventory-free relatedness annotation**, multilingual **Word-in-Context** representation learning, and **ordinal/ranking-aware** optimization. The model’s central claim is not merely that multilingual WiC can be solved with a larger encoder, but that WiC tasks are better treated as instances of a single graded semantic relatedness problem whose structure is already implicit in the DURel scale.

Source: https://www.emergentmind.com/topics/xl-durel