---
title: Word Sense Disambiguation Benchmarks
url: https://www.emergentmind.com/topics/word-sense-disambiguation-benchmarks
type: topic
---

# Word Sense Disambiguation Benchmarks

Word sense disambiguation (WSD) benchmarks are standardized evaluation datasets and protocols that provide quantitative, replicable platforms to measure and compare WSD systems. These benchmarks play a pivotal role in driving methodological progress by offering gold-annotated corpora, clearly defined task formulations, and unified evaluation metrics. Benchmark corpora are available for multiple languages, domains, and sense inventories, and address both supervised and unsupervised paradigms, all-words and lexical-sample setups, rare-sense (few/zero-shot) evaluation, and specialized tasks such as target sense verification.

## 1. Historical Development and Major Benchmark Families

**All-Words and Lexical Sample Benchmarks in English:**  
The principal tradition in English WSD evaluation is rooted in the Senseval and SemEval shared tasks. The "all-words" setting requires disambiguation of every eligible word in running text. The unified evaluation framework by Raganato et al. (2017) comprises five key test sets—Senseval-2, Senseval-3, SemEval-2007, SemEval-2013, and SemEval-2015—together covering 7,253 test instances and over 4,000 sense types, with SemCor as the standard supervised training corpus, annotated with WordNet 3.0 senses [1910.00194]. Lexical sample tasks restrict the evaluation to a predefined subset of ambiguous words, allowing controlled fine-grained discrimination, as in Senseval-2/3 and corresponding Swedish and Persian datasets [1802.09059, 2410.22827, 2107.01540].

**Multilingual and Non-English Benchmarks:**  
The expansion beyond English includes the SBU-WSD-Corpus for Persian, annotated with FarsNet senses [2107.01540], the SALMA corpus for Arabic with dual sense inventories and graded annotations [2310.19029], and recent Swedish benchmarks mapped to the SALDO inventory [2410.22827]. Cross-lingual benchmarks such as XL-WSD, constructed on BabelNet, provide multi-language support and gloss translation mechanisms [2503.08662].

**Few-Shot and Low-Shot Benchmarks:**  
To mitigate the bias toward high-frequency senses in standard training corpora, recent efforts include FEWS—a Wiktionary-extracted benchmark with precise few- and zero-shot evaluation splits (5,000 examples each per dev/test), far exceeding the rare-sense coverage of classical datasets [2102.07983]. MetricWSD’s experimental protocol targets rare-sense disambiguation using frequency-stratified metrics and episodic few-shot learning [2104.12677].

**Target Sense Verification and Specialized Setups:**  
Benchmarks such as WiC-TSV recast WSD as inventory-agnostic binary classification ("does sense s match target use in c?"), supporting both in-domain and domain-transfer evaluation, and enabling flexible adaptation beyond fixed inventories [2004.15016]. Generative and selection tasks for LLMs, such as those in the extended XL-WSD framework, test models’ ability to produce or select correct definitions given context [2503.08662].

## 2. Dataset Construction, Annotation Protocols, and Sense Inventories

Benchmark construction emphasizes both linguistic diversity and replicability. SemCor and its derivatives employ hand-annotated Brown corpus documents with WordNet sense tagging [1910.00194, 2402.13302]. The SBU-WSD-Corpus (Persian) samples news articles with maximal average ambiguity to ensure challenge, using the SAMP interface for FarsNet annotations; annotation quality is measured via Cohen’s κ = 0.83, with detailed per-POS and ambiguity statistics [2107.01540]. SALMA’s Arabic corpus employs a graded scoring scheme for multi-label sense relevance per token, supporting both classification and ranking paradigms, evaluated with linear/quadratic weighted kappa, MAE, and RMSE [2310.19029].

In resource-rich languages, domain, genre, and polysemy coverage are prioritized—e.g., Swedish Eukalyptus covers eight domains and employs coarse SALDO senses [2410.22827]. FEWS mines Wiktionary for maximal sense coverage with strict filtering for polysemy, achieving 71,391 sense types and 131,278 annotated instances across >300 domains [2102.07983].

## 3. Evaluation Metrics and Protocols

**F₁ score** dominates all-words and lexical sample benchmarks, defined as the micro-averaged harmonic mean of precision and recall over all test instances:
\[
F_1 = 2 \cdot \frac{\mathrm{Precision} \cdot \mathrm{Recall}}{\mathrm{Precision}+\mathrm{Recall}}
\]
where
\[
\mathrm{Precision} = \frac{TP}{TP + FP}, \quad \mathrm{Recall} = \frac{TP}{TP + FN}
\]
Metrics such as accuracy become equivalent to F₁ in forced-choice (single-label) settings [1910.00194, 2104.12677, 2102.07983]. Lexical-sample and few-shot benchmarks explicitly stratify performance by sense frequency (few-/zero-shot splits), reporting granular accuracy for each subset [2102.07983, 2104.12677]. Non-binary settings (e.g., multi-label in SALMA) adopt graded agreement metrics—linear/quadratic weighted kappa, MAE, RMSE [2310.19029].

Specialized settings use tailored metrics:  
- **Generative tasks** (produce definitions): evaluated with ROUGE-L and BERTScore [2503.08662].  
- **Sense selection tasks** (multiple-choice): primary accuracy, with optional precision, recall, F₁ [2503.08662].  
- **Target sense verification** (binary): accuracy and F₁ on positive/negative matches [2004.15016].

## 4. Major Benchmark Results and Model Advancements

**State-of-the-art Progression:**  
- Pre-trained Transformers (BERT, Bi-Encoder, neurosymbolic methods) have systematically increased F₁ on unified benchmarks to 80–86% [1910.00194, 2307.16663, 2105.10146].
    - BERT-based models with linear projection and gating achieve 74.1 F₁ avg (up to 76.2) on all-words test sets [1910.00194].
    - Bi-Encoders with triplet and hypernym-pretraining reach 80.6 F₁ [2105.10146].
    - Neurosymbolic nested-ball methods surpass the long-held 80% glass ceiling, reaching average F₁ ≈ 85 [2307.16663].
    - Gloss selection and data augmentation (wordnet examples) push fine-tuned BERT to 71.9 F₁, +1.5 over GlossBERT [2009.11795].
- Non-parametric few-shot methods (MetricWSD) yield 75.1 F₁ overall—3.5–6.6 point gain on rare senses [2104.12677].
- Low-shot WSD (FEWS): best neural bi-encoders reach ≈73% accuracy overall, with humans at ~80% [2102.07983].

**Low-Resource and Multilingual Benchmarks:**  
- Persian SBU-WSD-Corpus: SVM and MLP yield ~72.4–72.7 F₁, best knowledge-based system (Basile-14) at 67.8 [2107.01540].
- Arabic SALMA: ArabGlossBERT achieves 84.2% (Modern) and 77.6% (Ghani) accuracy under TSV [2310.19029].
- Swedish SENSEVAL-2: best LLM (Claude 3.5 Sonnet + definitions) achieves 0.855 accuracy, but BERT+LR still dominates (0.931) [2410.22827].

**LLM Benchmarks and WiC-TSV:**  
- Zero-shot LLMs on XL-WSD (multiple-choice) yield 53–75% accuracy (mid-to-large scale), but fine-tuned Llama3.1-8B surpasses SOTA: F₁ = 0.847 [2503.08662].
- WiC-TSV: BERT-L reaches 76–78 F₁, with an 8–10 point gap to human upper bound (85.3%) [2004.15016].

## 5. Benchmarks for Rare Senses and Domain Adaptation

Conventional benchmarks are highly skewed toward frequent senses. FEWS and MetricWSD directly address this by balancing few- and zero-shot evaluation and simulating rare-sense episodes. On FEWS, all models except gloss-informed bi-encoders perform well below human upper bound (80%); e.g., knowledge-based Lesk+emb reaches only 44% [2102.07983]. MetricWSD’s balanced episodic sampling yields 68.7 F₁ on ≤10-examples-per-word and 56.3 F₁ on low-frequency senses, compared to 65.2/49.7 for BERT-classifier [2104.12677].

Domain adaptation and transfer are explored via WiC-TSV's multi-domain splits and few-shot protocols. Performance of non-English and specialty-domain models on SBU-WSD-Corpus (Persian), SALMA (Arabic), and Swedish Eukalyptus confirms that coverage, sense inventory quality, and human definitions are critical for robust disambiguation [2107.01540, 2310.19029, 2410.22827].

## 6. Limitations, Ongoing Challenges, and Future Directions

Despite significant advances, current benchmarks reveal persistent bottlenecks:
- **Sense inventory completeness and granularity:** Difficulties arise in sense inventory mapping across resources (WordNet, FarsNet, Ghani, SALDO, BabelNet), especially for rare or domain-specific senses.
- **Low-resource and cross-lingual evaluation:** Recent efforts such as SALMA (Arabic), SBU-WSD-Corpus (Persian), and XL-WSD (multilingual) expand coverage, but evaluation in many languages remains limited [2310.19029, 2107.01540, 2503.08662].
- **Rare-sense generalization:** Even the strongest neural models lag behind humans on novel/zero-shot senses, as demonstrated on FEWS and MetricWSD’s frequency-stratified evaluation [2102.07983, 2104.12677].
- **Inventory-agnostic and binary verification tasks:** WiC-TSV enables evaluation without dependence on full sense inventories, but does not test complete inventory discrimination [2004.15016].
- **Contextualization and prompt design in LLMs:** Swedish and XL-WSD experiments confirm that explicit, human-written sense definitions in prompts significantly boost zero-shot accuracy, but are a bottleneck for porting and domain adaptation [2410.22827, 2503.08662].
- **Metric reporting:** There is a movement toward frequency-stratified, few/zero-shot metrics in published results, with recommendations to always report separate metrics for rare senses [2102.07983].

Future benchmarks will likely emphasize few-/zero-shot evaluation, domain transfer, multilingual expansion, and integration of inventory-independent settings—complementing all-words and lexical-sample paradigms with generative, selection, and sense-verification tasks.

---

**References:**  
- "Improved Word Sense Disambiguation Using Pre-Trained Contextualized Word Representations" [1910.00194]  
- "Non-Parametric Few-Shot Learning for Word Sense Disambiguation" [2104.12677]  
- "FEWS: Large-Scale, Low-Shot Word Sense Disambiguation with the Dictionary" [2102.07983]  
- "Word Sense Disambiguation as a Game of Neurosymbolic Darts" [2307.16663]  
- "Persian-WSD-Corpus: A Sense Annotated Corpus for Persian All-words Word Sense Disambiguation" [2107.01540]  
- "SALMA: Arabic Sense-Annotated Corpus and WSD Benchmarks" [2310.19029]  
- "How Well Do Large Language Models Disambiguate Swedish Words?" [2410.22827]  
- "WiC-TSV: An Evaluation Benchmark for Target Sense Verification of Words in Context" [2004.15016]  
- "Exploring the Word Sense Disambiguation Capabilities of Large Language Models" [2503.08662]  
- "Enhancing Modern Supervised Word Sense Disambiguation Models by Semantic Lexical Resources" [2402.13302]

Source: https://www.emergentmind.com/topics/word-sense-disambiguation-benchmarks