---
title: 'SWORDS: Lexical Substitution in Context'
url: https://www.emergentmind.com/topics/substituting-words-with-synonyms-swords
type: topic
---

# SWORDS: Lexical Substitution in Context

Searching arXiv for recent and foundational work on SWORDS, lexical substitution, and context-aware synonym replacement.
“Substituting WORDs with Synonyms (SWORDS)” denotes a family of problems and methods concerned with replacing a target word with an alternative that preserves meaning under contextual constraints. In contemporary NLP, the term is closely associated with lexical substitution: given a context \(c\) and a target word \(w\), a system must return substitutes \(w'\) that fit the specific sentence rather than merely appearing in a context-free thesaurus [2106.04102]. The label also refers to two concrete research artifacts: SWORDS, the Stanford Word Substitution Benchmark for lexical substitution [2106.04102], and SWORDS, the code-switching algorithm introduced in UniCoM, where selected words are replaced with translation-equivalent words from another language while preserving sentence semantics and matrix-language word order [2508.15244]. Across these uses, the central issue is the same: synonym substitution is only reliable when sense, syntax, usage, and broader contextual relevance are explicitly modeled [2502.04173].

## 1. Task definition and conceptual scope

Lexical substitution is formalized as the task of mapping a context \(c\) and a target word \(w\) to a ranked list of substitutes \(w'\) that can replace \(w\) in \(c\) without breaking meaning or acceptability [2106.04102]. In SWORDS, each triple \((c,w,w')\) carries a score equal to the fraction of annotators who would actually consider using \(w'\) to replace \(w\) as the author of the sentence, turning lexical substitution into a graded contextual appropriateness problem rather than an exercise in dictionary lookup [2106.04102].

A central clarification in recent work is that lexical substitution is not identical to synonym substitution. “Lexical Substitution is not Synonym Substitution” argues that the substitute should not necessarily be only synonymous, but should also fit well into the surrounding context of the target word while preserving the sentence’s grammatical structure [2502.04173]. This distinction is reinforced by the SWORDS benchmark paper, which contrasts context-sensitive substitution with context-free thesaurus relations and positions lexical substitution as a task for writing assistance rather than as a mere synonym lookup problem [2106.04102].

This suggests a useful technical distinction between lexical relatedness and substitutability. A word may be semantically related, or even listed as a synonym in a lexical resource, yet still be contextually inappropriate, stylistically odd, or syntactically ill-formed in a given sentence [2106.04102]. The adversarial-attack literature makes the same point negatively: many purported “synonym substitution attacks” rely on replacements that are ungrammatical or do not preserve the original sentence’s semantics, showing that surface-level synonymy claims are often insufficient [2210.02844].

## 2. Benchmarking with SWORDS

SWORDS was introduced to address two limitations of earlier lexical substitution benchmarks: low substitute coverage and weak quality control [2106.04102]. Existing resources such as SemEval-2007 and CoInCo relied on human recall as the only source of substitutes, which underrepresents the alternatives that are hardest for humans to think of and most useful for writing assistance [2106.04102].

The benchmark reframes lexical substitution as a classification task over a predefined candidate set. For each \((c,w)\) pair, candidates are drawn from Roget’s Thesaurus for the target part of speech and augmented with human-generated substitutes from CoInCo; annotators then judge each triple \((c,w,w')\) with a binary decision indicating whether they would actually consider using the substitute as the author of the sentence [2106.04102]. The final score is the fraction of positive labels, usually out of 10 judgments, and operational thresholds define **acceptable** substitutes as score \(> 50\%\), **conceivable** substitutes as score \(> 0\%\), and **inconceivable** substitutes as score \(= 0\%\) [2106.04102].

The resulting dataset contains 1,132 contexts, 1,132 targets, 68,683 unique substitutes, and 375,855 total labels [2106.04102]. It averages 60.7 substitutes per target, including 21.4 conceivable and 4.1 acceptable substitutes, and therefore has 4.1x more conceivable substitutes per target and 1.8x more acceptable substitutes per target than the overlapping CoInCo estimate [2106.04102]. When equalizing the number of substitutes, SWORDS yields an average score of 50.7% versus 34.4% for CoInCo relabelings, which the benchmark authors summarize as 1.5x higher average appropriateness [2106.04102].

The benchmark also makes explicit that substitutes may be words or phrases and are represented as lemmas, so inflection is deliberately separated from contextual appropriateness [2106.04102]. This broad coverage creates a stricter evaluation setting for systems that must retrieve not just obvious substitutes but also plausible, less recalled alternatives.

## 3. Context-aware modeling of substitutes

Recent systems treat contextual fit as the core modeling problem. An unsupervised lexical simplification and substitution method with context augmentation generates candidates by combining contextual–decontextual embedding similarity with substitutes proposed by T5 or mT5 on additional sampled contexts of the target word [2311.00310]. Given a target word \(x\) in context \(c_x\), the method first computes a contextualized embedding \(f(x,c_x)\), constructs multi-prototype decontextualized embeddings \(f^k(y)\) for vocabulary items, and scores candidates by
\[
S(y \mid x, c_x) = \max_k \mathrm{cos}\bigl(f^k(y), f(x,c_x)\bigr)
\]
with an additional fastText similarity term when subword segmentation is noisy [2311.00310].

Its main novelty is context augmentation. For the target word \(x\), the method samples 300 sentences containing \(x\) from monolingual corpora, clusters their contextualized embeddings with \(K\)-means using \(K=4\), masks \(x\) in those sampled sentences, and uses T5 or mT5 to generate single-word substitutes [2311.00310]. Cluster weights \(w_k\) are defined by overlap with the top similarity-based candidates from the original target context, and the augmented-context score is
\[
\tilde{S}(y \mid x, c_x) = \sum_k w_k \sum_{c'_x \in C_{x,k}} \mathrm{I}(y \mid c'_x)
\]
where \(\mathrm{I}(y \mid c'_x)\) indicates whether T5 or mT5 generated \(y\) for the masked sentence \(c'_x\) [2311.00310].

Candidates are then reranked by four signals: embedding similarity, LM perplexity or generation probability, word frequency, and the augmented-context score \(\tilde{S}\) [2311.00310]. Weighted rank aggregation produces the final list, and for lexical substitution on SWORDS the simplicity term is disabled and the candidate pools are enlarged to \(M_1=30\) and \(M_2=50\) [2311.00310]. The method achieves a state-of-the-art result on the SWORDS lexical substitution dataset and substantially outperforms other unsupervised systems across English, Portuguese, and Spanish on TSAR-2022 [2311.00310].

Other context-aware architectures integrate external lexical knowledge. LexSubCon combines a BERT proposal score based on a mix-up embedding of the target, gloss similarity from WordNet definitions selected by GlossBERT, sentence-level semantic similarity computed with a fine-tuned sentence similarity model, and a candidate validation score based on contextual impact [2107.05132]. Its final score is a linear combination
\[
\text{score}(x_c) = w_p s_p(x_c) + w_g s_g(x_c) + w_{\text{sen}} s_{\text{sen}}(x_c) + w_v s_v(x_c)
\]
with \(w_p=0.05\), \(w_g=0.05\), \(w_{\text{sen}}=1.0\), and \(w_v=0.5\) [2107.05132]. This architecture exemplifies a broader trend: contextual substitution quality improves when local fit, sense-level similarity, and whole-sentence semantic preservation are modeled jointly.

## 4. Sense discrimination, lexical resources, and synonym structure

Sense awareness is a prerequisite for reliable substitution. Resource-oriented work has long emphasized that a synonym is only safe when the active sense is known. A WSD-specific WordNet organizes senses of polysemous words by clue words rather than by broad synsets and hypernym trees, precisely to avoid “ambiguity in ambiguity” caused by non-discriminative higher-level relations [1409.3512]. In that model, each sense \(s_i\) is associated with a clue-word collection \(C(s_i)\), and overlap with context words supports sense selection before any synonym choice is made [1409.3512].

Vector-based WSD follows a similar logic. An \(\varepsilon\)-filtration method for Russian defines synsets \(syn_k^w\) for a target word \(w^*\), filters both sentence and synset vectors by cosine thresholds, and chooses the synset with maximal set-level proximity \(\tilde{K}_k(\varepsilon)\) [1805.09559]. The practical implication is that synonyms should be selected from the synset corresponding to the resolved sense rather than from an undifferentiated synonym list [1805.09559].

Graph-based synset induction addresses the same problem at resource-construction time. Watset builds a synonym graph, applies local clustering to ego networks to induce sense-specific nodes, disambiguates neighbors, and then performs global clustering to obtain synsets [1704.07157]. Its local-global meta-clustering yields fuzzy clustering at the word level while keeping individual sense nodes in hard clusters, which is exactly the structure needed for sense-aware lexical substitution [1704.07157].

Theoretical work further sharpens the distinction between word-level relatedness and sense-level identity. “Synonymy = Translational Equivalence” treats synsets as equivalence classes of absolute synonymy of senses and multi-synsets as equivalence classes of semantic equivalence across languages [2004.13886]. Under that view, word-level substitution is valid when the relevant senses belong to the same synset; near-synonymy at the word level is insufficient unless the active sense is fixed [2004.13886].

## 5. Limits of naive synonym substitution

A recurrent finding across several lines of work is that naive synonym replacement is unreliable. In adversarial NLP, four widely used substitution methods generate large fractions of invalid substitution words that are ungrammatical or do not preserve the original sentence’s semantics [2210.02844]. For PWWS on AG-News, only 20.2% of 26,600 substitutions were matched-sense synonyms, whereas 75.4% were mismatched-sense synonyms and about 3.8% were morphological substitutions [2210.02844]. For counter-fitted GloVe \(k\)-nearest neighbors and MLM-based generators, a 30-word candidate set contains far fewer than one true context-appropriate synonym on average, with most candidates falling into an “others” category rather than matched-sense synonymy [2210.02844].

Embedding-based and sentence-embedding constraints only partly solve this. Word embedding cosine similarity can filter random unrelated words and antonyms, but it cannot reliably separate matched-sense from mismatched-sense synonyms and often overfavours morphological variants that are ungrammatical in context [2210.02844]. This result complements the benchmark-side critique in SWORDS, where many CoInCo substitutes become unacceptable under stricter contextual judgments and benchmark scores are shown to reflect ease of recall rather than appropriateness [2106.04102].

Recent lexical substitution work reaches a similar conclusion from a generation perspective. ConCat, which concatenates a masked sentence and the original sentence as input to RoBERTa, is motivated by the observation that standard masked-language-model substitution either overfits to the target word or produces context-fitting words that drift semantically [2502.04173]. On LS07, CoInCo, and SWORDS, ConCat improves over a recreated dropout baseline on best, best-mode, oot, oot-mode, \(P@1\), and \(P@3\), and human evaluation shows that respondents prefer its substitutes overall [2502.04173]. The paper also documents potential pitfalls in CoInCo, including context-poor sentences, noisy labels, and substitutes that are inappropriate in context [2502.04173].

A plausible implication is that “synonym substitution” is best treated as a constrained inference problem over sense, context, syntax, and usage, not as a lexical lookup problem. The benchmark, generation, and adversarial literatures all converge on this point, even though they operationalize it differently [2106.04102] [2502.04173] [2210.02844].

## 6. Applications beyond benchmarked lexical substitution

The same substitution machinery appears in domains outside writing assistance. In robustness research, synonym replacement is used both to attack and to defend classifiers. Randomized Substitution and Vote treats synonym-substitution attacks as optimized replacement sequences and detects adversarial examples by randomly substituting words with synonyms, generating \(k\) perturbed texts, aggregating logits, and flagging disagreement between the original prediction and the voted prediction [2109.05698]. Because the method requires no architecture modification or extra training, it treats synonym substitution as a probe of local decision-boundary fragility [2109.05698].

A complementary defense, the Synonym Encoding Method, clusters synonyms in counter-fitted embedding space and maps each cluster to a unique code word before the classifier’s input layer [1909.06723]. Its encoder \(E\) is intended to make many nearby synonym variants collapse to the same representation, so that for many \(x' \in V_\epsilon(x)\), \(E(x') \approx E(x)\) [1909.06723]. Here synonym substitution is not used to produce alternatives for humans but to smooth model behavior against word-level perturbations.

In text-to-SQL, synonym substitution exposes the reliance of schema linking on surface lexical overlap. Spider-Syn replaces schema-related words with manually selected synonyms that reflect real-world paraphrases, and exact-match accuracy drops sharply for GNN, IRNet, RAT-SQL, and RAT-SQL\(_{\text{BERT}}\) when explicit correspondence between questions and schema names is removed [2106.01065]. Manual and automatic schema synonym annotations partly restore performance, showing that substitution robustness depends on explicitly representing lexical alternatives [2106.01065].

The term SWORDS is also used in multilingual speech generation. In UniCoM, SWORDS creates intra-sentential code-switching speech by replacing selected words in a matrix-language sentence with semantically equivalent words from an embedded language while conditioning on part of speech and preserving original sentence semantics and language-specific word order [2508.15244]. Word-level translation pairs are extracted by GPT-4o-mini, aligned to speech segments with MMS-FA, and substituted in both text and audio; in CS-FLEURS, substitution is limited to nouns, verbs, and interjections, with at most three substitutions per sentence [2508.15244]. This is not lexical substitution in the benchmark sense, but it preserves the same operational core: meaning-preserving replacement under contextual and structural constraints.

## 7. Resource enrichment and future directions

Reliable substitution systems depend on the quality of lexical resources. “Grouping Synonyms by Definitions” groups French synonyms by dictionary senses by comparing definition indices from TLFi and synonym dictionaries, reaching at best precision 67% and recall 71% against a lexicographer-annotated gold standard [0909.3445]. The work shows that dictionary-based sense partitioning is feasible and that gloss overlap remains competitive for synonym-to-sense assignment [0909.3445].

Arabic resource construction has moved toward graded synonymy. A benchmark and scoring algorithm for enriching Arabic synonyms assigns candidates fuzzy values in \([0,1]\) using cyclic paths in synonym and translation graphs and evaluates agreement against four linguists with RMSE and MAE [2302.02232]. The scoring function
\[
Fuzzy(f_i) = \theta_1 \cdot P_i + \theta_2 \cdot Q_i
\]
models synonymy strength rather than binary membership, which is directly relevant to ranked substitution systems that need thresholds or graded preferences rather than categorical synonym labels [2302.02232].

Sense-aware embeddings provide another avenue. The SWSDS model for Chinese first applies SememeWSD to label a polysemous target with a sense ID, then retrieves the top 10 synonyms of that sense from OpenHowNet and averages their vectors to form a sense embedding [2206.14388]. On LCQMC, this improves semantic similarity accuracy from 67.9% to 71.9%, suggesting that synonym sets can serve as robust semantic representatives once sense has been resolved [2206.14388].

At the same time, distributional work on near synonyms cautions against assuming full interchangeability. A Random Forest trained on Hindi word embeddings can distinguish Sanskrit-origin from Perso-Arabic-origin near synonyms with average accuracy of about 88%, and around 95% after removing the most error-prone words, indicating that usage patterns preserve etymological and cultural signals even when words share dictionary-level meaning [2604.01425]. This suggests that contextual appropriateness is shaped not only by denotation and syntax but also by register, cultural association, and historical lexical strata.

Taken together, these findings define the current research frontier. High-quality SWORDS systems require broad candidate coverage, explicit sense discrimination, context-sensitive ranking, and lexical resources that encode more than coarse synonymy [2106.04102] [2311.00310] [2302.02232]. They also require evaluation protocols that do not confuse easy recall with appropriateness and do not treat lexical substitution as a synonym lookup task by default [2106.04102] [2502.04173].

Source: https://www.emergentmind.com/topics/substituting-words-with-synonyms-swords