Contextualized Stopwords in NLP

• Contextualized stopwords are tokens whose stopwordhood is defined by the surrounding context, allowing adaptive filtering beyond static lists.
• They are identified using diverse methods such as statistical metrics, contextual embedding geometry, and task-driven performance tests.
• Applications span news categorization, patent text analysis, and social media processing, enhancing both retrieval and sentiment analysis tasks.

Contextualized stopwords (CS) are tokens whose stopwordhood is conditioned by context rather than assumed to be fixed at the language level. In the literature summarized here, “context” may mean document domain, news category, retrieval setting, downstream task, embedding neighborhood, or corpus-specific usage. This contrasts with static stopword lists, which treat frequent function words as uniformly uninformative across all texts. Under a contextualized view, a token may behave like a stopword in one setting and as a content-bearing signal in another: “million” may be non-discriminative in one news category but informative in another, “goal” may be conventional within sports, “not” is often indispensable for sentiment analysis, and “no” can be crucial in medical retrieval (Turki et al., 2024, DiPietro, 2022, Hofstätter et al., 2022).

1. Conceptual foundations

Static stopword lists are language-wide inventories of frequent function words such as articles, pronouns, and prepositions, typically hand-curated or frequency-based and applied uniformly regardless of document domain or topic. Contextualized stopwords extend this notion by conditioning stopwordhood on context: a word may be semantically low-value across all categories, semantically weak only within a specific category, or removable only relative to a particular task and model. The distinction between domain-agnostic stopwords and category-specific stopwords is explicit in categorized news corpora, where text categorization is used to separate words with high cross-category dispersion from words that are frequent only within a particular topical slice (Turki et al., 2024).

This contextualization emerged because generic stopword inventories repeatedly fail in specialized corpora. Engineering texts contain patent boilerplate such as “said,” “preferably,” and “respectively” that are not captured by general English lists; Arabic social-media sentiment uses Egyptian dialect function words absent from Modern Standard Arabic lists; Stack Overflow contains software-engineering tokens that are frequent and minimally discriminative within that domain; and Twitter introduces highly repetitive platform conventions and event-driven bursts that distort purely frequency-based heuristics [(Sarica et al., 2020); (Medhat et al., 2014); (Fan et al., 2023); (Choy, 2012)].

A second conceptual shift is that some work defines CS not only lexically but representationally. In contextual embedding spaces, punctuation, common function words, and some tense-dominated verb usages can align with dominant structural directions rather than semantic ones. Under this view, CS are tokens whose contextual embeddings primarily encode syntax, style, or other structural regularities that are unhelpful for semantic tasks (Rajaee et al., 2021). Topic modeling and neural retrieval extend the idea further by treating CS as tokens whose contextualized embeddings exhibit low self-similarity across occurrences or whose learned document-side gates mark them as non-essential to relevance scoring (Ma et al., 2024, Hofstätter et al., 2022).

2. Forms of contextualization

The literature does not impose a single operational definition. Instead, CS is instantiated through several distinct but compatible criteria.

Operational context	Primary signal	Representative papers
Category-conditioned corpora	High dispersion and entropy across categories, or category-skewed frequency	(Turki et al., 2024)
Domain- or class-specific text mining	Low discriminative value relative to labels or decision boundaries	(Alshanik et al., 2020, Fan et al., 2023)
Task-specific model behavior	Minimal degradation after deleting a token from the corpus	(DiPietro, 2022)
Contextual embedding geometry	High alignment with local structural dominant directions	(Rajaee et al., 2021)
Corpus-aware topic modeling	Low self-similarity across contextual occurrences	(Ma et al., 2024)
Neural retrieval	Learned whole-word gate sets a contextual representation to zero	(Hofstätter et al., 2022)
Unsupervised spacing statistics	Strong Brody-distribution fit of inter-occurrence spacing	(Łobodziński, 2021)

Category-conditioned CS separates domain-agnostic stopwords from category-specific stopwords. The former are consistently frequent and weakly informative across categories; the latter are frequent or conventional only within a category and may carry little discriminative signal there while remaining informative elsewhere. This formulation is particularly explicit in categorized African-language news corpora (Turki et al., 2024).

Domain- and task-conditioned CS instead depend on supervision or performance. In binary classification, words near a class-separating hyperplane are treated as domain-common and weakly discriminative; in transformer-based sentiment analysis, words are ranked by the performance drop caused by deleting them, so stopwordhood becomes specific to the dataset, tokenizer, model, and metric (Alshanik et al., 2020, DiPietro, 2022).

Embedding-conditioned CS reframes stopwordhood geometrically. A token may be frequent and structurally useful to the LLM yet undesirable for semantic similarity, topic-word selection, or late-interaction retrieval because its contextualized representation tracks sentence style, topic-following behavior, or query-irrelevant document context rather than stable semantic content (Rajaee et al., 2021, Ma et al., 2024, Hofstätter et al., 2022).

3. Quantitative formulations

In categorized corpora, the main statistical quantities are category-conditional probabilities

$$P(w\mid c) = \frac{\text{count}(w,c)}{\sum_{w'} \text{count}(w',c)},\qquad P(c\mid w) = \frac{\text{count}(w,c)}{\sum_{c'} \text{count}(w,c')},$$

category entropy

$H_c(w) = - \sum_{c} P(c\mid w)\,\log P(c\mid w),$

and dispersion across categories

$D(w) = \frac{|\{c:\, \text{count}(w,c)>0\}|}{|C|}.$

Higher $H_c(w)$ and high $D(w)$ indicate domain-agnostic behavior, while category-specific stopwords are characterized by high relative frequency within a category and lower cross-category dispersion. The same framework incorporates TF, DF, IDF, and mutual information, and the details block gives a composite stopwordness score

$$S_{\text{stop}(w)} = \alpha\,\overline{\text{TF}(w)}+\beta\,\overline{1/\text{IDF}(w)}+\gamma\,\overline{H_c(w)}-\delta\,\text{ContentPOS}(w),$$

together with a contrastive category-specific score $S_{\text{cat}(w,c)}$ and Jaccard overlap

$J(A,B) = \frac{|S_A \cap S_B|}{|S_A \cup S_B|}.$

These quantities operationalize whether a word is globally common, locally conventional, or still content-bearing (Turki et al., 2024).

A different family of formulations uses geometry. In class-conditioned feature spaces, the hyperplane method defines

$H = \{ x \in \mathbb{R}^k : w^T x + b = 0 \},$

with $w = \text{Center}_A - \text{Center}_B$ and $H_c(w) = - \sum_{c} P(c\mid w)\,\log P(c\mid w),$0, and then ranks tokens by their distance to $H_c(w) = - \sum_{c} P(c\mid w)\,\log P(c\mid w),$1,

$H_c(w) = - \sum_{c} P(c\mid w)\,\log P(c\mid w),$2

Tokens near the hyperplane are treated as domain-common CS because they occur similarly across classes. In contextual embedding spaces, isotropy is measured through the partition function

$H_c(w) = - \sum_{c} P(c\mid w)\,\log P(c\mid w),$3

and isotropy index

$H_c(w) = - \sum_{c} P(c\mid w)\,\log P(c\mid w),$4

with local versions computed after clustering and centering. Local PCA then removes dominant structural directions via

$H_c(w) = - \sum_{c} P(c\mid w)\,\log P(c\mid w),$5

This formalizes CS as tokens whose contextual embeddings strongly project onto cluster-specific structural components (Alshanik et al., 2020, Rajaee et al., 2021).

Task- and corpus-conditioned methods define stopwordhood through behavioral tests. In sentiment analysis, the importance of a word is the performance loss incurred when it is removed:

$H_c(w) = - \sum_{c} P(c\mid w)\,\log P(c\mid w),$6

with aggregate drop $H_c(w) = - \sum_{c} P(c\mid w)\,\log P(c\mid w),$7 for a set $H_c(w) = - \sum_{c} P(c\mid w)\,\log P(c\mid w),$8 and shrinkage

$H_c(w) = - \sum_{c} P(c\mid w)\,\log P(c\mid w),$9

In topic modeling, CAST computes contextualized occurrence embeddings $D(w) = \frac{|\{c:\, \text{count}(w,c)>0\}|}{|C|}.$0, a corpus mean $D(w) = \frac{|\{c:\, \text{count}(w,c)>0\}|}{|C|}.$1, and average pairwise cosine self-similarity

$D(w) = \frac{|\{c:\, \text{count}(w,c)>0\}|}{|C|}.$2

then filters tokens with $D(w) = \frac{|\{c:\, \text{count}(w,c)>0\}|}{|C|}.$3. In ColBERTer, each whole-word passage representation receives a learned scalar gate

$D(w) = \frac{|\{c:\, \text{count}(w,c)>0\}|}{|C|}.$4

and words with $D(w) = \frac{|\{c:\, \text{count}(w,c)>0\}|}{|C|}.$5 are removed from the index. In the Random Matrix Theory approach, contextualized stopwords are words whose inter-occurrence spacing distributions are well fit by the Brody distribution

$D(w) = \frac{|\{c:\, \text{count}(w,c)>0\}|}{|C|}.$6

with classification based on low mean-square fitting error (DiPietro, 2022, Ma et al., 2024, Hofstätter et al., 2022, Łobodziński, 2021).

4. Multilingual and domain-specific extraction

One of the clearest corpus-level demonstrations of CS is the categorized stopword extraction study on nine African languages plus French. Using MasakhaNEWS, the African Stopwords Project, and MasakhaPOS, the unified candidate list was formed from crowdsourced or TF-IDF-derived stopword lists plus words tagged AUX, PRON, CCONJ, SCONJ, DET, and PART. After tokenization, punctuation removal, and lowercasing, most languages achieved more than 80% stopword detection success in the development set. Lower coverage appeared in languages with morphology or orthography that hinder token-level recognition: French had 466 stopwords found (67.5%), Igbo 49 (70.0%), Luganda 90 (62.1%), and Yoruba 62 (38.8%), whereas Hausa reached 319 (97.0%), Swahili 151 (96.8%), and Kirundi 55 (93.2%). Over 40% of stopwords were common across all categories, and less than 15% were unique to a single category; Somali, for example, had 92.0% of stopwords common across all categories and 0.0% unique to one category. The paper also highlights context-sensitive examples such as Hausa milyan “million,” Swahili yasio “not,” and Luganda teyali “it was not,” illustrating that numerals, polarity markers, and temporal or aspectual auxiliaries may or may not act as stopwords depending on category and downstream use (Turki et al., 2024).

Domain-specific list construction predates this categorized framing but reaches similar conclusions. In technical language processing over USPTO utility patents, 6,559,305 patents, 30,265,976 sentences, and 781,156,082 tokens were mined to identify engineering-specific uninformative terms. After phrase detection, lemmatization, and removal of known generic stopwords, four criteria—TF, IDF, a modified TF-IDF, and entropy—were ensembled, and two engineering experts curated 62 statistically identified uninformative terms. Together with 25 earlier validated terms, this yielded a final engineering stopword list of 87 items. The curation explicitly retained high-frequency technical nouns such as “method,” “system,” “apparatus,” and “device,” because they were judged informative in engineering contexts rather than contextualized stopwords (Sarica et al., 2020).

Software engineering corpora show the same task dependence. A Stack Overflow corpus of 10,000 answered questions across four tags was used to construct two 200-word software-engineering stopword lists, one TF-IDF-based and one Poisson-based. When these lists were plugged into three replicated software-engineering tools, the TF-IDF list improved 17 out of 19 evaluation measures, whereas using no stopwords improved only 4 and was worse on 12. In the app-reviews classifier, the SE-specific list improved several precision, recall, and F1 figures over a general list; in RACK query recommendation, the TF-IDF list improved Top-10, MRR@10, MAP@10, and MR@10 over baseline (Fan et al., 2023).

Social-media corpora underscore the importance of dialect, script, and platform artifacts. In Arabic sentiment analysis over movie-related Facebook, Twitter, and review corpora, a 620-word Egyptian dialect list and a 1061-word corpus-based list were generated, and the paper reports that general lists containing Egyptian dialect words gave better performance than MSA-only stopword lists. In Sinhala Facebook data, a decade-long corpus of 1,500,917 posts from 533 pages and a Sinhala-only subcorpus of 364,402 posts were used to derive a frequency-based stopword list via z-scores between $D(w) = \frac{|\{c:\, \text{count}(w,c)>0\}|}{|C|}.$7 and $D(w) = \frac{|\{c:\, \text{count}(w,c)>0\}|}{|C|}.$8 after singleton removal. The paper explicitly notes that this lenient rule can include category-heavy political content such as “minister,” “president,” or personal names, which is precisely why a contextualized rather than universal interpretation is required [(Medhat et al., 2014); (Wijeratne et al., 2020)].

5. Contextualized stopwords in neural representation learning

In contextual word representations, CS is tied to anisotropy and local structural dominance. The cluster-based isotropy study shows that globally, BERT, RoBERTa, and GPT-2 are extremely anisotropic in non-input layers, but local isotropy improves markedly after k-means clustering and zero-meaning each cluster. The paper reports, for example, BERT baseline $D(w) = \frac{|\{c:\, \text{count}(w,c)>0\}|}{|C|}.$9 improving to $H_c(w)$0 at $H_c(w)$1, and RoBERTa baseline $H_c(w)$2 improving to $H_c(w)$3. Clusters rich in punctuation and stopwords encode structural and syntactic features rather than semantics; removing local dominant directions improves semantic tasks. On STS-B, BERT-base improved from 47.4 to 66.0 with cluster-based removal, RoBERTa-base from 56.0 to 71.4, and GPT-2 from 24.8 to 62.3. The same work shows that verb representations are heavily organized by tense, and local deprojection reduces the undesirable pattern in which different-tense same-meaning instances are farther apart than same-tense different-meaning instances (Rajaee et al., 2021).

CAST imports a related idea into topic modeling by defining functional tokens as those with low self-similarity across contextualized occurrences. Candidate centroid words are contextualized on the corpus rather than treated as isolated strings, and tokens with $H_c(w)$4 are removed from the candidate pool. On 20NewsGroups, $H_c(w)$5 worked well; examples below the threshold included “good” at 0.297 and “<pad>” at 0.299, whereas “armenian” at 0.833, “genocide” at 0.781, and “israeli” at 0.732 were retained. CAST-MPNET reached NPMI = 0.18, LLM-TC = 3.88, and LLM-TD = 3.8 on 20NewsGroups, and on unprocessed 20NewsGroups with stopwords retained it outperformed BERTopic, whose noisy-data topics could consist largely of stopwords such as “the”, “to”, “and”, “is”, “of”.

Neural retrieval operationalizes CS even more directly. ColBERTer aggregates subwords into unique whole-word vectors and learns a scalar gate over each passage word. Words with $H_c(w)$6 are dropped from the index, so contextualized stopwords are document words deemed non-essential for query–document scoring. This is not a fixed lexicon: even a high-IDF word may be removed if it is unhelpful in its local context, whereas a low-IDF token such as “no” may be retained because it matters for relevance. Empirically, ColBERTer retains about 29–36% fewer vectors than BOW² alone, yields about 2.5× fewer vectors than ColBERT in one configuration, and in the 1-dimensional Uni-ColBERTer setting reaches index storage parity with plaintext at approximately 3.3 GB versus 3.05 GB plaintext while maintaining strong effectiveness. The example query “does doxycycline contain sulfa” illustrates the contextual nature of the gate: “doxycycline,” “sulfa,” “containing,” and “no” are kept with high scores, whereas contextually non-essential words may be removed or downweighted (Hofstätter et al., 2022).

6. Evaluation, misconceptions, and unresolved issues

A recurrent misconception is that stopwords are synonymous with fixed lists of function words. The literature surveyed here rejects that simplification. Context-sensitive examples include numerals, polarity markers, adverbs, tense or aspectual auxiliaries, generic verbs and adjectives, patent boilerplate, social-media artifacts, and even political names if the corpus and thresholding scheme encourage them. At the same time, several studies show that frequent words can remain highly informative: in sentiment analysis, the most important SST2 tokens include “not,” “but,” “n't,” “never,” “is,” “no,” “without,” “too,” “and,” and “a”; in PubMed journal classification, some classic stopwords such as “who,” “should,” and “whom” can be discriminative; and in engineering patent text, terms like “method,” “system,” and “device” were explicitly retained (DiPietro, 2022, Alshanik et al., 2020, Sarica et al., 2020).

Evaluation is also heterogeneous. In the African-language categorization study, “detection success rate” is not precision or recall but the percentage of unified stopword candidates observed in the development set after preprocessing; the paper separately recommends precision, recall, and F1 when a gold list or human judgments are available (Turki et al., 2024). Topic-model evaluation can likewise be misleading: CAST notes that BERTopic may return topics of stopwords on noisy data and still receive moderately good NPMI because stopwords co-occur frequently, which is why the paper supplements co-occurrence-based evaluation with LLM-based coherence and diversity measures (Ma et al., 2024).

Several unresolved issues recur across the literature. Morphological richness, agglutination, cliticization, and orthographic variation reduce token-level detection coverage in languages such as Yoruba, Igbo, and Luganda, and apostrophe removal affects French clitics (Turki et al., 2024). Domain shift remains a major problem: lists derived from news need not transfer to conversational social media or legal text, and CS learned on SST2 or MS MARCO may not generalize without re-estimation (DiPietro, 2022, Hofstätter et al., 2022). Model-based pruning can remove useful syntax, discourse, tense, or query-relevant cues if thresholds are too aggressive, which is why the papers repeatedly recommend human review, downstream sanity checks, significance testing, and, in multilingual settings, morphology-aware preprocessing, subword models, or contextual embeddings as future directions (Turki et al., 2024, Rajaee et al., 2021, Ma et al., 2024).

Taken together, these studies define contextualized stopwords not as a single list but as a family of corpus-, category-, task-, and model-dependent mechanisms for identifying tokens that are frequent, structurally dominant, weakly discriminative, or otherwise non-essential relative to a specific analytical objective. The central implication is methodological rather than lexical: stopwordhood is best treated as a conditional property of usage.