African Stopwords Project

• African Stopwords Project is a large-scale initiative that systematically curates stopword lists for diverse African languages and dialects.
• It employs hybrid methodologies combining linguistic, statistical, and crowdsourced strategies to refine text preprocessing for NLP applications.
• The project addresses resource gaps by standardizing stopword resources, boosting performance in tasks such as information retrieval and sentiment analysis.

The African Stopwords Project is a large-scale initiative dedicated to the systematic curation, extraction, and validation of stopword lists across a wide range of African languages and dialects. Stopwords—high-frequency function words, pronouns, particles, and other grammatical markers carrying minimal semantic content—are essential for robust text preprocessing in information retrieval, NLP pipelines, and text classification tasks. The project operates at the intersection of computational linguistics, language documentation, and practical NLP, providing foundational resources long available for European and Asian languages but nearly absent for most African tongues. Its hybrid methodologies combine linguistic, statistical, crowdsourced, and domain-adaptive strategies, addressing unique typological, morphological, and sociolinguistic characteristics of African languages (Emezue et al., 2023, Turki et al., 2024).

1. Project Objectives, Scope, and Rationale

The African Stopwords Project aims to address the severe resource gap in annotated stopword lists for African languages. These languages, typically low-resource, exhibit considerable morphological diversity, agglutination, and dialectal variability, which complicate the direct transfer of stopword methodologies from high-resource languages such as English or French. In addition to enhancing basic NLP tasks—preprocessing, IR, sentiment/topic classification—the project has a descriptive linguistic component, revealing cross-linguistic function-word patterns including shared grammatical markers across language families (e.g., Bantu noun class concords, Afro-Asiatic particles) (Emezue et al., 2023, Medhat et al., 2015).

Key motivations include:

• Enabling efficient feature extraction and vocabulary reduction in African language corpora, thus improving downstream model performance.
• Providing standardized, readily-available stopword files for corpora, toolkits (e.g., NLTK, spaCy), and shared NLP tasks.
• Illuminating intra-continental function word overlap and code-switching phenomena, particularly relevant for dialect-rich environments like Egyptian Arabic social media or Nigerian Pidgin news (Emezue et al., 2023, Turki et al., 2024).

2. Language Coverage and Typological Features

The current coverage spans at least ten African languages, with planned expansion to over thirteen. The languages represent diverse families—Bantu, Chadic, Cushitic, Creole, Germanic-derived Afrikaans, and Yoruboid. Stopword selection criteria must respect morphosyntactic properties unique to each language. Table 1 below presents example languages, family, and salient function word types.

Language	Major Family	Function-Word Types
Hausa	Afro-Asiatic, Chadic	Subject markers, postpositions
Kirundi	Bantu	Noun-class prefixes, concords
Swahili (kiSwahili)	Bantu	Agglutinative prefixes, clitics
Yoruba	Benue–Congo, Yoruboid	Preverbal pronouns, particles
Somali	Afro-Asiatic, Cushitic	Article suffixes, pronouns
Nigerian Pidgin	Creole/English	Simplified, high-frequency func words
Afrikaans	Germanic	Articles, prepositions

Representative stopword samples: “die, en, van” (Afrikaans), “da, ne, a” (Hausa), “ku, mu, ya” (Kinyarwanda), “ni, a, kwa” (Swahili) (Emezue et al., 2023).

Notably, languages with rich agglutination (Swahili, Yoruba, Igbo) require handling of affixal variants and subword units. Bantu noun class concords (e.g., "ba-", "i-", "mu-") and compound pronouns are particularly frequent and semantically minimal (Emezue et al., 2023, Turki et al., 2024).

3. Data Collection, Preprocessing, and Corpus Construction

A credible stopword list presupposes representative corpora reflecting actual language use. The project utilizes news (MasakhaNEWS), social media (Facebook, Twitter), web forums, and review sources, emphasizing domains with high incidence of code-switching, neologisms, dialectal forms, and transliterations (e.g., Franco-Arab in Egyptian OSN). Preprocessing incorporates noise removal (URLs, emoticons, non-script content), normalization (orthographic variants, translation of English in-mixes, spell correction), and tokenization. For Arabic dialect projects, explicit attention is given to converting Latinized script (“de7k” → “ضحك”) and mapping dialectal forms to MSA or English for cross-verification (Medhat et al., 2015, Medhat et al., 2014).

Corpus statistics from Egyptian Arabic work illustrate typical data scale and annotation effort: Twitter (781 tweets), Facebook (1,261 comments), and review texts (32) together span nearly 25,000 tokens and 4,818 unique types post-cleanup (Medhat et al., 2014).

4. Methodologies for Stopword Identification and Validation

The primary methodology comprises a multi-phase pipeline integrating statistical, linguistic, and, where feasible, text-categorical strategies (Medhat et al., 2015, Emezue et al., 2023, Turki et al., 2024):

4.1 Frequency-Based Ranking (Statistical)

• Compute word term frequency $\mathrm{TF}(w) = \mathrm{freq}(w)$ and select the top $N$ high-frequency types as candidate stopwords.
• Optionally, compute document frequency $\mathrm{DF}(w)$ and inverse document frequency $$\mathrm{IDF}(w) = \log\left(\frac{N_{\text{docs}}}{\mathrm{DF}(w)}\right)$$ to downweigh rare function words or domain-locked terms (Medhat et al., 2015, Medhat et al., 2014).

4.2 Semantic and Domain Filtering (Manual + Cross-Linguistic)

• Remove candidate nouns/verbs with content semantics (POS-tagging, bilingual lookup).
• For each candidate $w$, check presence in standard (e.g., MSA, ISO, English) stopword lists; map dialect forms to standard equivalents for double-checking.
• Retain only words meeting: high corpus frequency, semantic vacuity in isolation, generalizability across domains (Medhat et al., 2015, Emezue et al., 2023).

4.3 Morphological Expansion

• For morphologically rich languages (Arabic, Bantu, Swahili), systematically expand stopword bases by generating valid affixal forms (prefixes—definite article, conjunctions, prepositions; suffixes—object and possessive pronouns) and unifying orthographic variants (e.g., “أ”/“إ”, “ي”/“ى”) (Medhat et al., 2015, Medhat et al., 2014).

4.4 Hybrid and Categorical Filtering

• Construct unified candidate lists from linguistic rules (POS tags: AUX, PRON, DET, CONJ) and statistical/crowdsourced sources.
• Leverage labeled text categorization (MasakhaNEWS categories) to compute coverage score $S_w = \frac{1}{|C|} \sum_{c\in C} I_{w,c}$, with $I_{w,c}=1$ if $w$ is present in category $c$.
• Retain candidates appearing in all domains (domain-agnostic), maximizing recall without sacrificing generality (Turki et al., 2024).

5. Evaluation, Metrics, and Empirical Results

Stopword list efficacy is primarily evaluated via downstream NLP tasks, with a secondary emphasis on cross-domain recall and precision (Medhat et al., 2015, Medhat et al., 2014, Turki et al., 2024):

Classification Performance

• Sentiment analysis and topic classification are standard tasks for comparative experiments.
• Typical metrics: accuracy, precision, recall, $F_1$-measure:

$N$0

• Classifiers: Naïve Bayes (NB), Decision Tree (DT), tested with unigram and bigram features (Medhat et al., 2015, Medhat et al., 2014).

Key Empirical Findings

• Domain-agnostic, hybrid, or dialect-specific stopword lists consistently outperform monolingual (e.g., MSA-only) lists in informal, noisy OSN data (ED-only: NB accuracy gain from ∼49% to ∼58% on Facebook; ED removal: NB gain of 5–10% on Twitter corpora) (Medhat et al., 2015).
• Detection success rate for domain-agnostic stopwords exceeds 80% for most covered languages; lower rates observed for agglutinative languages (e.g., Yoruba 38.8%) due to boundary segmentation issues (Turki et al., 2024).
• <15% of candidate stopwords are truly category-unique, restricting negative impact from over-pruning (Turki et al., 2024).

6. Challenges and Linguistic Considerations

African languages exhibit rich morphological and register diversity absent in many high-resource languages. Principal challenges include:

• Agglutinative Morphology: Rich prefixation/suffixation in Bantu, Yoruba, and Swahili necessitate expansion beyond lemma-level stopwords, ideally requiring morphological analyzers or subword tokenization to prevent under-segmentation (Turki et al., 2024).
• Code-Switching and Transliteration: Real-world texts (especially social media) mix dialects, standards, and scripts (e.g., Franco-Arabic). Pipelines must handle normalization, script conversion, and cross-reference with major-language stopword lists for maximum coverage (Medhat et al., 2015, Medhat et al., 2014).
• Context-Dependence: Some high-frequency words are semantically non-neutral in certain domains (e.g., proper names, adverbs). Context-sensitive and domain-specific filtering is essential to prevent loss of model-relevant information (Turki et al., 2024).

7. Recommendations, Future Directions, and Resource Sharing

Best practices articulated in project outputs comprise:

1. Construct representative, multi-domain corpora reflecting real-world and informal registers.
2. Apply three-phase list-building: frequency ranking, semantic filtering/validation, affixal expansion.
3. Combine automatic and manual curation, reserving human review for ambiguous cases.
4. Merge dialect and standard stopword lists to support mixed-register data.
5. Evaluate lists on downstream NLP tasks across multiple classifiers and feature granularities.
6. Open-source all validated lists under permissive licenses and integrate with major NLP toolkits and pipelines (Emezue et al., 2023, Medhat et al., 2015, Medhat et al., 2014).

Future plans include scaling to over 13 African languages, integrating statistical candidate extraction via TF–IDF, entropy, IG, and KL-divergence, and involving large-scale expert and crowdsourced review. For morphologically rich languages, incorporation of subword models (BPE) or neural context-aware embedding approaches (e.g., mBERT, XLM-R) is recommended to improve recall/precision balance in stopword detection. Fine-grained, context- or domain-sensitive stopword lists—adjusted via coverage-score thresholds—are identified as a next research milestone (Turki et al., 2024, Emezue et al., 2023).

Public resources, including fully curated lists, annotation guidelines, and workflows, are available in the Masakhane Preprocessor repository, facilitating reproducibility and iterative improvement (Emezue et al., 2023).

---

Key References:

• "The African Stopwords project: curating stopwords for African languages" (Emezue et al., 2023)
• "Text Categorization Can Enhance Domain-Agnostic Stopword Extraction" (Turki et al., 2024)
• "Egyptian Dialect Stopword List Generation from Social Network Data" (Medhat et al., 2015)
• "Corpora Preparation and Stopword List Generation for Arabic data in Social Network" (Medhat et al., 2014)