Subword TF–IDF (STF-IDF)

• Subword TF–IDF (STF-IDF) is a machine-learned multilingual IR framework that uses subword tokenization to overcome language-specific heuristics and manage OOV tokens.
• The approach trains a SentencePiece BPE model on a large multilingual corpus, computes TF-IDF with optional sublinear scaling, and applies L2 normalization for cosine similarity retrieval.
• Empirical results show STF-IDF achieves state-of-the-art performance across 12 languages, matching or exceeding traditional TF-IDF baselines without manual stop word removal or stemming.

Subword TF–IDF (STF-IDF) is a fully machine-learned information retrieval framework that replaces traditional word-based tokenization and associated language-specific heuristics with subword tokenization, thereby enabling robust and extensible multilingual search. STF-IDF eliminates manual preprocessing such as stop word removal and stemming, offering higher accuracy, particularly in multilingual and morphologically rich scenarios. The method inherently supports out-of-vocabulary (OOV) handling and achieves state-of-the-art retrieval performance across diverse languages (Wangperawong, 2022).

1. End-to-End STF-IDF Pipeline

The STF-IDF methodology follows a structured pipeline:

1. Subword Tokenizer Training: A large multilingual corpus—using sources such as Wikipedia dumps for the top 100 character-based languages—serves as the training data. Language sampling probability for each language $\ell$ is $p_\ell = D_\ell / \sum_i D_i$, where $D_\ell$ is the monolingual data size. Temperature rescaling with $T=5$ is applied to upsample low-resource languages: $p'_\ell \propto p_\ell^{1/T}$. SentencePiece BPE is trained using character coverage $0.9995$ and vocabulary size $V = 128{,}000$.
2. Document and Query Processing: After training, the subword tokenizer segments input text (document or query) into subword tokens $s_1, s_2, ..., s_m$. Term frequencies $tf_{s,d}$ are computed for each subword.
3. IDF Computation: For document collection size $N$, document frequency $p_\ell = D_\ell / \sum_i D_i$0, and inverse document frequency $p_\ell = D_\ell / \sum_i D_i$1.
4. TF–IDF Weighting: For each subword $p_\ell = D_\ell / \sum_i D_i$2 in document $p_\ell = D_\ell / \sum_i D_i$3, calculate $p_\ell = D_\ell / \sum_i D_i$4 (and analogously for queries).
5. Normalization and Vector Construction: Optionally, sublinear scaling $p_\ell = D_\ell / \sum_i D_i$5 is applied if $p_\ell = D_\ell / \sum_i D_i$6, zero otherwise. Each document/query is represented as a raw vector $p_\ell = D_\ell / \sum_i D_i$7 ($p_\ell = D_\ell / \sum_i D_i$8 is the subword vocab size), then normalized by the $p_\ell = D_\ell / \sum_i D_i$9 norm: $D_\ell$0.
6. Retrieval by Cosine Similarity: Similarity is computed as $D_\ell$1, with ranking by descending similarity.

2. Mathematical Definitions and Weighting

STF-IDF formally defines its weighting scheme as follows:

• Term Frequency: $D_\ell$2 is the count of subword $D_\ell$3 in document $D_\ell$4.
• Inverse Document Frequency: $D_\ell$5, or smoothed variant $D_\ell$6.
• TF–IDF Weight: $D_\ell$7.
• Sublinear Scaling (optional): $D_\ell$8 for $D_\ell$9; $T=5$0 otherwise.
• Vector Normalization: All document/query vectors are normalized to the unit $T=5$1 norm.

This weighting structure obviates the need for frequency threshold-based stop word lists, since frequent tokens inherently receive low $T=5$2 and thus negligible TF-IDF importance.

3. Indexing, Retrieval, and Algorithmic Implementation

The core algorithm can be summarized as follows:

1. Tokenizer Training: Train a SentencePiece BPE model with sampled multilingual data, character coverage $T=5$3, vocabulary size $T=5$4, and temperature-based upsampling.
2. Inverted Index Construction: For each document $T=5$5, subwords $T=5$6 are counted to generate $T=5$7. Document frequencies $T=5$8 are updated. For each subword $T=5$9 in $p'_\ell \propto p_\ell^{1/T}$0, its posting list (storing $p'_\ell \propto p_\ell^{1/T}$1 where $p'_\ell \propto p_\ell^{1/T}$2) is maintained.
3. Vectorization: Sparse vectors $p'_\ell \propto p_\ell^{1/T}$3 are assembled for all documents, then $p'_\ell \propto p_\ell^{1/T}$4-normalized.
4. Query Processing: Tokenize query, compute $p'_\ell \propto p_\ell^{1/T}$5, apply weighting, L2 normalize to form $p'_\ell \propto p_\ell^{1/T}$6.
5. Similarity Search: Candidate documents are selected as the union over subword posting lists. Each candidate $p'_\ell \propto p_\ell^{1/T}$7 is scored via $p'_\ell \propto p_\ell^{1/T}$8 (cosine similarity). Top $p'_\ell \propto p_\ell^{1/T}$9 are returned.

This structure allows efficient sparse retrieval over large multilingual corpora.

4. Elimination of Heuristics and Multilingual Capabilities

STF-IDF eliminates traditional IR heuristics:

• Word Splitting: Subword model learns meaningful token boundaries, removing the need for whitespace-based segmentation.
• OOV Handling: Unseen words are decomposed into familiar subword units, ensuring all tokens are represented even for rare or novel forms.
• Stop Words: Frequent subwords automatically receive low $0.9995$0 scores, rendering explicit stop lists unnecessary.
• Stemming and Morphology: Morphological relationships are implicitly captured, as variants share subword components.
• Unified Multilingual Vocabulary: A single subword vocabulary represents all supported scripts and languages, inherently supporting code-switching and mixed-language text without language detection or separate pipelines.

A plausible implication is that linguistic preprocessing is functionally subsumed by data-driven subword modeling, reducing complexity and support burden in multilingual IR systems.

5. Empirical Evaluation and Comparative Results

The STF-IDF approach is empirically validated on XQuAD paragraph retrieval, a parallel dataset across 12 languages (en, es, de, el, ru, tr, ar, vi, th, zh, hi, ro) with 240 Wikipedia paragraphs and 1190 queries per language. Accuracy is defined as the proportion of queries for which the top-1 retrieved paragraph contains the answer.

A summary of experimental results:

Method	English	Spanish	German	Greek	Russian	Turkish	Arabic	Vietnamese	Thai	Chinese	Hindi	Romanian
Word-based (baseline)	84.2%	—	—	—	—	—	—	—	—	—	—	—
Word + stop-word	83.9%	—	—	—	—	—	—	—	—	—	—	—
Word + stemming	84.9%	—	—	—	—	—	—	—	—	—	—	—
Word + stop + stem	85.2%	—	—	—	—	—	—	—	—	—	—	—
STF-IDF (no heuristics)	85.4%	85.8%	84.9%	81.3%	82.9%	80.1%	77.1%	84.5%	83.5%	82.4%	80.9%	85.0%

STF-IDF matches or exceeds the traditional TF-IDF baselines for English and achieves at least 80% accuracy in 10 additional languages, all without stop word removal or stemming (Wangperawong, 2022).

6. Key Implementation Parameters and Open Source Resources

Critical implementation details for STF-IDF include:

• Subword Tokenizer: SentencePiece BPE, $0.9995$1 tokens, character coverage $0.9995$2, temperature sampling $0.9995$3.
• Term Weighting: Choices between raw $0.9995$4 or sublinear scaling ($0.9995$5); always $0.9995$6.
• Vector Normalization: $0.9995$7 norm.
• Indexing: Inverted index via postings lists of $0.9995$8 for each subword.
• Retrieval: Cosine similarity, computed as sparse dot product over subword overlap.
• Reference Implementation: Open-source code and a demo notebook are provided at https://github.com/artitw/text2text (Wangperawong, 2022).

This configuration yields a generalizable, language-agnostic IR system that is efficient, robust to OOV input, and empirically state-of-the-art for multilingual paragraph retrieval.

7. Contextual Significance and Implications

By obviating manual linguistic preprocessing and directly leveraging subword tokenization, STF-IDF provides a scalable solution to multilingual search challenges where traditional TF-IDF would require extensive hand-tuning per language. The architecture demonstrates that a single, unified, machine-learned pipeline can achieve strong cross-lingual retrieval performance, simplifying deployment and maintenance for global IR systems (Wangperawong, 2022).

A plausible implication is that future IR architectures may increasingly favor statistical tokenization and language-agnostic representations, further reducing the need for language-dependent engineering in large-scale retrieval settings.