--- title: Subword TF–IDF (STF-IDF) url: https://www.emergentmind.com/topics/subword-tf-idf-stf-idf type: topic --- # Subword TF–IDF (STF-IDF) Subword TF–IDF (STF-IDF) is a fully machine-learned information retrieval framework that replaces traditional word-based tokenization and associated language-specific heuristics with subword tokenization, thereby enabling robust and extensible multilingual search. STF-IDF eliminates manual preprocessing such as stop word removal and stemming, offering higher accuracy, particularly in multilingual and morphologically rich scenarios. The method inherently supports out-of-vocabulary (OOV) handling and achieves state-of-the-art retrieval performance across diverse languages [2209.14281]. ## 1. End-to-End STF-IDF Pipeline The STF-IDF methodology follows a structured pipeline: 1. **Subword Tokenizer Training**: A large multilingual corpus—using sources such as Wikipedia dumps for the top 100 character-based languages—serves as the training data. Language sampling probability for each language $\ell$ is $p_\ell = D_\ell / \sum_i D_i$, where $D_\ell$ is the monolingual data size. Temperature rescaling with $T=5$ is applied to upsample low-resource languages: $p'_\ell \propto p_\ell^{1/T}$. SentencePiece BPE is trained using character coverage $0.9995$ and vocabulary size $V = 128{,}000$. 2. **Document and Query Processing**: After training, the subword tokenizer segments input text (document or query) into subword tokens $s_1, s_2, ..., s_m$. Term frequencies $tf_{s,d}$ are computed for each subword. 3. **IDF Computation**: For document collection size $N$, document frequency $df_s = |\{d : tf_{s,d} > 0\}|$, and inverse document frequency $idf_s = \log (N / df_s)$. 4. **TF–IDF Weighting**: For each subword $s$ in document $d$, calculate $tfidf_{s,d} = tf_{s,d} \times idf_s$ (and analogously for queries). 5. **Normalization and Vector Construction**: Optionally, sublinear scaling $tf'_{s,d} = 1 + \log(tf_{s,d})$ is applied if $tf_{s,d} > 0$, zero otherwise. Each document/query is represented as a raw vector $v \in \mathbb{R}^V$ ($V$ is the subword vocab size), then normalized by the $L_2$ norm: $v \leftarrow v/\lVert v\rVert_2$. 6. **Retrieval by Cosine Similarity**: Similarity is computed as $sim(q,d) = v_q \cdot v_d$, with ranking by descending similarity. ## 2. Mathematical Definitions and Weighting STF-IDF formally defines its weighting scheme as follows: - **Term Frequency**: $tf_{s,d}$ is the count of subword $s$ in document $d$. - **Inverse Document Frequency**: $idf_s = \log \frac{N}{|\{ d : s \in d\}|}$, or smoothed variant $idf_s = \log \left(1 + \frac{N}{df_s}\right)$. - **TF–IDF Weight**: $tfidf_{s,d} = tf_{s,d} \times idf_s$. - **Sublinear Scaling** (optional): $tf'_{s,d} = 1 + \log(tf_{s,d})$ for $tf_{s,d} > 0$; $0$ otherwise. - **Vector Normalization**: All document/query vectors are normalized to the unit $L_2$ norm. This weighting structure obviates the need for frequency threshold-based stop word lists, since frequent tokens inherently receive low $idf$ and thus negligible TF-IDF importance. ## 3. Indexing, Retrieval, and Algorithmic Implementation The core algorithm can be summarized as follows: 1. **Tokenizer Training**: Train a SentencePiece BPE model with sampled multilingual data, character coverage $0.9995$, vocabulary size $128{,}000$, and temperature-based upsampling. 2. **Inverted Index Construction**: For each document $d$, subwords $s$ are counted to generate $tf_{s,d}$. Document frequencies $df_s$ are updated. For each subword $s$ in $d$, its posting list (storing $(d, w_{s,d})$ where $w_{s,d} = tfidf_{s,d}$) is maintained. 3. **Vectorization**: Sparse vectors $v_d$ are assembled for all documents, then $L_2$-normalized. 4. **Query Processing**: Tokenize query, compute $tf_{s,q}$, apply weighting, L2 normalize to form $v_q$. 5. **Similarity Search**: Candidate documents are selected as the union over subword posting lists. Each candidate $d$ is scored via $v_q \cdot v_d$ (cosine similarity). Top $k$ are returned. This structure allows efficient sparse retrieval over large multilingual corpora. ## 4. Elimination of Heuristics and Multilingual Capabilities STF-IDF eliminates traditional IR heuristics: - **Word Splitting**: Subword model learns meaningful token boundaries, removing the need for whitespace-based segmentation. - **OOV Handling**: Unseen words are decomposed into familiar subword units, ensuring all tokens are represented even for rare or novel forms. - **Stop Words**: Frequent subwords automatically receive low $idf$ scores, rendering explicit stop lists unnecessary. - **Stemming and Morphology**: Morphological relationships are implicitly captured, as variants share subword components. - **Unified Multilingual Vocabulary**: A single subword vocabulary represents all supported scripts and languages, inherently supporting code-switching and mixed-language text without language detection or separate pipelines. A plausible implication is that linguistic preprocessing is functionally subsumed by data-driven subword modeling, reducing complexity and support burden in multilingual IR systems. ## 5. Empirical Evaluation and Comparative Results The STF-IDF approach is empirically validated on XQuAD paragraph retrieval, a parallel dataset across 12 languages (en, es, de, el, ru, tr, ar, vi, th, zh, hi, ro) with 240 Wikipedia paragraphs and 1190 queries per language. Accuracy is defined as the proportion of queries for which the top-1 retrieved paragraph contains the answer. A summary of experimental results: | Method | English | Spanish | German | Greek | Russian | Turkish | Arabic | Vietnamese | Thai | Chinese | Hindi | Romanian | |----------------------------|----------|---------|--------|-------|---------|---------|--------|------------|-------|---------|-------|----------| | Word-based (baseline) | 84.2% | — | — | — | — | — | — | — | — | — | — | — | | Word + stop-word | 83.9% | — | — | — | — | — | — | — | — | — | — | — | | Word + stemming | 84.9% | — | — | — | — | — | — | — | — | — | — | — | | Word + stop + stem | 85.2% | — | — | — | — | — | — | — | — | — | — | — | | STF-IDF (no heuristics) | 85.4% | 85.8% | 84.9% | 81.3% | 82.9% | 80.1% | 77.1% | 84.5% | 83.5% | 82.4% | 80.9% | 85.0% | STF-IDF matches or exceeds the traditional TF-IDF baselines for English and achieves at least 80% accuracy in 10 additional languages, all without stop word removal or stemming [2209.14281]. ## 6. Key Implementation Parameters and Open Source Resources Critical implementation details for STF-IDF include: - **Subword Tokenizer**: SentencePiece BPE, $V = 128,000$ tokens, character coverage $0.9995$, temperature sampling $T=5$. - **Term Weighting**: Choices between raw $tf$ or sublinear scaling ($1 + \log tf$); always $idf = \log(N / df)$. - **Vector Normalization**: $L_2$ norm. - **Indexing**: Inverted index via postings lists of $(\text{doc\_id}, \text{weight})$ for each subword. - **Retrieval**: Cosine similarity, computed as sparse dot product over subword overlap. - **Reference Implementation**: Open-source code and a demo notebook are provided at https://github.com/artitw/text2text [2209.14281]. This configuration yields a generalizable, language-agnostic IR system that is efficient, robust to OOV input, and empirically state-of-the-art for multilingual paragraph retrieval. ## 7. Contextual Significance and Implications By obviating manual linguistic preprocessing and directly leveraging subword tokenization, STF-IDF provides a scalable solution to multilingual search challenges where traditional TF-IDF would require extensive hand-tuning per language. The architecture demonstrates that a single, unified, machine-learned pipeline can achieve strong cross-lingual retrieval performance, simplifying deployment and maintenance for global IR systems [2209.14281]. A plausible implication is that future IR architectures may increasingly favor statistical tokenization and language-agnostic representations, further reducing the need for language-dependent engineering in large-scale retrieval settings. Source: https://www.emergentmind.com/topics/subword-tf-idf-stf-idf