---
title: Class-Based TF-IDF (c-TF-IDF)
url: https://www.emergentmind.com/topics/class-based-tf-idf-c-tf-idf
type: topic
---

# Class-Based TF-IDF (c-TF-IDF)

Class-based TF-IDF (c-TF-IDF) is a term weighting scheme for topic modeling that generalizes standard TF-IDF from single documents to clusters, or "classes," of semantically similar documents. This approach was introduced in the context of the BERTopic model, where documents are first embedded using transformer-based models, clustered based on semantic similarity, and then described by high-scoring terms extracted using c-TF-IDF. The central aim of c-TF-IDF is to create coherent and distinctive topic representations by measuring term salience within a cluster relative to its prevalence across all clusters [2203.05794].

## 1. Formal Definition and Mathematical Formulation

c-TF-IDF operates over a collection of $D$ documents partitioned into $C$ clusters (topics), with $V$ the vocabulary of unique terms. Key definitions:

- $tf_{t, d}$: raw count of term $t$ in document $d$
- $tf_{t, c} = \sum_{d \in \mathcal{C}_c} tf_{t, d}$: aggregated frequency of $t$ in cluster $c$
- $T_c = \sum_{t\in V} tf_{t, c}$: total tokens in cluster $c$
- $A = \frac{1}{C}\sum_{c=1}^C T_c$: mean tokens per cluster
- $tf_t = \sum_{c=1}^C tf_{t, c}$: total occurrence of $t$ over corpus

The c-TF-IDF score for term $t$ in cluster $c$ is:

$$
W_{t,c}^{\mathrm{c\text{-}TF\text{-}IDF}} = tf_{t,c} \cdot \log \left( 1 + \frac{A}{tf_t} \right)
$$

No additional smoothing is applied, and the log ensures non-negativity. Optionally, each class vector $W_{:,c}$ may be $L_1$-normalized.

## 2. Conceptual Motivation and Distinction from Standard TF–IDF

Standard TF-IDF quantifies how important a term is to a single document versus the entire corpus, using inverse document frequency (idf): $idf_t = \log(N/df_t)$, where $df_t$ is the number of documents containing $t$. In c-TF-IDF, the focus shifts to measuring a term’s prominence within a cluster of documents (pseudo-document), using global frequency $tf_t$ in lieu of document frequency, and scaling by average class length for class normalization.

The table below summarizes the main contrasts:

| Quantity                        | Standard TF-IDF             | c-TF-IDF                                      |
|----------------------------------|-----------------------------|------------------------------------------------|
| Document unit                   | single doc $d$              | cluster/pseudo-doc $c$                         |
| Term frequency                  | $tf_{t, d}$                 | $tf_{t, c} = \sum_{d \in c} tf_{t, d}$         |
| Inverse frequency               | $\log(N/df_t)$              | $\log(1 + A/tf_t)$                             |
| Normalization                   | $L_2$ or $L_1$ (postweight) | optional $L_1$ (pre-smoothing/dynamic models)   |

c-TF-IDF thus emphasizes terms that are frequent in one cluster yet infrequent globally, supporting highly specific topic descriptions.

## 3. Algorithmic Outline

The extraction of topic representations using c-TF-IDF in the BERTopic pipeline follows these steps:

1. **Embedding**: Each document $d$ is embedded with a pre-trained transformer model (e.g., SBERT).
2. **Dimensionality Reduction**: Embeddings are reduced with UMAP.
3. **Clustering**: HDBSCAN assigns cluster labels $\ell_d \in \{1\ldots C\}$.
4. **Term Aggregation**: For each cluster $c$:
    - Compute $tf_{t,c}$ by summing term counts for all $d$ with $\ell_d=c$.
    - Compute cluster size $T_c$.
5. **Global Stats**: Calculate $A = \frac{1}{C}\sum_{c=1}^C T_c$ and $tf_t = \sum_{c=1}^C tf_{t,c}$.
6. **Weight Calculation**: Compute $W_{t,c} = tf_{t,c} \cdot \log(1 + A/tf_t)$ for all $t,c$.
7. **(Optional) Normalization**: Apply $L_1$ normalization to each $W_{:,c}$.
8. **Topic Representation**: For each $c$, select the top $n$ terms by $W_{t,c}$ as topic descriptors.

Key hyperparameters include $top\_n$ (topic words per topic), $min\_cluster\_size$, and minimum term frequency thresholds [2203.05794]. Preprocessing recommendations include stopword removal, non-alphabetic token filtering, and lemmatization.

## 4. Integration in BERTopic and Practical Recommendations

Within BERTopic, c-TF-IDF serves as the final stage to derive representative keywords for each topic discovered via clustering of semantic embeddings. The procedure leverages:

- Pre-trained SBERT variants (e.g., all-mpnet-base-v2) for embeddings.
- UMAP for dimensionality reduction.
- HDBSCAN for density-based clustering with adjustable granularity (typical min_cluster_size: 5–15).
- Preprocessing steps to reduce noise and sparsity, including lemmatization and rare-term filtering.
- Post-processing methods such as merging small topics by c-TF-IDF vector proximity for more robust topic sets.

In dynamic topic modeling, global idf terms can be reused across time slices, while $tf_{t,c}$ is recalculated to reflect evolving clusters. Smoothing of topic vectors is accomplished by temporal averaging of normalized c-TF-IDF scores.

## 5. Empirical Evaluation of c-TF-IDF

Empirical results reported for c-TF-IDF in BERTopic demonstrate strong topic coherence across multiple benchmarks:

- On 20 Newsgroups: NPMI = 0.166 (vs. LDA 0.058, NMF 0.089, Top2Vec–SBERT 0.068, CTM 0.096)
- On BBC News: 0.167 (vs. LDA 0.014, NMF 0.012, Top2Vec–SBERT –0.027, CTM 0.094)
- On Trump tweets: 0.066 (vs. LDA –0.011, NMF 0.009, Top2Vec–Doc2Vec –0.169, CTM 0.009)

Across a range of embedding models (USE, Doc2Vec, MiniLM, MPNET), c-TF-IDF maintains stability in both coherence and topic diversity. In dynamic topic modeling scenarios, BERTopic with c-TF-IDF outperforms LDA Sequence, achieving higher NPMI coherence (for example, 0.079 vs. 0.009 on Trump data).

A plausible implication is that clustering contextual embeddings followed by c-TF-IDF scoring produces interpretable topics that are more coherent than those produced by classical topic models, and that this framework is robust to changes in document embedding models [2203.05794].

## 6. Applicability, Limitations, and Usage Recommendations

Practical usage of c-TF-IDF centers on topic modeling pipelines seeking fast, coherent, and semantically consistent topic representations from clustered embeddings. Key recommendations include:

- Selecting embedding models fine-tuned for semantic similarity.
- Keeping clustering hyperparameters fixed for comparability.
- Employing aggressive stopword, noise, and rare-term filtering for stability.
- Optionally merging undersized topics iteratively based on c-TF-IDF similarity.
- Leveraging c-TF-IDF in dynamic topic modeling by persistent idf terms and time-local $tf_{t,c}$ computation.

No evidence is reported regarding controversies or significant conceptual limitations specific to the c-TF-IDF approach in topic modeling; however, its dependence on high-quality document embeddings and clustering quality underlines the importance of upstream components. This suggests that effectiveness may be constrained by the suitability of embedding and clustering choices for a given corpus or application context [2203.05794].

Source: https://www.emergentmind.com/topics/class-based-tf-idf-c-tf-idf