c-TF-IDF: Class-Based TF-IDF Explained

• Class-based TF-IDF (c-TF-IDF) is a method that aggregates documents into classes to capture term distinctiveness across topic clusters.
• It uses modified logarithmic scaling on aggregated frequency counts to produce clear and interpretable topic-word distributions.
• The approach underpins neural topic modeling in BERTopic by enabling dynamic topic representation and statistical enrichment through class-level weighting.

Class-based TF-IDF (c-TF-IDF) denotes a family of term-weighting procedures in which the basic unit of weighting is no longer an individual document but a class-level aggregation, such as a topic cluster, label, or other partition of a corpus. In this setting, documents assigned to the same class are treated collectively, and term salience is defined by how well a term characterizes that class relative to the remaining classes. In BERTopic, c-TF-IDF is the mechanism that converts clusters obtained from contextual embeddings into interpretable topic-word distributions (Grootendorst, 2022). A complementary line of work places class-level weighting in a statistical testing framework by treating each class as a “mega-document” and relating class-based TF-ICF and c-TF-IDF to the negative logarithm of a one-tailed Fisher’s exact test $p$-value for term enrichment (Sheridan et al., 21 Jul 2025). A third, explicitly supervised perspective replaces inverse-frequency heuristics with class-distribution certainty measures such as troenpy-based Positive Class Frequency (PCF), yielding class-aware weights that are conceptually similar to c-TF-IDF but grounded in a different information-theoretic construction (Zhang, 2023).

1. Conceptual Basis

The classical TF-IDF scheme weights a term by combining its frequency within a document and its rarity across documents. In BERTopic, the same intuition is transferred from the document level to the class level: a class $c$ is defined as a cluster of documents concatenated into a single “class document,” and terms are weighted by their importance to that aggregate rather than to any one document (Grootendorst, 2022).

This shift changes the interpretive target. Standard TF-IDF asks which terms are important to a document. c-TF-IDF asks which terms are important to a class, topic, or cluster. In topic modeling, this means that the objective is not merely to recover frequent words, but to recover words that distinguish one cluster from others. The result is a topic-word distribution that can be used for top-word extraction, topic comparison, and downstream operations such as topic merging and temporal smoothing (Grootendorst, 2022).

A useful generalization appears in the Fisher-based account. There, c-TF-IDF is treated as the class-level analogue of document-level TF-IDF: one aggregates all documents in a class, defines class-level counts, and then asks whether a term is unusually concentrated in that class relative to the rest of the corpus. This suggests that c-TF-IDF is not only a representational heuristic but also a proxy for class-wise enrichment (Sheridan et al., 21 Jul 2025).

2. Formalizations and Weighting Equations

BERTopic introduces c-TF-IDF through a direct class-level modification of standard TF-IDF. Standard TF-IDF is recalled as

$$W_{t,d} = tf_{t,d} \cdot \log\left(\frac{N}{df_t}\right),$$

where $t$ is a term, $d$ is a document, $N$ is the number of documents in the corpus, $tf_{t,d}$ is the term frequency of $t$ in document $d$, and $df_t$ is the number of documents containing $c$0 (Grootendorst, 2022).

BERTopic then defines class-based TF-IDF as

$c$1

where $c$2 is a class or topic cluster, $c$3 is the frequency of term $c$4 in the concatenated class document, $c$5 is the frequency of $c$6 across all classes, and $c$7 is the average number of words per class. The logarithm is modified by adding 1 so that the weight remains positive (Grootendorst, 2022).

This BERTopic formulation differs from the class-level form discussed in the Fisher-based exposition. There, after aggregating documents by class, one defines class-level term counts

$c$8

and class lengths

$c$9

and then introduces a class-based TF-ICF,

$$W_{t,d} = tf_{t,d} \cdot \log\left(\frac{N}{df_t}\right),$$0

with $$W_{t,d} = tf_{t,d} \cdot \log\left(\frac{N}{df_t}\right),$$1 the total occurrences of term $$W_{t,d} = tf_{t,d} \cdot \log\left(\frac{N}{df_t}\right),$$2 in the whole collection and $$W_{t,d} = tf_{t,d} \cdot \log\left(\frac{N}{df_t}\right),$$3 the total number of term tokens in the collection (Sheridan et al., 21 Jul 2025).

The same exposition also states that c-TF-IDF is often written as

$$W_{t,d} = tf_{t,d} \cdot \log\left(\frac{N}{df_t}\right),$$4

where $$W_{t,d} = tf_{t,d} \cdot \log\left(\frac{N}{df_t}\right),$$5 is the number of classes and $$W_{t,d} = tf_{t,d} \cdot \log\left(\frac{N}{df_t}\right),$$6 is the number of classes in which term $$W_{t,d} = tf_{t,d} \cdot \log\left(\frac{N}{df_t}\right),$$7 appears (Sheridan et al., 21 Jul 2025). The literature therefore contains more than one formalization. BERTopic uses an aggregated frequency-based logarithmic factor involving $$W_{t,d} = tf_{t,d} \cdot \log\left(\frac{N}{df_t}\right),$$8 and $$W_{t,d} = tf_{t,d} \cdot \log\left(\frac{N}{df_t}\right),$$9, whereas the Fisher-based account distinguishes between c-TF-ICF, which is token-frequency based, and a common c-TF-IDF form based on inverse class frequency.

3. Role in BERTopic and Neural Topic Modeling

In BERTopic, c-TF-IDF is the final stage of a three-part pipeline: document embedding, document clustering, and topic representation. The embedding stage uses the Sentence-BERT framework. The resulting document embeddings are reduced in dimensionality with UMAP and then clustered with HDBSCAN, which models clusters using a soft-clustering approach and allows noise to be modeled as outliers (Grootendorst, 2022).

Once clusters are available, BERTopic constructs a class document for each cluster by concatenating all documents assigned to that cluster. c-TF-IDF is then computed on these class documents, producing a class-by-term matrix of weights. Terms with the largest $t$0 values define the topic representation. This procedure is meant to overcome the centroid-based perspective used in other embedding-based topic models, where representative words are taken to be those nearest to a cluster centroid in embedding space. BERTopic instead derives topic-word distributions directly from class-level term statistics (Grootendorst, 2022).

The same machinery supports topic reduction and dynamic topic modeling. BERTopic iteratively merges the least common topic with its most similar one by comparing c-TF-IDF representations. For temporal analysis, it computes a local representation

$t$1

where $t$2 indexes a timestep and $t$3 is the term frequency restricted to documents at that timestep. The global IDF component is reused, so local topic representations can be created without embedding and clustering documents again. For linearly evolving topics, each topic-time c-TF-IDF vector is L1-normalized and then averaged with the previous timestep’s normalized vector (Grootendorst, 2022).

Empirically, BERTopic is reported to generate coherent topics and remain competitive across benchmarks involving classical models and more recent clustering-based approaches. The paper further notes that c-TF-IDF supports metadata-conditioned topic representations beyond time, including author and journal (Grootendorst, 2022).

4. Statistical Interpretation Through Fisher’s Exact Test

A statistical justification of class-based weighting is provided by the Fisher-based account of TF-IDF. At the document level, the paper constructs a $t$4 contingency table for a term $t$5 and document $t$6, with counts for occurrences of $t$7 inside and outside the document. It then considers the upper-tail hypergeometric $t$8-value for testing whether the term is over-represented in that document. The weight

$t$9

is therefore a significance score: larger values correspond to smaller $d$0-values and stronger over-representation (Sheridan et al., 21 Jul 2025).

The main theoretical result is that, under mild regularity conditions, TF-ICF is the dominant component of $d$1. Under additional idealized assumptions, TF-IDF itself is connected to $d$2, and in a limiting case it converges to TF-IDF in the limit of an infinitely large document collection (Sheridan et al., 21 Jul 2025).

This document-level derivation is lifted to the class level by treating each class as one aggregated document. For term $d$3 and class $d$4, one builds an analogous contingency table using $d$5, $d$6, $d$7, and $d$8, computes a class-level one-sided Fisher $d$9-value, and defines

$N$0

Under class-level analogues of the same conditions, the paper states that

$N$1

so c-TF-ICF can be interpreted as an approximation to a class-wise enrichment statistic (Sheridan et al., 21 Jul 2025).

This interpretation is especially important because it clarifies what a high c-TF-IDF-like score means: the term is not merely frequent in a class, but unexpectedly concentrated there relative to a null model of random allocation of term tokens.

5. Relation to Other Class-Aware Weighting Schemes

A distinct but closely related class-aware line is developed through troenpy, a quantity defined as the expectation of Positive Information,

$N$2

in deliberate contrast to Shannon entropy, which is the expectation of Negative Information (Zhang, 2023). Troenpy is described as measuring the certainty, commonness, and similarity of the underlying distribution.

On this basis, the paper defines Positive Class Frequency (PCF) for a class-label distribution $N$3 as

$N$4

and then, for a term $N$5, uses the difference $N$6 as a term-level class-based weight reflecting the certainty gain induced by the presence of $N$7. Multiplying this by a smoothed classical IDF,

$N$8

yields $N$9, and TF-PI is used as a supervised alternative to TF-IDF (Zhang, 2023).

Although that paper does not mention c-TF-IDF explicitly, it characterizes PCF and Expected Class Information Bias (ECIB) as class-aware weighting schemes. The conceptual overlap is direct: c-TF-IDF and PCF both use aggregated class statistics to determine whether a term is characteristic of particular classes. The difference is that c-TF-IDF is an inverse-frequency construction, whereas PCF is motivated by a certainty gain in the class-label distribution, and ECIB is described as the expected odds ratio of the information quantity entropy and troenpy (Zhang, 2023).

The empirical claims in that work reinforce the practical importance of class-aware weighting. TF-PI uniformly outperforms TF-IDF across all datasets in the reported $tf_{t,d}$0NN setting, with an average overall error reduction of $tf_{t,d}$1 and a $tf_{t,d}$2 error reduction on R8. In logistic regression, adding ECIB features and binary term features to TF-PI further reduces errors on most datasets. A plausible implication is that c-TF-IDF belongs to a broader family of class-conditional weighting methods in which label structure, not only document rarity, is the key signal (Zhang, 2023).

6. Extensions, Misconceptions, and Limitations

A common simplification is to treat c-TF-IDF as nothing more than “ordinary TF-IDF after clustering.” BERTopic’s formulation is more specific than that. It concatenates documents by cluster, computes class-level term frequencies, and uses $tf_{t,d}$3 rather than the classical $tf_{t,d}$4. The change is not only procedural but semantic: the target of weighting is class distinctiveness, not document distinctiveness (Grootendorst, 2022).

Another misconception is that class specificity can be captured solely by counting in how many classes a term appears. The Fisher-based exposition provides a counterexample: if a term appears in both classes but is heavily concentrated in one, a class-count factor such as $tf_{t,d}$5 can be zero, whereas c-TF-ICF and the direct Fisher score remain large because they capture concentration rather than mere presence or absence. This shows that c-TF-IDF and c-TF-ICF are not interchangeable in all regimes (Sheridan et al., 21 Jul 2025).

The main limitations are also explicit in the literature. In BERTopic, the topic representation itself does not directly account for contextual information, because the final topic-word representation is generated from bags-of-words. The words in a topic representation merely sketch the importance of words in a topic, and redundant terms may remain; Maximal Marginal Relevance is suggested as a possible remedy but is not explored. BERTopic also inherits a single-topic assignment assumption from the clustering stage, even though HDBSCAN soft clustering can approximate topic distributions (Grootendorst, 2022).

The Fisher justification depends on idealized assumptions for its cleanest equivalences: equal lengths, strong burstiness patterns, sufficiently large counts, and globally rare terms. The paper also notes finite-sample issues, the roughness of approximations for very small counts, and the simplicity of the null model. It suggests generalized hypergeometric models, Bayesian Fisher variants, and alternative tests such as likelihood ratio and $tf_{t,d}$6 as possible extensions (Sheridan et al., 21 Jul 2025).

Taken together, these results position c-TF-IDF as a central class-level weighting idea rather than a single invariant formula. In neural topic modeling it is an interpretable topic-representation mechanism; in the Fisher framework it is an approximate enrichment statistic; and in troenpy-based work it is one member of a broader class of label-aware weighting schemes that explicitly model certainty gain and class bias.