Inverse Cardinality Weighting for Text Categorization

• Inverse Cardinality Weighting is a term-weighting method that measures a term's rarity based on its presence in a limited number of predefined categories.
• The unsupervised tf.icf variant replaces traditional idf to enhance classification accuracy, notably improving macro and micro-F1 scores with similarity-based classifiers.
• The supervised formulation combines icf with relevance frequency, yielding superior performance in binary text categorization by emphasizing both category dispersion and class relevance.

Searching arXiv for the specified paper to ground the article in the cited source. arXiv search query: (Wang et al., 2010) Inverse cardinality weighting, in the cited formulation called inverse category frequency ($\mathrm{icf}$), is a term-weighting principle for text categorization that replaces document-level rarity with category-level rarity. For a term $t$, it measures how many predefined categories contain that term in the training set and assigns higher weight to terms occurring in fewer categories. In the paper "Inverse-Category-Frequency based supervised term weighting scheme for text categorization" (Wang et al., 2010), this idea appears in two forms: an unsupervised tf.icf scheme and an icf-based supervised term weighting scheme that combines $\mathrm{icf}$ with relevance frequency. The paper argues that the intuition behind $\mathrm{idf}$ for text categorization is less reasonable than in information retrieval, and reports that the proposed approaches are superior or comparable to six supervised term weighting schemes and three traditional schemes in terms of macro-F1 and micro-F1 (Wang et al., 2010).

1. Formal definition

Let $C$ be the set of all predefined categories, with $|C|$ the number of categories. For each term $t$, the category frequency is defined as

$$cf(t) = \left| \{\, c \in C : t \text{ occurs in at least one training document of category } c \,\} \right|.$$

The inverse category frequency of $t$ is then

$\mathrm{icf}(t)=\log\frac{|C|}{cf(t)}.$

Because $t$0, $t$1. In analogy with $t$2 in information retrieval, terms appearing in fewer categories get higher $t$3 (Wang et al., 2010).

Under this formulation, inverse cardinality weighting is the use of the cardinality of the set of categories containing a term rather than the number of documents containing that term. The cardinality being inverted is therefore category-level, not document-level. This gives the method a direct interpretation as idf at the category level, a characterization stated explicitly in the summary of the paper.

2. Unsupervised term weighting: tf.icf

The unsupervised construction substitutes $t$4 for the usual $t$5 factor and leaves $t$6 raw:

$t$7

For numerical stability, a smoothed variant is often used:

$t$8

The scheme favors terms occurring in fewer categories, rather than fewer documents (Wang et al., 2010). The paper positions this substitution as a direct response to the mismatch between information retrieval and text categorization: in information retrieval, $t$9 down-weights terms that occur in many documents, whereas in text categorization the relevant dispersion is argued to be across categories.

A common misconception is that the standard $\mathrm{icf}$0 rarity correction is automatically the most appropriate default for classification. The paper disputes this point on task-specific grounds: a term may occur in many documents yet remain highly discriminative if those documents belong to only one or two categories. In that setting, $\mathrm{icf}$1 preserves the contribution of category-specific terms that $\mathrm{icf}$2 may undervalue.

3. Supervised weighting: combining icf with relevance frequency

The supervised formulation assumes category-label information and distinguishes one positive category $\mathrm{icf}$3 from all others. For a term $\mathrm{icf}$4, define

• $\mathrm{icf}$5 as the number of positive-class training documents containing $\mathrm{icf}$6,
• $\mathrm{icf}$7 as the number of negative-class training documents containing $\mathrm{icf}$8.

Using the relevance frequency factor of Lan et al.,

$\mathrm{icf}$9

the paper combines $\mathrm{idf}$0 and $\mathrm{idf}$1 into a supervised term-weighting scheme. The recommended form multiplies $\mathrm{idf}$2 by a single log combining the two signals:

$\mathrm{idf}$3

Equivalently, it may be viewed as

$\mathrm{idf}$4

up to constant offsets in the logs (Wang et al., 2010).

This construction joins two different discriminative criteria. The $\mathrm{idf}$5 term increases when a term is more characteristic of the positive class, while $\mathrm{idf}$6 increases when a term is confined to fewer categories overall. The resulting weight is therefore sensitive both to positive-vs.-negative relevance and to dispersion across the full category set.

4. Rationale in text categorization

The paper’s central rationale is that, in text categorization, what really matters is whether a term is spread across many categories. A term that appears only in one or two categories is highly discriminative, even if it appears in many documents of those categories. On this view, $\mathrm{idf}$7 captures term dispersion across categories rather than term dispersion across documents (Wang et al., 2010).

This reframes rarity from a retrieval-oriented notion to a classification-oriented one. Replacing $\mathrm{idf}$8 by $\mathrm{idf}$9 focuses the model on category-level rarity, which is more directly relevant to classification. The paper further states that term weighting in text categorization should exploit category-level dispersion, and presents $C$0 as a simple, intuitive replacement for $C$1.

A plausible implication is that inverse cardinality weighting is especially well aligned with settings where within-class term frequency is high but cross-class spread is low. The paper’s reported results are consistent with that interpretation, particularly for similarity-based classifiers and imbalanced corpora, although the summary does not present a separate ablation isolating those conditions.

5. Experimental setting and reported results

The empirical evaluation covers three corpora:

• Reuters-21578 ("ModApte" split): single-label, 52 classes, skewed;
• 20 Newsgroups: 20 classes, roughly balanced;
• la12 (TREC-derived): multi-class, moderate size.

The classifiers are

• kNN with $C$2 and cosine similarity,
• Centroid-based classifier with cosine similarity,
• Support Vector Machine (LIBSVM) with linear and RBF kernels; parameters $C$3 chosen by cross-validation for multi-class, default for binary.

The evaluation metrics are Precision ($C$4), Recall ($C$5), $C$6, macro-averaged $C$7, and micro-averaged $C$8 (Wang et al., 2010).

For unsupervised weighting on multi-class text categorization, the paper compares $C$9, $|C|$0, $|C|$1, and $|C|$2. Across all three corpora and all four weighting functions, tf.icf consistently matches or outperforms tf.idf and pure tf and idf. The largest gains are reported with similarity-based classifiers, with more modest but still positive gains for SVM. On Reuters-52, centroid+tf.icf lifts macro-F1 by $|C|$3 pp over tf.idf, kNN by $|C|$4 pp, and SVM by about 0.3 pp.

For supervised weighting on binary text categorization, the paper compares nine schemes, including tf.rf, prob-based, $|C|$5, IG, and GR. The new icf + rf scheme yields the best or near-best macro-F1 and micro-F1 on Reuters-52, 20 Newsgroups and la12. Against tf.rf and the prob-based method, the icf-based scheme improves micro-F1 by up to $|C|$6 pp for SVM and especially shines with kNN and centroid. Two concrete examples given are: on Reuters-52 with SVM, icf-based achieves $|C|$7 micro-F1 vs. $|C|$8 for tf.rf; on la12 with centroid, $|C|$9 vs. $t$0.

6. Significance, recommendations, and limitations

The paper presents several direct implications. It states that unsupervised tf.icf is recommended as a new default for multi-class tasks, and that for binary tasks, combining $t$1 with $t$2 yields further gains. The reported benefits are largest with similarity-based classifiers (kNN, centroid) and in imbalanced settings; SVM gains are more modest, but SVM remains state-of-the-art overall (Wang et al., 2010).

The computational profile is also part of the argument. The summary states that there are no extra hyperparameters beyond those already in tf.idf/rf; $t$3 and category counts are precomputed once on the training set. In that sense, the proposed weighting is intended as a structurally simple replacement rather than a more elaborate modeling framework.

The limitations and extensions named in the summary are narrow and technical. It notes that further tuning of the offset constants, such as the “2+” inside logs, may yield small gains. It also identifies two directions left to future work: exploring $t$4 together with document-length normalization, and exploring it within probabilistic models (e.g. BM25-style). These remarks suggest that inverse cardinality weighting is best understood not as a complete classification architecture but as a reusable weighting component whose main contribution is to recast rarity at the category level.