IDF-Weighted Jaccard Similarity

• IDF-weighted Jaccard similarity is a metric that extends the classic index by weighing terms using TF–IDF to capture term informativeness.
• It employs sampling methods like Consistent Weighted Sampling and MinHash variants to approximate similarity efficiently in large-scale text collections.
• Applications include scalable text alignment and near-duplicate detection, offering improved performance and reduced computational overhead.

IDF-weighted Jaccard similarity generalizes classic Jaccard similarity to the context of weighted sets or real-valued vectors, weighting each element's contribution according to its informativeness—typically via inverse document frequency (IDF) or a related function. This weighted extension has become fundamental in information retrieval, large-scale text alignment, and similarity search, where simple binary set presence does not capture the relative importance of terms or features. Numerous sampling-based, sketching, and indexing frameworks have been specifically designed for its efficient approximation and scalable deployment in realistic, large-scale settings.

1. Mathematical Formulation

IDF-weighted Jaccard similarity is defined for weighted sets or vectors, most often in the context of TF–IDF vector representations. For two documents $A$, $B$ with term set $\Sigma$, the weight of a term $t$ in $A$ is $w(t,A) = \text{tf}(t, A)\cdot\text{idf}(t)$. The similarity is then

$$J_{w}(A,B)\;=\; \frac{\sum_{t\in A\cup B}\min\bigl(w(t,A),\,w(t,B)\bigr)} {\sum_{t\in A\cup B}\max\bigl(w(t,A),\,w(t,B)\bigr)}$$

This reduces to the classic Jaccard index $|A \cap B|/|A \cup B|$ when $w(t,\cdot)\in \{0,1\}$.

For probability-normalized weighted vectors (e.g., in (Ertl, 2019, Moulton et al., 2018)), the "probability Jaccard" similarity coincides with the above $$\sum_i \min(w_i^A, w_i^B)/\sum_i \max(w_i^A, w_i^B)$$ when weight vectors are normalized to sum to one.

In practical retrieval systems and experimental works, IDF weight is typically computed as $B$0 or with variants including smoothing, e.g., $B$1 (Bakiyev, 2022).

2. Efficient Estimation: Consistent Weighted Sampling and MinHash

Computing the exact weighted Jaccard similarity requires $B$2 time per pair, prohibitive for large collections. The standard solution is to approximate similarity via randomized sketching techniques with collision probabilities matching $B$3.

Consistent Weighted Sampling (CWS) and its variants generalize MinHash:

• ICWS (Improved CWS): For each nonzero-weight term $B$4, sample $B$5, $B$6, and $B$7. Given $B$8,
  • $B$9
  • $\Sigma$0
  • The tuple $\Sigma$1 is the hash value; the minimum $\Sigma$2 over $\Sigma$3 is the weighted minhash for $\Sigma$4 (Zhang et al., 30 Aug 2025, Wu et al., 2018).
• I$\Sigma$5CWS: Addresses statistical dependence issues in ICWS by separating the randomization for the indices that produce $\Sigma$6 and $\Sigma$7 (hence $\Sigma$8), thereby restoring the joint independence required for theoretical guarantees. This further improves the accuracy and theoretical soundness of the estimator (Wu et al., 2017, Wu et al., 2018).
• ProbMinHash: Efficiently computes $\Sigma$9-length signatures so that $t$0 for each $t$1, via both uncorrelated and correlated sampling regimes; offers an amortized time significantly better than previous approaches for large $t$2 (Ertl, 2019).

For all these methods, the unbiased estimator,

$t$3

converges to $t$4 with variance $t$5 for independent hash components.

3. Indexing and Substring Alignment: The MONO Framework

For alignment and near-duplicate detection among all substrings of a document, direct sketching of the $t$6 possible substrings is impractical. The MONO framework (Zhang et al., 30 Aug 2025) leverages the property that nearby substrings share the same CWS hash, partitioning the $t$7 grid of substrings into $t$8 "compact windows" (where $t$9 is maximum term frequency):

1. Window Generation: Uses active-key detection to only emit new hash minima when necessary, and partitions index space via monotonic skyline updates.
2. Indexing: Builds inverted lists for each hash value.
3. Query: For a query $A$0, retrieves candidate substrings via their indices and verifies similarity by aggregating over multiple hash functions.
4. Complexity: Index construction in $A$1 time and space; query latency proportional to the number of inverted list hits.

This achieves speedups up to $A$2 and index size reductions of $A$3 compared to prior algorithms for substring alignment (Zhang et al., 30 Aug 2025).

4. Algorithmic Variants and Practical Considerations

A spectrum of weighted MinHash-type algorithms exist, each targeting different runtime, memory, or statistical trade-offs:

Algorithm Category	Principle	Complexity
Quantization-based	Explicit binary expansion	$A$4
Active-index-based (CWS)	Active positions, skips	$A$5 (or $A$6)
ICWS/PCWS/I$A$7CWS	Direct analytical sampling	$A$8 per sketch
ProbMinHash	Bulk hash evaluation	$A$9

• 0-bit CWS: drops the $w(t,A) = \text{tf}(t, A)\cdot\text{idf}(t)$0 component for compactness, with negligible accuracy loss.
• Practical CWS (PCWS): reduces required random draws per index, increasing speed by ~20% (Wu et al., 2018).
• ProbMinHash 3/4: introduces dependencies between signature components for lower estimator variance (Ertl, 2019).
• All methods preserve unbiasedness in $w(t,A) = \text{tf}(t, A)\cdot\text{idf}(t)$1 estimation, but their memory and speed properties vary with the specifics of sampling, hash aggregation, and whether a dense or sparse representation is required.

Implementation guidance emphasizes stable computation in the log domain (to accommodate large IDF weights and low-frequency terms), careful seeding of random number generators for reproducibility, and typical sketch lengths $w(t,A) = \text{tf}(t, A)\cdot\text{idf}(t)$2–$w(t,A) = \text{tf}(t, A)\cdot\text{idf}(t)$3 to achieve sub-$w(t,A) = \text{tf}(t, A)\cdot\text{idf}(t)$4 mean squared error per estimate (Wu et al., 2018, Wu et al., 2017).

5. IDF-weighted Jaccard in Extended Text Similarity and Language-specific Applications

Applications in natural language processing and information retrieval often require domain-specific modifications. For instance, in the context of Kazakh-language documents, synonym expansion can be incorporated before TF–IDF weighting:

• Each position is assigned the conventional TF–IDF weight. If zero, look up synonyms and use any with nonzero TF–IDF in the document (Bakiyev, 2022).
• The similarity formula can appear variably; some works adopt direct $w(t,A) = \text{tf}(t, A)\cdot\text{idf}(t)$5, which preserves the nonnegativity and weighting of match terms.
• Experiments demonstrate small but measurable gains in recall and discriminability when domain-specific synonymy is integrated, especially for languages with limited existing resource coverage.

6. Optimality and Theoretical Guarantees

Sampling-based estimators achieve strong theoretical guarantees:

• Consistency: All CWS, ICWS, I$w(t,A) = \text{tf}(t, A)\cdot\text{idf}(t)$6CWS, and ProbMinHash estimators are scale-invariant: normalization of input vectors does not affect the collision probability.
• Pareto Optimality: P-MinHash is proven "maximally consistent," matching the optimal achievable collision probability for any sampling-based LSH (no other scheme offers higher collision rate for every pair without penalizing more-similar pairs) (Moulton et al., 2018).
• Tight Worst-case Bounds for Alignment: MONO establishes lower/upper bounds for substring indexing that are proven tight: any such scheme on text length $w(t,A) = \text{tf}(t, A)\cdot\text{idf}(t)$7 and max frequency $w(t,A) = \text{tf}(t, A)\cdot\text{idf}(t)$8 must use at least $w(t,A) = \text{tf}(t, A)\cdot\text{idf}(t)$9 space in the worst case (Zhang et al., 30 Aug 2025).

7. Empirical Performance and Recommended Settings

Empirical results across large corpora—books, web text, and language-specific news datasets—uniformly support the scaling laws and practical gains predicted theoretically:

• Up to $$J_{w}(A,B)\;=\; \frac{\sum_{t\in A\cup B}\min\bigl(w(t,A),\,w(t,B)\bigr)} {\sum_{t\in A\cup B}\max\bigl(w(t,A),\,w(t,B)\bigr)}$$0 speedups in index construction for alignment tasks (Zhang et al., 30 Aug 2025).
• Index sizes and query latencies improved by $$J_{w}(A,B)\;=\; \frac{\sum_{t\in A\cup B}\min\bigl(w(t,A),\,w(t,B)\bigr)} {\sum_{t\in A\cup B}\max\bigl(w(t,A),\,w(t,B)\bigr)}$$1 and up to $$J_{w}(A,B)\;=\; \frac{\sum_{t\in A\cup B}\min\bigl(w(t,A),\,w(t,B)\bigr)} {\sum_{t\in A\cup B}\max\bigl(w(t,A),\,w(t,B)\bigr)}$$2, respectively.
• Estimator variance tightly controlled; with $$J_{w}(A,B)\;=\; \frac{\sum_{t\in A\cup B}\min\bigl(w(t,A),\,w(t,B)\bigr)} {\sum_{t\in A\cup B}\max\bigl(w(t,A),\,w(t,B)\bigr)}$$3, subpercent MSE is routine (Wu et al., 2018).
• Larger vocabularies and sparse representation benefit particularly from I$$J_{w}(A,B)\;=\; \frac{\sum_{t\in A\cup B}\min\bigl(w(t,A),\,w(t,B)\bigr)} {\sum_{t\in A\cup B}\max\bigl(w(t,A),\,w(t,B)\bigr)}$$4CWS and PCWS, which maintain low collision estimator variance with fast runtime (Wu et al., 2017).
• Synonym-augmented TF–IDF Jaccard similarity increases mean similarity of genuinely related document pairs by several percentage points in language-specific settings (Bakiyev, 2022).

Altogether, IDF-weighted Jaccard similarity, with state-of-the-art sampling and indexing schemes, constitutes a mature, theoretically sound, and empirically validated approach for fine-grained, scale-aware similarity estimation in large text and feature vector collections.