---
title: IDF-Weighted Keyword Overlap
url: https://www.emergentmind.com/topics/idf-weighted-keyword-overlap
type: topic
---

# IDF-Weighted Keyword Overlap

IDF-weighted keyword overlap refers to the quantification and approximation of semantic similarity between texts by weighting n-gram or token overlap according to the inverse document frequency (IDF) of each token. This approach generalizes the classic unweighted Jaccard similarity, integrating term frequency-inverse document frequency (TF-IDF) schemes prominent in information retrieval, and lies at the core of advanced text alignment and near-duplicate detection frameworks. Algorithms that accurately and efficiently estimate this measure for all substrings of large corpora enable robust detection of semantic similarity, down-weighting spurious matches on frequent vocabulary and emphasizing rare, information-carrying terms [2509.00627].

## 1. Definition of IDF-Weighted Jaccard Similarity

IDF-weighted Jaccard similarity extends the classical set-based Jaccard metric by accounting for both token frequency and global discriminativity. Given token sets $A$ and $B$, each token $t$ is assigned an inverse document frequency

$$\operatorname{idf}(t) = \log\left(\frac{N}{N_t}\right)$$

where $N$ is the corpus size and $N_t$ the number of documents containing $t$. For a segment $A$,

$$w_t^A = \operatorname{tf}(t, A) \cdot \operatorname{idf}(t)$$

with $\operatorname{tf}(t, A)$ denoting the term frequency of $t$ in $A$. The IDF-weighted Jaccard similarity is then

$$J_w(A, B) = \frac{\sum_{t\in A\cap B}\min(w_t^A, w_t^B)}{\sum_{t\in A\cup B} \max(w_t^A, w_t^B)}$$

This metric smoothly discounts ubiquitous tokens and accentuates rare, content-discriminative vocabulary. Because IDF weights are corpus-level constants, they can be incorporated into precomputed token weightings per segment [2509.00627].

## 2. Consistent-Weighted Sampling (CWS) for Efficient Estimation

Classic min-hash sketches estimate unweighted Jaccard similarity and are insufficient for the weighted case. To extend to $J_w$, improved consistent-weighted sampling (CWS) [Ioffe '10] is utilized. For each token $t$, CWS associates three shared random variables: $r_t \sim \operatorname{Gamma}(2,1)$, $c_t \sim \operatorname{Gamma}(2,1)$, and $\beta_t \sim \operatorname{Uniform}(0,1)$. For segment $A$ and $w_t^A$,

\[
\begin{align*}
y_t &= \exp\left\{ r_t \cdot (\lfloor \log w_t^A / r_t + \beta_t \rfloor - \beta_t)\right\} \\
a_t &= c_t / (y_t \cdot \exp(r_t))
\end{align*}
\]

The sketch $h(A)$ is taken as the triple $(t^*, y^*, a^*)$ where $a^* = \min_t a_t$. With $k$ independent hash sketches,

\[
\mathbb{P}[ h_i(A) = h_i(B)] = J_w(A, B)
\]
\[
\widehat{J}_w(A, B) = \frac{1}{k} \sum_{i=1}^k 1\{ h_i(A) = h_i(B) \}
\]

Thus, the CWS-based sketch enables unbiased, sublinear estimation of IDF-weighted overlap in $O(1)$ time per nonzero token [2509.00627].

## 3. Indexing and Querying All Substrings: MONO Framework

Indexing all $O(n^2)$ substrings under weighted Jaccard is intractable by exhaustive representation. The MONO framework efficiently builds and queries such indices using "compact windows": axis-aligned rectangles $[a..b]\times[c..d]$ of substring intervals $T[a..b]$ through $T[c..d]$ sharing identical CWS hash values.

Active keys are constructed for each token $t$ by processing all pairs $(p, q)$, $T[p]=T[q]=t$, considering the number of occurrences $x$ in $T[p..q]$. Only pairs $(p,q)$ where the minimal hash is achieved for the first $x$ occurrences are retained. The number of active keys is $O(n + n\log f)$ in expectation, with $f$ as the maximal token frequency.

Compact windows are generated by sorting active keys and maintaining a 2D skyline to form non-overlapping rectangular partitions. For $k$ hashes, $k$ sets of windows are maintained in inverted indices, mapping hash values to tuples $(\text{T-ID}, a, b, c, d)$ [2509.00627].

## 4. Optimal Index Construction and Theoretical Bounds

The MONO partitioning algorithm produces an expected $O(n + n\log f)$ windows per text, which is theoretically tight. Any hash-based method must produce at least $\Omega(n + n\log f)$ distinct windows in the worst case—demonstrated via coupon-collector arguments on adversarial texts consisting of repeated tokens in blocks of length $f$—to guarantee sensitivity to all distinct weighted min-hash segmentations [2509.00627].

This optimality ensures subquadratic space and index construction costs, a substantial improvement over previous approaches requiring $\Omega(nk)$ or worse.

## 5. Threshold Queries and Retrieval

The threshold-query mechanism proceeds in three steps:

1. Compute the $k$-sketch of query $Q$ using CWS.
2. Retrieve $k$ inverted lists for the corresponding hash values $v_1,\dots,v_k$.
3. Run a two-dimensional plane-sweep (IntervalScan) algorithm over the compact windows to identify all substrings $(i,j)$ covered by at least $\lceil k\theta \rceil$ lists—i.e., substrings whose estimated similarity exceeds threshold $\theta$.

Each matching substring $T[i..j]$ is reported with polylogarithmic overhead in $n$, plus the output size, ensuring fast query execution suitable for large-scale duplicate detection and semantic search [2509.00627].

## 6. Empirical Results and Comparative Performance

On real-world corpora (PAN, OpenWebText), MONO achieves up to $26\times$ faster index construction than the AllAlign MultiSet-MinHash method, yields indices up to $30\%$ smaller, and improves query latency by up to $3\times$. The practical $f$ values—maximum local term frequency per text—are typically small, making the extra $n\log f$ factor minor in operational settings [2509.00627].

| Approach                | Index Construction | Index Size   | Query Latency |
|-------------------------|-------------------|--------------|---------------|
| MONO (CWS + windows)    | up to 26× faster  | 30% smaller  | up to 3× faster|
| MultiSet-MinHash (prev) | —                 | —            | —             |

## 7. Context and Applications

IDF-weighted keyword overlap provides a principled, efficiently computable method for near-duplicate alignment and similarity search under realistic IR-weighted metrics. By combining TF-IDF weighting, consistent-weighted sampling sketches, and an optimal partitioning and index scheme, the MONO framework establishes the first guaranteed subquadratic solution for substring alignment in large corpora under weighted Jaccard similarity [2509.00627]. A plausible implication is broader applicability to tasks in document forensics, plagiarism detection, and semantic clustering, where approximate set similarity under importance-weighted scoring is critical.

Source: https://www.emergentmind.com/topics/idf-weighted-keyword-overlap