--- title: MinHash–Jaccard Criterion url: https://www.emergentmind.com/topics/minhash-jaccard-criterion type: topic --- # MinHash–Jaccard Criterion The MinHash–Jaccard criterion underpins a widely used family of techniques for efficient set and similarity search in large-scale data mining and information retrieval. It is centered on the equivalence between the probability that two sets, polygons, or weighted objects produce the same MinHash value, and their Jaccard (resemblance) similarity. This criterion immediately yields unbiased estimators for Jaccard similarity and enables locality-sensitive hashing (LSH) constructions, ANN search, and a variety of advanced sketching and sampling methodologies. The criterion has been extended from unweighted sets to weighted multisets, continuous measures, and even domain-specific objects such as polygons in spatial databases [2511.16576][1811.04633][1706.05698][2109.03337][1704.04370][1206.2082][2511.16576][1809.04052]. ## 1. Formal Definition of the MinHash–Jaccard Criterion Let $A$ and $B$ be two objects (typically finite sets, multisets, or measurable regions) in a universe $U$. Their Jaccard similarity is defined as $$ J(A, B) = \frac{|A \cap B|}{|A \cup B|}. $$ For polygons ($A, B \subseteq \mathbb{R}^2$), we have $$ J(A, B) = \frac{\operatorname{Area}(A \cap B)}{\operatorname{Area}(A \cup B)}. $$ A MinHash function $h$ maps sets to signatures such that $$ \Pr[h(A) = h(B)] = J(A, B). $$ For binary sets, a classical MinHash applies a random permutation $\pi$ to $U$ and sets $h_\pi(S) = \arg\min_{x \in S} \pi(x)$. For polygons, MinHash is instantiated by sampling uniform random points in a bounding rectangle and counting the number of samples until the first falls within the object [2511.16576]. ## 2. Collision Probability and Estimation Properties The core theoretical result is that for independent hash functions (or sampling seeds), the collision probability for any single MinHash coordinate equals the Jaccard similarity: $$ \Pr[h_i(A) = h_i(B)] = J(A, B), $$ for all $i$ [2511.16576][2109.03337][1811.04633]. Given $k$ independent MinHash components, define $X_i$ as the indicator that the $i$th coordinate matches, so $$ \widehat{J} = \frac{1}{k}\sum_{i=1}^k X_i $$ provides an unbiased estimator: $$ \mathbb{E}[\widehat{J}] = J(A, B), \quad \operatorname{Var}(\widehat{J}) = \frac{J(1-J)}{k}. $$ Concentration bounds follow from Hoeffding's or Chernoff's inequality. For any $\varepsilon > 0$, $$ \Pr\left[|\widehat{J} - J| \geq \varepsilon\right] \leq 2 \exp(-2k\varepsilon^2) $$ [2511.16576][1704.04370][1811.04633]. This property extends to generalized criteria, e.g., generalized Jaccard for weighted sets, polygonal intersection for spatial objects, and probability distribution analogues [1811.04633][1809.04052]. ## 3. Algorithmic Realizations: Classical, PolyMinHash, and Advanced Schemes ### Standard MinHash For $k$ hash functions, sketch each set $S$ as $(h_1(S), ..., h_k(S))$ with $h_j(S) = \min_{x \in S} h_j(x)$. Jaccard estimation reduces to fraction of coordinate-wise collisions [1811.04633]. ### PolyMinHash for Area-based Similarity For a polygon $P$ within bounding rectangle $B$: - For each of $k$ seeds, repeatedly sample $(x, y) \sim \mathrm{Uniform}(B)$ - Count attempts until a sampled point lands in $P$ - $h_i(P)$ is the attempt number for the $i$th seed Sigantures are compared via collision counts; collision probability matches area-based Jaccard [2511.16576]. See pseudocode in original for details. ### Weighted and Probabilistic Extensions Weighted MinHash via Consistent Weighted Sampling (CWS) and variants produces unbiased estimators of weighted Jaccard similarity: $$ J_w(w, w') = \frac{\sum_i \min(w_i, w'_i)}{\sum_i \max(w_i, w'_i)} $$ Implementations such as ICWS, PCWS, 0-bit CWS, and I$^2$CWS provide efficiency/accuracy trade-offs in large-scale settings [1811.04633]. Probability distribution MinHash generalizes the collision probability using the “maximally consistent sampling” criterion [1809.04052]. ### Variance-Reduced and Efficient Schemes - **SuperMinHash:** Uses permutation-based value shifts for variance reduction (factor $\alpha(m, u)<1$, can halve the variance when $|A \cup B| \ll m$) and achieves $O(1)$ amortized insertion time for large sets [1706.05698]. - **Circulant MinHash (C-MinHash):** Employs one or two permutations with circulant shifts so that the estimator is unbiased (two-permutation) or has negligible bias (one-permutation), while provably achieving lower variance than independent MinHash [2109.03337][2109.04595]. - **Dimension-Independent/Distributed:** Strategies such as DISCO (MinHashSampleMap) yield unbiased estimators while reducing communication in MapReduce settings by O(N·L·k) → O(D·k·log(Dk)) without loss in accuracy [1206.2082]. - **Fast Sketching:** “Mixture” sketches construct $t$-length signatures in $O(t\log t + |A|)$ time but guarantee the same collision and concentration properties as classical MinHash [1704.04370]. ## 4. Extensions: Weighted, Continuous, and Spatial Objects The MinHash–Jaccard criterion generalizes beyond finite sets: - **Weighted Sets (Generalized Jaccard):** Estimators and collision probability retain the form $\Pr[h(A) = h(B)] = J_w(w, w')$ for weighted objects via CWS and its improved variants. All key approaches, including quantization, active index sampling, and CWS, yield unbiased (or asymptotically unbiased) estimators, with per-coordinate complexities between $O(n)$ and $O(\sum_i \log w_i)$ depending on the method [1811.04633]. - **Probability Distributions:** The generalized collision formula for nonnegative vectors (including probability distributions) is $$ J(x, y) = \sum_{i:\,x_i>0,\,y_i>0} \frac{1}{\sum_j \max(x_j/x_i,\,y_j/y_i)} $$ which reduces to classical Jaccard when $x, y$ are binary. This is Pareto-optimal among LSHs based on sampling [1809.04052]. - **Spatial Domains:** PolyMinHash adapts MinHash to polygonal objects by replacing set membership with geometric inclusion and uniform permutation with uniform random sampling in $\mathbb{R}^2$ [2511.16576]. The collision probability still exactly recovers area-based Jaccard. ## 5. Empirical Analysis and Search Trade-offs Precision and runtime in MinHash–Jaccard-based systems hinge on signature length $k$: - Increasing $k$ reduces estimator variance and hence false positive rates, but increases computation and space costs [2511.16576][1706.05698][1811.04633]. - In PolyMinHash, with $m=1$, $70$–$75\%$ data pruning at high recall is achieved ($2$–$3\times$ speedup). With $m=3$, up to $89\%$ pruning and $4$–$5\times$ speedup at recall $0.88$–$0.93$. With $m=5$, $98\%$ pruning, up to $6$–$7\times$ speedup but recall drops to $0.60$–$0.71$ [2511.16576]. - SuperMinHash and C-MinHash consistently yield strictly lower estimator variance than classical MinHash. SuperMinHash achieves up to $2\times$ lower variance for $|A \cup B|