---
title: 'MinHash LSH: Efficient Similarity Search'
url: https://www.emergentmind.com/topics/minhash-lsh
type: topic
---

# MinHash LSH: Efficient Similarity Search

MinHash LSH (Locality Sensitive Hashing) is a pivotal randomized algorithmic framework for approximate similarity search in high-dimensional binary and set-valued data. At its core, MinHash LSH efficiently approximates the Jaccard similarity between sets or equivalent binary vectors via random hashing and enables sublinear-time retrieval of near-duplicate or highly similar pairs. The method has seen extensive application in deduplication, plagiarism detection, web-scale indexing, and is foundational in both theory and in multiple industrial-scale systems.

## 1. Fundamental Principles of MinHash LSH

MinHash LSH exploits the fact that the Jaccard similarity between two sets $X, Y$ is equal to the probability that a random permutation of the universe $U$ produces the same minimum element in both $X$ and $Y$:
$$
J(X, Y) = \frac{|X \cap Y|}{|X \cup Y|}\,, \quad \Pr[ h_\pi(X) = h_\pi(Y) ] = J(X, Y)
$$
where $h_\pi(X)$ denotes the smallest element of $X$ under permutation $\pi$ [2102.08942, 1411.3787].

For practical deployment, instead of true random permutations, one typically uses $k$ independent hash functions $h_j: U \rightarrow [M]$, and constructs a $k$-length signature:
$$
H(X) = (\hat{h}_1(X), \ldots, \hat{h}_k(X))\,, \quad \hat{h}_j(X) = \min \{ h_j(i) : i \in X \}
$$
The Jaccard similarity is then estimated as the fraction of positions where the two signatures coincide.

## 2. Classical Banding for Locality-Sensitive Hashing

The signature matrix is partitioned into $b$ bands of $r$ rows each ($k = br$). For each band $t$, the $r$-tuple is treated as a key; any two sets that agree in a band are candidate matches. The probability that two sets with Jaccard similarity $s$ become candidates is:
$$
P_{\mathrm{candidate}}(s) = 1 - (1 - s^r)^b
$$
This sharply amplifies the gap between high- and low-similarity pairs, enabling efficient approximate nearest neighbor search [2102.08942, 1603.07410].

## 3. Algorithmic Variants and Extensions

### 3.1 Asymmetric Minwise Hashing

Standard MinHash is biased towards smaller sets when set overlap (inner product) or containment is the desired measure. Asymmetric Minwise Hashing (MH-ALSH) removes this bias by transforming each set into longer binary vectors via asymmetric padding:
$$
P'(x) = [x; \, 1^{M-|x|};\, 0^{|x|}] ;\quad Q'(q) = [q; 0^M]
$$
and further through double composition, producing $P''(x), Q''(q) \in \{0,1\}^{D+2M}$. The Jaccard resemblance after transformation is
$$
R''(x, q) = \frac{\langle x, q \rangle}{2M - \langle x, q \rangle}
$$
making collision probability monotonic in set overlap. This yields strictly better theoretical guarantees for sublinear search in the sparse-binary regime and dominates other LSH methods for set containment [1411.3787].

### 3.2 Generalization to Probability Distributions

For positive vectors or probability distributions $x,y \in \mathbb{R}_+^n$, MinHash LSH has been extended to a generalized similarity measure:
$$
J(x, y) := \sum_{i: x_i>0, y_i>0} \frac{1}{\sum_{j=1}^n \max(x_j/x_i, y_j/y_i)}
$$
A sampled hash $H(\cdot)$ satisfies $\Pr[H(x)=H(y)] = J(x, y)$, and the definition reduces exactly to set-Jaccard in the binary case [1809.04052]. Two algorithms are provided:
- For sparse vectors: generate exponential weights and pick argmin.
- For dense/continuous distributions: A*-sampling over a proposal measure.

This extension is scale-invariant and more sensitive to support differences than earlier weighted MinHash schemes.

### 3.3 Sub-logarithmic Space: HyperMinHash

Standard MinHash requires $O(\log n)$ bits per hash. HyperMinHash reduces this to $O(\log\log n)$ by a floating-point encoding of the minimum ("exponent" plus "mantissa"), providing mergeability and enabling Jaccard estimation with $O(\epsilon^{-2}(\log\log n + \log(1/(t\epsilon))))$ space for target Jaccard $t$ and error $\epsilon$. HyperMinHash supports streaming updates and unions, handling sets of size up to $10^{19}$ with moderate memory on commodity hardware [1710.08436].

### 3.4 Variance Reduction: SuperMinHash

SuperMinHash introduces negative dependence among signature coordinates for further variance reduction. For union sizes $u < m$, the variance factor $\alpha(m,u)\ll1$ yields up to $2\times$ tighter confidence intervals for the Jaccard estimator and accelerates signature generation, especially when $|S| \gg m$ [1706.05698].

### 3.5 Braun-Blanquet Similarity and Chosen Path LSH

MinHash is suboptimal on equal-size sets or for Braun-Blanquet similarity. The Chosen Path scheme achieves a lower $\rho$ parameter by enforcing collision properties tuned directly for $B(x, y) = |x \cap y| / \max(|x|, |y|)$, outperforming MinHash, especially when all sets are size-$t$ [1612.07710].

## 4. Application Domains and Large-Scale Frameworks

MinHash LSH forms the basis of scalable deduplication and similarity search infrastructure:

- **Dataset Deduplication** : FED accelerates MinHash LSH dramatically on GPU clusters using a rolling 32-bit hash and pipelined kernels, maintaining consistent deduplication quality ($J>0.95$) at up to $100\times$ speedups compared to optimized CPU baselines. Hash evaluation and band grouping become $O(1)$ per shingle, supporting near-interactive deduplication of trillion-token corpora [2501.01046].
- **Internet-Scale Domain Search** : LSH Ensemble indexes massive sets using MinHash sketches and partitioned LSH tables, supporting set containment queries robust to domain size skew typical of web-scale data. Equi-depth partitioning approximates optimality for power-law size distributions [1603.07410].
- **Text, Graph, and Malware Clustering** : Application examples cover malware clustering and name deduplication, where MinHash (with banding) achieves order-of-magnitude speedups and high recall [2102.08942].
- **Cardinality and Unions** : HyperMinHash enables efficient join cardinality and Jaccard estimation for data streams and multi-set unions under memory constraints [1710.08436].

## 5. Theoretical Performance: Collision Probabilities and ρ-values

The efficiency of MinHash LSH is fundamentally characterized by the $(S_0, c S_0, p_1, p_2)$-sensitivity framework:
$$
\rho = \frac{\ln(1 / p_1)}{\ln(1 / p_2)}
$$
For MinHash under Jaccard, $p_1 = S_0$, $p_2 = c S_0$. For cosine similarity (binary vectors), crucial inequalities link the MinHash and SimHash probability curves:
$$
S^2 \leq R \leq \frac{S}{2 - S}
$$
yielding $\rho_{\text{MinHash}} (S_0, c) = \frac{\ln S_0^2}{\ln(c S_0/(2 - c S_0))}$, which is strictly smaller than the cosine-LSH $\rho$ of SimHash for high similarity [1407.4416].

For set containment and overlap, Asymmetric Minwise Hashing and Chosen Path schemes yield strictly improved $\rho$-values, particularly for sparse or equal-size input sets [1411.3787, 1612.07710].

## 6. Implementation, Parameter Selection, and Practical Considerations

Efficient implementation of MinHash LSH leverages the following key aspects:
- **Signature Construction**: For set $X$, $k$-length signatures via minimum hash values; computational cost is $O(k|X|)$.
- **Banding Parameters**: Choice of $b$ bands and $r$ rows offers a tunable trade-off between recall and false positives. Empirical settings are typically $k = 100\mbox{–}200$, $r = 5\mbox{–}10$, $b = k/r$ [2102.08942].
- **GPU Optimization**: Optimized GPU pipelines exploit rolling (non-cryptographic) hash functions and parallel signature computation [2501.01046].

For probability distributions, the P-MinHash algorithm for sparse or dense data ensures $O(\text{nnz}(x))$ or optimal expected iterations [1809.04052]. In all cases, for high-dimensional and sparse data, MinHash LSH and its variants are broadly preferred due to empirical and theoretical performance advantages.

## 7. Relationship to Other Similarity Measures and LSH Families

MinHash LSH specializes in Jaccard similarity and outperforms SimHash for cosine similarity on sparse/binary data. It is not optimal for all similarity metrics; for Braun-Blanquet similarity or strictly equal-size sets, dedicated schemes achieve strictly better performance. MinHash LSH remains foundational, however, and is extensible both to weighted/probability-vector inputs and compressed or mergeable sketches [1407.4416, 1809.04052, 1710.08436].

A plausible implication is that, unless attention is restricted to dense, real-valued data or specific alternative similarity metrics, MinHash LSH is canonical and usually optimal for large-scale approximate set similarity search, both in theory and in current system deployments.

Source: https://www.emergentmind.com/topics/minhash-lsh