---
title: 'b-Bit MinHash: Compressed Minwise Hashing'
url: https://www.emergentmind.com/topics/b-bit-minhash
type: topic
---

# b-Bit MinHash: Compressed Minwise Hashing

b-Bit MinHash, or **b-bit minwise hashing**, is a compressed form of minwise hashing for estimating set resemblance and for building similarity-preserving representations of massive sparse binary data. For sets \(S_1,S_2\subseteq \Omega=\{0,1,\dots,D-1\}\), its target similarity is the **resemblance** or Jaccard similarity,
\[
R(S_1,S_2)=\frac{|S_1\cap S_2|}{|S_1\cup S_2|}=\frac{a}{f_1+f_2-a},
\]
with \(f_1=|S_1|\), \(f_2=|S_2|\), and \(a=|S_1\cap S_2|\). Classical MinHash stores the minimum under each random permutation; b-Bit MinHash stores only the lowest \(b\) bits of each such minimum, and then corrects analytically for the extra collisions introduced by truncation. The 2009 paper that established the method framed it as a storage-efficient alternative to ordinary MinHash, while later work showed that the same representation induces a valid positive definite kernel, an explicit sparse linear feature map, and effective large-scale learning algorithms for linear SVM and logistic regression [0910.3349] [1105.4385].

## 1. Origins and core construction

Classical minwise hashing applies a random permutation \(\pi:\Omega\to\Omega\) and records
\[
z_1=\min(\pi(S_1)),\qquad z_2=\min(\pi(S_2)).
\]
Its defining identity is
\[
\Pr\bigl(z_1=z_2\bigr)=R.
\]
With \(k\) independent permutations \(\pi_1,\dots,\pi_k\), the standard estimator is
\[
\hat R_M=\frac{1}{k}\sum_{j=1}^k 1\{\min(\pi_j(S_1))=\min(\pi_j(S_2))\},
\]
with variance
\[
\operatorname{Var}(\hat R_M)=\frac{1}{k}R(1-R).
\]
The difficulty is storage: in the standard implementation each minhash is typically stored as a 64-bit value, so large \(k\) and large collections make sketch storage and repeated I/O expensive [0910.3349].

b-Bit MinHash changes only the stored representation. If \(e_{1,i}\) and \(e_{2,i}\) are the \(i\)-th lowest bits of \(z_1\) and \(z_2\), then instead of testing full equality of \(z_1\) and \(z_2\), it tests whether the lowest \(b\) bits match. This reduces storage from 64 bits per sample to \(b\) bits per sample, a factor of \(64/b\) reduction [1106.0967]. The original theoretical paper further quantified the tradeoff in the sparse least favorable regime \(r_1,r_2\to 0\): for resemblance at least \(0.5\), using \(b=1\) may reduce storage space by at least a factor of \(21.3\) relative to 64-bit MinHash, or \(10.7\) relative to 32-bit MinHash [0910.3349].

The method was developed for **binary** or set-valued data in extremely high-dimensional sparse regimes. That specialization is essential: its most direct similarity target is resemblance, and much of its later learning theory, empirical evaluation, and systems engineering is explicitly about sparse binary vectors, shingled text, and similarly structured data [1105.4385] [1205.2958].

## 2. Collision law, unbiased estimation, and variance

The central theoretical result of b-Bit MinHash is that the probability of a \(b\)-bit collision remains an affine function of resemblance. Let
\[
z_1=\min(\pi(S_1)),\qquad z_2=\min(\pi(S_2)),
\]
and define \(r_1=f_1/D\), \(r_2=f_2/D\). For large \(D\),
\[
P_b=\Pr\left(\prod_{i=1}^b 1\{e_{1,i}=e_{2,i}\}=1\right)=C_{1,b}+(1-C_{2,b})R,
\]
where
\[
C_{1,b}=A_{1,b}\frac{r_2}{r_1+r_2}+A_{2,b}\frac{r_1}{r_1+r_2},\qquad
C_{2,b}=A_{1,b}\frac{r_1}{r_1+r_2}+A_{2,b}\frac{r_2}{r_1+r_2},
\]
with
\[
A_{1,b}=\frac{r_1(1-r_1)^{2^b-1}}{1-(1-r_1)^{2^b}},\qquad
A_{2,b}=\frac{r_2(1-r_2)^{2^b-1}}{1-(1-r_2)^{2^b}}.
\]
Thus truncation does not destroy identifiability of \(R\); it changes only the intercept and slope of the collision law [0910.3349] [1106.0967].

Using \(k\) independent permutations, the empirical \(b\)-bit collision rate is
\[
\hat P_b=\frac{1}{k}\sum_{j=1}^k \left\{\prod_{i=1}^b 1\{e_{1,i,\pi_j}=e_{2,i,\pi_j}\}\right\},
\]
and the unbiased resemblance estimator is
\[
\hat R_b=\frac{\hat P_b-C_{1,b}}{1-C_{2,b}}.
\]
Its variance is
\[
\operatorname{Var}(\hat R_b)=\frac{1}{k}\frac{P_b(1-P_b)}{(1-C_{2,b})^2}
=\frac{1}{k}\frac{\left[C_{1,b}+(1-C_{2,b})R\right]\left[1-C_{1,b}-(1-C_{2,b})R\right]}{(1-C_{2,b})^2}.
\]
As \(b\to\infty\), \(A_{1,b},A_{2,b},C_{1,b},C_{2,b}\to 0\), so \(P_b\to R\) and b-Bit MinHash recovers ordinary MinHash [0910.3349].

In the sparse limit emphasized repeatedly in later work, the collision law simplifies to
\[
P_b=\frac{1}{2^b}+\left(1-\frac{1}{2^b}\right)R,
\]
so the extra collision probability becomes the intuitive random-match floor \(2^{-b}\) [1108.3072]. The large-\(D\) approximation is a standing assumption of the theory, but the later learning paper notes that the approximation is highly accurate even for fairly small \(D\) [1106.0967].

## 3. Positive-definite kernel and explicit feature map

A major later development was the realization that b-Bit MinHash is not only a similarity estimator but also a valid kernel construction. For sets \(S_1,\dots,S_n\), let \(z_i=\min(\pi(S_i))\) under one permutation. The following matrices are all positive definite:

1. the resemblance matrix \(\mathbf R\), with
   \[
   R_{ij}=\frac{|S_i\cap S_j|}{|S_i|+|S_j|-|S_i\cap S_j|},
   \]
2. the minwise hashing matrix \(\mathbf M\), with
   \[
   M_{ij}=1\{z_i=z_j\},
   \]
3. the b-bit minwise hashing matrix \(\mathbf M^{(b)}\), with
   \[
   M^{(b)}_{ij}=\prod_{t=1}^b 1\{e_{i,t}=e_{j,t}\}.
   \]

The proof is constructive. For b-Bit MinHash,
\[
M^{(b)}_{ij}=\sum_{t=0}^{2^b-1}1\{z_i\bmod 2^b=t\}\,1\{z_j\bmod 2^b=t\},
\]
so each object is represented by a one-hot vector over \(2^b\) categories, and \(\mathbf M^{(b)}\) is a Gram matrix. The resemblance matrix is positive definite because \(R_{ij}=\Pr(M_{ij}=1)=E(M_{ij})\), and expectation preserves positive definiteness [1105.4385].

This observation yields the explicit linearization used in large-scale learning. For each of the \(k\) retained \(b\)-bit values \(t_s\in\{0,\dots,2^b-1\}\), encode \(t_s\) as a one-hot vector in \(\mathbb R^{2^b}\), then concatenate the \(k\) blocks. The resulting expanded feature map has dimension
\[
2^b\times k
\]
and exactly \(k\) ones. Inner products in that expanded space count \(b\)-bit collisions and therefore approximate resemblance. This is the basis for integrating b-Bit MinHash with linear SVM, logistic regression, and related linear learners while approximating a nonlinear resemblance kernel [1106.0967].

The resulting preprocessing-and-learning pipeline is simple: start from sparse binary vectors, generate \(k\) minhashes, retain only the lowest \(b\) bits, store the compact representation, expand each code to a one-hot block at runtime, and train a linear solver such as LIBLINEAR. The 2011 learning paper reports that only a very minimal modification of LIBLINEAR was needed [1105.4385].

## 4. Large-scale learning, empirical performance, and systems practice

The learning papers evaluate b-Bit MinHash primarily on the **webspam** dataset, with \(n=350{,}000\), \(D=16{,}609{,}143\), and about 24GB in LIBSVM format, and later on an expanded **RCV1** dataset of about 200GB with about \(10^9\) dimensions [1106.0967] [1108.3072].

| Dataset | Scale | Reported outcome |
|---|---:|---|
| webspam | \(350{,}000\) examples, \(16{,}609{,}143\) dimensions, about 24GB | \(b\ge 8\) and \(k\ge 100\)–\(150\) gave test accuracy very close to the original data |
| webspam | \(k=200\), \(b=8\) | about 70MB effective storage versus roughly 24GB original input |
| expanded RCV1 | about 200GB, about \(10^9\) features | with \(k=30\), \(b=12\), linear SVM and logistic regression already exceeded 90% test accuracy |
| expanded RCV1 | \(k\ge 300\), \(b=12\) | test accuracy exceeded 95% for both linear SVM and logistic regression |

On webspam, the reported speedups were substantial. For linear SVM, training time dropped from about 100 seconds on the original data to about 3–7 seconds near \(C=1\), excluding preprocessing; loading original data took about 12 minutes versus about 10 seconds for hashed data; and testing time dropped from about 100–200 seconds to about 1–2 seconds. For logistic regression, training time dropped from about 1000 seconds to roughly 30–50 seconds when \(b\le 8\). The same study reports that direct nonlinear SVM training with the resemblance kernel was still running after a week, whereas using b-bit hashed representations made the learning task feasible, with \(b=8\), \(k\ge 200\) yielding test accuracy essentially matching the best linear baseline on the original data [1106.0967].

Comparisons with **Vowpal Wabbit (VW)** and random projections are a second major empirical theme. The theory paper shows that VW has the same variance as random projections when the sign distribution is \(\{-1,+1\}\), and that on binary data b-Bit MinHash is usually far more accurate at equal storage [1106.0967]. On webspam, 8-bit MinHash with \(k=200\) achieved about the same test accuracy as VW with \(k=10^6\) [1106.0967]. On the 200GB expanded RCV1 dataset, merely 30 b-bit hashed values per data point achieved similar accuracies to VW with \(2^{14}\) hashed values per data point [1108.3072].

The main practical obstacle is preprocessing, because one must compute many minima per example. The 2012 systems paper addressed three deployment bottlenecks: preprocessing cost, applicability to online learning, and replacement of full random permutations by simple hash functions. It reported GPU preprocessing speedups of 20–80, making preprocessing substantially smaller than data loading time, and showed that simple 2-universal and 4-universal hash families produced learning results very similar to those obtained with fully random permutations on sparse data. In online learning, where data are loaded for many epochs, the compressed representation reduced per-epoch loading time enough to yield training-time ratios of 10.05 on webspam and 28.91 on expanded RCV1, with loading-time ratios of 8.95 and 29.07, respectively [1205.2958].

## 5. Statistical refinements and theoretical scope

The original resemblance estimator uses only whether two retained \(b\)-bit values are equal. Later work showed that this discards information. For each permutation, the pair \((u_{1,b},u_{2,b})\) of retained \(b\)-bit values lies in a \(2^b\times 2^b\) contingency table. The 2011 estimator paper therefore proposed a hierarchy of maximum-likelihood estimators: the full estimator \(\hat s_{b,f}\), using all \(2^b\times 2^b\) cells; \(\hat s_{b,do}\), using the \(2^b\) diagonal cells plus \(P_{b,<}\) and \(P_{b,>}\); \(\hat s_{b,d}\), using the diagonal cells plus one lumped off-diagonal cell; and the practical three-cell estimator \(\hat s_{b,3}\), using only
\[
P_{b,=},\qquad P_{b,<},\qquad P_{b,>}.
\]
The paper’s main empirical conclusion is that the biggest gains occur when resemblance is low but containment is high, and that for b-bit hashing the full MLE can improve over the equality-only estimator by 5-fold to 100-fold, while the three-cell estimator often preserves most of that gain [1108.0895].

A separate line of theory studied b-bit min-wise hashing as a statistically analyzable dimension-reduction method for sparse regression and classification. For a sparse design matrix \(X\in\mathbb R^{n\times p}\) with average number of nonzero entries per row \(q\), the 2013 theory paper showed that ordinary least squares, ridge regression, and \(\ell_2\)-penalized logistic regression on the hashed design can achieve vanishing prediction error asymptotically as long as
\[
q\|\beta^*\|_2^2/n\rightarrow 0.
\]
The same paper also showed that regression on the reduced design can approximate not only ordinary linear models but also more flexible structures, including row-normalized signals and interaction models, while retaining non-asymptotic error bounds driven by sparsity rather than by the ambient dimension \(p\) [1308.1269].

The b-bit viewpoint also reappears in later privacy work. The 2020 local-differential-privacy paper explicitly builds on Li and König’s range compression: if \(B=2^b\), then each range-\(B\) MinHash entry is a \(b\)-bit value, and in the experimentally favored regime \(B=2\) the released sketch is effectively a privatized 1-bit MinHash signature. That paper treats range-\(B\) MinHash as a generalization of b-Bit MinHash and derives unbiased estimators that debias both compression collisions and privacy noise [2008.08134].

## 6. Limitations, misconceptions, and later developments

b-Bit MinHash is specialized rather than universal. Its strongest results are for **binary** or naturally binarized data, **very high-dimensional sparse** vectors, and resemblance-style similarity. Its formulas assume large \(D\), though later work reports that the approximation is highly accurate even for smaller \(D\). Preprocessing requires generating multiple minhashes, and true random permutations on huge universes are expensive, which is why later systems work relied on universal hash families and GPU acceleration [1106.0967] [1205.2958].

A persistent misconception is that b-Bit MinHash is merely a static similarity-estimation trick. That is only partly correct. It is also a positive definite kernel with an explicit finite-dimensional feature map, which is why it can support large-scale supervised learning [1105.4385]. The opposite misconception is that because it stores only a few bits, it should be naturally suited to streaming or sketch algebra. Later papers explicitly reject that interpretation. HyperMinHash states that unlike comparable Jaccard fingerprints in sub-logarithmic space such as b-bit MinHash, it retains MinHash’s features of streaming updates, unions, and cardinality estimation; it also recommends b-bit MinHash when the goal is only a one-time Jaccard fingerprint of a set [1710.08436]. MaxLogHash similarly presents b-bit MinHash as highly memory-efficient for static high-similarity estimation but unsuitable for streaming sets [1905.08977]. SetSketch makes the same contrast in another form, noting that b-bit minwise hashing reduces MinHash values to a few bits in a finalization step, but after finalization the sketch cannot be further aggregated or merged [2101.00314].

Several later variants address operational gaps rather than replacing the core method outright. **Pb-Hash** partitions a large \(B\)-bit hash into \(m\) chunks of \(b\) bits rather than discarding most bits; its analysis shows an accuracy loss from correlation, but often not much for \(m=2\sim 4\) [2306.15944]. **C-MinHash** rigorously reduces the number of permutations needed for full MinHash from \(K\) to two via a circulant construction; that result is directly relevant to upstream hash generation, although it does not analyze b-bit truncation itself [2109.03337]. A dynamic threshold filter for MinHash-style estimators was also later proposed for image similarity search and explicitly stated to extend to b-Bit Minwise Hashing whenever the estimator satisfies a binomial distribution [1807.02895].

The resulting historical pattern is consistent. b-Bit MinHash remains the canonical compressed MinHash method for static sparse binary data, especially when the task is Jaccard-style fingerprinting or resemblance-based learning. Later work mainly clarifies what is gained by this aggressive truncation—dramatic storage reduction, strong learning performance, and explicit kernel structure—and what is lost—mergeability, streaming updates, and part of the statistical information available in full hashes.

Source: https://www.emergentmind.com/topics/b-bit-minhash