Consistent-Weighted Sampling (CWS)

• Consistent-Weighted Sampling (CWS) is a randomized sketching method that extends MinHash to weighted sets, accurately estimating weighted Jaccard similarity.
• Core variants such as ICWS, I²CWS, and SCWS optimize computation speed and storage while ensuring unbiased similarity estimations with proven runtime improvements.
• CWS is widely applied in large-scale search, feature hashing, privacy-preserving analytics, and deep learning, demonstrating practical efficiency and scalability.

Consistent-Weighted Sampling (CWS) is a randomized sketching technique designed to estimate the Jaccard similarity between weighted sets (nonnegative vectors) efficiently. By extending the concept of MinHash—originally restricted to binary sets—CWS enables accurate, scalable Jaccard similarity estimation for high-dimensional or massive-scale weighted data. This is accomplished via hash codes constructed to guarantee that the collision probability between two codes equals their weighted Jaccard similarity, supporting core applications in large-scale learning, nearest-neighbor search, and de-duplication.

1. Mathematical Foundations: Weighted Jaccard Similarity and Sketching

Given two nonnegative vectors $x, y \in \mathbb{R}_+^n$, the generalized (weighted) Jaccard similarity is defined as:

$$J(x, y) = \frac{\sum_{k=1}^n \min(x_k, y_k)}{\sum_{k=1}^n \max(x_k, y_k)}$$

Direct computation is cost-prohibitive for high-dimensional or sparse data. CWS addresses this by producing $K$ independent hash codes $h(x)$ such that the probability of code collision satisfies:

$\Pr[h(x) = h(y)] = J(x, y)$

An unbiased estimator of $J(x, y)$ is then obtained by the mean collision rate over $K$ samples, with variance $O(1/K)$. The time complexity to compute all $K$ codes for a sparse vector is $O(K \cdot \mathrm{nnz}(x))$, a critical scalability property (Wu et al., 2017).

2. Core CWS Algorithms and Variants

CWS encompasses several algorithmic realizations, each tailored to optimize speed, storage, or statistical properties:

Classical CWS and ICWS

Improved Consistent Weighted Sampling (ICWS) provides constant-time computation for each weight independently of its value. For each coordinate $$J(x, y) = \frac{\sum_{k=1}^n \min(x_k, y_k)}{\sum_{k=1}^n \max(x_k, y_k)}$$0, ICWS samples:

• $$J(x, y) = \frac{\sum_{k=1}^n \min(x_k, y_k)}{\sum_{k=1}^n \max(x_k, y_k)}$$1
• $$J(x, y) = \frac{\sum_{k=1}^n \min(x_k, y_k)}{\sum_{k=1}^n \max(x_k, y_k)}$$2
• $$J(x, y) = \frac{\sum_{k=1}^n \min(x_k, y_k)}{\sum_{k=1}^n \max(x_k, y_k)}$$3

Computes:

• $$J(x, y) = \frac{\sum_{k=1}^n \min(x_k, y_k)}{\sum_{k=1}^n \max(x_k, y_k)}$$4
• $$J(x, y) = \frac{\sum_{k=1}^n \min(x_k, y_k)}{\sum_{k=1}^n \max(x_k, y_k)}$$5
• $$J(x, y) = \frac{\sum_{k=1}^n \min(x_k, y_k)}{\sum_{k=1}^n \max(x_k, y_k)}$$6
• $$J(x, y) = \frac{\sum_{k=1}^n \min(x_k, y_k)}{\sum_{k=1}^n \max(x_k, y_k)}$$7

Hash code is the pair $$J(x, y) = \frac{\sum_{k=1}^n \min(x_k, y_k)}{\sum_{k=1}^n \max(x_k, y_k)}$$8 where $$J(x, y) = \frac{\sum_{k=1}^n \min(x_k, y_k)}{\sum_{k=1}^n \max(x_k, y_k)}$$9 (Wu et al., 2017).

Independence Violations and I²CWS

ICWS's derivation assumes $K$0 and $K$1 are independent. The construction actually introduces dependence, since both $K$2 and $K$3 are functions of the same $K$4, which propagates to $K$5. This dependence leads to estimator bias and inferior mean squared error. The I²CWS algorithm rectifies this by generating the requisite random variates for $K$6 and $K$7 independently, strictly separating their sources of randomness and thereby restoring the unbiasedness of the estimator (Wu et al., 2017).

SCWS: Simplified 0-Bit CWS

SCWS, or Simplified 0-Bit Consistent Weighted Sampling, further accelerates CWS by eliminating the $K$8-coordinate and floor operations, directly using $K$9 with pre-drawn $h(x)$0. This yields over $h(x)$1 runtime improvement with no systematic loss in accuracy or WJS estimation bias relative to ICWS or 0-Bit ICWS (Raff et al., 2018).

Bin-wise CWS (BCWS) and DP Extensions

BCWS leverages the “one permutation plus densification” paradigm: a single global permutation splits the feature set into $h(x)$2 bins, applying CWS within each. Densification addresses empty bins, providing unbiasedness with dramatically reduced computation—$h(x)$3 rather than $h(x)$4, empirically up to $h(x)$5 times faster on sparse data. BCWS supports differentially private extensions (DP-BCWS) via randomized response on output indices, retaining strong utility at practical privacy budgets, and underpins privacy-preserving sketching for machine learning (Li et al., 2023).

Unweighted Reduction Schemes

Alternative “fast, small, easy” CWS schemes reduce weighted sets to unweighted multisets with negligible bias. One-permutation MinHash is then applied, providing $h(x)$6 hash operations per input element, $h(x)$7 overall runtime, and practical effectiveness for massive-scale applications without sacrificing accuracy for “interesting” similarities (Haeupler et al., 2014).

Variant	Key Feature	Complexity per Sample
Classical CWS	Full (i*, t*) output	$h(x)$8
ICWS	O(1) per coordinate	$h(x)$9
I²CWS	Strict y⊥a by design	$\Pr[h(x) = h(y)] = J(x, y)$0
SCWS	0-bit, simplified	$\Pr[h(x) = h(y)] = J(x, y)$1, small const.
BCWS	Binned + densification	$\Pr[h(x) = h(y)] = J(x, y)$2
Reduct. (Haeupler et al., 2014)	Unweighted mapping	$\Pr[h(x) = h(y)] = J(x, y)$3

3. Theoretical Properties: Unbiasedness, Variance, and Consistency

For all canonical CWS variants (under appropriate independence conditions), the hash collision probability recovers weighted Jaccard similarity exactly:

$\Pr[h(x) = h(y)] = J(x, y)$4

Thus, for $\Pr[h(x) = h(y)] = J(x, y)$5 independent hashes, the estimator

$\Pr[h(x) = h(y)] = J(x, y)$6

yields $\Pr[h(x) = h(y)] = J(x, y)$7, with $\Pr[h(x) = h(y)] = J(x, y)$8. Consistency is defined as producing identical hashes for features with identical weighted counts, ensured via careful seeding of the pseudorandom number generators and discretization/floor tricks.

In the case of some fast reduction-based methods (Haeupler et al., 2014), the estimator exhibits negligible bias $\Pr[h(x) = h(y)] = J(x, y)$9 and $J(x, y)$0 variance.

4. Practical Implementations and Applications

CWS enables a wide range of high-impact applications:

• Large-Scale Search and Learning: Efficient similarity estimation for non-binary data, feature hashing for machine learning, and duplicate detection (Wu et al., 2017, Raff et al., 2018).
• Privacy-Preserving Sketching: By applying differential privacy machinery to outputs (e.g., DP-BCWS), CWS supports private aggregation and classification without substantial loss in accuracy at realistic privacy budgets (Li et al., 2023).
• Low-Precision and Resource-Efficient Models: Binary one-hot encodings and 8-bit count-sketches on CWS output drastically reduce model size and enable first-layer computation in neural networks to use only additions instead of multiplications (see GCWSNet) (Li et al., 2022).
• Sub-Linear Approximate Nearest Neighbor (ANN) Search: Fast similarity sketches for hashing-based ANN indexing structures.

Notably, empirical evaluations confirm SCWS achieves over $J(x, y)$1 the speed of ICWS—up to $J(x, y)$2 in some benchmark scenarios—while matching the accuracy on WJS estimation, classification, and $J(x, y)$3-NN precision (Raff et al., 2018). BCWS achieves up to $J(x, y)$4-fold overall speedups (Li et al., 2023).

5. Extensions: Generalized CWS, Neural Hashing, and DP

The CWS framework generalizes naturally to encompass alternative kernels. Generalized Consistent Weighted Sampling (GCWS) is defined for the powered-GMM (pGMM) kernel:

$J(x, y)$5

where $J(x, y)$6 is a doubled non-negative vector representing signed data. GCWS enables efficient and numerically stable hash-based approximations for generalized similarity kernels, aids in sparse binary feature generation, and allows deep neural networks to use highly efficient hash-based first-layer computations (“GCWSNet”) (Li et al., 2022).

Differential privacy is achieved in BCWS by randomized response mechanisms. In practice, DP-BCWS supports competitive classification accuracy ($J(x, y)$795–98%) under $J(x, y)$8-DP constraints for $J(x, y)$9–$K$0, as demonstrated on datasets such as MNIST and DailySports (Li et al., 2023).

6. Empirical Evaluation: Datasets, Metrics, Performance

Empirical studies cover a range of tasks:

• WJS Estimation Quality: Across high-dimensional synthetic and real-world data, I²CWS reduces MSE by 5–30% over ICWS, with largest gains at low fingerprint lengths (Wu et al., 2017).
• Classification: Evaluations on RCV1, KDD, and other benchmarks show that all high-quality CWS variants yield nearly identical classification accuracy. SCWS, BCWS, and binary-reduced approaches match or exceed the baseline, with orders-of-magnitude speedups (Raff et al., 2018, Li et al., 2022).
• Retrieval and ANN: In top-$K$1 retrieval (e.g., Webspam, URL datasets), I²CWS attains 4–7% improvements in precision@K and MAP@K versus ICWS; speedups from SCWS and BCWS are consistently observed.
• Privacy-Utility Tradeoff: DP-BCWS retains $K$2 classification accuracy under practical DP regimes (Li et al., 2023).
• Model Efficiency: Binary GCWS features and 8-bit count-sketches reduce model size by up to $K$3 without measurable loss in accuracy (Li et al., 2022).

Typical benchmarking datasets include LIBSVM a9a, cod-rna, covtype, MNIST, RCV1, URL, and others, covering a variety of dimensions and sparsity patterns.

7. Future Directions and Open Questions

Current research continues to address formal tail-bounds for SCWS error rates, extensions to streaming or drifting data, and improved densification strategies for extremely sparse data or massive sketch sizes. Alternative randomization schemes and privacy mechanisms remain active subjects, particularly in the context of federated and private learning. CWS’s integration with deep learning (GCWSNet), hardware-efficient architectures, and large-scale privacy-preserving analytics highlights ongoing interest in further accelerating, generalizing, and securing CWS-based sketching in data-intensive settings (Li et al., 2022, Li et al., 2023, Raff et al., 2018, Wu et al., 2017).