Limited Independence Suffices for Large-k Min-wise Hashing

Abstract: Min-wise hashing and its $k$-min-wise variant are standard tools in similarity estimation, sampling, sketching, and streaming. A $k$-min-wise family requires every prescribed $r$-subset of a fixed set, for $r\le k$, to appear as the $r$ smallest hash values with approximately the fully random probability, up to multiplicative error $δ$. Previous analyses show that $O(\log(1/δ)+k\log\log(1/δ))$-wise independence suffices. Consequently, for $k=Θ(\log N)$ and $δ=N<sup>{-c}$, the standard polynomial construction uses $O(k\log N\log\log N)$ seed bits. Recent work of Chen, Huang, and Li achieves the optimal $O(k\log N)$ seed length for $k=\log<sup>{O(1)}N$, but only with almost-polynomial error $2^{-O(\log N/\log\log N)}$, leaving open whether polynomially small error is possible with the same seed length. We prove that the standard $s$-wise independent polynomial hash family is $k$-min-wise with multiplicative error $δ$ for $s=O(k+\log(1/δ)).$ Thus, when $k=Ω(\log(1/δ))$, only $O(k)$-wise independence is required. In particular, for $k=Θ(\log N)$ and $δ=N<sup>{-c}$, this gives an explicit family with seed length $O(k\log N)$, matching the support-size lower bound up to constant factors. The proof conditions on the prescribed bottom set and bounds the error only after averaging over the random threshold given by its largest hash value, rather than controlling every threshold separately.

• The paper achieves optimal seed length of O(k log N) for large-k min-wise hashing using s-wise independent polynomial hash families.
• It introduces an averaged-error analysis that eliminates the previous O(k log log N) seed blowup by controlling error over three threshold regimes.
• The findings enable resource-optimal hashing in data sketching and derandomized sampling, with significant implications for similarity estimation.

Limited Independence for Large-$k$ Min-wise Hashing: An Expert Overview

Problem Statement and Background

Min-wise hashing, especially in its $k$-min-wise variant, is integral to similarity estimation, data sketching, and streaming. A $k$-min-wise hash family ensures that for any subset $X \subseteq [N]$ and any $r \le k$, every $r$-subset of $X$ is approximately uniformly likely—up to a multiplicative error $\delta$—to comprise the $r$ lowest hash values of $X$. The $k$0 case corresponds to classical (min-wise) hashing.

The technical challenge addressed in this work focuses on constructing explicit $k$1-min-wise hash families with optimal seed length—specifically, $k$2—while ensuring a polynomially small error ($k$3) in the regime $k$4. Previous polynomial-based constructions and limited independence arguments ([FPS11]) achieved this only up to an $k$5 seed-length blowup factor. The more recent rectangle-PRG construction ([CHL26]) achieves optimal seed length for $k$6, but its error is only almost polynomially small.

Main Contributions

The main technical result is a sharp quantitative analysis of $k$7-wise independent polynomial hash families for $k$8-min-wise hashing. The author demonstrates that, for $k$9, the polynomial hash family is $k$0-min-wise with multiplicative error $k$1. This removes the previously unavoidable $k$2 seed length for the regime $k$3 and $k$4, achieving a seed length of $k$5—optimal up to constants. Notably, the construction is explicit (using standard $k$6-wise independent hash polynomials).

The approach deviates from previous threshold-by-threshold error control. Instead, the analysis considers the average error over the distribution of the random threshold induced by the hash values of the prescribed bottom set, exploiting the order-statistic distribution present in the problem.

Theoretical Results

Main Theorem:

A standard $k$7-wise independent polynomial hash family with $k$8 is $k$9-min-wise with multiplicative error $X \subseteq [N]$0. For $X \subseteq [N]$1, the independence requirement is $X \subseteq [N]$2, so for $X \subseteq [N]$3 and $X \subseteq [N]$4, the polynomial family achieves optimal seed length $X \subseteq [N]$5.

Support-size Lower Bound:

Any $X \subseteq [N]$6-min-wise family must have support size at least $X \subseteq [N]$7, i.e., seed length at least $X \subseteq [N]$8 for $X \subseteq [N]$9. The construction is, therefore, seed-length optimal in the logarithmic-$r \le k$0 regime.

Analytical Methodology

The crux of the analysis lies in decomposing the bottom-set event based on the maximum hash value among $r \le k$1. Conditioning on the entire hash vector $r \le k$2, the hash values on $r \le k$3 retain $r \le k$4-wise independence. Rather than demanding tight control at every threshold, the analysis averages the error, weighted by the true order-statistic distribution.

Three threshold regimes are considered:

• Small expected hit count ($r \le k$5): Estimation via inclusion-exclusion and higher-moment Bonferroni inequalities yields exponentially small additive errors.
• Middle expected count: Instead of direct estimation, monotonicity is used to reduce to the small regime, followed by averaging.
• Large expected count: Standard moment bounds yield rapidly decaying probabilities.

This three-tiered division allows removal of the $r \le k$6 loss and validates the optimal seed-length construction for large $r \le k$7.

Implications

Practical Impact

• Resource-optimal minhashing: Large-scale applications (e.g., in similarity search across web-scale corpora) can now use seed-optimal explicit constructions for bottom-$r \le k$8 sketches, even when many queries demand polynomially small error.
• Derandomization: The result pushes the boundary between existential and explicit constructions for $r \le k$9-min-wise independence, harmonizing theoretical optimality with practical feasibility.
• Generality: When $r$0, one can perform sampling without replacement with nearly uniform guarantees using minimal randomness, enabling robust reuse of hash functions in sketching algorithms.

Theoretical Consequences

• Replaces previous constructions: The analysis shows previous combinatorial rectangle PRG-based explicit constructions as non-essential for this parameter regime.
• Averaged-error techniques: The analytic approach—averaging over random thresholds—may extend to future work on limited independence in settings with rare events, especially where tight tail bounds are infeasible or suboptimal.

Open Problems and Future Directions

Two core questions remain:

• Low-$r$1, low-error regime: It remains open whether explicit $r$2-min-wise hash families with polynomial error and seed-length $r$3 can be constructed for $r$4, especially for constant $r$5.
• Explicit relative-error PRGs for rare events: Can one obtain explicit generators with optimal seed for rare-event (e.g., pointwise no-hit) probabilities with multiplicative error?

Progress on these questions would further tighten the connection between limited independence, pseudorandom generators for combinatorial rectangles, and derandomized sampling primitives.

Conclusion

This work completes the landscape for explicit $r$6-min-wise hashing in the large-$r$7 regime, demonstrating that standard $r$8-wise independence suffices for achieving polynomially small error with optimal seed length—matching lower bounds up to constants. The methodology of focusing on averaged errors rather than uniform pointwise guarantees may foster further advances in derandomization and limited independence. Important avenues remain open in the small-$r$9/low-error regime, where the techniques herein might inspire future explicit constructions.

Reference:

"Limited Independence Suffices for Large-$X$0 Min-wise Hashing" (2607.10255)