Non-minimal k-perfect hashing: Tight lower bounds and an application to fast static hash tables

Abstract: A minimal perfect hash function (minimal PHF) is a data structure mapping a static set of $n$ keys to $n$ bins without collisions. Two natural generalizations are minimal $k$-PHFs where $n$ keys are mapped to $n/k$ bins of capacity $k$ each, and (non-minimal) PHFs with load factor $α < 1$ where the number of bins is increased by a factor of $1/α$, resulting in spare capacity. While there has been a recent surge of interest in perfect hashing generally, non-minimal $k$-PHFs have not been systematically studied despite a natural use case of speeding up static hash tables: The idea is that a small cache-resident $k$-PHF maps each key $x$ to a cache-line-sized bin of capacity $k$ where $x$ resides. Ideally, this yields a branchless lookup operation with a single cache miss working at high load factors for positive and negative queries alike. Our main theoretical contribution is to determine tight space lower bounds for $k$-PHFs for all pairs of $α \in (0,1]$ and $k \geq 1$. It turns out that combining $α < 1$ and $k \geq 2$ drastically reduces the space of $k$-PHFs, e.g. for $(k,α) = (16,0.8)$ the space lower bound is $0.027$ bits per key while for $(k,α) = (16,1.0)$ and $(k,α) = (1,0.8)$ the lower bounds are higher by factors of $\approx 8$ and $\approx 32$, respectively. On the practical side, we develop a $k$-PHF based on PtrHash and tune it for use in static hash tables. Empirically, our implementation produces $k$-PHFs of size roughly $50\%$ above the lower bound. A static hash set based on this $k$-PHF is consistently at least as fast as other hash sets for negative and mixed queries. On two of the three tested architectures it achieves up to $1.5\times$ speedup for large $n\geq 30M$ where a $1$-PHF does not fit in cache.

• The paper derives tight lower bounds for non-minimal k-perfect hashing using analytic combinatorics and large deviation analysis.
• It introduces a practical k-PtrHash construction that achieves space usage within 50% of the information-theoretic lower bound and supports over 300 million queries per second.
• The research demonstrates that cache-friendly, static hash table designs based on k-PHF significantly outperform traditional schemes in large-scale and negative-query dominated applications.

Non-minimal $k$-Perfect Hashing: Tight Lower Bounds and Fast Static Hash Tables

Introduction and Motivation

This work provides a systematic analysis of non-minimal $k$-perfect hash functions (PHFs), presenting both new tight space lower bounds and practical algorithms for their construction and application in static hash tables. A $k$-PHF for a static set of $n$ keys maps each key to a bin such that no bin contains more than $k$ keys. While minimal PHFs ($k$=1 and load factor $\alpha$=1) have a long history, their non-minimal $k$-PHF counterparts with $k\geq2$ and load factor $\alpha<1$ have not received comparable attention despite strong practical motivation for high-throughput, cache-friendly static lookup data structures.

The practical impetus derives from the ability to use $k$0-PHFs as key-indexing mechanisms for large static sets: placing all keys in bins (ideally of cache line size) enables extremely efficient querying using a single memory access and SIMD or branchless search within a bin. The reduction in hash structure size (relative to minimal PHFs) allows retaining data locality for larger $k$1 and high-load factors ($k$2 close to $k$3), thereby achieving significant performance benefits over traditional schemes in cache-constrained scenarios.

Theoretical Results: Tight Space Lower Bounds

The first main contribution is the derivation of tight space lower bounds for all combinations of $k$4 and load $k$5. The analysis generalizes and unifies previously disparate bounds by extending the brute-force seed-search argument—previously known to be optimal only in minimal cases—to arbitrary $k$6 and $k$7, using analytic combinatorics and large deviation analysis. This yields the following summary:

• Brute-force Construction Optimality: Under a large-universe assumption, the brute-force approach—searching for a hash function seed such that no bin exceeds $k$8 keys—is proved to be space-optimal for all $k$9 and $k$0. The space is roughly $k$1 bits per key, with $k$2 the probability that a random function is $k$3-perfect.
• Closed-form and Asymptotic Bounds: For minimal PHFs ($k$4, $k$5), space must be at least $k$6 bits/key; for larger $k$7 or smaller $k$8, closed-form and asymptotic analyses reveal a dramatic reduction, with space decaying exponentially as slack increases. For example, $k$9 yields a lower bound of $n$0 bits/key—over $n$1 times smaller than $n$2.
• Distributional Insight: The number of keys in a bin asymptotically follows a truncated Poisson distribution conditioned on the global load, leading to precise threshold phenomena and two distinct regimes depending on the slack parameter $n$3 relative to $n$4.

Figure 2: Space lower bound per key for various $n$5 and load factors $n$6, illustrating exponential decay as $n$7 and slack increase.

These results both generalize prior special-case lower bounds and provide quantitative regimes where using larger $n$8 and smaller $n$9 yields sharply lower storage requirements.

Algorithmic Results: Efficient $k$0-PHFs and High-Throughput Hash Sets

The second primary contribution is a practical extension of the PtrHash technique to the $k$1-PHF case, yielding a $k$2-PHF with tunable $k$3 and $k$4, and near-lower-bound space consumption. The bucket placement paradigm partitions the key set and greedily seeks a placement of keys in bins without exceeding the $k$5-key per bin limit, with an enhanced scoring function designed to maximize future placement flexibility at each step.

Strong empirical results demonstrate:

• Space Efficiency: The implemented $k$6-PtrHash construction consistently achieves space within $k$7 of the information-theoretic lower bound.
• Construction and Query Throughput: The approach achieves over 300 million queries per second, outperforming state-of-the-art static hash set constructions on large real-world sets.
• Cache Fit and Scalability: For large sets ($k$8M), the reduction in hash structure size enables the $k$9-PHF (with $k$0 corresponding to cache line size) to remain cache-resident beyond the point where $k$1 PHFs cease to fit, resulting in up to $k$2 faster queries for negative and mixed queries across multiple CPU architectures.

Figure 4: Query throughput and space usage of the $k$3-PHF compared to best-in-class baselines as $k$4 increases.

Figure 1: 1- and 12-thread throughput for non-prefetched and prefetched static hash set queries on an Intel Skylake CPU.

Figure 3: 1- and 128-thread throughput for hash set queries on Intel Xeon CPU; benefits of $k$5-PHF scale to high thread counts.

Figure 5: Effect of scaling threads (1 vs. 192) on query throughput for static hash sets on an AMD EPYC CPU.

Implications and Future Work

On the theoretical front, these tight lower bounds provide a comprehensive framework for analyzing new static hashing schemes and yield benchmarks that any practical algorithm must approach to be considered optimal with respect to space.

Practically, the $k$6-PHF construction enables creation of static hash sets that are both highly space- and query-efficient, especially in read-heavy and negative-query-dominated applications (e.g., sequencing read classification, k-mer indexing, large-scale lookup tasks). The findings directly enable engineering of hash sets that remain fast at previously-intractable scales, and branchless/SIMD query routines further enhance real-world utility.

Possible directions for future research include the design of $k$7-PHF schemes minimizing construction time under parallel, distributed, or external memory models; amortizing preprocessing cost across multiple related key sets; and extending the probabilistic and combinatorial analysis to schemes with variable bin sizes or even more flexible capacity constraints. Given the rapidly growing deployments of static hashing in AI and genomics pipelines, further empirical study to tailor $k$8 and $k$9 to hardware cache architectures and application access patterns appears warranted.

Conclusion

This work closes longstanding gaps in the theory and practice of non-minimal $\alpha$0-perfect hashing, providing both tight space lower bounds for all $\alpha$1 and load factors and a new class of practical $\alpha$2-PHFs that are competitive to the lower bound and deliver cache-efficient, high-throughput static hash sets. The results have immediate applicability for practitioners seeking scalable, performant, and memory-efficient static set representations in settings where query throughput is paramount.