PtrHash: Minimal Perfect Hashing at RAM Throughput (2502.15539v2)

Abstract: Given a set $K$ of $n$ keys, a minimal perfect hash function (MPHF) is a collision-free bijective map $\mathsf{H_{mphf}}$ from $K$ to ${0, \dots, n-1}$. This work presents a (minimal) perfect hash function that first prioritizes query throughput, while also allowing efficient construction for $10^9$ or more elements using 2.4 bits of memory per key. Both PTHash and PHOBIC first map all $n$ keys to $n/\lambda < n$ buckets. Then, each bucket stores a pilot that controls the final hash value of the keys mapping to it. PtrHash builds on this by using 1) fixed-width (uncompressed) 8-bit pilots, 2) a construction algorithm similar to cuckoo-hashing to find suitable pilot values. Further, it 3) uses the same number of buckets and slots for each part, with 4) a single remap table to map intermediate positions $\geq n$ to $<n$, 5) encoded using per-cacheline Elias-Fano coding. Lastly, 6) PtrHash support streaming queries, where we use prefetching to answer a stream of multiple queries more efficiently than one-by-one processing. With default parameters, PtrHash takes 2.0 bits per key. On 300 million string keys, PtrHash is as fast or faster to build than other MPHFs, and at least $2.1\times$ faster to query. When streaming multiple queries, this improves to $3.3\times$ speedup over the fastest alternative, while also being significantly faster to construct. When using $10^9$ integer keys instead, query times are as low as 12 ns/key when iterating in a for loop, or even down to 8 ns/key when using the streaming approach, just short of the 7.4 ns inverse throughput of random memory accesses.

PtrHash: Minimal Perfect Hashing at RAM Throughput

The paper presents PtrHash, a novel minimal perfect hash function (MPHF) design, emphasizing query throughput while retaining construction efficiency and manageable memory usage. PtrHash offers an innovative approach compared to previous solutions like PTHash and PHOBIC, by primarily focusing on throughput optimization via the novel use of fixed-width 8-bit pilots. These pilots significantly simplify query operations, achieving high-speed performance inherently suitable for large datasets.

Core Contributions

PtrHash introduces several foundational elements for constructing efficient MPHFs:

• Fixed Number of Buckets and Slots: Each part of the data set is allocated a uniform number of buckets and slots, simplifying indexing strategies.
• 8-bit Pilots: Using uncompressed 8-bit pilots allows for direct storage and faster lookup times, thus minimizing the computational overhead typical of more complex encoding methods.
• Pilot Search Inspired by Cuckoo Hashing: The pilot values are optimized using a strategy akin to cuckoo hashing, ensuring robustness in collision resolution by intelligently displacing colliding buckets.
• Single Remap Table: The intermediate positional data exceeding the target range is mapped utilizing a unified remap table, further enhanced by Elias-Fano coding for efficient storage.
• Prefetching for Streaming Queries: To handle streaming data efficiently, PtrHash employs anticipatory prefetching strategies, significantly enhancing responsive query capabilities.

Performance and Results

PtrHash demonstrates remarkable efficiency. On benchmarks with 300 million string keys, it yields results indicating competitive construction times and exceptional query speeds. For instance, PtrHash is found to be approximately 1.75 times faster than competing MPHFs during standard query tests, escalating to 3.1 times the speed when leveraging streaming queries. On top of this, the query time is reduced to as low as 12 nanoseconds per key when iterating over integer keys, and a significant 8 nanoseconds per key in streaming mode, with performances approaching 10% of the theoretical memory-bound throughput limits.

With parameters aiming to balance compactness and speed, PtrHash constructs its data structure in 62 nanoseconds per key, while occupying only 2.12 bits per key, indicating a key design decision to trade minor increases in space for substantial gains in query speed.

Theoretical and Practical Implications

PtrHash operates at high efficiency levels, making it highly suited for applications demanding rapid access to large static datasets such as bioinformatics, networking infrastructure, and large database management. By optimizing memory access patterns and minimizing computation overhead, PtrHash makes strides toward saturating memory throughput, a valuable feat in any high-performance computing application.

Future explorations may continue to explore optimizing the balance between memory usage and construction speed further, and potentially extend analysis into probabilistic models explaining the efficacy of PtrHash's pilot selection strategy.

Conclusion

PtrHash emerges as a robust and innovative solution within the domain of MPHFs, offering a balance of high throughput, efficient memory use, and relatively quick construction times. Its design demonstrates how paying careful attention to hardware capabilities and algorithmic efficiency can achieve system performance near theoretical maximums, providing a promising direction for further work on scalable data structure solutions in big data environments.