Binary Fuse Filters: Fast and Smaller Than Xor Filters (2201.01174v1)

Abstract: Bloom and cuckoo filters provide fast approximate set membership while using little memory. Engineers use them to avoid expensive disk and network accesses. The recently introduced xor filters can be faster and smaller than Bloom and cuckoo filters. The xor filters are within 23% of the theoretical lower bound in storage as opposed to 44% for Bloom filters. Inspired by Dietzfelbinger and Walzer, we build probabilistic filters -- called binary fuse filters -- that are within 13% of the storage lower bound -- without sacrificing query speed. As an additional benefit, the construction of the new binary fuse filters can be more than twice as fast as the construction of xor filters. By slightly sacrificing query speed, we further reduce storage to within 8% of the lower bound. We compare the performance against a wide range of competitive alternatives such as Bloom filters, blocked Bloom filters, vector quotient filters, cuckoo filters, and the recent ribbon filters. Our experiments suggest that binary fuse filters are superior to xor filters.

• The paper presents a novel filter design that reduces storage overhead to within 13% of the theoretical minimum, surpassing XOR filters.
• It employs 3-wise and 4-wise hashing schemes to achieve over twice the construction speed of XOR filters while maintaining efficient query performance.
• Experimental results show that binary fuse filters outperform competing probabilistic filters in both speed and space efficiency, benefiting high-performance data systems.

Overview of Binary Fuse Filters

The paper, "Binary Fuse Filters: Fast and Smaller Than XOR Filters," by Thomas Mueller Graf and Daniel Lemire, presents an advancement in the domain of probabilistic filters used for approximate set membership. By introducing binary fuse filters, the authors aim to address the limitations of existing filter structures, particularly focusing on reducing storage requirements while maintaining query speed.

Probabilistic Filter Background

Probabilistic filters like Bloom and cuckoo filters are essential data structures for efficiently checking membership of elements in large datasets, allowing a small probability of false positives. These filters are particularly useful in applications where minimizing expensive operations, such as disk or network accesses, is critical. Traditional Bloom filters tend to use about 44% more memory than their theoretical lower bound, indicating room for optimization.

Innovation and Contribution

The xor filter, a recent development, demonstrated improvements by maintaining storage within 23% of the theoretical optimal. However, binary fuse filters take this a step further:

• Storage Efficiency: Binary fuse filters improve storage efficiency to within 13% of the theoretical lower bound, making them more space-efficient than xor filters. They achieve this by partitioning the storage into smaller segments and using efficient hashing strategies.
• Construction Speed: Remarkably, the construction of binary fuse filters is more than twice as fast as that of xor filters, addressing a key limitation of existing xor-based approaches.

The proposed filters use a 3-wise hashing scheme, and a further 4-wise variant is introduced, reducing storage requirements to about 8% above the theoretical minimum. This sacrifice in query speed is only modest, providing a beneficial trade-off for applications where space is at a premium.

Experimental Evaluation

The authors conducted extensive experiments comparing binary fuse filters with several competitive alternatives, including Bloom, blocked Bloom, vector quotient, cuckoo, and ribbon filters. The findings were significant:

• Performance Superiority: Binary fuse filters consistently outperformed xor filters in terms of both speed and storage efficiency, suggesting they could supplant xor filters in most practical scenarios.
• Query and Construction Time: The new filters exhibit significantly improved construction times without a noticeable impact on query speed, marking an advancement in the practical utility of probabilistic filters.

Implications and Future Developments

The implications of this research are notable in areas requiring efficient data processing, such as databases, networking, and large-scale data analysis. The decrease in storage overhead can lead to reduced resource consumption, which is critical in environments with strict performance and space constraints.

Future research could explore further optimizations, such as bulk updates to the binary fuse filter, enhancing its flexibility. Additionally, examining scalability across distributed systems and multi-threaded environments would be a valuable extension, potentially incorporating advanced hardware features like AVX-512.

In conclusion, binary fuse filters represent a solid advancement in the efficient management of set membership queries, offering practical improvements over existing methodologies. Their introduction into the landscape of probabilistic filters presents significant opportunities for enhanced data processing capabilities in various domains.