Rateless Bloom Filters: Adaptive AMQs

• Rateless Bloom Filters are adaptive AMQs that dynamically grow without preset capacity while tightly controlling false positives.
• They employ a streaming model that incrementally refines filter slices based on a cost-benefit criterion to optimize communication overhead.
• Hybrid protocols combining RBFs with rateless IBLTs further minimize bandwidth by isolating residual differences for efficient set reconciliation.

Rateless Bloom Filters (RBFs) are a class of approximate membership query (AMQ) data structures and protocols that generalize classic Bloom filters by removing fixed-capacity constraints, enabling adaptive, unbounded growth while tightly controlling false positive probability and communication overhead. RBFs have emerged to address challenges in environments such as distributed set reconciliation and scalable data filtering, where both element cardinality and set differences are unknown or highly variable and element size may not be uniform. These structures have been investigated as central components in applications ranging from large-scale replica synchronization to flexible join/anti-join operations in databases.

1. Technical Motivation and Design Constraints

Traditional Bloom filters, as well as cuckoo filters and similar AMQ variants, require pre-allocation of filter size based on the expected set cardinality and desired false positive probability. This prior configuration creates fundamental limitations: inserts beyond the designated capacity either fail or cause the false positive probability (fpp) to increase rapidly, and resizing typically entails expensive rebuilding and network transfer. Moreover, in protocols such as set reconciliation, optimal performance requires exact knowledge of the difference cardinality $d$—the size of the symmetric set difference—which is rarely known in practice, particularly after major network disruptions.

For messaging protocols, especially those operating over possibly divergent or variable-sized datasets, the inability to configure filters dynamically without sacrificing optimality constitutes a significant bottleneck. A plausible implication is that robust rateless filtering mechanisms can prevent costly misconfigurations and bandwidth waste when $d$ is misestimated.

2. Structure and Operation of Rateless Bloom Filters

The defining property of an RBF is its ratelessness: it supports an ongoing stream of insertions and emits a progressively refined filter representation. In contrast to classic filters with a static bit-array and fixed hash function count, modern RBFs are implemented via partitioned or streamed Bloom filters, emitting successive "slices" (each typically using one hash function) that incrementally limit the false positive region.

• The sender can stream slices partitioned by hash functions; each successive slice increases discriminatory power.
• At any stage, the receiver can decide to halt slice reception when the additional filter information no longer yields a beneficial cost reduction compared to direct reconciliation, applying an adaptive cost-benefit stopping criterion.

The adaptive streaming is governed by the cost model:

$|{\rm new\_TN}| < \frac{m}{C_{\rm elem}}$

where $|{\rm new\_TN}|$ is the number of newly confirmed true negatives per slice, $m$ the size of a slice (in bits), and $C_{\rm elem}$ the expected cost to reconcile an unresolved element directly.

The total membership identification communication is:

$${\rm Cost}(A \setminus B) = |{\rm BF}_B| + C_{\rm elem} \cdot |{\rm FP}_{A \setminus B}|$$

This structure allows for dynamic bandwidth adjustment without needing to estimate $d$ up-front, tracking near-optimal communication overhead across the entire regime of $d$.

3. Hybrid Protocols: RBFs with Rateless Invertible Bloom Lookup Tables

State-of-the-art reconciliation protocols, such as those in "Rateless Bloom Filters: Set Reconciliation for Divergent Replicas with Variable-Sized Elements" (Gomes et al., 31 Oct 2025), interleave RBFs with additional streaming data structures that resolve the remaining false positives. The two-stage protocol is:

1. Bidirectional RBF Streaming: Each side streams slices, with both parties partitioning their elements according to observed filter memberships. Streaming stops when the cost-benefit threshold is reached.
2. Rateless IBLT Reconciliation: The smaller residual sets, mostly consisting of the actual differences, are reconciled using a rateless IBLT process, which—like the RBF—is streamed until decoding succeeds, again without knowledge of $d$.

This design leverages the RBF for cheap exclusion of most differences and limits expensive set-difference enumeration to a reduced subset, thus minimizing bandwidth and computational overhead for large, divergent sets and variable-sized elements.

4. Mathematical Analysis and Performance Tradeoffs

RBFs match the efficiency of optimally parameterized static Bloom filters without prior knowledge of $d$. The maximum information density for each slice with one hash function is:

$m = \frac{n}{\ln(2)}$

where $n$ is the number of elements inserted. RBF communication cost remains within a few percent of the static optimum—this is achieved adaptively without configuration. If $d$ is underestimated in static protocols, performance degrades rapidly; RBFs are resilient against such misconfigurations.

For classic AMQ structures, the optimal number of hash functions for a filter is:

$h = \frac{m}{n} \ln 2$

False positive probability approaches:

$p = \left(1 - e^{-hn/m}\right)^h$

For scalable or rateless constructions, space per item achieves $O(\lg(1/\varepsilon) + \lg\lg n)$ bits per inserted element, which meets the asymptotic lower bound for growable AMQs (Apple, 2021).

Computationally, RBFs incur slight increases in encoding/decoding time (streamed, less cache-local), but for set sizes typical in practice ($\sim$100k elements), transfer savings outweigh computation costs.

5. Comparison with Related Filters and Surveyed Variants

The closest concept in the taxonomies covered by "Shed More Light on Bloom Filter's Variants" (Patgiri et al., 2019) is the Scalable Bloom Filter (SBF), which chains together multiple Bloom filters, appending new filters as occupancy or error rates require. SBFs can handle unknown cardinalities, but unlike modern RBFs, they may require more conservative parameter ramping and do not implement adaptive streaming or cost-based termination.

Variant	Scalability (Unknown n)	Automatic Growth	False Positive Bound
Standard Bloom Filter	No	No	Fixed (if $n$ known)
Scalable Bloom Filter	Yes	Yes	Yes
Rateless Bloom Filter	Yes	Yes	Yes

Taffy filters (Apple, 2021) present independent but strongly related advances in rateless/dynamic AMQ design, supporting unbounded insertions, optimal space efficiency, and mathematically controlled $fpp$ via either split block (TBF), cuckoo (TCF), or DySECT (MTCF) architectures. Lookup cost varies ($O(\lg n)$ for TBF, $O(1)$ for TCF/MTCF), but all structures are designed to grow without sacrificing performance.

6. Application Domains and Empirical Performance

RBFs and hybrid RBF-rateless IBLT protocols excel in set reconciliation for distributed systems experiencing unpredictable divergence. Noteworthy scenarios:

• Replica repair after network partition: The protocol achieves over 20% reduction in communication cost (sometimes 30%+ over computationally tractable static schemes) for Jaccard similarity below 85%.
• Variable-size element synchronization: Fully supported, as all elements are streamed without needing to predigest into fixed-length digests.
• Large/long-lived blacklists: As in security settings (e.g. password breach sets), RBFs remove the need for early size estimation, growing seamlessly to accommodate load spikes.

For datasets with very high similarity (Jaccard $>$97.5%), classic $O(d)$ schemes prove slightly more efficient, as RBF streaming incurs an $O(n)$ initial cost. The break-even threshold is approximately 85% similarity, making RBF-based methods most effective for substantial and unpredictable set differences.

7. Challenges, Variants, and Future Directions

Although not explicitly analyzed in all surveyed literature (Patgiri et al., 2019), RBFs resolve the core challenge of Bloom filter adaptivity under uncertainty, aligning with the scalable Bloom filter in goals but diverging in economics and operational model. Taffy filters (Apple, 2021) and protocols that combine streaming RBFs with rateless IBLT demonstrate robust empirical and theoretical performance, but further work is required for API standardization, efficient hardware implementations, and integration into high-throughput distributed systems.

A plausible implication is that optimal filter adaptivity, streaming, and hybridization with reconciliation structures will become standard practice in scalable, resource-aware AMQ design for cloud, database, and security applications. This suggests a general move toward composable modular protocols which—for any set cardinality or divergence—achieve near-optimal space, time, and communication overhead without costly misconfiguration.