Indistinguishable Bloom Filter (IBF)

• Indistinguishable Bloom Filter (IBF) is a randomized data structure designed for dynamic sets, supporting efficient insertion, deletion, and recoverable extraction below a critical load threshold.
• It leverages multiple hash functions and per-cell fields such as count, idSum, and hashSum to enable iterative peeling for both full and partial element recovery.
• IBFs are applied in set reconciliation, network telemetry, and privacy-aware data outsourcing, balancing space efficiency with communication overhead.

An Indistinguishable Bloom Filter (IBF), more widely known as an Invertible Bloom Filter, is a randomized data structure designed to represent dynamic sets (or multisets) compactly, supporting efficient insertion, deletion, and, with high probability, recovery (“extraction”) of the encoded set elements. IBFs extend classical Bloom filters’ approximate membership property with invertibility: they can enumerate their contents if the structural load—determined by the number of stored elements relative to the number of cells—remains below a critical threshold. These properties underpin applications in set reconciliation protocols, straggler identification, network telemetry, and privacy-aware data outsourcing. The efficiency and reliability of IBFs rest on the interplay between their hash-based indexing mechanisms and the random hypergraph models that characterize their behavior under extraction, both for full and partial listing of stored items (Goodrich et al., 2011, Kubjas et al., 2020, Houen et al., 2022).

1. Data Structure, Fields, and Hashing Mechanisms

An IBF is an array $F[1..m]$ of $m$ cells. Each cell maintains three fields:

• Count (integer): net tally of insertions minus deletions hashed to this cell
• idSum (universe element, typically XOR-sum of bit-string keys): aggregate of inserted values
• hashSum (checksum, modulo a small range $C$): sum of a short hash $B(x)$ of each inserted key $x$

The IBF uses $k$ independent hash functions $H_1, ..., H_k$ mapping the key universe $X$ into coordinates $[m]$. In practice, hash function outputs are distributed to ensure near-uniform load balancing.

Parameter definitions:

• $n$: number of elements to be stored
• $m$: number of IBF cells
• $k$ ($=h$ in some works): number of hash functions
• $\alpha = m/n$: storage overhead ratio

The design selects $k$ and $m$ to trade off between space and extraction success. The critical threshold $c_k$, arising from the random $k$-hypergraph induced by the hash assignments, demarcates the region where extraction succeeds with high probability (Goodrich et al., 2011, Kubjas et al., 2020). For example, $c_3 \approx 1.222$, $c_4 \approx 1.295$, $c_5 \approx 1.425$ for $k = 3,4,5$.

2. Algorithms: Insertion, Deletion, and Extraction

Insertion and Deletion

For a key $x$:

• Insert($x$): For $i = 1,...,k$, update $F[H_i(x)]$ by incrementing count, XOR-ing $x$ to idSum, and adding $B(x)$ to hashSum.
• Remove($x$): Same locations are updated in reverse (decrement, XOR, subtract).

Extraction

Extraction proceeds through iterative “peeling.” For each cell:

1. Detect singleton cells: those with $|\text{count}|=1$ and whose hashSum matches the idSum ($\text{hashSum} = B(\text{idSum})$).
2. Recover the corresponding element, remove it (reverse Insert), and continue.
3. Repeat until no singletons remain.

This process is mathematically equivalent to peeling vertices of degree one in a random $k$-uniform hypergraph. Extraction succeeds if no stopping set (non-empty $2$-core) remains (Goodrich et al., 2011, Kubjas et al., 2020).

3. Failure Probabilities and Threshold Analysis

Full Extraction

The fundamental threshold for successful extraction is set by the random hypergraph core phenomenon:

• If $m/n > c_k$, with probability $1 - o(1)$, all $n$ elements are extracted: the random hypergraph is peelable.
• If $m/n < c_k$, extraction fails with high probability due to stopping sets (Goodrich et al., 2011).

The value $c_k$ is defined as the infimum of all $\alpha$ for which the following has no fixed point $x \in (0,1)$:

$x = 1 - e^{-k \alpha x^{k-1}}$

For $k=3$, $c_3\approx1.222$; for $k=4$, $c_4\approx1.295$; for $k=5$, $c_5\approx1.425$.

The failure probability for $m \geq (c_k+\varepsilon)\,n$ decays polynomially: $O(n^{-(k-2)})$ (Kubjas et al., 2020). For strict finite-size bounds, exact counting of extractable configurations can yield precise lower and upper failure bounds (Kubjas et al., 2020):

$$P(\text{Extracted} \geq e) \geq \sum_{\ell=e}^{n} \frac{A(\ell)}{N_H^{kn}}$$

where $A(\ell)$ counts the number of matrices from which at least $\ell$ elements are extractable, and $N_H=m/k$.

Partial Extraction and Under-Provisioning

When $\alpha < c_k$, partial extraction becomes relevant. Even at $\alpha \approx 1.0$, partial extraction succeeds for a substantial fraction—merely $50\%$ recovery may be feasible for $k=3$ (Kubjas et al., 2020). This motivates multi-round or iterative reconciliation protocols.

4. Iterative Set Reconciliation Protocols

IBFs support efficient iterative reconciliation when storage overhead is insufficient for full extraction. In such protocols:

1. Party A encodes its set as an IBF and transmits to B.
2. B constructs an IBF of its own set, subtracts, and extracts elements from the difference.
3. Only a fraction of the symmetric difference is recovered each round (dictated by partial extraction bounds).
4. Unrecovered elements remain for subsequent rounds, with freshly initialized IBFs.

Each iteration acts as a Markov chain step: the remaining set difference shrinks by a random variable with distribution determined by $A_{d_i}(e)$ from the extraction analysis. Standard hitting-time arguments and explicit upper bounds demonstrate that only $O(\log d)$ rounds are typically required to fully reconcile even when $\alpha \approx 1$ (Kubjas et al., 2020).

5. Comparisons and Variants: Lookup Tables and Minimal IBF Designs

The Invertible Bloom Lookup Table (IBLT) is a generalization supporting key–value pairs, with extended fault-tolerance against extraneous deletes, duplicate keys, and keys with multiple values (Goodrich et al., 2011). Each cell in the IBLT may include additional hash-based checksums (e.g., keyHashSum and valueHashSum) to guard against “poisoning” or false singleton detection. The listing threshold and analytic core behavior are governed by the same hypergraph principles as the basic IBF.

The Simple Set Sketch is a minimalistic IBF variant, using a single XOR field per bucket and implicit "quotienting" via bucket index for singleton detection, eschewing explicit counts and per-cell checksums. The load threshold for successful full recovery drops: at $k=3$, the peelable threshold is $\alpha_0 \approx 0.81$ (thus, $n \leq 0.81\, m$) (Houen et al., 2022). This approach yields superior space efficiency at the cost of slightly increased risk of anomalous, but easily correctable, decoding mistakes.

Structure Version	Overhead Threshold ($\alpha = m/n$)	Per-Cell Fields
Standard IBF	$c_3 \approx 1.222$	Count, idSum, hashSum
IBLT	$c_3 \approx 1.222$	Count, keySum, valueSum, keyHashSum, valueHashSum
Simple Set Sketch	$\approx 0.81$	XOR aggregate only

6. Applications and Empirical Performance

IBFs and IBLTs are fundamental in set reconciliation: minimizing communication for database or file synchronization by transmitting only a sketch of the symmetric difference (Goodrich et al., 2011, Kubjas et al., 2020). Specific use cases include:

• Distributed deduplication
• Network flow tracking, where insertions/deletions correspond to flow start/stop events (Goodrich et al., 2011)
• Oblivious selection and retrieval in privacy-preserving outsourced data settings (Goodrich et al., 2011)

Numerical and simulation results confirm the sharpness of the $c_k$ listing threshold: empirical failure rates agree with leading-order theoretical analysis. For IBLT, with $k=5$, $m/n \approx 1.425$ marks the transition to near-certain successful listing (Goodrich et al., 2011). In the presence of extraneous deletes and duplicates, or a nontrivial poisoned key fraction ($\gamma$), the robust checksum scheme continues to enable extraction of almost all valid items, contingent on $k$ and $m/n$ (Goodrich et al., 2011).

7. Storage-Communication Trade-offs and Design Recommendations

When full extraction in a single round is nonessential, setting $k=3$, $\alpha \approx 1.0$ allows for recovery of $20-50\%$ of the items with negligible risk, drastically reducing communication overhead compared to provisioning for full extraction in one shot (Kubjas et al., 2020). Iterative peeling rounds, each operating on a newly constructed IBF, complete the reconciliation at low cost. For one-way or client-server synchronization, this yields a fast initial round and completion within two to three passes. For channels with tight constraints, repeated partial extractions optimize total data transferred versus latency.

A rule of thumb: $k=3$ is optimal for one-round overheads in the $[1.0,1.3]$ range, $k=4$ for $[1.3,1.6]$, and for high-probability full extraction, $\alpha$ should be chosen above $c_k$ (Kubjas et al., 2020).

---

Indistinguishable Bloom Filters thus combine the theoretical efficiency of random graph-based coding with practical, robust algorithms for dynamic set representation, supporting both retrieval and difference reconciliation under rigorous probabilistic guarantees (Goodrich et al., 2011, Kubjas et al., 2020, Houen et al., 2022).