SizeFilter: Size-Adaptive Data Structures

• SizeFilter is a size-adaptive approximate membership data structure that adjusts its space usage based on the current set size, ensuring no false negatives.
• It employs stage-dependent prefix lengths and limited-independence hashing to achieve optimal space with an unavoidable n log log n overhead while maintaining a false positive rate at most ε.
• The structure supports dynamic growth via deamortized capacity doubling and extends to an exact dictionary variant using (1+o(1)) n log(u/n) bits for efficient key-value storage.

Searching arXiv for recent and canonical papers on “SizeFilter” and closely related size-adaptive filter structures. SizeFilter is a size-adaptive approximate membership data structure for a dynamic set $S \subseteq [u]$, where the space usage depends on the current set size $n = |S|$ rather than on a fixed upper bound $N$. In the formulation developed by Liu, Yin, and Yu, a SizeFilter supports insert(x) and lookup(x) with no false negatives and false positive rate at most $\epsilon$, while matching the information-theoretically optimal leading constant in the unavoidable $n \log\log n$ overhead incurred by size adaptivity. The same framework also yields an exact dictionary whose space tracks the current size as $(1+o(1))\,n\log(u/n)$ bits, with constant worst-case update and query time with high probability (Liu et al., 2020).

1. Model and semantic scope

The underlying problem is the dynamic membership problem on the universe $[u] = \{0,1,\dots,u-1\}$. A dictionary answers membership exactly. A filter is an approximate membership structure: if $x \in S$, lookup(x) must always return YES; if $x \notin S$, it may return YES with probability at most $\epsilon$ (Liu et al., 2020).

The defining feature of SizeFilter is size adaptivity. Classical dictionaries and filters are usually dimensioned against an upper bound $n = |S|$0, so their space is tied to capacity rather than occupancy. A size-adaptive structure instead makes space depend on the current $n = |S|$1. In this model, worst-case $n = |S|$2 time “with high probability” means that, for any sequence of operations of polynomial length in $n = |S|$3, the probability that any operation exceeds the stated time is at most $n = |S|$4 for some constant $n = |S|$5. The paper strengthens this further: there is a global failure probability $n = |S|$6, for some constant $n = |S|$7, over the precomputed random bits; conditioned on no failure, every operation runs in worst-case $n = |S|$8 time and the query error guarantee holds (Liu et al., 2020).

This formulation separates two notions that are often conflated. One is dynamic growth, where a filter can expand without immediate failure. The other is occupancy-sensitive succinctness, where the bit complexity continuously tracks the current $n = |S|$9. SizeFilter addresses the second problem directly.

2. Lower bounds and the cost of adaptivity

The central lower bound, originating in prior work summarized by Liu, Yin, and Yu, states that any dynamic filter whose space depends on the current size $N$0 must use at least

$N$1

This establishes an unavoidable additive $N$2 term when the structure is required to track the current set size (Liu et al., 2020).

That lower bound contrasts with the classical fixed-capacity bound

$N$3

which is optimal when a capacity bound $N$4 is known in advance. The gap is not an artifact of construction quality; it is the price of size adaptivity itself. A common misconception is that one should be able to match the fixed-capacity bound merely by growing a conventional filter as needed. The lower bound shows that this is impossible in the size-adaptive regime (Liu et al., 2020).

The paper resolves two specific open issues left by earlier work: the leading constant in the $N$5 term, and the possibility of worst-case $N$6 insertion time with high probability while retaining succinct space tied to the current $N$7. For all $N$8 and $N$9, the filter uses

$\epsilon$0

thereby matching the lower bound up to the $\epsilon$1 term and attaining the optimal leading constant in both terms (Liu et al., 2020).

3. Structural design of the filter

At the highest level, the construction applies a global $\epsilon$2-wise independent hash function

$\epsilon$3

with $\epsilon$4, and organizes updates into stages indexed by $\epsilon$5, where stage $\epsilon$6 is active when

$\epsilon$7

When a key $\epsilon$8 is inserted during stage $\epsilon$9, the structure stores an $n \log\log n$0-bit prefix of $n \log\log n$1. Queries compute $n \log\log n$2 and return YES iff some stored prefix is a prefix of $n \log\log n$3, which guarantees zero false negatives (Liu et al., 2020).

The stage-dependent prefix length is

$n \log\log n$4

The extra $n \log\log n$5 term is the mechanism by which the global false positive rate is balanced across stages. For $n \log\log n$6, the analysis uses the bound

$n \log\log n$7

so the overall false positive rate is at most $n \log\log n$8 (Liu et al., 2020).

The construction splits prefix handling into two regimes. Very short prefixes are represented by truth tables: for prefixes of length $n \log\log n$9, a bitmap over all $(1+o(1))\,n\log(u/n)$0-bit strings indicates whether any stored prefix matches them. This costs $(1+o(1))\,n\log(u/n)$1 bits in stage $(1+o(1))\,n\log(u/n)$2 and yields $(1+o(1))\,n\log(u/n)$3 query time. Longer prefixes are handled by a novel exact membership structure for long strings, described as “subtables + adaptive fingerprints + data blocks” (Liu et al., 2020).

Within a stage of capacity $(1+o(1))\,n\log(u/n)$4, the longer-string structure randomly partitions the universe into $(1+o(1))\,n\log(u/n)$5 buckets using limited independence. Each bucket contains $(1+o(1))\,n\log(u/n)$6 items with high probability. Inside each bucket, the structure maintains adaptive fingerprints, navigators, and data blocks. Adaptive fingerprints store, for each item, a shortest distinguishing prefix of length at least $(1+o(1))\,n\log(u/n)$7; navigators form a compact pointer array indexed by fingerprint rank; data blocks group $(1+o(1))\,n\log(u/n)$8 items and encode them in almost-static succinct form, while a dynamic block absorbs fresh insertions before background reorganization (Liu et al., 2020).

Growth is handled by capacity doubling with deamortization. As $(1+o(1))\,n\log(u/n)$9 approaches the current capacity $[u] = \{0,1,\dots,u-1\}$0, a new structure of capacity $[u] = \{0,1,\dots,u-1\}$1 is built in the background. The transfer from old to new proceeds in $[u] = \{0,1,\dots,u-1\}$2 work per insertion, the structure remains queryable throughout, and there is no need to keep multiple full copies. The model includes extendable arrays supporting $[u] = \{0,1,\dots,u-1\}$3-time grow and shrink, which makes this continuous rebuilding compatible with space tracking by the current $[u] = \{0,1,\dots,u-1\}$4 (Liu et al., 2020).

4. Space and time guarantees

The principal asymptotic guarantees are compactly summarized below.

Structure	Space	Time
Filter	$[u] = \{0,1,\dots,u-1\}$5 bits	worst-case $[u] = \{0,1,\dots,u-1\}$6 insertion and lookup w.h.p.
Dictionary	$[u] = \{0,1,\dots,u-1\}$7 bits	worst-case $[u] = \{0,1,\dots,u-1\}$8 insertion and lookup w.h.p.

These guarantees hold for $[u] = \{0,1,\dots,u-1\}$9, and the dictionary can additionally store $x \in S$0-bit values with an extra $x \in S$1 bits per key-value pair (Liu et al., 2020).

The filter’s space bound can be read elementwise: summing the stage costs yields an effective per-key budget of approximately

$x \in S$2

plus lower-order terms $x \in S$3. The exact-membership component for long strings contributes, per key,

$x \in S$4

bits inside a stage of capacity $x \in S$5, which is the local ingredient behind the current-size dictionary bound (Liu et al., 2020).

The failure analysis depends on $x \in S$6-wise independence with $x \in S$7. That level of independence suffices for Chernoff-style concentration in the load-balancing arguments for subtables, in the adaptive fingerprint uniqueness argument, and in the compact encodings of headers and identities. These local concentration events combine into a global failure probability at most $x \in S$8 for some constant $x \in S$9 (Liu et al., 2020).

A second misconception concerns the meaning of “constant time.” The guarantee is not merely expected amortized. Conditioned on the preprocessing randomness not failing, every insertion and lookup runs in worst-case $x \notin S$0 time.

5. Exact dictionary variant

The same paper presents a size-adaptive exact dictionary using

$x \notin S$1

which matches the optimal space dependence on the current size $x \notin S$2 and supports worst-case $x \notin S$3 insertions and lookups with high probability (Liu et al., 2020).

The dictionary construction uses the same subtable and data-block machinery as the filter, but the key technical obstacle is implementing the required randomization without assuming a fully random permutation of $x \notin S$4 with small description and constant evaluation time. The paper bypasses that obstacle with a Feistel-style permutation combined with three $x \notin S$5-wise independent hash functions:

• $x \notin S$6,
• $x \notin S$7,
• $x \notin S$8.

The Feistel permutation is

$x \notin S$9

where $\epsilon$0 is split into left and right parts with $\epsilon$1 and $\epsilon$2. This ensures that subtable loads and adaptive fingerprints behave like the random case needed for worst-case $\epsilon$3 performance with high probability, without requiring a fully random succinct permutation (Liu et al., 2020).

This dictionary result is significant because it shows that the size-adaptive phenomenon is not limited to approximate membership. Exact dictionaries can also be made occupancy-sensitive and succinct, provided one is willing to adopt the same deamortized staged architecture and limited-independence analysis.

6. Constraints, comparisons, and open questions

The results are established in the regime $\epsilon$4. The paper gives two reasons for this restriction: truth-table overhead for short prefixes remains $\epsilon$5 there, and the subtable sizes and bucket loads concentrate tightly enough under limited independence to sustain the deamortized worst-case guarantees. For smaller $\epsilon$6, the text suggests simpler baseline filters, such as a single small Bloom-like filter sized to a conservative $\epsilon$7 or a two-level scheme, albeit with either higher space per element or weaker guarantees (Liu et al., 2020).

This restriction matters for interpretation. SizeFilter is not a uniformly optimal construction across all parameter regimes; it is a sharp optimality result for the large-$\epsilon$8 regime. The open problems listed in the paper therefore remain structurally important: removing the condition $\epsilon$9, reducing the lower-order $n = |S|$00 redundancy in the dictionary, and weakening the randomness assumptions while preserving worst-case $n = |S|$01 time with high probability (Liu et al., 2020).

In the broader literature, related dynamic filters pursue adjacent objectives rather than identical ones. “Taffy filters” grow without inserts failing and without meaningfully increasing the false positive probability, and achieve $n = |S|$02 bits per element, with a Taffy Cuckoo Filter providing $n = |S|$03 lookup and a layered Taffy Block Filter providing $n = |S|$04 lookup (Apple, 2021). “Smaller and More Flexible Cuckoo Filters” reduces space for online-insertion filters to approximately $n = |S|$05 bits for target false positive rate $n = |S|$06 using overlapping windows (Schmitz et al., 9 May 2025). These results are closely related in purpose, but they address growability and fixed-target approximate membership rather than the specific information-theoretic problem of making space follow the current $n = |S|$07 with the optimal leading constant. This suggests that SizeFilter occupies a distinct point in the design space: it is not merely dynamic, but dynamically succinct in the strong current-size sense.

The lasting contribution of SizeFilter is therefore twofold. It identifies the exact asymptotic surcharge imposed by size adaptivity—an unavoidable $n = |S|$08 term per key—and it gives a concrete dynamic construction meeting that barrier while preserving worst-case constant-time operations with high probability (Liu et al., 2020).