Compressed Tabulation-Weighted Treap

• The paper introduces a compressed treap that encodes both keys and structure near the information-theoretic minimum with only O(log U) bits of redundancy.
• It employs a tabulation-weighted prioritization using composite hash functions to ensure history independence and maintain BST and heap properties.
• The dynamic FID built on this treap supports efficient rank, select, insert, and delete operations via recursive entropy coding and advanced lookup tables.

A compressed tabulation-weighted treap is a highly space-efficient, history-independent variant of randomized binary search trees designed to enable succinct dynamic dictionaries with minimal redundancy. It serves as the main technical building block in dynamic data structures supporting rank, select, insert, and delete operations, achieving optimal expected time and sublinear space overhead, as established in (Kuszmaul et al., 22 Oct 2025). This structure realizes a regime where both the content and navigation of tree-based dictionaries can be encoded within a few bits over the information-theoretic minimum, thus fundamentally bypassing the classical tree-structure bottleneck.

1. Structure and Definition

A compressed tabulation-weighted treap consists of three pivotal components:

• Tabulation-weighted prioritization: Node priorities are assigned via a composite tabulation hash function over key segments. Element $x \in [U]$ is decomposed into $(x_\text{high}, x_\text{mid}, x_\text{low})$, and its priority $h(x)$ is defined as $A_{f(x_\text{high})}[x_\text{midlow}]$, where $A_i$ are pre-built random permutation tables and $f(\cdot)$ is a pairwise-independent hash. The construction ensures, for fixed weight $h(x)$, the remaining bits are uniformly distributed and reconstructible.
• Treap ordering: The tree follows the classical treap rule: heap order by priority, BST order by keys. The root of every subtree is the key with maximal $h(x)$ among its elements.
• Compression and encoding: Tree structure is encoded recursively, using entropy coding and extensive lookup tables, leveraging the limited number of possible weight profiles and recoverability of key bits from ranks and weights.

The design guarantees space usage of $\log\binom{U}{n} + O(\log U)$ bits, where the first term is the information-theoretic lower bound for storing a set of $n$ keys from $[U]$.

2. Construction and Encoding Mechanism

The compressed tabulation-weighted treap is built as follows:

• Chunk partitioning: The universe $[U]$ is partitioned into ranges corresponding to superchunks, subchunks, and fine-grained segments, aligning with the decomposition of each key into high, mid, and low bits.
• Tabulation arrays: Arrays $A_i$ provide random permutations on $[W]$ for $W = \poly(n_{\max})$. For each chunk, weights are assigned and tracked so that pivot identification and subtree construction are history-independent and conducive to compression.
• Recursive encoding: At each subtree, encoding comprises:
  1. The pivot's tabulation weight,
  2. High-mid bits, as determined by the weight function,
  3. Pivot's rank among its keys, followed by recursive compression of left and right children. This structure enables explicit enumeration and tabulation for small intervals, while for large intervals, entropy-optimal coding is applied.

• RAM model transition: Word-RAM instantiation uses virtual memory adapters, predefined lookup tables, and entropy encoders, with all tables sized at $\poly(n_{\max}) + \polylog U$ words.

3. Key Properties and Complexity

Operation	Expected Time	Space (bits)	Notes
insert/delete	$O(\log^3 n)$		Compressed treap only
rank/select	$O(\log^2 n)$
—	—	$\log\binom{U}{n} + O(\log U)$	Redundancy $= O(\log U)$; $O(1)$ bits if $n,U$ provided

• Space efficiency: Achieves redundancy $O(\log U)$ bits or $O(1)$ when $n$ and $U$ are externally supplied.
• Time bounds: Insert/delete in $O(\log^3 n)$ and rank/select in $O(\log^2 n)$ expected time. Used as small blocks within larger FID schemes, these bounds improve to $O(1 + \log n/\log w)$.
• Static and dynamic table requirements: Construction is predicated on precomputed lookup tables of size $\poly(n)$ and $\polylog U$; tables are reusable and independent of dynamic input.
• Temporary space during operations: $O(n \log U)$ bits for updates or rebuilds, amortized over operations.

4. Tree-Structure Bottleneck and History Independence

Traditional dynamic search trees face a bottleneck: all possible binary tree shapes with $n$ nodes constitute an exponential space, inherently requiring $\Omega(n)$ bits of structure representation. The compressed tabulation-weighted treap circumvents this as follows:

• Unique shape by keyset and hash: Given the set of keys and the static tabulation arrays, the treap's structure is determined unambiguously, independent of operation history.
• Integrated encoding: By encoding keys and structure simultaneously and exploiting uniformity and small support distributions due to tabulation weighting, the number of possibilities per subtree compresses rapidly, reducing overhead to $O(\log U)$ or less.
• Statistical guarantees: With support properties of tabulation hash, the structure can be navigated and reconstructed with minimal metadata, breaking the $\Theta(n)$ bits barrier for dynamic trees.

5. Role in Succinct Dynamic Dictionaries

The compressed tabulation-weighted treap forms the backbone of the first dynamic fully indexable dictionary (FID) achieving

$O(1 + \log n/\log \log U)$

expected amortized time for insert, delete, rank, and select, with space

$$\log\binom{U}{n} + \frac{n}{2^{(\log n)^{\Omega(1)}}} + \operatorname{polylog} U$$

bits, establishing truly sublinear ($o(n)$) redundancy. Previous dynamic solutions imposed at least $O(1)$ bits/key redundancy; this construction shatters that separation, resolving the open question by Pibiri and Venturini (Kuszmaul et al., 22 Oct 2025). A plausible implication is that future FID designs may universally adopt history-independent tree encodings combined with tabulation-weighted pivots.

6. Lookup Tables and Their Organization

To facilitate compression and efficient operations, the structure utilizes:

• Tabulation arrays: $\poly(n)$ arrays of $W^4$ entries, each for random permutations.
• Weight profile enumeration: For small subintervals, all possible pivot/structure profiles are precomputed and tabulated.
• Entropy coding tables: Distributions for encoding pivot variables are stored for efficient entropy-optimal encoding/decoding.
• Adapters in RAM: Variable-length addressing and combination handled by lookup tables.

All table construction is static, bounded in word space, and their existence does not affect dynamic operation time or memory complexity.

7. Theorems and Guarantees

• Proposition ([Lemma~\ref{lem:treap}] in (Kuszmaul et al., 22 Oct 2025)):
  • There exists a dynamic FID based on the compressed tabulation-weighted treap supporting:
  • Space: $\log\binom{U}{n} + O(\log U)$ bits,
  • Insert/delete: $O(\log^3 n)$ expected time,
  • Rank/select: $O(\log^2 n)$ expected time,
  • Static tables: $\poly(n), \polylog U$ words,
  • Intermediate space: up to $O(n \log U)$ bits during an operation.
• Main Theorem:
  • A dynamic FID supporting all dictionary operations in expected $O(1 + \log n / \log \log U)$ time, using only $\log\binom{U}{n} + n / 2^{(\log n)^{\Omega(1)}} + \polylog U$ bits, establishing sublinear redundancy.

8. Mathematical Formulas

• Space: $\text{Space}(n, U) = \log \binom{U}{n} + O(\log U)$
• Per-key redundancy: $$r = \frac{R}{n} = \frac{1}{2^{(\log n)^{\Omega(1)}}}$$
• Operation time: $$\text{Insert/Delete: } O(\log^3 n),\ \text{Rank/Select: } O(\log^2 n)$$

Conclusion

The compressed tabulation-weighted treap provides a dynamic dictionary structure that achieves optimal time-complexity for insert/delete/rank/select with space arbitrarily close to the information-theoretic minimum, by uniquely determining the tree shape from the keyset and static tabulation functions and compressing the recursive encoding via entropy encoding and lookup tables. This resolves a longstanding open question in succinct data structures, bypassing the tree-structure bottleneck and establishing a new baseline in dynamic dictionary design (Kuszmaul et al., 22 Oct 2025).