--- title: Dynamic External Hashing url: https://www.emergentmind.com/topics/dynamic-external-hashing type: topic --- # Dynamic External Hashing Dynamic external hashing refers to the class of hash-based dictionary data structures that efficiently support insertions, deletions, and queries for a dynamic set of items stored in external memory, typically disk. The primary objective is to minimize the number of costly block transfers (I/Os) between main memory and external storage, in accordance with the standard Aggarwal–Vitter external-memory (I/O) model, while ensuring near-constant query times and amortized update costs. Dynamic external hashing departs from traditional comparison-based paradigms, leveraging hashing techniques and buffering to break through prior lower bounds for external dictionary operations [0811.3062, 1104.2799, 1805.09423]. ## 1. External-Memory Model and Problem Definition The Aggarwal–Vitter I/O model characterizes systems by an internal memory of size $M$ words and an unbounded disk divided into blocks of $B$ words. Each atomic key-value item occupies a single word, and each I/O consists of transferring an entire block between disk and RAM. Operations are measured by I/Os, with computation in RAM assumed to be free [0811.3062]. The canonical dynamic external hashing problem is to maintain a dynamic set of up to $n$ items and support: - $\mathrm{Insert}(x)$: insert item $x$. - $\mathrm{Delete}(x)$: remove item $x$. - $\mathrm{Lookup}(x)$: determine if $x$ is present and retrieve its value. Hashing in this setting traditionally focuses on minimizing disk accesses, since random access in external memory is orders of magnitude slower than in RAM. ## 2. Query–Insertion Tradeoff and Optimal Bounds A central insight in dynamic external hashing is the inherent tradeoff between insertion and query costs, parameterized by the block size $B$ and a tunable parameter $\lambda$ known as the growth or branching factor. The optimal tradeoff, established in the cell-probe model, asserts that any hashing-based dictionary achieving an amortized insertion cost $O(\lambda/B)$ must incur a query cost of at least $\Omega(\log_\lambda N)$ I/Os, where $N$ is the total number of stored items [1805.09423, 1104.2799]. This result answers an open question posed by Jensen and Pagh regarding the achievable limits of buffering [0811.3062]. Specifically, for any $\lambda\ge \max\{\log\log N,\, \log_{M/B}(N/B)\}$, the following bounds are attainable and tight: \[ t_u = O\left(\frac{\lambda}{B}\right),\qquad t_q = O\bigl(\log_\lambda N\bigr) \] where $t_u$ denotes amortized I/Os per insertion and $t_q$ denotes expected I/Os per query [1805.09423, 1104.2799]. Attempting to drive $t_q$ to $1+O(1/B^c)$ for any $c>1$ in the presence of buffering provably prevents $o(1)$ insertion time: the memory buffer becomes useless and insertions cost approximately one I/O each [0811.3062]. In contrast, if $c<1$, buffering yields $o(1)$ amortized insertion cost. ## 3. Core Techniques: Buffering and Layered Hashing Dynamic external hashing leverages memory buffering, multi-level layout, and novel hash-based addressing to achieve optimal tradeoffs: - **Buffer Trees and Hierarchical Hashing:** Traditional buffer trees batch updates and direct them down a multi-way tree. Hash-based equivalents use a static $M$-ary tree, with each node storing keys either directly or in a recursively nested hash "gadget" manageable in cache [1104.2799]. When a node overflows, its entries are pushed to children, with the fan-out and flush frequency determined by the buffer size. - **Indivisibility Paradigm and Comparison Barrier:** Prior external-memory dictionaries (e.g., B-trees) treated keys as indivisible atoms and relied on comparison-based routing, incurring $O((1/B)\log_M N)$ update cost. The use of hash-based splitting and recombination of bit-codes circumvents this barrier, permitting bulk movement and sub-logarithmic IO cost for both insert and query [1104.2799]. - **Multi-Code Hashing:** Keys are expanded into tuples (e.g., page-hash, distribution-hash, shadow-hash) to enable hierarchical placement and precise log-based lookup, with recursive structure mirroring that of LSM/size-tiered compaction [1104.2799, 1805.09423]. ## 4. State-of-the-Art Data Structures Recent advances have resulted in a family of structures matching the optimal insertion–query tradeoff: | Structure Name (as in literature) | Key Features | Bounds Achieved | |-----------------------------------|--------------------------------------------|---------------------------------------------| | IP Hash Table [1805.09423, 1104.2799] | Complex, optimal curve for all practical $\lambda$ | $O(\lambda/B)$ insert, $O(\log_\lambda N)$ query | | Bundle of Arrays Hash Table (BOA) [1805.09423] | Simpler; based on size-tiered LSM | Optimal for narrower $\lambda$ | | Bundle of Trees Hash Table (BOT) [1805.09423] | Mixes BOA’s simplicity with full optimality | Matches IP Hash Table bounds | | Cache-Oblivious BOT (COBOT) [1805.09423] | First cache-oblivious optimal hash table | Same as BOT for all $\lambda$ | | Fully De-Amortized Cuckoo Hashing [1107.4378] | O(1) worst-case I/Os, negligible failure | Cache-oblivious; $\Theta(n)$ space | Details of these structures’ internal designs vary. For example, BOA and BOT leverage size-tiered structures and recursive partitioning, while cache-oblivious variants use layout techniques (van Emde Boas or contiguous array) to remain agnostic to unknown $B$ and $M$ at run time [1805.09423]. ## 5. Lower Bounds and Theoretical Limits Lower-bound proofs in the cell-probe and external-memory models demonstrate that for block size $B$ and cache size $M$, no data structure achieving amortized insertion cost $O(\lambda/B)$ can support queries in $o(\log_\lambda N)$ expected I/Os [1104.2799, 0811.3062]. The proof technique typically divides updates into epochs and analyzes the "footprint"—blocks accessed by queries—showing that otherwise one could encode information too efficiently, violating information-theoretic constraints. When buffer utilization targets ultra-fast queries ($c>1$ in $1+O(1/B^c)$), buffering ceases to offer an advantage; insertions cost nearly one I/O each, matching the no-buffer case [0811.3062]. ## 6. Algorithmic Details: Insertion and Query Procedures High-performance dynamic external hash tables use multi-level gadgets or hash forests. In the Iacono–Pătraşcu design, insertions append to a log at each gadget, with bulk "flush" operations recursively distributing entries into smaller gadgets or subtrees [1104.2799]. Query operations scan tails, then recursively probe smaller gadgets using partial hashes, with each recursion yielding $O(1)$ true matches and negligible false positives due to high-independence hash families. These operations can be formalized with gadget pseudocode; the IO cost recurrence solves to $O(\lambda/B)$ insert and $O(\log_\lambda N)$ query [1104.2799]. Cache-oblivious and de-amortized cuckoo hashing variants use nested tables and explicit queues/stashes to ensure O(1) worst-case I/O for all operations with overwhelming probability, regardless of $B$ or $M$, and with negligible failure probability [1107.4378, 1805.09423]. ## 7. Comparison with Prior Art and Extensions Dynamic external hashing distinguishes itself from B-trees and buffer trees, which both incur higher update costs due to comparison-based limits: - **Buffer Trees:** $O((\lambda/B)\log N)$ update and $O(\log_\lambda N)$ query [1104.2799]; hashing-based structures remove the logarithmic update factor. - **B-Trees:** $O(1)$ update and $O(\log_B N)$ query; no sublogarithmic queries for updates faster than $1/B$. - **Indivisibility and Key Truncation:** Breaking indivisibility enables bypassing the comparison barrier, achieving asymptotic improvements when parameters allow. - **Cache-Oblivious Hashing:** Recent BOT and COBOT structures extend optimal dynamic hashing to cache-oblivious settings [1805.09423]. These advances address the dictionary problem in external memory while achieving linear space usage ($\Theta(N)$ words, with $o(N)$ overhead) and negligible probability of failure over polynomial operation sequences [1107.4378, 1805.09423]. --- Dynamic external hashing has thus defined and closed the optimality frontier for the dynamic dictionary problem in external memory, introducing practical schemes that balance hash-based randomization, hierarchical buffering, and structured recursion to achieve I/O bounds tightly coupled to the tunable parameter $\lambda$ [1805.09423, 1104.2799, 0811.3062]. Source: https://www.emergentmind.com/topics/dynamic-external-hashing