Lock-Free Linear-Probing Hash Table

• Lock-free linear-probing hash table is a concurrent open-addressing dictionary that scans sequentially to insert and retrieve keys while ensuring system-wide progress.
• It employs atomic primitives like 2-word CAS and bounded metadata to manage collisions and enable wait-free lookups, contrasting with lock-based approaches.
• Its advanced resizing and migration techniques reclaim deletion space without full rebuilds, preserving a compact memory layout and efficient cache locality.

A lock-free linear-probing hash table is a concurrent open-addressing dictionary that stores keys, or key-value pairs, in a single array and resolves collisions by scanning forward from the hashed home position until a suitable cell is found. In the concurrent setting, the central objective is to retain the compact memory layout and cache locality of sequential linear probing while providing linearizability and system-wide progress without lock-based serialization. Research on the topic spans minimal 2-word-CAS designs with tombstones and migration-based resizing, newer space-efficient algorithms with wait-free lookups and safe tombstone reuse, and adjacent variants that clarify what is specific to strict linear probing as opposed to hopscotch, Robin Hood, or bounded multi-index probing (Maier et al., 2016, Attiya et al., 15 Jun 2026).

1. Definition and structural model

In the basic formulation, the table is a circular array of cells, and a key $k$ is associated with a home position determined by a hash function. The canonical probing rule is standard linear probing with wrap-around: operations inspect cells at or after the home position until they find the key, an insertion position, or evidence of absence. In the bounded “folklore table,” each cell is a 128-bit aligned pair storing one machine-word key and one machine-word value, with two distinguished key values reserved as empty_key and del_key (Maier et al., 2016).

The resulting state space is intentionally small. A slot is either empty, occupied, or a tombstone. In this family of algorithms, elements never move in-place once inserted; deletion changes only the key field to del_key and leaves the value word untouched. This preserves the ordinary linear-probing invariant that all keys belonging to a cluster are stored in the first free slot at or after their hash position, and that searches must scan across tombstones just as they scan across occupied cells (Maier et al., 2016).

The more recent space-efficient formulation keeps the same single-array, increment-by-one probe discipline but replaces the minimal tombstone-only state with a richer set of per-cell tags. Its cells can encode EMPTY, TOMBSTONE, COLLIDED, and DELETED, together with present copies of a key tagged as tentative, final, or revalidate; a CAS-only variant also introduces a marked reservation state $\langle\langle v, j\rangle,\text{marked}\rangle$. This design still performs no relocation of other keys and avoids backward-shift compaction, but it uses these bounded states to reconcile duplicate inserts and to reclaim deletion space safely without rebuilding the table (Attiya et al., 15 Jun 2026).

A useful point of orientation is that strict lock-free linear probing differs from related open-addressing schemes. Hopscotch confines keys to a fixed-size neighborhood and uses per-bucket bitmaps and relocation, rather than a pure forward scan, while the MPI-based distributed DHT described in later work uses a fixed small set of candidate indices derived from hash slices and states explicitly that it is “not linear probing” (Kelly et al., 2019, Lübke et al., 19 Apr 2025).

2. Core concurrent algorithms

In the classical lock-free design, Find is read-only. Starting at $i=h_c(k)$, it scans forward until it encounters either a cell whose key equals $k$ or a cell whose key is empty_key. A hit returns the copied <key,value> pair; a miss linearizes at the read that sees empty_key. Because x86 lacks an atomic 2-word load, the key and value may be read non-atomically, but the correctness argument is by case analysis: seeing empty_key during a racing insert is linearizable as “not yet inserted,” and seeing the new value after reading a matching key is linearizable as a post-update read (Maier et al., 2016).

Updates are unified as insertOrUpdate(k,d,up). The operation probes linearly, reads the current cell, and then either tries a 2-word CAS from an empty cell to <k,d>, or performs atomicUpdate when the key is already present. The default atomicUpdate is a CAS from the current pair to <k, up(k,current.value,d)>, while common specializations use a single-word store for overwrite and fetch-and-add for increment. Insert linearizes at the successful CAS that writes <k,d> into an empty slot; update linearizes at the successful atomic update of the value (Maier et al., 2016).

Deletion in this design is deliberately simple. It changes the key to del_key with an atomic store or CAS of the 2-word cell. The slot remains on the probe path, so later operations continue to scan over it. This preserves correctness but makes tombstone accumulation a first-order performance issue until a later migration compacts the table (Maier et al., 2016).

The 2026 space-efficient design uses a more elaborate scan discipline. Each operation begins with forward_scan(v) from $h(v)$ and may then execute backward_scan(v,end) over the discovered run. lookup(v) is wait-free: it performs at most one forward and one backward scan, touches at most $m$ cells per scan, and at most once per encountered tentative copy attempts to change tentative to revalidate. It returns true when it observes final or revalidate, or when it successfully writes revalidate; otherwise it returns false after the bounded scans (Attiya et al., 15 Jun 2026).

insert(v) in that algorithm first searches for an existing copy. If none is validated, it probes for EMPTY or TOMBSTONE, writes a tentative copy, and then validates it by scanning the run from $h(v)$. Earlier copies or already-final copies force withdrawal of the new tentative; later tentative copies may be eliminated by writing COLLIDED. If validation reaches EMPTY without conflict, the tentative is promoted to final, which is the insertion linearization point. delete(v) overwrites final with TOMBSTONE, or tentative-like states with DELETED, and the owner later cleans DELETED or COLLIDED to TOMBSTONE (Attiya et al., 15 Jun 2026).

3. Atomic primitives, ABA avoidance, and progress

The minimal lock-free table relies on 2-word CAS over the <key,value> cell. Lookups are plain reads; modifications use CAS and, in some specializations, fetch-and-add on the value. The design does not introduce version tags. Instead, races are resolved because any intervening change to either word causes the full-cell CAS to fail. The paper states that the layout avoids pointer-based ABA issues in practice because the compare operand is the full 2-word cell, nonempty keys are not recycled back to empty_key, and migration never modifies the source table in-place (Maier et al., 2016).

Its progress guarantee is lock-freedom for modifications and wait-freedom for reads in the bounded table. Lookups never write or wait. Update loops retry on CAS failure, but each failure implies some competing modification succeeded, so global progress holds. Individual starvation remains possible under sustained contention, which is consistent with ordinary lock-free semantics (Maier et al., 2016).

The space-efficient design makes the same separation sharper: lookup is explicitly wait-free, while insert and delete are lock-free. In the LL/SC variant, ABA is prevented by properly paired LL/SC on a single cell, and the algorithm never maintains overlapping reservations on different addresses. In the CAS variant, ABA-like confusion is prevented by the marked state $\langle\langle v,j\rangle,\text{marked}\rangle$, whose owner identifier records which insert currently claims the tentative copy. Only the owner or the clobbering operation may advance that state (Attiya et al., 15 Jun 2026).

The lock-free proof structure there depends on bounded interference. Each insert or lookup can trigger at most one revalidation of a given tentative copy; once a tentative becomes COLLIDED or DELETED, only its owner completes cleanup to TOMBSTONE. Consequently, if an insert or delete does not complete, infinitely many other operations must complete, which establishes system-wide lock-freedom. Lookup is wait-free because it performs only bounded scans and bounded single-cell modifications (Attiya et al., 15 Jun 2026).

4. Deletion semantics and reclamation of space

Deletion is one of the principal fault lines in concurrent linear probing. In the 2016 design, deletion creates tombstones by writing del_key. This preserves the probe sequence, but tombstones are not recycled immediately and accumulate until migration rebuilds the probe structure in a new table. The paper treats migration as the mechanism that removes tombstones, compacts live entries, and reestablishes a clean array state (Maier et al., 2016).

The 2026 design addresses precisely this weakness. It states that tombstones are reused safely under concurrency and that the table is able to reclaim space used by deleted elements without rebuilding the table. The mechanism is local: deleting a final copy writes TOMBSTONE directly, whereas deleting a tentative-like copy writes DELETED; the owner then cleans DELETED to TOMBSTONE. COLLIDED is handled similarly. Tombstones are subsequently reusable by arbitrary keys (Attiya et al., 15 Jun 2026).

This difference has algorithmic consequences. Earlier tombstone-based designs, including some prior concurrent open-addressed tables, avoid general tombstone reuse or require rebuilds to eliminate them; the newer design explicitly identifies safe tombstone reuse with bounded metadata as a central contribution. A plausible implication is that it narrows one of the traditional gaps between the concurrency requirements of linear probing and its compact sequential form, because deletion no longer forces periodic rebuilding solely for reclamation (Attiya et al., 15 Jun 2026).

The absence of external memory reclamation is also notable. Because these structures use fixed arrays and store keys in-place, there are no per-entry heap nodes and no need for hazard pointers or epoch schemes for entries. The same observation appears, in a different open-addressing context, in lock-free hopscotch hashing, where inline storage and physical deletion likewise avoid per-entry reclamation machinery (Kelly et al., 2019).

5. Resizing, migration, and generalization

The fastest bounded table in the 2016 work does not support dynamic size adaptation, but the paper’s main systems contribution is to “lift these limitations in a provably scalable way.” Growth is triggered when occupancy reaches about $60\%$, with capacity doubled by default. Size tracking is approximate rather than exact: thread-local successful inserts and deletions are flushed to global counters, yielding an estimate $S=I-D$ with error $\langle\langle v, j\rangle,\text{marked}\rangle$0 in the regime where $\langle\langle v, j\rangle,\text{marked}\rangle$1 (Maier et al., 2016).

Migration for growth uses a cluster-preserving property of linear probing. If $\langle\langle v, j\rangle,\text{marked}\rangle$2 is a maximal cluster in the source table and the growth factor is $\langle\langle v, j\rangle,\text{marked}\rangle$3, then sequential migration maps all keys from that cluster into $\langle\langle v, j\rangle,\text{marked}\rangle$4 in the target table. Since disjoint source clusters map to disjoint target intervals, they can be copied in parallel without fine-grained synchronization. The source is partitioned into fixed-size blocks, threads claim blocks with fetch-and-add, and each migrates those clusters whose first cell lies in its block (Maier et al., 2016).

Shrinking is more delicate because the same cluster property does not hold for $\langle\langle v, j\rangle,\text{marked}\rangle$5. The paper therefore uses a two-phase scheme: first, threads migrate all keys whose scaled hash falls into their assigned target block; second, after a barrier, a smaller boundary set is inserted concurrently using the ordinary atomic insertion algorithm. Two consistency strategies are described for migration: asynchronous marking, which marks source cells before copying and prevents further updates to them, and a semi-synchronized exclusion protocol inspired by RCU, based on a global growing flag and per-thread busy flags (Maier et al., 2016).

The more recent space-efficient table does not implement resizing. Insert returns ABORT only after a full traversal without finding EMPTY or TOMBSTONE, and the paper treats repeated aborts as a signal for an external rebuild using known techniques. Its correctness and amortized bounds do not rely on rebuilding, because tombstone reuse already solves the deletion-reclamation problem in the steady state (Attiya et al., 15 Jun 2026).

6. Correctness properties and complexity bounds

Both principal designs are linearizable, but they formalize linearization differently. In the 2016 table, lookups linearize at the read of a matching key or at the read of empty_key; inserts and updates linearize at the successful CAS or other atomic update. The correctness argument for non-atomic lookup reads is semantic rather than acquire/release based: returned results correspond to either the pre-update or post-update state of a racing operation (Maier et al., 2016).

The 2026 design develops a per-key linearizability proof. It proves a structural invariant: if cell $\langle\langle v, j\rangle,\text{marked}\rangle$6 contains $\langle\langle v, j\rangle,\text{marked}\rangle$7, then every cell between $\langle\langle v, j\rangle,\text{marked}\rangle$8 and $\langle\langle v, j\rangle,\text{marked}\rangle$9 is non-empty, so all copies of a key lie in a single run. It also proves that at most one cell contains $i=h_c(k)$0 at any time. Lookup returning true linearizes at the read of final or revalidate, or at the successful write of revalidate; insert linearizes at the promotion of its own tentative copy to final, or at a synchronized DELETED overwrite in the cases identified by the proof; delete linearizes at its successful overwrite to TOMBSTONE or DELETED (Attiya et al., 15 Jun 2026).

Load factor enters through the standard parameter $i=h_c(k)$1. The 2016 work recommends target load factors around $i=h_c(k)$2, achieved either by pre-initializing to at least twice the expected size or by triggering growth near $i=h_c(k)$3 occupancy. The paper attributes its high throughput in part to keeping probe sequences short and cache-friendly in that range (Maier et al., 2016).

The 2026 work states expected amortized step complexity per operation of $i=h_c(k)$4, where $i=h_c(k)$5 is point contention per key, under a scheduler that does not overlap two inserts of the same key for the complexity analysis. Equivalently, it gives

$i=h_c(k)$6

It also recalls the classical sequential formulas for linear probing: successful searches require approximately $i=h_c(k)$7 probes, while unsuccessful searches require approximately $i=h_c(k)$8 probes (Attiya et al., 15 Jun 2026).

7. Variants, comparisons, and recurring misconceptions

A common misconception is to treat all concurrent open-addressing tables with short probe sequences as “lock-free linear probing.” The literature draws sharper boundaries. Lock-free hopscotch hashing remains open-addressed and cache-local but is not strict linear probing: it maintains a fixed neighborhood invariant, uses per-bucket bitmaps and relocation, and employs K-CAS for atomic hopping. Its searches are bounded to bitmap-selected neighborhood positions rather than a plain consecutive scan (Kelly et al., 2019).

Another misconception is to equate distributed bounded-probe DHTs with linear probing. The MPI-based distributed hash table proposed for HPC uses a fixed set of candidate indices derived from slices of a 64-bit hash; writes probe those candidates and overwrite the last candidate on full collision. The paper explicitly states that the design “does not use linear probing,” and its consistency mechanism is checksum-based optimistic validation over MPI one-sided RMA rather than cellwise CAS or LL/SC over a shared-memory array (Lübke et al., 19 Apr 2025).

History-independent concurrency introduces a different axis of design. The Robin Hood table of 2025 is lock-free, linearizable, and state-quiescent history-independent, but it is not plain one-key-per-cell linear probing. Each cell stores two elements and two bits: a value, a lookahead, and a mark in $i=h_c(k)$9. The paper argues that with a 1-cell lookahead and $k$0 metadata, only Robin Hood’s age-based priority order allows a reader to conclude absence from two consecutive cells, and it proves impossibility results for stronger wait-free guarantees under the corresponding history-independence requirements (Attiya et al., 26 Mar 2025).

The comparison that most directly bears on strict lock-free linear probing is between the 2016 and 2026 designs. The earlier design achieves very high performance with 2-word CAS, read-only lookups, tombstones, and scalable migration, but it accepts periodic rebuilding as the mechanism for tombstone cleanup and does not make growth fully lock-free (Maier et al., 2016). The later design preserves the one-array linear-probing layout while adding enough bounded metadata to provide wait-free lookup, lock-free insert and delete, and safe tombstone reuse without rebuilds, with only constant extra bits per cell under LL/SC or logarithmic extra bits under CAS (Attiya et al., 15 Jun 2026).

From these results, a consistent picture emerges. Strict lock-free linear probing is attractive because it preserves the compact memory layout and spatial locality of sequential linear probing, but concurrency forces trade-offs among metadata size, deletion semantics, resizing protocol, and lookup progress. The research record shows several stable points in that design space: minimal-CAS tables with migration-based rebuilding, richer bounded-state tables with safe tombstone reuse and wait-free lookups, and adjacent non-linear-probing schemes that improve other dimensions by giving up the exact consecutive-scan model (Maier et al., 2016, Attiya et al., 15 Jun 2026).