d‑HNSW: Adaptations of HNSW for ANN Search

Updated 5 July 2026

d‑HNSW is a term denoting various modifications of the base HNSW algorithm to meet specific retrieval and system objectives.
The dense retrieval variant integrates Lucene’s HNSW within Anserini, unifying sparse and dense search with near-exact recall and scalable query throughput.
Other adaptations reengineer memory architectures, graph topologies, and update strategies to improve performance, scalability, and maintenance in ANN tasks.

d‑HNSW is not a single standardized algorithmic name, but an overloaded designation applied to several HNSW‑derived or HNSW‑adapted systems. In recent arXiv literature it has denoted, among other things, dense retrieval using Lucene’s HNSW indexes inside the Anserini/Pyserini stack, an RDMA‑based disaggregated-memory vector search engine, a de‑hierarchized or dimension‑aware reading of HNSW, a dual‑branch HNSW with LID‑driven insertion and skip bridges, a dynamic-update interpretation of HNSW, and a distribution‑aware adaptive search strategy that changes $ef$ per query (Ma et al., 2023, Liu et al., 17 May 2025, Munyampirwa et al., 2024, Nguyen et al., 23 Jan 2025, Xiao et al., 2024, Zhang et al., 7 Dec 2025). This plurality of usage suggests that the term functions primarily as a contextual label for “modified HNSW” rather than as a unique canonical method.

1. Terminological scope and relation to standard HNSW

HNSW itself is a multi-layer proximity graph for approximate nearest neighbor search, with a dense base layer containing all elements and progressively sparser upper layers whose membership probabilities decay exponentially with level. Search proceeds from a top-layer entry point by greedy descent, and the principal runtime controls are $M$ , $efConstruction$ , and $efSearch$ (Malkov et al., 2016).

Usage of “d‑HNSW”	Core idea	Representative paper
Dense retrieval	HNSW over dense text embeddings inside Lucene/Anserini	(Ma et al., 2023)
Disaggregated memory	HNSW redesigned for RDMA-based remote memory	(Liu et al., 17 May 2025)
De-hierarchized / dimension-aware	Flat NSW or hierarchy removal in high dimension	(Munyampirwa et al., 2024)
Dual-branch	Two HNSW branches with LID-based insertion and skip bridges	(Nguyen et al., 23 Jan 2025)
Dynamic-update	HNSW adapted for long update sequences and unreachable-point mitigation	(Xiao et al., 2024)
Distribution-aware	Query-adaptive $ef$ selection using similarity distributions	(Zhang et al., 7 Dec 2025)

The common substrate across these usages is the original HNSW design: a graph-based ANN index that combines proximity-graph navigation with a skip-list-like hierarchy (Malkov et al., 2016). What changes from one “d‑HNSW” usage to another is the system objective. In one line of work the “d” refers to dense retrieval; in another it refers to disaggregation; in others it is associated with de‑hierarchization, dual branches, dynamics, or distribution awareness. A plausible implication is that the literature treats HNSW as a flexible systems primitive whose graph-search core can be reinterpreted at multiple layers of the stack.

2. Dense retrieval in Lucene and the Anserini/Pyserini ecosystem

In "Anserini Gets Dense Retrieval: Integration of Lucene's HNSW Indexes" (Ma et al., 2023), “d‑HNSW” informally means dense retrieval using HNSW graphs inside Lucene. The central contribution is to make Lucene’s HNSW vector index a first-class option for dense retrieval so that both dense and sparse search live in a single Lucene-based stack. This eliminates the earlier two-stack pattern in which sparse retrieval used Lucene while dense retrieval relied on Faiss.

The setting is standard bi-encoder retrieval. Queries and documents are mapped to dense vectors $\mathbf{q}, \mathbf{d} \in \mathbb{R}^d$ , and retrieval is cast as nearest-neighbor search under inner product or cosine similarity. Lucene’s implementation expects L2-normalized vectors and therefore effectively uses cosine similarity. The paper exposes this capability through two new Anserini classes, IndexHnswDenseVectors and SearchHnswDenseVectors, mirroring the BM25-style indexing and search pattern already familiar to Anserini users (Ma et al., 2023).

The system-level significance is unification. Sparse retrieval still uses Lucene inverted indexes through IndexCollection and SearchCollection, while dense retrieval uses Lucene HNSW vector indexes through IndexHnswDenseVectors and SearchHnswDenseVectors. Hybrid retrieval can therefore be implemented inside one Lucene-based runtime rather than by stitching together Lucene and Faiss. The paper is explicit that this is not a new ANN algorithm; it is an integration pattern.

Because Lucene’s HNSW requires L2-normalized vectors, the authors train cosDPR-distil, a dense retriever aligned to cosine similarity. On MS MARCO passage, the reported effectiveness of Lucene HNSW is nearly indistinguishable from flat exact search with that model: flat exact search gives MRR@10 $= 0.390$ and Recall@1k $= 0.980$ , whereas Lucene HNSW gives MRR@10 $= 0.389$ and Recall@1k $= 0.974$ (Ma et al., 2023). On BEIR, average nDCG@10 is $M$ 0 for flat dense retrieval and $M$ 1 for HNSW. Operationally, the paper also reports that Lucene HNSW is slower to build than Faiss HNSW but scales well in multi-threaded querying, with 16-thread search at $M$ 2 reaching 373 QPS versus 164.5 QPS for Faiss HNSW in the reported setup (Ma et al., 2023).

Within this usage, d‑HNSW therefore denotes dense vector retrieval using Lucene’s native HNSW support, chiefly valued for backend unification, hybrid-retrieval simplification, and reproducible experimentation.

3. Disaggregated-memory d‑HNSW

In "Efficient Vector Search on Disaggregated Memory with d-HNSW" (Liu et al., 17 May 2025) and its later system formulation "d-HNSW: A High-performance Vector Search Engine on Disaggregated Memory" (Fang et al., 13 Mar 2026), d‑HNSW denotes a redesign of HNSW-based vector search for RDMA-enabled disaggregated memory. The compute pool contains CPU-rich nodes with limited DRAM acting as a cache, while the memory pool provides a large RDMA-registered remote address space. The system preserves HNSW’s greedy graph-search behavior, but reorganizes index structure, memory layout, and execution so that pointer chasing does not translate into an RDMA round-trip at each graph step.

The core mechanism is a split between a small cached representative index and large remote sub-indexes. In the 2025 system, the representative structure is a meta-HNSW built from a uniform random sample of 500 vectors; each bottom-layer node of this meta-HNSW defines a cluster and serves as the entry point for a sub-HNSW built over the vectors assigned to that cluster (Liu et al., 17 May 2025). Query processing runs the meta-HNSW entirely in local DRAM, selects the top relevant sub-HNSW clusters, fetches those subgraphs from remote memory, and then runs ordinary HNSW search locally on the fetched subgraphs. The 2026 paper generalizes this with balanced clustering, a lightweight representative index, an RDMA-friendly graph layout, query-aware data loading, and a pipelined execution model that overlaps RDMA transfers with computation (Fang et al., 13 Mar 2026).

The remote-memory layout is central. Sub-HNSW data are stored in contiguous groups with shared overflow regions so that dynamic insertions do not fragment each cluster into multiple RDMA fetches. Query-aware batching deduplicates cluster loads across multiple queries, and NIC doorbell batching amortizes network round trips (Liu et al., 17 May 2025). In the 2026 version, the system additionally leaves internal gaps inside serialized sub-index components and uses epoch-based rebuilds when overflow regions are exhausted (Fang et al., 13 Mar 2026).

The reported effect is large. On realistic hardware with three compute nodes and one memory node, the 2025 paper reports that d‑HNSW outperforms a naive disaggregated HNSW by up to 117× in latency while maintaining recall as 0.87 in dataset SIFT1M@1 (Liu et al., 17 May 2025). The 2026 system reports up to $M$ 3 query throughput improvement while maintaining a high recall of 94% (Fang et al., 13 Mar 2026). These designs define d‑HNSW as a systems architecture for remote-memory ANN, not as a change to HNSW’s search criterion.

A related but distinct disaggregated design is SHINE, described as a scalable HNSW index in disaggregated memory that preserves the full HNSW graph rather than partitioning it logically. SHINE stores a global HNSW across memory nodes, uses caches on compute nodes, and “logically combine[s] the caches of the compute nodes” through balanced clustering and adaptive routing to reduce the cache segmentation penalty while preserving the same accuracy as a single-machine HNSW (Widmoser et al., 23 Jul 2025). This suggests that disaggregated-memory d‑HNSW has already bifurcated into at least two strategies: partitioned sub-HNSWs with representative routing, and graph-preserving remote HNSW with cache specialization.

4. De-hierarchized and topology-modified interpretations

A different line of work uses the d‑HNSW label in a structural sense. "Down with the Hierarchy: The 'H' in HNSW Stands for 'Hubs'" (Munyampirwa et al., 2024) studies when the hierarchy can be dropped entirely. The paper introduces FlatNav, effectively HNSW without upper layers: one keeps the base-layer graph and runs the same greedy or beam search over that single layer. On high-dimensional real datasets, FlatNav and HNSW are reported to be practically indistinguishable in p50 and p99 latency-versus-recall curves, while using less memory; on BigANN 100M, peak memory is 183 GB for HNSW and 113 GB for FlatNav (Munyampirwa et al., 2024).

The explanatory mechanism is the Hub Highway Hypothesis: in high-dimensional spaces, the base graph develops a small set of well-connected, heavily traversed hub nodes that play the same navigational role as HNSW’s upper layers. The paper proposes a practical rule of thumb: for $M$ 4, hierarchy remains useful; for $M$ 5, a flat navigable small-world graph is often sufficient (Munyampirwa et al., 2024). In this interpretation, d‑HNSW can mean de‑hierarchized HNSW or dimension-aware HNSW.

"Dual-Branch HNSW Approach with Skip Bridges and LID-Driven Optimization" (Nguyen et al., 23 Jan 2025) introduces another structural meaning. There, d‑HNSW refers to a dual-branch HNSW, also called HNSW++, that maintains two separate HNSW-style branches above a shared base layer. Nodes are assigned to branches and levels using normalized Local Intrinsic Dimensionality, high-LID points are preferentially promoted to upper layers, and skip bridges allow search to jump directly to layer 0 when a high-LID, near-query entry point is detected (Nguyen et al., 23 Jan 2025). The paper reports recall improvements of 18% in NLP and up to 30% in CV tasks, construction-time reductions of up to 20%, and maintained inference speed (Nguyen et al., 23 Jan 2025).

These two usages are conceptually opposite. The de‑hierarchized reading removes layers because high-dimensional hub structure makes them redundant, whereas the dual-branch reading adds extra hierarchical structure to improve connectivity, outlier capture, and robustness to local minima. This suggests that d‑HNSW, in structural usage, is best understood as an umbrella for topology re-interpretations of HNSW rather than a single topological doctrine.

Adjacent work reinforces that broader point. QuIVer is not an HNSW variant, but it is explicitly positioned against HNSW and Vamana-style graph indexes and shows that graph construction, pruning, and navigation can be performed entirely in a 2-bit sign–magnitude binary space for cosine-native, low-effective-dimensional data (Xiao et al., 4 May 2026). A plausible implication is that the space of “modified HNSW” research is increasingly organized around data-distribution assumptions, not only around graph engineering.

5. Dynamic-update and maintenance-oriented d‑HNSW

Another recurring interpretation treats d‑HNSW as dynamic HNSW: an HNSW variant intended for long-running insertion, deletion, and update workloads. "Enhancing HNSW Index for Real-Time Updates: Addressing Unreachable Points and Performance Degradation" (Xiao et al., 2024) analyzes the standard replaced_update pathway and identifies two failures under long update sequences: poor update efficiency and the unreachable points phenomenon. The paper defines an unreachable point as a point that possesses outgoing edges but lacks incoming edges in all layers of the navigable small-world graph (Xiao et al., 2024).

The mechanism is structural degradation under delete-and-reinsert cycles. If the only incoming edges to some node are dropped during local repair, that node may remain in the graph yet become unreachable from the entry point. On SIFT, under 3000 iterations of 5% random deletion and reinsertion, the number of unreachable points grows monotonically to 3–4% of the index, and recall drops by about 3%; increasing efSearch cannot recover these nodes because they are structurally inaccessible (Xiao et al., 2024).

The proposed response is MN‑RU (Mutual Neighbor Replaced Update), together with variants MN‑RU $M$ 6, $M$ 7, $M$ 8, and MN‑THN‑RU. Instead of repairing all one-hop neighbors and their large two-hop candidate sets, MN‑RU restricts repair to mutually connected neighbors of the deleted point and builds smaller candidate sets from original neighbors, the deleted point’s neighbors, and the incoming replacement point (Xiao et al., 2024). The paper reports that MN‑RU variants are 2–4× faster than the baseline HNSW replaced-update procedure and significantly suppress the growth of unreachable points across full-coverage, random, and new-data scenarios.

A backup-index mechanism complements this. The system periodically enumerates reachable points in the main HNSW, builds a second HNSW over the unreachable set, and answers queries with a dualSearch that merges results from the main and backup indexes (Xiao et al., 2024). Although the paper notes that the term d‑HNSW is not explicitly used there, the design fits the practical meaning of a dynamic or update-capable HNSW and has become part of the semantic range attached to the label.

6. Distribution-aware adaptive search

A further meaning of d‑HNSW is distribution-aware HNSW. "Distribution-Aware Exploration for Adaptive HNSW Search" (Zhang et al., 7 Dec 2025) leaves the HNSW index structure unchanged but dynamically chooses $M$ 9 per query to approximately meet a declarative target recall. The underlying claim is that a static, globally configured $efConstruction$ 0 is distribution-agnostic and therefore produces both over-searching on easy queries and under-searching on hard ones.

The method, Ada‑ef, models the full distance list between a query and the database as approximately Gaussian for inner product, cosine similarity, and cosine distance. Offline, it computes a mean vector and covariance matrix for the database. Online, it estimates the query-specific distance-distribution parameters, collects a small list of early distances from a 2-hop base-layer exploration, bins those distances by global quantiles, and computes a weighted difficulty score: $efConstruction$ 1 where $efConstruction$ 2 counts how many collected distances fall into bin $efConstruction$ 3, and $efConstruction$ 4 emphasizes the smallest-distance bins (Zhang et al., 7 Dec 2025). That score is then mapped, through an offline calibration table, to the smallest $efConstruction$ 5 that meets the target recall for similar-score queries.

This design preserves HNSW’s top-down search and base-layer best-first search; only the choice of $efConstruction$ 6 changes. Empirically, the paper reports that compared with state-of-the-art learning-based adaptive approaches, Ada‑ef achieves the target recall while reducing online query latency by up to 4×, offline computation time by 50×, and offline memory usage by 100× (Zhang et al., 7 Dec 2025). It is also update-friendly because the statistics and sampled query ground truth can be incrementally maintained after insertions and deletions.

In this usage, d‑HNSW no longer refers to storage architecture or graph topology. It refers to a search policy that is aware of the similarity distribution induced by the embedding space. This broadens the term beyond “where the graph lives” or “how the graph is wired” into “how search effort is adapted.”

7. Comparative perspective and significance

Across these lines of work, d‑HNSW denotes at least four distinct intervention layers on top of standard HNSW: retrieval-stack integration, systems architecture, graph topology, and query-time control. Dense Lucene d‑HNSW modifies the software stack while keeping Lucene’s HNSW intact (Ma et al., 2023). Disaggregated-memory d‑HNSW restructures storage, routing, and execution to make HNSW viable over RDMA (Liu et al., 17 May 2025, Fang et al., 13 Mar 2026). De‑hierarchized and dual-branch d‑HNSW reinterpret the graph topology itself (Munyampirwa et al., 2024, Nguyen et al., 23 Jan 2025). Dynamic and distribution-aware d‑HNSW alter maintenance and search policy, respectively (Xiao et al., 2024, Zhang et al., 7 Dec 2025).

The common benchmark for all of them remains the original HNSW formulation: a hierarchical navigable small-world graph with bounded degree, random exponential level assignment, greedy descent, and an $efConstruction$ 7-controlled base-layer search (Malkov et al., 2016). What the d‑HNSW literature demonstrates is that HNSW has become less a single immutable ANN index than a design substrate from which multiple specialized systems can be derived.

This suggests a useful editorial reading of the term. In contemporary usage, d‑HNSW is best treated not as a singular named algorithm but as a family label for context-specific HNSW adaptations. Any technical discussion of d‑HNSW therefore requires immediate disambiguation: dense retrieval in Lucene, disaggregated memory, de‑hierarchization, dual-branch topology, dynamic maintenance, or distribution-aware exploration are materially different objects, even though all inherit the navigable small-world search core.