BatANN: Distributed Disk-Based ANN Search

• BatANN is a distributed, disk-based ANN search system designed for massive vector datasets using a global proximity graph and an efficient baton-passing protocol.
• It partitions the global graph with a neighborhood-aware algorithm to minimize inter-server I/O and maintain logarithmic search efficiency.
• Performance evaluations demonstrate low latency (<6ms) and high recall (0.95@10) with near-linear throughput scaling across multiple servers.

BatANN is a distributed, disk-based approximate nearest neighbor (ANN) search system designed to support high-throughput, low-latency vector search over datasets too large to fit in memory or on a single server. By integrating a single global proximity graph, neighborhood-aware partitioning, and a baton-passing protocol that transfers full query state across servers, BatANN ensures logarithmic search efficiency, minimal inter-server I/O, and near-linear throughput scaling as the number of servers increases. It is the first open-source distributed disk-based vector search system to operate over a single global graph (Dang et al., 10 Dec 2025).

1. Problem Setting and Motivation

Modern information-retrieval applications—including retrieval-augmented generation (RAG), large-scale image search, and recommendation—require efficient $k$-nearest neighbor (k-NN) queries in high-dimensional embedding spaces. A brute-force (exact) search has complexity $O(N)$ or worse due to dimensionality, making it impractical for large $N$. Graph-based ANN methods, empirically achieving $O(\log N)$ convergence using proximity graphs, are the dominant scalable solution.

As dataset sizes approach billions of points, global index structures exceed DRAM capacity. Disk-based systems (e.g., DiskANN, Starling) store full embeddings and neighbor lists on SSDs, retaining quantized vector summaries (PQ codes) in memory to prune candidate sets. However, a single-server design is ultimately limited by SSD IOPS and bandwidth. Distributing storage and computation across multiple machines can raise query-per-second (QPS) throughput, but naïve sharding ("scatter–gather") strategies lose logarithmic search efficiency and waste I/O. Prior distributed solutions based on global graphs incur high inter-server latency due to round-trip communication per graph hop.

Target system metrics are Recall@k (e.g., 0.95 at $k=10$), throughput (QPS) at fixed recall, and end-to-end latency below 6 ms even at very high QPS (Dang et al., 10 Dec 2025).

2. Index Structure and System Architecture

2.1 Global Proximity Graph (Vamana)

BatANN constructs a single global proximity graph $G = (V, E)$ using the Vamana algorithm. Each node in $V$ (representing a datapoint) maintains up to $R$ out-edges ($R=64$ in practice), selected by a diversifying heuristic to improve graph navigability. Formally, each node $i$ retains a neighbor set $O(N)$0, $O(N)$1, designed so that greedy traversal from any starting point converges to the true nearest neighbors in $O(N)$2 steps.

Indexing details:

• On-disk: Each vector's full (float) embedding and its neighbor ID list are stored, typically compressed to fit within a 4 KB SSD sector.
• In-memory: Product-quantized (PQ) codes, consuming 32 bytes per 128-dimensional vector, guide the search. Overall DRAM usage is $O(N)$3 bytes.

2.2 Distributed Graph Partitioning

The global graph is partitioned into $O(N)$4 server shards using a neighborhood-aware graph partitioner [arXiv (Gottesbüren et al., 2024)], minimizing off-server beam-search hops. Empirical results indicate that, with 10 servers, only 20–25% of graph hops cross server boundaries.

Each server is provisioned as follows:

• Replicated PQ codes for all $O(N)$5 vectors.
• Local SSD storage for embeddings and neighbor lists of its partition.
• An in-memory “head” graph for rapid starting-point selection, consisting of a 1% random sample of $O(N)$6.

3. The Baton-Passing Protocol

3.1 Motivation

Previous distributed global-graph approaches incur at least two network round-trips per off-server neighbor access (request + response). BatANN’s key innovation is to transfer ("pass") the entire query’s state to the owning server, which then continues the beam search locally. This asynchronous baton-passing approach reduces per-hop network overhead by nearly half and aligns with modern high-throughput networking and SSD architectures.

3.2 Query State and Protocol

A query’s “baton” (state $O(N)$7) includes:

• Beam $O(N)$8 (size $O(N)$9, each entry $N$0node, approx-dist$N$1),
• Explored set $N$2,
• Full-precision result list for reranking,
• Parameters $N$3 and the query embedding $N$4.

For typical $N$5, message size is $N$6–$N$7 KB.

The distributed beam-search protocol operates as follows:

1. The client hashes $N$8 to a designated server $N$9, which performs an in-memory head-index search and initializes $O(\log N)$0.
2. While unexplored beam nodes remain:
   • Select up to $O(\log N)$1 unexplored nodes.
   • If any are local: issue concurrent IO_uring SSD reads and update beam/explored set.
   • If all are remote: select the node $O(\log N)$2 with minimum distance, serialize $O(\log N)$3, and transfer the baton to the server owning $O(\log N)$4. That server resumes execution.
3. Once all beam nodes are explored, the holding server returns top-$O(\log N)$5 results to the client.

Each baton-passing step incurs latency $O(\log N)$6 (serialization, $O(\log N)$7s) plus $O(\log N)$8 (TCP, $O(\log N)$9s on 25 GbE) per inter-server hop.

4. Complexity, Efficiency, and Optimizations

4.1 Computational Cost

On a single server holding $k=10$0 points, greedy beam search converges in $k=10$1 steps. Retaining a single global graph ensures total hops remain $k=10$2 regardless of $k=10$3. Each step issues up to $k=10$4 concurrent SSD reads, and with high-random-IOPS NVMe SSDs ($k=10$5300K), increasing $k=10$6 has negligible impact on per-step read cost.

4.2 Throughput Scaling

Because per-query disk I/O and distance computation are nearly constant irrespective of $k=10$7, distributing the same workload enables near-linear scalability. Throughput on $k=10$8 servers can be modeled as $k=10$9, where $G = (V, E)$0 accounts for inter-server hop overhead.

4.3 Locality and Algorithmic Heuristics

Graph partitioning (arXiv (Gottesbüren et al., 2024)) yields 75–88% of hops that remain local, limiting baton passes. When $G = (V, E)$1, a heuristic additionally processes all local candidates before considering baton passing, further reducing inter-server communication.

4.4 Batching and Caching

Each thread processes a fixed set of 8 queries in a pipelined, interleaved manner to overlap SSD I/O and computation. This yields 20–30% higher single-server throughput. Query embedding $G = (V, E)$2 is cached after the initial transmission and is reused until final result acknowledgment, eliminating duplication.

5. Experimental Evaluation

5.1 Datasets and Experimental Setup

The system is evaluated on 10-node CloudLab c6620 clusters (28-core Intel Xeon Gold, 128 GB DRAM, 25 GbE) using standard Big-ANN benchmarks:

• BIGANN: 100M and 1B SIFT 128-dim (uint8), L2 distance,
• MSSPACEV: 100M and 1B SpaceV 100-dim (int8),
• DEEP: 100M 96-dim float conv-net features.

5.2 Baselines

• ScatterGather: Identical graph and partitioning, but each subgraph is queried independently then merged (no global graph).
• Single-server baselines: DiskANN, PipeANN, CoroSearch (for microbenchmarking).

Index built using Vamana with $G = (V, E)$3, $G = (V, E)$4, $G = (V, E)$5 (PipeANN for 100M; ParlayANN for 1B).

5.3 Performance Highlights

At Recall@10 = 0.95:

• For 100M points, 10 servers, BatANN achieves 6.21–6.49$G = (V, E)$6 QPS improvement vs. ScatterGather (e.g., 6.49$G = (V, E)$7 on DEEP).
• For 1B points, 10 servers, BatANN achieves 5.10$G = (V, E)$8 (BIGANN), 2.5$G = (V, E)$9 (MSSPACEV).
• BatANN’s total disk I/O and distance calculations remain within 1% of single-server levels; in contrast, ScatterGather’s I/O and compute scale proportionally with $V$0.
• QPS scales as $V$1 at fixed recall with 10 servers; ScatterGather saturates after a few shards.

Dataset	Size	Servers	QPS Speedup (BatANN vs ScatterGather)	Mean Latency (ms)
BIGANN	100M, 1B	10	6.21–6.49× (100M), 5.10× (1B)	$V$26
MSSPACEV	100M, 1B	10	6.21–6.49× (100M), 2.5× (1B)	$V$36
DEEP	100M	10	6.49×	$V$46

At saturating QPS rates, BatANN maintains mean latency $V$5 ms (rising only 13% from 5 to 10 servers), whereas ScatterGather’s tail latency degrades rapidly above 6K QPS.

5.4 Beam Width Ablation

Ablating beam width from $V$6 to $V$7 reduces total and inter-server hops by approximately 4$V$8, yielding over 2$V$9 higher QPS and about half the latency at 0.95 recall@10, with no extra CPU or I/O cost.

6. Limitations and Future Directions

• Message size: Each baton currently sends the entire result list ($R$0 nodes); streaming only the beam state and piggybacking partial results is a proposed future optimization.
• Disk layout: Incorporating locality-aware disk placement (as in Starling’s 4 KB-aligned strategy) and dynamic pipeline width (as in PipeANN) could further minimize I/O and latency.
• Dynamic updates: Efficient distributed point insert/delete without full repartitioning remains unresolved; multi-node in-place update mechanisms have yet to be generalized.
• Fault tolerance: There is no current replication or fail-over; integration with state machine replication systems such as Derecho is a suggested extension.
• Network optimizations: Although commodity TCP suffices for multi-KB baton messages, the architecture would readily enable RDMA or CXL deployment, whose benefits are yet to be quantified.
• Multitenancy: Resource sharing and scheduling for multiple distinct vector search databases per cluster remain open research issues.

A plausible implication is that BatANN’s methodology may serve as a blueprint for distributed sublinear-time search over massive graph-structured data, given continued progress on distributed dynamic indexing and system-level robustness (Dang et al., 10 Dec 2025).