FlashANNS: GPU-Accelerated Out-of-Core ANNS

• FlashANNS is a GPU-accelerated, SSD-resident graph-based ANNS system designed for billion-scale vector retrieval by overlapping SSD I/O with GPU computation.
• It employs a dependency-relaxed asynchronous pipeline and warp-level concurrent SSD access to minimize latency and maximize throughput.
• The system balances graph degree selection with compute and storage constraints to effectively optimize performance in large-scale retrieval scenarios.

Searching arXiv for the cited FlashANNS paper and related baseline systems to ground the article with current identifiers. FlashANNS is a GPU-driven, SSD-backed, out-of-core graph approximate nearest neighbor search system for billion-scale vector search, introduced to address the storage-compute bottleneck in settings where the dataset and graph index do not fit in main memory (Xiao et al., 14 Jul 2025). It targets the common retrieval regime in information retrieval, recommendation systems, and retrieval-augmented generation, where high-dimensional query embeddings must be matched against corpora containing hundreds of millions to billions of vectors. The system’s central premise is that prior out-of-core ANNS designs do not jointly optimize SSD I/O and distance computation, leading either storage or compute to remain idle. FlashANNS addresses this by combining a dependency-relaxed asynchronous pipeline, warp-level concurrent SSD access, and computation-I/O balanced graph degree selection within a GPU-centered execution model (Xiao et al., 14 Jul 2025).

1. Problem setting and bottleneck characterization

Approximate nearest neighbor search is used in embedding retrieval pipelines across search, recommendation, and RAG. In the out-of-core setting, vectors and graph nodes are stored on SSDs and retrieved on demand during search. The challenge is not simply bulk data access: high-recall graph-based ANNS requires iterative traversal in which each step expands neighbors, computes distances, updates candidate structures, and determines which node to visit next. This produces fine-grained, latency-sensitive, data-dependent storage accesses (Xiao et al., 14 Jul 2025).

FlashANNS is motivated by what the paper calls the storage-compute bottleneck. Existing systems suffer from two limitations: failing to overlap SSD accesses with distance computation, and extended I/O latency caused by a suboptimal I/O stack (Xiao et al., 14 Jul 2025). In strict best-first traversal, the loop follows a serialized fetch-compute-fetch pattern: a node is fetched from SSD, distances are computed, the next candidate is chosen, and only then can the next SSD access be issued. This dependency chain prevents pipelining.

The pressure of this bottleneck is intensified by scale. The paper notes that indices can reach up to $6\times$ raw data size, so a 400GB dataset may imply multi-terabyte memory requirements (Xiao et al., 14 Jul 2025). SSDs provide capacity in the 10TB–100TB range and are therefore attractive for billion-scale corpora, but graph traversal produces many small random reads whose latency characteristics interact poorly with conventional search loops. FlashANNS is designed specifically for this regime.

2. Relation to prior out-of-core ANNS systems

The paper situates FlashANNS against SPANN, DiskANN, and FusionANNS as representative out-of-core baselines (Xiao et al., 14 Jul 2025). The comparison is organized around the granularity of search and the degree to which I/O and compute are coordinated.

SPANN is described as a cluster-based system that stores posting lists on SSD and centroids in memory. Query processing first finds relevant centroids, then reads the corresponding posting lists, then computes exact distances for all candidates. The paper characterizes this as inherently coarse-grained because centroids are imperfect surrogates. On SIFT1B, at recall@10 of 81%, 85%, 90%, and 95%, candidate vector counts are larger than DiskANN by $1.14\times$, $1.37\times$, $1.78\times$, and $2.34\times$, respectively, increasing both SSD traffic and compute (Xiao et al., 14 Jul 2025).

FusionANNS improves cluster-based search by keeping PQ codes on the GPU and using quantized prefiltering before SSD fetches. According to the paper, the coarse cluster structure still causes excessive candidate generation, and FusionANNS incurs $1.36\text{–}1.70\times$ more SSD page accesses than DiskANN on SIFT1B (Xiao et al., 14 Jul 2025). The paper further reports that FusionANNS splits work asymmetrically across CPU and GPU: CPU-side tasks take $2.7\times\text{–}4.9\times$ more time per query than GPU prefiltering on SIFT1B, leaving the GPU underutilized (Xiao et al., 14 Jul 2025).

DiskANN is identified as the strongest prior graph-based out-of-core baseline. It stores full-precision graph nodes on SSD and PQ-compressed vectors in memory, which reduces candidate counts and SSD accesses relative to cluster-based methods. FlashANNS nevertheless identifies three limitations in DiskANN: CPU-bound execution, I/O amplification from subpage access, and lack of I/O-compute overlap due to synchronous traversal dependencies (Xiao et al., 14 Jul 2025). Even at 54 cores, the measured SSD bandwidth reaches only 2.85 GB/s, less than 15% of a modern SSD system’s approximately 20 GB/s peak (Xiao et al., 14 Jul 2025). At graph degree 64, each vertex retrieval is only 384B, comprising $64$ neighbor IDs at $4$B each plus 128B metadata, which is only 9.37% of a 4KB SSD read unit; thus 90.63% of the read bandwidth is wasted per operation (Xiao et al., 14 Jul 2025).

This comparison establishes the design space in which FlashANNS operates. Unlike SPANN and FusionANNS, it does not rely on coarse posting-list retrieval as the primary search primitive. Unlike DiskANN, it moves the graph search engine to the GPU and explicitly pipelines SSD I/O with distance computation (Xiao et al., 14 Jul 2025).

3. System architecture and data placement

FlashANNS is presented as a GPU-accelerated, SSD-based graph ANNS framework in which the GPU is the main search engine while graph nodes remain out of core on SSDs (Xiao et al., 14 Jul 2025). The architecture combines GPU-based distance computation and query-parallel graph traversal, SSD-resident graph and vector-node data, host-side management of the SSD I/O path, and an asynchronous pipeline that overlaps fetching and computation.

The paper describes three main mechanisms: dependency-relaxed pipelined I/O-compute processing, warp-level GPU-SSD direct I/O, and sampling-based graph-degree selection (Xiao et al., 14 Jul 2025). These mechanisms are organized around explicit data placement. On SSD reside full-precision vector nodes and complete neighbor lists. On the GPU reside query processing state, distance computation kernels, per-warp buffers, and completion signaling structures. On CPU or host memory resides the I/O request array managed by the host side of the custom I/O stack (Xiao et al., 14 Jul 2025).

Query execution is warp-centric. A query is assigned to a dedicated GPU warp of 32 threads. One leader thread manages the result priority queue, the candidate min-max heap, and I/O synchronization, while all 32 threads participate in distance computation and data movement (Xiao et al., 14 Jul 2025). The search begins from an entry point in the graph, although the exact initialization rule is not specified. Processing then iterates through frontier expansion, SSD fetches for upcoming graph nodes, double-buffered overlap between compute and I/O, and continued priority-driven traversal until termination (Xiao et al., 14 Jul 2025).

The paper repeatedly emphasizes 4KB as the SSD access granularity. Per-warp buffer slots are 4KB each, and graph node layout is tuned with awareness of this unit (Xiao et al., 14 Jul 2025). This design choice is central both to the I/O stack and to the graph-degree selection strategy.

4. Dependency-relaxed asynchronous pipeline

The central algorithmic innovation is a dependency-relaxed asynchronous pipeline that breaks the strict adjacent-step dependency of best-first graph traversal (Xiao et al., 14 Jul 2025). In the strict form, computation in step $i$ depends on data fetched in step $1.14\times$0, and the result of step $1.14\times$1 determines what SSD data to fetch for step $1.14\times$2. FlashANNS relaxes this by allowing one-step staleness: candidate selection for step $1.14\times$3 can depend on step $1.14\times$4 instead of strictly on step $1.14\times$5. This permits SSD retrieval for step $1.14\times$6 and distance computation for step $1.14\times$7 to execute concurrently (Xiao et al., 14 Jul 2025).

The paper states that one-step staleness is chosen because it is sufficient to expose the available overlap, because more than one step of staleness brings no clear further overlap benefit, and because multi-step staleness may hurt search quality by steering traversal using overly stale neighbor information (Xiao et al., 14 Jul 2025). Implementation uses double buffering: one buffer is consumed by the current GPU traversal computation while the other is filled by SSD requests for the next phase, with the buffers alternating in ping-pong fashion (Xiao et al., 14 Jul 2025).

A formal bound is given for path inflation under this relaxation. For strict search path $1.14\times$8 of length $1.14\times$9, and relaxed path $1.37\times$0, with step deviation $1.37\times$1, the paper states

$1.37\times$2

with base case

$1.37\times$3

By induction,

$1.37\times$4

and after $1.37\times$5 steps the maximal cumulative detour depth $1.37\times$6 satisfies

$1.37\times$7

The intended conclusion is that single-step relaxation cannot increase path length by more than $1.37\times$8 (Xiao et al., 14 Jul 2025).

Empirically, the observed increase is much smaller. The paper reports that the baseline best-first traversal averages 64.1 steps, while the relaxed version averages 67.1 steps, a 4.7% increase (Xiao et al., 14 Jul 2025). It attributes the modest gap to two observations: only 24.3% of baseline steps, or 15.6 out of 64.1, depended critically on immediate predecessor freshness, and the underlying Vamana graph includes long edges that can still provide useful navigational shortcuts under slightly stale decisions (Xiao et al., 14 Jul 2025).

The pipeline effect is substantial in the reported ablations. On SIFT1B with a 250-degree graph and 4 SSDs, the measured overlap ratio is 36%–47%, GPU utilization rises from 36.3% to 79.8%, and end-to-end QPS improves by 33.6%–46.6% over the non-pipelined version (Xiao et al., 14 Jul 2025).

5. Warp-level concurrent SSD access

FlashANNS complements pipelined traversal with a warp-level concurrent SSD access mechanism designed to prevent the I/O path itself from reintroducing serialization (Xiao et al., 14 Jul 2025). The design builds on CAM, described in the paper as “Asynchronous GPU-Initiated, CPU-Managed SSD Management for Batching Storage Access,” but changes the synchronization granularity from kernel-level to per warp (Xiao et al., 14 Jul 2025).

The motivation is tail-latency sensitivity. Native CAM uses kernel-level global synchronization, so all warps in a kernel wait for the slowest SSD request before progressing. In the random-read workload of graph ANNS, this creates cross-warp blocking. FlashANNS instead gives each warp its own request state, completion status, and 4KB destination buffer, so a warp can resume as soon as its own SSD request completes, independent of other warps (Xiao et al., 14 Jul 2025).

The custom I/O stack has three structures: a CPU-hosted I/O request array, a GPU-resident completion signal array, and GPU buffer space of

$1.37\times$9

Each array element corresponds to a warp (Xiao et al., 14 Jul 2025). The workflow is that a warp writes a request into its designated slot, the CPU aggregates 4KB random-read requests and dispatches them to SSD, the data are transferred into that warp’s GPU buffer slot, and the corresponding completion flag is updated. At synchronization points, the warp checks its own completion signal and resumes computation if ready (Xiao et al., 14 Jul 2025).

The paper describes this mechanism as a lock-free I/O stack with warp-level concurrency control, but does not specify exact queue structures, memory registration details, DMA protocol details, use of io_uring or SPDK, request coalescing algorithms beyond CPU aggregation, or precise NVMe command fusion mechanics (Xiao et al., 14 Jul 2025). The architecture is therefore explicit at the systems level while leaving several low-level implementation details unspecified.

The measured throughput gain from warp-level access is clear. On SIFT1B with 4 SSDs and a 250-degree graph, throughput improves by 43%–68% relative to kernel-level synchronized I/O (Xiao et al., 14 Jul 2025).

6. Computation-I/O balanced graph degree selection

Graph degree in FlashANNS is treated as a joint compute-storage control variable rather than solely an index-quality parameter (Xiao et al., 14 Jul 2025). Higher degree increases computation per step because more neighbors must be examined, but it also tends to improve graph navigability, reducing the number of traversal steps. In the out-of-core setting there is an additional storage-side effect: as long as node data remain under the 4KB SSD read size, increasing degree can improve useful payload per SSD read without increasing per-access I/O cost proportionally (Xiao et al., 14 Jul 2025).

This addresses the subpage inefficiency highlighted earlier. At degree 64, a node occupies only 384B, so most of a 4KB SSD read is underutilized (Xiao et al., 14 Jul 2025). Increasing degree packs more useful graph data into each access. However, if degree is raised too far, the system becomes compute-bound on the GPU. The optimal degree therefore depends on the available SSD bandwidth and the GPU’s computational capacity (Xiao et al., 14 Jul 2025).

FlashANNS uses a two-phase profiling and selection process. First, before full indexing, it constructs lightweight sample graph indices with varying degrees; these preserve data type characteristics but omit actual neighbor relationships. It then measures stage latencies for compute and SSD I/O under the target hardware. Second, it selects the graph degree that best balances the pipeline stages, qualitatively maximizing overlap and minimizing idle time (Xiao et al., 14 Jul 2025). The paper does not provide a closed-form optimization objective or an explicit formula for the selector.

The reported DEEP1B profiling illustrates the balancing principle. With 1 SSD, a 150-degree graph yields I/O latency $1.78\times$0 compute latency, while a 250-degree graph yields $1.78\times$1, so the workload remains I/O-bound and higher degree remains beneficial. With 2 SSDs, the corresponding ratios are $1.78\times$2 and $1.78\times$3, making 250-degree nearly balanced. With 4 SSDs, the ratios become $1.78\times$4 and $1.78\times$5, indicating that 250-degree has become compute-bound and the optimum lies between 150 and 250 (Xiao et al., 14 Jul 2025).

The ablation results are consistent with this interpretation. On DEEP1B, with 1–2 SSDs, higher-degree graphs perform better because the workload remains I/O-bound; with 8 SSDs, lower degree becomes preferable due to compute pressure. Specifically, under 8 SSDs, a 150-degree graph outperforms 250-degree by 13% at 96% recall@10 (Xiao et al., 14 Jul 2025). This suggests that FlashANNS treats degree as a hardware-aware systems parameter rather than a fixed indexing constant.

7. Experimental results, scope, and limitations

The evaluation uses a single server with 2 × Intel Xeon Gold 5320 at 2.20 GHz, 52 threads total, 768 GB DDR4 DRAM, an NVIDIA A100 80GB PCIe GPU, and 8 × 3.84TB Intel P5510 NVMe SSDs under a PCIe 4.0 x16 topology on Ubuntu 22.04 LTS (Xiao et al., 14 Jul 2025). Three billion-scale datasets are reported: SIFT1B with 1B vectors of 128D uint8 and 10,000 queries, DEEP1B with 1B vectors of 96D float32 and 10,000 queries, and SPACEV1B with 1B vectors of 100D int8 and 29,300 queries (Xiao et al., 14 Jul 2025). The principal metric is throughput versus recall@10, with emphasis on performance at $1.78\times$6 recall@10.

In single-SSD configuration, at $1.78\times$7 recall@10, FlashANNS achieves $1.78\times$8 higher throughput than the state-of-the-art baselines (Xiao et al., 14 Jul 2025). At 98% recall@10, it is roughly comparable to FusionANNS on SIFT1B and SPACEV1B, at $1.78\times$9 throughput, while on DEEP1B it achieves $2.34\times$0 higher throughput than FusionANNS, which the paper attributes to FusionANNS’s CPU exact-distance burden on float32 data (Xiao et al., 14 Jul 2025).

In multi-SSD settings, FlashANNS scales with additional storage bandwidth. On SIFT1B at 95% recall@10, throughput improves over baselines by $2.34\times$1 with 2 SSDs, $2.34\times$2 with 4 SSDs, and $2.34\times$3 with 8 SSDs (Xiao et al., 14 Jul 2025). The abstract reports a broader multi-SSD throughput improvement range of $2.34\times$4 (Xiao et al., 14 Jul 2025). The paper also compares FlashANNS with an in-memory variant of FusionANNS and states that FlashANNS still outperforms it in the reported SIFT experiment, although exact numeric values are not given in the provided text (Xiao et al., 14 Jul 2025).

The paper further reports an out-of-core scalability demonstration by expanding DEEP1B to a 30B-vector dataset of 1,073 GB, where FlashANNS maintains a good throughput-recall tradeoff, though detailed numbers are not included in the excerpt (Xiao et al., 14 Jul 2025). The same text refers to this as “exabyte-scale,” but 1,073 GB is approximately 1 TB; the correct interpretation is therefore TB-scale. This discrepancy is textual rather than architectural.

Several implementation details remain unspecified in the available description. These include the exact graph build algorithm and full Vamana parameters, the exact stopping condition for search, precise low-level SSD I/O mechanisms, memory footprint and caching policy, and exact formulas for graph-degree optimization (Xiao et al., 14 Jul 2025). Accordingly, the design is clearly articulated at the systems and algorithmic levels, while some engineering details remain high level.

FlashANNS is best understood as a co-designed out-of-core ANNS engine in which graph traversal, asynchronous storage access, and hardware-aware degree tuning are jointly arranged to keep both SSDs and GPU compute active (Xiao et al., 14 Jul 2025). A plausible implication is that its main contribution is not merely the use of GPU acceleration or SSD residency in isolation, but the explicit removal of coordination bottlenecks between them.