RemoteANN: A Native Vector Search Engine for Cloud Object Storage

Abstract: Vector search engines built on cloud object storage have drawn growing attention from both academia and industry. These systems typically adopt a disaggregated storage-compute architecture, persisting the primary vector payload in object stores (e.g., Amazon S3) while retaining compact index metadata on compute nodes. However, conventional vector indexing structures are not natively tailored to the pronounced network latency and per-request access costs of cloud object stores, resulting in network I/O amplification, degraded query throughput, and higher API expenses. To resolve these challenges, we present RemoteANN, a vector search engine explicitly co-designed for cloud object storage across indexing and querying. To mitigate the dominant network I/O bottleneck, RemoteANN couples an Inverted File (IVF) index with RaBitQ quantization, further optimized by inter-cluster and intra-cluster data reordering for improved data locality. Building on this tailored data layout, RemoteANN's query engine combines cluster pruning, fine-grained Range-GET requests, request merging, and local NVMe SSD caching, dramatically reducing discrete storage requests and network transfer volume without sacrificing accuracy. To address the index construction bottleneck, we further develop QGKMeans, which substantially reduces construction latency without compromising clustering quality. Extensive evaluations on real-world datasets demonstrate that, compared with representative baselines, RemoteANN achieves up to a $5.9\times$ throughput speedup while issuing up to $16.1\times$ fewer object storage requests at 97% target recall.

• The paper introduces RemoteANN, achieving up to $16.1 imes$ fewer object-storage requests and $5.9 imes$ higher throughput than baselines when searching near-neighbors in cloud-based vector data, demonstrating it is the speed and cost leader at approx. 97% recall.
• RemoteANN employs IVF with RaBitQ built for Cloud, leveraging locality-aware vector placements, analytical pruning, fine-grained Range-GET, request merging, and NVMe caching, to optimize index and query processing with cloud specifications.
• The system includes QGKMeans for fast index construction, ensuring operational feasibility at large scales by employing intelligent centroid assignment that matches state-of-the-art clustering speed while retaining recall performance in benchmarks.

Problem setting and system thesis

RemoteANN addresses approximate nearest-neighbor search when the vector payload is stored in cloud object storage and the query engine runs on a separate compute instance. This deployment model offers elasticity, durability, and storage-cost advantages, but it changes the dominant performance objective relative to in-memory or local-SSD ANN systems. Object storage imposes substantial time-to-first-byte latency and charges primarily by discrete retrieval requests. Consequently, an index that is acceptable on local storage can become inefficient when each cluster or graph hop induces a remote request.

The paper’s central thesis is that cloud-native vector search requires joint optimization of the ANN index, physical data layout, query execution, and storage hierarchy. RemoteANN therefore selects an IVF structure rather than a graph index, because IVF can retrieve candidate clusters with a bounded number of remote round trips. It combines IVF with RaBitQ quantization, locality-aware vector placement, analytical pruning, fine-grained Range-GET operations, request merging, and local NVMe caching. Index construction is accelerated separately through QGKMeans.

This is a systems paper rather than a proposal for a new ANN index family in isolation. Its contribution lies in aligning the index’s logical access pattern with the latency and pricing model of object storage. The evaluation reports up to $5.9\times$ higher throughput and up to $16.1\times$ fewer object-storage requests than representative baselines at approximately 97% recall (2610.25172).

Architecture and storage layout

RemoteANN uses a three-level hierarchy: DRAM on the compute node, local SSD, and remote object storage. IVF centroids and compact routing metadata are pinned in DRAM. The primary vector payload remains in object storage, allowing the system to retain the scalability and elasticity benefits of disaggregated storage.

The quantization layout is particularly important. Each vector is encoded using 9-bit RaBitQ quantization. The code is divided into a 1-bit prefix and an 8-bit suffix. The prefixes remain in DRAM and provide a coarse distance estimator; suffixes and high-precision payload data are stored remotely. Frequently accessed cluster data can additionally be cached on local NVMe SSD.

This arrangement separates two query requirements. Cheap, memory-resident information determines which clusters and vectors are unlikely to contribute to the result. Remote storage is accessed only for candidates that survive this filtering. The design consequently treats quantization not merely as a compression mechanism but as a mechanism for controlling remote I/O.

The paper’s warm-query assumption is material. It assumes that index metadata, including centroids and routing structures, is already resident in memory. Cold starts, in which the compute instance must first retrieve metadata from object storage, are excluded from the primary evaluation. The authors suggest workload-driven prefetching as a possible mitigation, but that mechanism is not implemented or evaluated.

Figure 1: RemoteANN separates DRAM-resident routing metadata from local-cache and cloud-object-storage payloads.

Index construction with QGKMeans

Approximate centroid assignment

IVF construction requires repeated assignment of vectors to centroids, and this becomes expensive when the number of clusters is large. RemoteANN introduces QGKMeans, which uses SymphonyQG as an approximate nearest-centroid accelerator. At each $k$-means iteration, QGKMeans builds an ANN structure over the current centroids and searches it with each data vector. The structure is rebuilt after centroid updates.

The resulting approximation is empirically controlled through clustering quality rather than through an exact assignment guarantee. Across the evaluated datasets, QGKMeans produces WCSS and downstream IVF recall comparable to Faiss, SuperKMeans, and SeededANN while reducing construction time. Representative results include:

Dataset	QGKMeans time	Fastest baseline time	QGKMeans recall
SIFT	6.2 s	8.9 s	0.941
MSong	9.2 s	10.1 s	0.992
Contriever	12.9 s	28.2 s	0.977
GIST	15.6 s	23.2 s	0.821
Cohere	204.2 s	1,128.4 s	0.947
Deep-1B	8,973.5 s	unavailable within 12 h	0.999

The strongest result is on Cohere, where QGKMeans is approximately $5.5\times$ faster than the fastest evaluated baseline while maintaining essentially equivalent clustering quality. On Deep-1B, QGKMeans completes in roughly 2.5 hours, whereas all compared alternatives exceed the stated 12-hour cutoff. The implication is that approximate centroid assignment can make flat IVF construction operationally feasible at large scale, although the evaluation does not establish how QGKMeans behaves under distribution drift or incremental index maintenance.

Inter-cluster locality

Conventional IVF layouts do not require a meaningful physical order among clusters because local random access is relatively cheap. That assumption fails for object storage. If query-selected clusters are scattered through a large object, merging their byte ranges may produce oversized requests containing substantial irrelevant data.

RemoteANN orders clusters according to an approximate minimum-weight Hamiltonian path over centroid distances. It evaluates nearest-neighbor and MST-doubling heuristics, as well as Hilbert and Z-order curves after PCA projection. The nearest-neighbor heuristic produces the shortest empirical centroid path on all six million-scale datasets, despite the MST method having the stronger worst-case approximation guarantee.

The full distance-matrix computation has $O(K^2d)$ cost for $K$ centroids in $d$ dimensions. To make the procedure practical for large $K$, the implementation recursively partitions centroids and applies localized ordering within each partition. With 262,144 centroids, the partitioned procedure completes in under one minute according to the paper.

Intra-cluster locality

Vectors within each cluster are sorted by distance to the cluster centroid. This ordering is designed for the later range-selection procedure. Under the RaBitQ estimator, vectors farther from the centroid have larger distance-estimation error bounds because the error term is scaled by their centroid-relative norm. Such vectors are therefore harder to prune and tend to remain in the candidate set. Placing them contiguously makes the surviving candidates occupy a smaller physical interval, reducing the byte span of a Range-GET.

Figure 2: Intra-cluster reordering concentrates vectors that survive quantization-based pruning into a smaller physical range.

Query processing

RemoteANN’s query pipeline first selects the top $n_{\text{probe}}$ IVF clusters using DRAM-resident centroids. It then estimates vector distances with the 1-bit RaBitQ prefixes, prunes clusters, computes candidate byte ranges, merges compatible requests, and retrieves data from either local NVMe or cloud object storage. Retrieved vectors are reranked using the available higher-precision information.

Analytical cluster pruning

RaBitQ supplies an unbiased distance estimator with a high-probability error bound. For each vector, RemoteANN derives lower and upper distance bounds. Since the exact top-$k$ distance is unknown before remote retrieval, it uses the $16.1\times$0-th smallest upper bound as a conservative threshold. A cluster is discarded if the lower bound of every vector in that cluster exceeds this threshold.

On the six million-scale datasets, conservative pruning reduces the number of remaining clusters substantially. For example, with 4,096 IVF clusters and OpenAI queries probing 240 clusters, only 13.9 clusters are retrieved on average while preserving a recall of 0.9716. The aggressive estimator-based threshold reduces this to 8.1 clusters with recall 0.9705, a recall loss of less than 0.5 percentage points relative to the conservative configuration.

The conservative mode is therefore the paper’s principal accuracy-preserving mechanism. The aggressive mode exposes an explicit cost-recall trade-off, but it is not theoretically guaranteed to preserve all valid neighbors. The authors also report that fewer than 0.01% of vectors violate the analytical RaBitQ bounds in their workloads; this empirical violation rate supports the observed recall, but it does not replace the probabilistic nature of the bound.

RemoteANN additionally performs dynamic pruning after requests have been dispatched. As completed clusters improve the current top-$16.1\times$1 threshold, pending clusters can be abandoned if their bounds become insufficient. This reduces work and straggler impact but cannot reduce object-storage request charges, because the requests have already been issued.

Range optimization

Downloading an entire IVF cluster is wasteful because only a small subset of vectors typically survives the coarse estimator. RemoteANN therefore identifies the minimum contiguous byte interval containing all unpruned vectors in a cluster and issues a Range-GET for that interval.

Range optimization reduces downloaded volume by at least 44% relative to full-cluster retrieval in the reported experiments. Intra-cluster reordering improves the reduction further:

Dataset	Random layout	Reordered layout
SIFT	55.8% of baseline volume	48.8%
MSong	46.3%	34.1%
Contriever	41.5%	35.4%
GIST	30.7%	19.3%
MXBAI	34.6%	27.4%
OpenAI	22.2%	19.8%

The values are normalized to downloading without range optimization, so the OpenAI result indicates that reordered range retrieval transfers only 19.8% of the corresponding unoptimized volume. This establishes that physical vector ordering is not an auxiliary implementation detail: it directly determines the effectiveness of the remote access primitive.

Request merging

Even after pruning and range selection, issuing one Range-GET per cluster can remain expensive because object-storage pricing is request-oriented. RemoteANN merges nearby ranges subject to a 600 KB maximum request size. A dynamic-programming algorithm sorts ranges by offset and optimizes lexicographically: it first minimizes the number of requests and then minimizes total transferred bytes among solutions with that minimum count.

The 600 KB threshold is empirical and workload-dependent. Larger thresholds can reduce request counts but may increase unnecessary transfer and latency. This parameter thus encodes a cost-latency policy rather than a universally optimal constant.

Figure 3: Nearby byte ranges are consolidated into bounded-size requests to reduce object-storage API operations.

Nearest-neighbor cluster ordering consistently improves merging compared with random placement. For example, at a fixed scan configuration, requests on SIFT fall from 60.3 under random ordering to 25.5 under nearest-neighbor ordering; on MSong they fall from 29.3 to 18.3. The improvement is smaller on high-dimensional OpenAI data, where requests decrease from 236.9 to 218.4, indicating that centroid proximity is an imperfect proxy for query-induced physical locality.

Local NVMe caching

RemoteANN optionally caches hot clusters on local NVMe SSD. The cache uses cluster-level allocation, 4 KB alignment, a preallocated backing file, TinyLFU-style admission, aging-LFU replacement, and asynchronous I/O through io_uring.

Cluster-level caching conflicts with fine-grained range retrieval. To resolve this conflict, the system downloads an optimized range only when it occupies less than a threshold fraction $16.1\times$2 of the cluster; otherwise, it fetches the complete cluster to populate the cache. This design sacrifices some immediate bandwidth efficiency for future cache reuse.

The cache evaluation shows monotonic growth in cluster hit ratio with capacity, but modest absolute coverage for the workloads tested. At 30% cache capacity, the hit ratio reaches 20.1% for Contriever and 13.1% for OpenAI. More importantly, caching does not improve throughput in the reported single-query experiments. Its value is therefore workload-dependent and primarily associated with reducing repeated remote access rather than with a demonstrated universal latency improvement.

Experimental evaluation

End-to-end performance

The evaluation uses SIFT, MSong, Contriever, GIST, MXBAI, and OpenAI at roughly million-vector scale, with additional experiments on 10-million-vector Cohere and the billion-vector Deep-1B dataset. RemoteANN is compared with Lance-RQ, an IVF-RaBitQ object-storage configuration, and RemoteSPANN, a SPANN implementation adapted to the same remote storage backend.

At 97% target recall, RemoteANN has the lowest median latency in all 18 dataset-concurrency configurations and the lowest 95th-percentile latency in 16. Relative to Lance-RQ, median latency improves by factors of 1.47--5.57 and 95th-percentile latency by factors of 1.19--5.15. The exceptions in tail latency occur on MSong at concurrency levels 1 and 4, where RemoteSPANN is slightly better.

Representative median and tail latencies at concurrency 16 are:

Dataset	RemoteANN $16.1\times$3	Lance-RQ $16.1\times$4	RemoteANN $16.1\times$5	Lance-RQ $16.1\times$6
SIFT	62.6 ms	314.1 ms	98.4 ms	343.0 ms
MSong	38.6 ms	188.9 ms	63.3 ms	269.5 ms
Contriever	35.3 ms	188.8 ms	54.1 ms	271.2 ms
GIST	118.5 ms	189.1 ms	221.4 ms	272.9 ms
MXBAI	33.5 ms	186.5 ms	51.7 ms	266.3 ms
OpenAI	44.3 ms	191.3 ms	70.4 ms	280.4 ms

The reported QPS-recall curves fluctuate rather than varying smoothly. The paper attributes this behavior to object-storage performance variability, which is consistent with the fact that remote request latency is not fully controlled by the ANN algorithm. This observation qualifies the throughput claims: they are strong under the tested AWS deployment, but cloud-service variability remains part of the end-to-end performance distribution.

Figure 4: RemoteANN maintains a stronger throughput-recall frontier than Lance-RQ and RemoteSPANN across concurrency levels and datasets.

Remote I/O and cost-relevant behavior

RemoteANN issues fewer Range-GET requests than both baselines at 97% recall:

Dataset	RemoteANN	Lance-RQ	RemoteSPANN
SIFT	18.5	97.1	58.5
MSong	10.7	57.6	39.9
Contriever	8.6	57.6	19.9
GIST	41.4	56.4	294.7
MXBAI	6.8	57.1	27.6
OpenAI	12.0	57.7	79.2
Cohere	18.6	26.2	300.0
Deep-1B	146.5	810.2	unavailable

The reduction is especially pronounced on GIST and Deep-1B relative to RemoteSPANN and Lance-RQ, respectively. These results substantiate the paper’s cost argument: throughput improvements are accompanied by fewer billable object-storage operations rather than achieved solely by consuming more remote bandwidth.

Construction time and memory

RemoteANN’s construction time is higher than Lance-RQ’s on every reported dataset because it uses flat rather than hierarchical $16.1\times$7-means and performs additional reordering and layout work. However, it is substantially faster than RemoteSPANN, with a maximum reported construction speedup of $16.1\times$8. On Deep-1B, RemoteANN requires 11,029 seconds compared with 5,176 seconds for Lance-RQ, while RemoteSPANN does not complete within the evaluation boundary.

RemoteANN and Lance-RQ have similar memory footprints because both retain compact quantized routing information, whereas RemoteSPANN requires substantially more memory for routing graphs, centroids, and replicated vector structures. The results demonstrate a deliberate trade-off: RemoteANN accepts additional one-time construction cost to reduce online remote I/O and query latency.

Figure 5: RemoteANN preserves its throughput and request-count advantages on Cohere and Deep-1B, including the billion-vector setting.

Ablation analysis

The ablation study isolates cluster pruning, range optimization, request merging, and caching. Pruning and range optimization provide the primary throughput gains without measurable recall degradation under the conservative threshold. Request merging reduces API operations but can slightly reduce throughput at high concurrency because merging may transfer extra bytes. This result is important because it rejects a simplistic claim that fewer requests always imply lower latency.

The reported request-cost reductions from merging reach approximately 30% on Contriever and 14% on OpenAI. Caching does not alter the measured QPS-recall curves in the selected experiments, although its hit ratio increases with cache capacity.

A raw-vector variant, RemoteANN-Raw, removes quantized remote payloads while retaining the query-side optimizations. It remains faster than both baselines but slower than standard RemoteANN. At concurrency 16, throughput is 305.2 versus 412.5 QPS on Contriever and 199.2 versus 298.2 QPS on OpenAI. This comparison indicates that the principal gains come from the co-designed layout and I/O pipeline, while RaBitQ contributes an additional reduction in payload and candidate-processing cost.

Figure 6: Cluster pruning and range optimization dominate throughput improvements, while request merging primarily reduces API operations and can impose a high-concurrency bandwidth cost.

Limitations and open questions

RemoteANN targets a static IVF index. Incremental insertion, deletion, rebalancing, and consistency under concurrent updates are not addressed. This limitation matters because physically reordered object layouts and cluster-level caching complicate mutation without periodic reconstruction. Systems such as SPFresh explicitly target incremental maintenance [(2610.25172); see also the paper’s discussion of SPFresh], but RemoteANN does not provide a comparable update protocol.

The warm-query assumption excludes metadata cold-start latency. In a serverless or highly bursty deployment, loading centroids and quantized routing metadata may dominate the first-query response time. The paper identifies prefetching as a possible solution but provides no quantitative analysis of its accuracy, resource overhead, or interaction with cache admission.

The empirical study is also tied to a particular AWS configuration, object-store implementation, 600 KB request threshold, 9-bit RaBitQ setting, and static query workloads. The extent to which the same ordering heuristics and threshold remain optimal across providers, regions, request pricing models, filtered queries, or nonstationary workloads remains open. Finally, the pruning guarantee is probabilistic through RaBitQ’s error bound, and the reported violation rate is empirical; stronger workload-independent end-to-end recall guarantees are not established.

Conclusion

RemoteANN presents a coherent cloud-native design for ANN search in which object-storage latency and request pricing are treated as first-class index-design constraints. Its main technical contribution is the integration of IVF, RaBitQ, QGKMeans, inter- and intra-cluster reordering, analytical pruning, range retrieval, request merging, and local caching into a single storage-aware pipeline.

Across the evaluated workloads, this co-design yields lower latency, higher throughput, and substantially fewer remote requests at approximately 97% recall. The strongest evidence supports the claim that physical layout and query-side I/O policy are as important as the underlying ANN approximation method in disaggregated vector search. The remaining central question is whether these techniques can be extended to mutable, cold-start-sensitive workloads without forfeiting the measured performance and cost advantages.