RemoteANN Explanation

• RemoteANN is an optimized retrieval engine designed for approximate nearest neighbor (ANN) search over cloud object storage, that focuses on reducing network latency, and object-storage requests, and optimising caching and request merging.
• The engine employs an Inverted File (IVF) index, 9-bit RaBitQ quantization, locality-aware cluster and vector ordering, vector-level pruning, and request merging in combination with NVMe local caching to deliver high throughput in real-world datasets by fetching clusters of related results through few object-requests.
• A single NVMe cache and online hardware environment RemoteANN reduced network-related request bottlenecks on evaluated datasets, showing significant throughput improvements, up to 5.9x higher throughput and up to 16.1x fewer object-storage requests at approximately 97% recall.

RemoteANN is a vector-search engine designed for approximate nearest-neighbor (ANN) retrieval over cloud object storage. It co-designs indexing, physical data layout, query execution, and caching for object stores whose access costs are dominated by network latency, time-to-first-byte (TTFB), and per-request charges. Its principal architecture combines an Inverted File (IVF) index, 9-bit RaBitQ quantization, locality-aware cluster and vector ordering, vector- and cluster-level pruning, fine-grained Range-GET operations, request merging, and local NVMe SSD caching. On evaluated real-world datasets, RemoteANN achieved up to $5.9\times$ higher throughput and up to $16.1\times$ fewer object-storage requests than representative baselines at approximately 97% recall (Gou et al., 18 Oct 2026).

1. Architectural scope and motivation

RemoteANN targets disaggregated storage-compute systems in which compute nodes execute queries and retain compact routing metadata, local DRAM and NVMe SSD provide metadata and caching capacity, and cloud object storage retains the large primary vector payload. This arrangement offers elasticity, durability, and comparatively low storage cost, but its access model differs fundamentally from local DRAM or SSD.

Object storage imposes TTFB and network-startup costs on every request, charges substantially by retrieval-request count, and performs poorly when an ANN algorithm generates many small or dependent accesses. An access pattern acceptable for local storage can therefore produce high latency, poor bandwidth utilization, and elevated API cost when transferred directly to object storage.

Graph indexes such as HNSW, NSG, and DiskANN repeatedly fetch nodes or neighbors during multi-hop traversal. Their dependent sequence of remote operations can accumulate TTFB:

$$\text{query}\rightarrow\text{fetch node}\rightarrow\text{choose neighbor}\rightarrow\text{fetch neighbor}\rightarrow\cdots$$

RemoteANN instead favors cluster-based search, especially IVF, because cluster selection is performed locally and the selected payload can be retrieved through a smaller number of logically consolidated remote-I/O operations. A naïve IVF implementation nevertheless creates a different problem: probing many clusters can generate one request per cluster, whereas downloading complete clusters transfers excessive data. RemoteANN addresses this tension through cluster pruning, locality-aware layout, Range-GET, request merging, and caching.

RemoteANN should be distinguished from the broader architectural use of “RemoteANN” in other supplied research contexts. “Edge-ANN: Storage-Efficient Edge-Based Remote Sensing Feature Retrieval” uses the term Edge-ANN for a storage-efficient tree index based on data-item anchors, with experiments on remote-sensing features (Lv et al., 21 Nov 2025). “RiftANN” denotes a graph-search system for passive RDMA-based memory disaggregation (Lin et al., 6 Oct 2026). Neither system is the cloud-object-storage-native engine described here.

2. Index representation and storage hierarchy

RemoteANN separates routing information, compact quantization metadata, cached data, and full-precision vectors across a storage hierarchy:

• DRAM stores IVF centroids, cluster metadata, byte offsets, routing structures, and 1-bit RaBitQ prefixes.
• Local NVMe SSD optionally stores frequently accessed clusters at cluster granularity.
• Cloud object storage stores full-precision vectors or high-precision payloads, together with remote RaBitQ suffixes.

The warm-query configuration assumes that IVF centroids and routing metadata are already resident in compute-node memory. Cold starts can be mitigated through workload-driven prefetching, although this is not evaluated in depth.

IVF routing

IVF partitions the vector collection into $K$ clusters with centroids $\{\mathbf c_1,\ldots,\mathbf c_K\}$. Given a query $\mathbf q$, RemoteANN computes local distances to the centroids and selects the closest $n_{\mathrm{probe}}$ clusters. Increasing $n_{\mathrm{probe}}$ generally improves recall while increasing computation, transferred data, and remote-access opportunities.

IVF is appropriate for object storage because its initial routing phase does not require remote access. Its cluster structure also provides a unit around which RemoteANN can perform pruning, range formation, request merging, and SSD caching.

RaBitQ representation

RemoteANN uses a 9-bit RaBitQ representation divided into a 1-bit prefix retained in DRAM and an 8-bit suffix stored remotely or cached locally. The prefix supplies a coarse distance estimator before remote payload retrieval, while the full-precision or higher-precision data support final candidate evaluation.

For residual vectors relative to an IVF centroid, the distance expression is decomposed into exact norm terms and a quantized inner-product term:

$$\|\mathbf{o}_r-\mathbf{q}_r\|^2 = \|\mathbf{o}_r-\mathbf c\|^2+ \|\mathbf q_r-\mathbf c\|^2 - 2\|\mathbf{o}_r-\mathbf c\| \|\mathbf q_r-\mathbf c\| \langle\mathbf o,\mathbf q\rangle.$$

RaBitQ estimates $\langle\mathbf o,\mathbf q\rangle$. If $16.1\times$0 is the estimated distance and $16.1\times$1 is the analytical error bound, RemoteANN uses

$16.1\times$2

The reported bound-violation rate was below $16.1\times$3 on the evaluated workloads and caused no material recall loss. The guarantee is nevertheless probabilistic rather than deterministic.

Physical layout

RemoteANN reorders data at two levels. Inter-cluster ordering places clusters likely to be selected together near one another in the object-storage file. The idealized objective is a minimum-weight Hamiltonian path over cluster centroids, but RemoteANN uses practical heuristics, including nearest-neighbor ordering, MST doubling, Hilbert ordering, and Z-ordering after PCA to three dimensions. The nearest-neighbor heuristic produced the shortest empirical paths among the evaluated alternatives and is the default layout strategy.

Within each cluster, vectors are sorted by Euclidean distance to the cluster centroid. Peripheral vectors have larger RaBitQ distance-estimation uncertainty and are more likely to survive pruning. Grouping them physically allows surviving candidates to occupy a more compact byte interval, improving Range-GET efficiency.

3. Query processing and remote I/O reduction

RemoteANN’s query pipeline consists of IVF routing, quantized pruning, cache lookup, range construction, request merging, remote retrieval, candidate generation, and final distance evaluation.

First, the query selects the nearest IVF clusters locally. The 1-bit RaBitQ prefixes then estimate vector distances within those clusters. Let $16.1\times$4 denote the $16.1\times$5-th smallest upper bound among candidate vectors. Since upper bounds are conservative,

$16.1\times$6

where $16.1\times$7 is the true distance of the $16.1\times$8-th result. A vector is eliminated when

$16.1\times$9

A cluster is eliminated when this condition holds for every vector in it. The system also supports an aggressive alternative based on the $$\text{query}\rightarrow\text{fetch node}\rightarrow\text{choose neighbor}\rightarrow\text{fetch neighbor}\rightarrow\cdots$$0-th smallest estimated distance, $$\text{query}\rightarrow\text{fetch node}\rightarrow\text{choose neighbor}\rightarrow\text{fetch neighbor}\rightarrow\cdots$$1. This prunes more aggressively but can reduce recall; the reported recall loss was below approximately $$\text{query}\rightarrow\text{fetch node}\rightarrow\text{choose neighbor}\rightarrow\text{fetch neighbor}\rightarrow\cdots$$2 at a 97% recall baseline.

RemoteANN performs dynamic pruning as responses arrive. It maintains a current exact candidate threshold, $$\text{query}\rightarrow\text{fetch node}\rightarrow\text{choose neighbor}\rightarrow\text{fetch neighbor}\rightarrow\cdots$$3, and can skip pending clusters whose lower bounds exceed that threshold. Because such requests may already have been dispatched, dynamic pruning reduces computation and straggler impact but does not necessarily reduce object-storage request count or API cost.

Range-GET formation

After vector-level pruning, surviving vectors within each cluster may occupy only a portion of the cluster’s physical byte layout. RemoteANN retrieves the minimum contiguous byte interval containing the surviving vectors rather than downloading the complete cluster. At a 97% target recall, range optimization reduced normalized downloaded volume to between 22.2% and 55.8% of the no-range baseline in the reported datasets. With intra-cluster ordering, the reported normalized volumes included 34.1% for MSong, 19.3% for GIST, and 19.8% for OpenAI.

Request merging

RemoteANN merges nearby byte ranges subject to a maximum merged-request size of 600 KB. Ranges are sorted by starting offset, and a dynamic program minimizes the number of merged requests first and total downloaded bytes second. Inter-cluster ordering is important because physically adjacent selected clusters create larger opportunities for consolidation.

At 600 KB, nearest-neighbor ordering reduced merged requests from 60.3 to 25.5 on SIFT and from 58.7 to 39.6 on MXBAI relative to random cluster order.

SSD caching

The optional NVMe cache operates at cluster granularity. Its implementation uses 4-KB alignment, a preallocated backing file, direct I/O, best-fit free-extent allocation, extent coalescing, TinyLFU-inspired admission, aging-LFU replacement, asynchronous io_uring I/O, and background cache insertion.

A threshold $$\text{query}\rightarrow\text{fetch node}\rightarrow\text{choose neighbor}\rightarrow\text{fetch neighbor}\rightarrow\cdots$$4, suggested as 0.8, determines whether to retrieve only an optimized range or the entire cluster for cache population. If the optimized range is less than $$\text{query}\rightarrow\text{fetch node}\rightarrow\text{choose neighbor}\rightarrow\text{fetch neighbor}\rightarrow\cdots$$5 of the cluster, RemoteANN fetches only the range; otherwise, it fetches the full cluster.

In the evaluated Contriever and OpenAI experiments, caching primarily reduced remote accesses rather than producing an observable throughput increase. At cache capacities equal to 10%, 20%, and 30% of remote data, the fraction of clusters served from cache was 8.5%, 14.6%, and 20.1% for Contriever, and 7.3%, 11.6%, and 13.1% for OpenAI.

4. QGKMeans and index construction

RemoteANN introduces QGKMeans to reduce the construction cost of IVF clustering. Standard flat $$\text{query}\rightarrow\text{fetch node}\rightarrow\text{choose neighbor}\rightarrow\text{fetch neighbor}\rightarrow\cdots$$6-means repeatedly assigns vectors to centroids through exhaustive comparisons. QGKMeans instead constructs a SymphonyQG ANN index over the current centroids and uses approximate nearest-centroid search for vector assignment.

Each iteration performs the following operations:

1. Construct a SymphonyQG index over the current centroids.
2. Query the index with every data vector.
3. Assign each vector to its approximate nearest centroid.
4. Recompute centroids.
5. Rebuild the SymphonyQG structure after centroid updates.
6. Repeat for 25 iterations in the evaluation.

The underlying clustering objective remains within-cluster sum of squares:

$$\text{query}\rightarrow\text{fetch node}\rightarrow\text{choose neighbor}\rightarrow\text{fetch neighbor}\rightarrow\cdots$$7

RemoteANN evaluates both WCSS and downstream IVF recall to ensure that faster approximate assignment does not materially degrade the partitioning relevant to retrieval. The reported WCSS and recall remained close to those of Faiss, SuperKMeans, and SeededANN.

QGKMeans was fastest in every displayed dataset. Reported construction times included 6.2 seconds on SIFT, 9.2 seconds on MSong, 12.9 seconds on Contriever, 15.6 seconds on GIST, 24.6 seconds on OpenAI, 204.2 seconds on Cohere, and 8,973.5 seconds on Deep-1B. RemoteANN’s construction is slower than Lance-RQ because it uses flat rather than hierarchical $$\text{query}\rightarrow\text{fetch node}\rightarrow\text{choose neighbor}\rightarrow\text{fetch neighbor}\rightarrow\cdots$$8-means and performs additional reordering and layout work; this is treated as a one-time cost exchanged for lower query-time I/O.

The evaluated cluster counts were $$\text{query}\rightarrow\text{fetch node}\rightarrow\text{choose neighbor}\rightarrow\text{fetch neighbor}\rightarrow\cdots$$9 for most datasets, $K$0 for Cohere, and $K$1 for Deep-1B. The principal construction limitation is that RemoteANN is designed for static IVF indexes. Efficient incremental insertion, deletion, and cluster rebalancing remain open problems, particularly because rewriting large object-store objects can be expensive.

5. Evaluation and performance

RemoteANN was evaluated on SIFT, MSong, Contriever, GIST, MXBAI, OpenAI, Cohere, and Deep-1B. The datasets span image, audio, and text embeddings, with dimensions from 96 to 1,536 and collection sizes from 769,382 vectors to one billion vectors. Searches return top-$K$2 results with $K$3.

The online serving environment used AWS EC2 m5dn.2xlarge instances with 8 vCPUs, 32 GB DRAM, 300 GB local NVMe SSD, and up to 25 Gbps network connectivity. The S3 bucket was located in the same AWS region. Baselines were Lance-RQ and RemoteSPANN; closed-source systems such as Turbopuffer and Amazon S3 Vectors were not directly evaluated.

Across concurrency levels of 1, 4, and 16, RemoteANN dominated both baselines in throughput-recall curves. At approximately 97% recall, it had the lowest median latency in all 18 dataset/concurrency configurations and the lowest $K$4 latency in 16 of 18 configurations. Relative to Lance-RQ, median latency was lower by $K$5–$K$6, and $K$7 latency was lower by $K$8–$K$9.

Representative median and tail latencies at 97% recall were:

Dataset and concurrency	RemoteANN $\{\mathbf c_1,\ldots,\mathbf c_K\}$0	Lance-RQ $\{\mathbf c_1,\ldots,\mathbf c_K\}$1
SIFT, 1	35.6 / 72.9 ms	86.8 / 111.3 ms
SIFT, 16	62.6 / 98.4 ms	314.1 / 343.0 ms
Contriever, 1	32.6 / 53.5 ms	70.8 / 97.8 ms
OpenAI, 16	44.3 / 70.4 ms	191.3 / 280.4 ms

The largest reported throughput improvement was $\{\mathbf c_1,\ldots,\mathbf c_K\}$2. Object-storage request reductions reached $\{\mathbf c_1,\ldots,\mathbf c_K\}$3 at 97% recall. Average requests per query included 18.5 for SIFT, 8.6 for Contriever, 12.0 for OpenAI, 18.6 for Cohere, and 146.5 for Deep-1B.

Index construction required 9 seconds and 0.03 GiB for SIFT, 20 seconds and 0.12 GiB for Contriever, 43 seconds and 0.23 GiB for OpenAI, 252 seconds and 1.72 GiB for Cohere, and 11,029 seconds and 32.40 GiB for Deep-1B. Compared with RemoteSPANN, RemoteANN achieved up to $\{\mathbf c_1,\ldots,\mathbf c_K\}$4 faster construction.

The ablation study attributes performance gains to the interaction of several components. Range-GET reduced transferred volume by at least 44% relative to disabling range optimization. Request merging sharply reduced API calls, although it could slightly reduce throughput at high concurrency by downloading gap bytes. The RemoteANN-Raw variant, which stored raw vectors rather than compact RaBitQ payloads, was slower than the complete system, supporting the contribution of quantized payloads in addition to layout and request optimizations.

6. Limitations, applications, and related systems

RemoteANN’s results depend on several deployment assumptions. The evaluation primarily concerns static indexes, warm queries, same-region object storage, empirical pruning parameters, and workload-dependent locality. Cold starts can incur substantial latency because centroids, prefixes, and routing structures must first be loaded. Cross-region transfer may dominate request costs. Object-store latency and throughput are variable, producing nonsmooth QPS-recall curves and complicating tail-latency control.

The parameters $\{\mathbf c_1,\ldots,\mathbf c_K\}$5, $\{\mathbf c_1,\ldots,\mathbf c_K\}$6, RaBitQ bit allocation, the 600-KB merge threshold, cache capacity, $\{\mathbf c_1,\ldots,\mathbf c_K\}$7, and conservative versus aggressive pruning require workload-specific tuning. Cache effectiveness depends on cluster popularity, request merging depends on physical locality, and range optimization depends on the effectiveness of the intra-cluster ordering. The reported RaBitQ bounds are not absolute guarantees, and aggressive pruning trades recall for lower I/O.

The system is most applicable to large, relatively static vector archives whose primary payloads are too large or too costly to retain entirely in local memory. Potential workloads include text, image, audio, and remote-sensing feature retrieval, provided that the application accepts approximate search and can tolerate object-store access variability. Its central systems principle is that physical layout becomes part of the ANN algorithm when storage is remote and request-priced: logical clusters and vectors must be arranged so that pruning leads to compact byte ranges and inexpensive consolidated requests.

RemoteANN differs from related systems in design emphasis:

• Edge-ANN replaces explicit high-dimensional tree hyperplanes with anchor IDs and an offset scalar, reducing index metadata storage for remote-sensing retrieval under RAM and disk constraints (Lv et al., 21 Nov 2025).
• RiftANN retains a graph index in passive RDMA-based memory and uses compact-vector navigation, selective exact verification, bounded RDMA pipelining, impact-aware traversal, and feedback-driven scheduling (Lin et al., 6 Oct 2026).
• RemoteANN targets cloud object storage specifically, where request count, TTFB, byte ranges, and API pricing motivate IVF-based routing, RaBitQ filtering, locality-aware layout, Range-GET, request merging, and NVMe caching (Gou et al., 18 Oct 2026).

The principal contribution of RemoteANN is therefore a systems-level co-design rather than an isolated new ANN primitive. IVF, quantization, caching, and request coalescing are individually established techniques; RemoteANN adapts them jointly to object storage. Under the evaluated AWS configuration and warm-query workload, this co-design produced the reported throughput and request-count improvements while preserving approximately 97% recall.