Unified and Efficient Approach for Multi-Vector Similarity Search

Abstract: Multi-Vector Similarity Search is essential for fine-grained semantic retrieval in many real-world applications, offering richer representations than traditional single-vector paradigms. Due to the lack of native multi-vector index, existing methods rely on a filter-and-refine framework built upon single-vector indexes. By treating token vectors within each multi-vector object in isolation and ignoring their correlations, these methods face an inherent dilemma: aggressive filtering sacrifices recall, while conservative filtering incurs prohibitive computational cost during refinement. To address this limitation, we propose MV-HNSW, the first native hierarchical graph index designed for multi-vector data. MV-HNSW introduces a novel edge-weight function that satisfies essential properties (symmetry, cardinality robustness, and query consistency) for graph-based indexing, an accelerated multi-vector similarity computation algorithm, and an augmented search strategy that dynamically discovers topologically disconnected yet relevant candidates. Extensive experiments on seven real-world datasets show that MV-HNSW achieves state-of-the-art search performance, maintaining over 90% recall while reducing search latency by up to 14.0$\times$ compared to existing methods.

• The paper introduces MV-HNSW, which natively indexes multi-vector data to optimize semantic search with high recall.
• It employs a novel symmetric edge-weight function and accelerated cluster-based similarity computation to significantly reduce search latency.
• Empirical evaluations demonstrate >90% recall and up to 14x latency reduction on large-scale datasets, ensuring robust scalability.

Unified and Efficient Graph-Based Indexing for Multi-Vector Similarity Search

Introduction and Problem Motivation

Multi-vector similarity search (MVSS) provides the foundation for fine-grained semantic retrieval in systems employing multi-vector representations, such as RAG-based frameworks and LLM-augmented search pipelines. Unlike traditional single-vector retrieval—which compresses entire objects into fixed-length embeddings, causing semantic dilution—MVSS works with objects segmented into multiple, typically token-level, vectors, enabling a more expressively granular match between queries and collections.

Prior MVSS solutions primarily adopt a filter-and-refine paradigm: heuristic single-vector indexes (e.g., HNSW, IVF, PQ, or product quantization) prune irrelevant candidates at the vector level, with subsequent refinement steps computing the intricate multi-vector similarity function (e.g., MaxSim) among the survivors. This strategy either incurs high computational cost (conservative filtering) or sacrifices recall (aggressive filtering), fundamentally due to the lack of native multi-vector indexing. The result is that the utility of intra-object correlations is unrealized and recall-efficiency tradeoffs remain suboptimal.

The MV-HNSW Framework: Native Multi-Vector Graph Index

The paper introduces MV-HNSW, a hierarchical navigable small world (HNSW)-style index that natively operates on multi-vector data and is optimized for unified multi-vector similarity search. In particular, MV-HNSW extends HNSW's single-vector graph-based paradigm by designing a principled multi-vector edge-weight function and by proposing algorithms that efficiently support similarity search across massive, high-dimensional, and structured multi-vector datasets.

The solution integrates three primary components:

1. Novel Edge-Weight Function: A symmetric, cardinality-robust, and query-consistent measure for connecting multi-vector graph nodes, overcoming the deficiencies of direct use of standard multi-vector operators for edge weights.
2. Accelerated Multi-Vector Similarity Computation: Efficient approximation of expensive multi-vector similarity functions using a cluster-based filter-and-refine algorithm, dramatically reducing both offline construction and online search overhead.
3. Augmented Graph Traversal: An adaptive search protocol that leverages precomputed auxiliary navigation tables (ANTs) to dynamically identify topologically disconnected, yet semantically relevant, multi-vectors bypassing standard locality constraints of the graph.

The high-level architecture is illustrated below.

Figure 1: Overview of the MV-HNSW solution integrating edge weight design, optimized construction, and augmented search.

Formalization of Unified Multi-Vector Similarity

The unified similarity function generalizes MaxSim and other multi-vector metrics. Given a query $Q=\{q_1,\dots,q_m\}$ and dataset multi-vector $V=\{v_1,\dots,v_n\}$, and a parameter $\gamma$ for $\gamma$-nearest neighbors, unified similarity is:

$$\operatorname{USim}(Q, V) = \sum_{q \in Q} w_q \cdot \frac{1}{\gamma} \sum_{v \in NN(q, V)} \text{dis}(q, v)$$

where $w_q$ is the token weight, and $dis$ is typically the inner product. This subsumes MaxSim (when $\gamma=1$ and $w_q=1$), weighted Chamfer (learnable $w_q$), and aggregate-$V=\{v_1,\dots,v_n\}$0NN similarities. Such a formulation necessitates both structural alignment and computational acceleration for practical deployment.

Graph Construction and Edge-Weight Function

A central innovation is the edge-weight function that overcomes critical failures of naively applying $V=\{v_1,\dots,v_n\}$1: asymmetry, cardinality bias, and lack of query consistency. MV-HNSW defines the edge between nodes $V=\{v_1,\dots,v_n\}$2 and $V=\{v_1,\dots,v_n\}$3 as the mean normalized bi-directional similarity:

$V=\{v_1,\dots,v_n\}$4

This function is rigorously proven to satisfy symmetry, cardinality-robustness, and query consistency—the necessary properties for effective navigation and locality-aware graph search. Index construction adopts the standard HNSW hierarchical insertion protocol, generalized to multi-vector datasets and integrating the new edge function.

Accelerated Similarity Computation

The major computation bottleneck in MVSS is evaluating $V=\{v_1,\dots,v_n\}$5, which involves $V=\{v_1,\dots,v_n\}$6 operations per pair of multi-vectors (with $V=\{v_1,\dots,v_n\}$7 the number of token vectors and $V=\{v_1,\dots,v_n\}$8 the dimension). To address this, the paper introduces a scalable filter-and-refine clustering approach:

• Query tokens are grouped by k-means into $V=\{v_1,\dots,v_n\}$9 clusters.
• Each cluster centroid retrieves $\gamma$0 nearest neighbors among data tokens.
• Candidate sets are refined for actual $\gamma$1 tokens using only the pruned neighbors, yielding high-accuracy approximations with sublinear cost. This pipeline is parallelizable and reduces index construction/online computation from $\gamma$2 per query-data pair to $\gamma$3.

Augmented Search Protocol

Even with a graph structure, strict graph locality may disconnect semantically relevant candidates, as high-recall neighbors might not be directly reachable. MV-HNSW resolves this using auxiliary navigation tables (ANTs), which for each token maintain a precomputed $\gamma$4-sized list of globally correlated multi-vectors. During query time, for each candidate multi-vector, ANTs are used to dynamically expand the candidate pool by “hopping” to relevant objects that would otherwise remain unreachable, thus bridging structural gaps.

Empirical Evaluation

Numerical Results: Recall and Efficiency

Extensive benchmarks are conducted on seven real-world datasets from LoTTE and MS MARCO (scale: hundreds of millions to billions of token vectors). MV-HNSW consistently achieves $\gamma$590% recall while reducing search latency by up to 14$\gamma$6 compared to state-of-the-art baselines such as ColBERT, ColBERTv2, XTR, and WARP.

Figure 2: Recall (Pooled) on large-scale datasets, MV-HNSW outperforms baselines in both recall and efficiency.

MV-HNSW exhibits near-logarithmic scaling in latency as dataset size expands, strong resilience to increases in query vector count and token dimensionality, and marginal search time growth as the required neighbor count $\gamma$7 increases.

Figure 3: Recall profile for MV-HNSW compared to other approaches across dataset sizes and query granularities.

Scalability

MV-HNSW is robust to variations in all critical parameters (dataset size $\gamma$8, query token count $\gamma$9, vector dimensionality $\gamma$0), maintaining high recall and low latency where all baselines degrade—especially for practical $\gamma$1 in large collections.

Ablation and Optimizations

Ablation studies demonstrate that:

• Clustered approximation for $\gamma$2 reduces construction time by $\gamma$3–$\gamma$4 at equivalent accuracy.
• Augmented search via ANT provides significant recall improvements, especially against highly-connected graph traversal getting trapped in local optima.

  Figure 4: Index construction speedup from clustered similarity computation over naive multi-vector pairwise methods.

  

  Figure 5: Recall improvements from the augmented search strategy with ANTs, resolving graph disconnectivity.

Index Construction Cost

The trade-off for MV-HNSW’s state-of-the-art online search performance is increased offline index construction time and storage, but the overhead remains within a practical operational envelope for billion-scale datasets.

Implications and Future Directions

This work demonstrates that graph-based indexing, with a formally-correct and multi-vector-aware adaptation, provides the sought-after balance in recall and efficiency for MVSS in RAG, large-scale IR, and LLM retrieval-augmentation. The theoretical framework for edge-weight design can be extended to other hierarchical or hybrid ANN indexes, and the cluster-based $\gamma$5 computation is a viable approach for other expensive pairwise operators.

The dynamic augmentation of the search process using ANTs also opens avenues for hybrid locality-globality navigation strategies more generally in structured retrieval. Future developments might further compress the index and ANTs or expand to cross-modal or hierarchical multi-vector settings, and address end-to-end system integration for real-time, multi-modal, and long-context retrieval domains.

Conclusion

The MV-HNSW framework constitutes a rigorous and practical solution for unified multi-vector similarity search, addressing the core structural and computational bottlenecks of prior approaches. It offers a robust, scalable, and efficient indexing method that is applicable to next-generation fine-grained retrieval systems and large-scale LLM hybrid architectures, providing a new paradigm for semantic search in settings with complex multi-vector representations.

Reference: "Unified and Efficient Approach for Multi-Vector Similarity Search" (2604.02815)