Multi-Vector Index Compression

• Multi-vector index compression is a technique that reduces the storage and computational footprint of neural retrieval models by compressing sets of high-dimensional token embeddings.
• It employs methods such as k-means quantization, product quantization, and attention-guided clustering to achieve significant memory reductions while preserving retrieval accuracy.
• These strategies enable scalable approximate nearest neighbor search across modalities and support efficient deployment in resource-constrained, real-time environments.

Multi-vector index compression encompasses a suite of algorithmic and architectural strategies designed to reduce the storage and computational footprint of late-interaction neural retrieval models that represent documents using sets of high-dimensional token/patch embeddings. These techniques are essential in scaling information retrieval systems across modalities (text, vision, video, audio) by enabling scalable approximate nearest neighbor (ANN) search, fast reranking, and efficient deployment in resource-constrained environments, all while minimizing impact on retrieval effectiveness.

1. Foundations and Motivations

Multi-vector retrieval frameworks, typified by models like ColBERT, store a set of token-level or patch-level vectors per document to enable fine-grained query-document interactions via late interaction (e.g., maxSim scoring). While achieving high recall and robustness, particularly on out-of-distribution queries, these approaches incur steep space and compute costs—often increasing index sizes by 5–20× compared to single-vector baselines and imposing quadratic or super-linear late-stage scoring costs (Clavié et al., 2024, Bach, 19 Jun 2025). Efficient multi-vector index compression addresses these bottlenecks using quantization, vector pooling, and structural compression methods, thereby unlocking their use in large-scale, real-time settings and supporting broader modalities (Qin et al., 24 Feb 2026).

2. Quantization and Compression Techniques

Quantization is a core technique for compressing multi-vector indices, leveraging clustering or non-uniform encoding to represent high-dimensional vectors with compact codes:

• K-Means Quantization: Embeddings are assigned to the nearest centroid among $K$ learnable codewords. Replacing a $D$-dimensional float32 vector ($4D$ bytes) with a single 1-byte centroid index can yield up to a $32\times$ storage reduction for $D=128$ (Bach, 19 Jun 2025). The objective is

$$\min_{\mathcal{C},\,\mathrm{assign}} \sum_{i=1}^N \|x_i - c_{\mathrm{assign}(i)}\|_2^2,$$

with each $x_i$ replaced by $q_i=\mathrm{assign}(i)\in\{0,\ldots,K-1\}$.

• Product and Neural Quantization: Multi-codebook schemes (product quantization, residual quantization) partition vectors and quantize each subvector separately or sequentially (Vallaeys et al., 6 Jan 2025). Neural extensions (QINCo2) condition codeword selection on previous reconstruction, improving rate–distortion, while pre-selection and beam search accelerate encoding. With 16-byte codes, QINCo2 achieves 30–40% lower MSE compared to prior art.
• Non-Uniform Vector Quantization (NVQ): Rather than relying on global codebooks, NVQ fits individualized nonlinear per-vector quantizers optimized via gradient-free search, providing up to $3.4\times$ reduction in storage compared to float32, with sub-1% recall loss (Tepper et al., 22 Sep 2025).

Further, binary encoding of quantized indices enables Hamming distance-based search. If $K$ cluster centroids, each index is mapped into $b=\lceil\log_2 K\rceil$ bits, supporting hardware-accelerated bitwise similarity (Bach, 19 Jun 2025).

3. Vector Pooling, Clustering, and Constant-Budget Compression

To reduce the number of vectors per document, constant-budget and clustering-based pooling methods coalesce sets of token or patch embeddings into a smaller, representative set, enabling significant compression:

• Token Pooling (Clustering): Token-level vectors $V=\{v_1,\ldots,v_n\}$ are compressed into $K$ pooled vectors $C=\{c_1,\ldots,c_K\}$ using k-means or hierarchical clustering, with each centroid stored as the average of its cluster (Clavié et al., 2024). This process enables 2–3× reduction ($P=2,3$) for $<1\%$ NDCG@10 drop.
• Hierarchical Pooling (Ward's linkage): Agglomerative clustering minimizes within-cluster variance until a constant budget $B$ centroids remain, each representing a cluster of original vectors. This is the default non-parametric approach for query-agnostic compression (Qin et al., 24 Feb 2026).
• Attention-Guided Clustering (AGC): AGC leverages externally or self-attention–derived saliency weights to select semantically salient embeddings as centroids, clusters remaining tokens to nearest centroids, and averages tokens within clusters weighted by attention (Qin et al., 24 Feb 2026). Empirical results indicate that AGC matches or outperforms other methods under heavy compression, maintaining $>97\%$ retrieval quality even with $80$–$95\%$ reduction in vector count.

Alternative constant-budget approaches include sequence resizing using MLPs (SeqResize) or insertion of learnable memory tokens (MemTok) within the encoder, but these often under-utilize budget capacity compared to AGC, especially at extreme compressions (Qin et al., 24 Feb 2026).

4. Structural and Lossless Index Compression

Beyond quantizing embedding vectors, compressing index metadata—especially vector IDs and auxiliary structures—can yield substantial gains for both IVF and graph-based indices:

• Random Order Coding (ROC) via ANS: For unordered lists (e.g., IVF inverted lists or graph neighbor sets), exploiting permutation invariance reduces the coding cost for $n$ IDs from $n \log_2 N$ to $n\,\log_2 N - \log_2 n!$ bits, enabling up to $7\times$ compression on billion-scale indices without loss in accuracy (Severo et al., 16 Jan 2025). ROC is based on bits-back coding and is entropy-optimal per list.
• Wavelet-Tree Encoding: Enables rank/select-based random access with small additional overhead, especially useful for very large $N$ and where cluster-label sequences exhibit skew (Severo et al., 16 Jan 2025).
• Compression of Quantized Codes: Conditional coding within clusters or per-position allows entropy coding of PQ-encoded vectors, yielding an additional $5$–$20\%$ reduction if the PQ output is non-uniform (Severo et al., 16 Jan 2025).

Such structural compression is complementary to vector quantization; combined, they can deliver $30\%$ reduction in end-to-end index size at billion-vector scale for IVF+QINCo+ROC (Severo et al., 16 Jan 2025).

5. Dynamic Pruning and Query-Time Acceleration

Dynamic pruning reduces the number of vectors engaged in computationally intensive late interaction:

• Attention-Guided Pruning: By computing attention saliency for each patch/token at query time, one can select the top $p\%$ of patches (e.g., $p=40\%,60\%$) for scoring, yielding up to $60\%$ reduction in late-interaction computations with $<2\%$ nDCG@10 loss (Bach, 19 Jun 2025).
• Hierarchical Patch Compression: In HPC-ColPali, patch embeddings are quantized, followed by attention-guided pruning, and optionally binarized for high-throughput Hamming similarity search under extreme constraints (Bach, 19 Jun 2025).

Query-time benefits include halving latency (e.g., $120\to60$ ms on ViDoRe) and supporting sub-linear search complexity under HNSW or Hamming-accelerated indexing.

6. Empirical Results and Trade-Offs

Across benchmarks (BEIR, ViDoRe, SEC-Filings, MSR-VTT), the following patterns are observed:

Method/Config	Storage Compression	nDCG@10 Rel. Drop	Latency Change	Dataset	Notes
Token/K-means/H-Pool P=2–3	2–3×	<1.5%	None/slight ↓	BEIR, LoTTe	Zero-architecture change, best up to P≈3
ConstBERT C=32 (vs. M≈180 ColBERT)	2×	0.95 ppt MRR	None	MSMARCO/BEIR	Fixed-size, optimal for OS paging
HPC-ColPali PQ-only K=256	32×	2%	–50%	ViDoRe	With pruning and binarization for latency
AGC, B=32–64 (vs. full index)	4–20× (vector  #)	0–3% (≤1% text)	None	BEIR, ViDoRe, MSR	Consistently outperforms other budgeted forms
NVQ (8-bit, d=1k, m=2–4)	3.1–3.4×	<1% recall@10	None	fMRI, text, etc.	Individualized, fast encoding
ROC for IDs (IVF, N=1e9, K=2^20)	3–7×	0	+10–20%	SIFT, Deep1M	Lossless, large gains for IVF, graphs

Key trade-offs include:

• Accuracy vs. Compression: Finer quantization/larger codebook reduces loss; more aggressive pooling or pruning increases savings but risks larger effectiveness drops past critical thresholds (e.g., $P>4$ or $C<16$) (Clavié et al., 2024, MacAvaney et al., 2 Apr 2025).
• Storage vs. Latency: Methods also drive lower query latency (e.g., up to 2× reduction with dynamic pruning, Hamming score acceleration) (Bach, 19 Jun 2025).
• Implementation Complexity: Zero-training (H-Pool), architecture-free (token pooling), and individualized (NVQ) methods are trivial to deploy, while AGC, learned pooling (ConstBERT), or QINCo2 require more sophisticated, end-to-end training (Qin et al., 24 Feb 2026, Vallaeys et al., 6 Jan 2025).

7. Practical Recommendations and Deployment Considerations

• Algorithm Selection: For tightest budgets and cross-modal performance, attention-guided clustering is optimal; for rapid integration or very large data, hierarchical pooling (H-Pool) or token pooling suffice (Qin et al., 24 Feb 2026, Clavié et al., 2024).
• Parameterization: Choose the vector budget ($B$, $K$, $C$) based on application constraints; empirical “knee points” (e.g., $C=32$ for ConstBERT, $P=2–3$ for pooling) balance memory and effectiveness.
• Latency/Throughput Optimization: Binary search, HNSW-based selection, and ROC-coded indices allow sub-linear search at massive scale; pairing with quantized codes and vector ID compression ensures system-wide efficiency (Bach, 19 Jun 2025, Severo et al., 16 Jan 2025).
• Multi-Tenant and Metadata Compression: For multi-user settings, approaches like Curator encode tenant-specific clustering subtrees via Bloom filters in a shared global clustering tree, maintaining per-tenant efficiency at shared-index memory cost (Jin et al., 2024).
• Lossless Hybridization: Combine lossy (quantization/pooling) and lossless (ID and structure coding) strategies for maximal space savings without search quality degradation.

Leading research consistently demonstrates that multi-vector index compression via clustering, quantization, pooling, pruning, and lossless coding strategies closes the gap in deployability between single-vector and late-interaction multi-vector retrieval—achieving order-of-magnitude savings in storage and compute with modest, quantifiable accuracy trade-offs (Qin et al., 24 Feb 2026, Bach, 19 Jun 2025, MacAvaney et al., 2 Apr 2025, Clavié et al., 2024).