---
title: Efficient Disk-Backed Late Interaction
url: https://www.emergentmind.com/topics/efficient-disk-backed-late-interaction-dli
type: topic
---

# Efficient Disk-Backed Late Interaction

Efficient Disk-backed Late Interaction (DLI) denotes a retrieval architecture in which document-side multi-vector representations are precomputed offline, persisted on disk or in CPU-addressable storage, and loaded selectively at query time so that late interaction scoring is applied only where it is most useful. The paradigm originates in ColBERT’s “contextualized late interaction,” which decouples query and document encoding, preserves token-level evidence, and replaces full cross-encoding with a MaxSim aggregation over contextualized token embeddings [2004.12832]. Subsequent systems refined the same idea through residual compression, centroid pruning, sparse first-stage retrieval, disk-persisted indexes, and multimodal page-level token storage, making late interaction viable for large text and document collections rather than only for in-memory re-ranking [2112.01488] [2205.09707] [2606.19960].

## 1. Origins in contextualized late interaction

The foundational formulation appears in ColBERT, which independently encodes queries and documents with a shared BERT encoder and delays interaction until a lightweight token-level scoring stage [2004.12832]. Queries are prepended with `[Q]` after `[CLS]`, documents with `[D]` after `[CLS]`, query sequences are padded up to a fixed length $N_q$ with BERT `[mask]` tokens, and contextualized hidden states are projected to an $m$-dimensional space and L2-normalized so that dot products equal cosine similarity. Document embeddings filter punctuation tokens to reduce storage and compute. In the experiments described for ColBERT, a typical setting is $m = 128$, with query augmentation to $N_q = 32$ embeddings and WordPiece tokenization throughout [2004.12832].

The key systems implication is that documents no longer need to be re-encoded per query. ColBERT explicitly exploits this by precomputing document token embeddings offline and storing them on disk or in CPU memory, while encoding the query once and reusing its embedding matrix across all candidate documents [2004.12832]. The paper reports that this architecture remains competitive with existing BERT-based ranking models while executing two orders-of-magnitude faster and requiring four orders-of-magnitude fewer FLOPs per query [2004.12832]. This is the central historical step from neural re-ranking toward DLI: token-level interaction is retained, but document-side transformer inference is removed from the online path.

The later literature extends the same decomposition rather than abandoning it. ColBERTv2 keeps the late interaction scorer but shrinks the space footprint through residual compression and denoised supervision [2112.01488]. PLAID keeps ColBERTv2’s compressed representation and accelerates search through centroid interaction and centroid pruning [2205.09707]. SLIM and SPLATE move first-stage candidate generation onto sparse inverted indexes while reserving exact late interaction for a second stage [2302.06587] [2404.13950]. Stellar and later visual-document retrievers apply the same architectural principle to multimodal token sequences stored on disk [2606.19960] [2602.03992].

## 2. Scoring functions and interaction operators

The canonical late interaction score is ColBERT’s MaxSim:
$$
s(Q, D) = \sum_{i=1}^{|E_q|} \max_{j \in [1, |E_d|]} E_{q_i}^\top E_{d_j}.
$$
With L2-normalized token embeddings, the inner product is cosine similarity [2004.12832]. ColBERT also supports an end-to-end retrieval variant based on squared L2 distance,
$$
s_{L2}(Q, D) = - \sum_i \min_j \lVert E_{q_i} - E_{d_j} \rVert_2^2,
$$
which is equivalent to MaxSim over negative squared L2 and was used because FAISS was faster with L2 in that setup [2004.12832]. ColBERTv2 preserves the same MaxSim scorer and treats compression and indexing as orthogonal to the underlying late interaction function [2112.01488].

ColBERTv2’s main change is representational rather than algebraic. Each document token vector $v$ is approximated as “nearest centroid + quantized residual”,
$$
v \approx \tilde{v} = C_t + \tilde{r},
$$
with storage cost of 4 bytes for the centroid ID and 16 or 32 bytes for the residual when $b \in \{1,2\}$ bits are used across $d = 128$ dimensions [2112.01488]. Query-time candidate generation uses an approximate lower-bound MaxSim obtained by probing centroids near each query token, while final ranking computes exact late interaction over the compressed per-candidate document representation [2112.01488].

A different line of work replaces dense token-space candidate generation with sparse lexical projections. SLIM maps each contextualized token vector $\mathbf{x}$ into a sparse vocabulary-aligned space through
$$
\phi(\mathbf{x}) = \log\big(1 + \mathrm{ReLU}(W^\top \mathbf{x} + \mathbf{b})\big),
$$
and then defines sparsified late interaction as
$$
s_{\text{SLIM}}(q, d) = \sum_{i=1}^{|q|} \max_{1 \le j \le |d|} \langle \phi(\mathbf{q}_i), \phi(\mathbf{d}_j) \rangle.
$$
Its first-stage retriever uses a linear interpolation between an upper and a lower bound of this score, reducing retrieval to a sparse dot product that is compatible with Lucene’s inverted indexes [2302.06587]. SPLATE performs a related decomposition by learning an MLM adapter on frozen ColBERTv2 token embeddings, aggregating token-to-vocabulary logits through a SPLADE-style transformation,
$$
w_v = \max_{i \in t} \log\Big(1 + \mathrm{ReLU}(w_{iv})\Big),
$$
and using the sparse dot product
$$
s_{\text{sparse}}(q,d) = \sum_{v \in \mathcal{V}} w_v(q)\, w_v(d)
$$
for candidate generation before exact ColBERTv2 MaxSim re-ranking [2404.13950].

Late interaction need not remain hand-crafted. LITE retains the similarity matrix $S = Q^\top D$ but replaces sum-max with a learnable scorer. In its separable variant, row-wise and column-wise MLP blocks transform the similarity matrix and the final score is
$$
s(q, d) = \mathbf{w}^\top {\sf vec}(\mathbf{S}'').
$$
The paper proves that LITE is a universal approximator of continuous scoring functions and reports that “small separable LITE” lowers latency and requires $0.25\times$ storage compared to ColBERT while improving MS MARCO passage re-ranking quality [2406.17968]. This suggests that DLI is compatible both with classical MaxSim and with more expressive late interaction operators, provided the document representation remains factorized and precomputable.

## 3. On-disk representation and index organization

The earliest DLI layout is straightforward dense persistence. ColBERT stores per-document token embedding matrices on disk, optionally in 16-bit or 32-bit format, and can batch them into 3D tensors for GPU re-ranking [2004.12832]. On MS MARCO, the reported footprint is 286 GiB for $m = 128$, 4 bytes/dim, cosine re-ranking with MRR@10 = 34.9; 154 GiB for $m = 128$, 2 bytes/dim, L2 end-to-end retrieval with MRR@10 = 36.0; 143 GiB for $m = 128$, 2 bytes/dim, L2 re-ranking with MRR@10 = 34.8; 54 GiB for $m = 48$, 4 bytes/dim, cosine with MRR@10 = 34.4; and 27 GiB for $m = 24$, 2 bytes/dim, cosine with MRR@10 = 33.9 [2004.12832]. The quality degradation under smaller $m$ and lower-precision storage is modest relative to the space reduction.

ColBERTv2 changes the storage unit from full-precision dense token vectors to centroid IDs plus residual codes. Its on-disk structures comprise per-token records, per-document grouped compressed token codes, and inverted lists per centroid [2112.01488]. On MS MARCO, the vanilla ColBERT index is 154 GiB, whereas ColBERTv2 reports 16 GiB for 1-bit residuals or 25 GiB for 2-bit residuals, plus about 4.5 GiB for inverted lists [2112.01488]. Centroids are chosen by k-means, with $|C|$ proportional to $16 \times \sqrt{n_{\text{embeddings}}}$ and rounded to the nearest power of two; the centroid ID is stored in 4 bytes [2112.01488].

PLAID preserves ColBERTv2’s compressed representation but reorganizes the online structures around centroid interaction. Its practical DLI design consists of a global centroid codebook, centroid-to-passage inverted lists, a passage directory, a centroid-ID stream, and a packed residual stream [2205.09707]. A particularly consequential engineering change is that PLAID stores passage IDs rather than embedding IDs in the centroid inverted lists; in MS MARCO v2 this reduced inverted-list space from 71 GB to 27 GB [2205.09707]. Reported end-to-end index sizes are 24.6 GiB for vanilla versus 21.6 GiB for PLAID on MS MARCO v1, 105.2 GiB versus 92.0 GiB on Wikipedia, 14.0 GiB versus 12.3 GiB on LoTTE pooled, and 246.0 GiB versus 202.2 GiB on MS MARCO v2 [2205.09707].

Sparse DLI systems adopt different layouts. SLIM indexes the sequence-level vector
$$
g_d := \max_{1 \le j \le |d|} \phi(\mathbf{d}_j)
$$
in Lucene as an impact-style inverted index, while storing the full token-level sparse matrices in SciPy CSR format for score refinement [2302.06587]. SPLATE adds a compact sparse index alongside the ColBERT index; on MS MARCO, the reported PISA index is approximately 2.2 GB, which the paper describes as negligible relative to ColBERT’s token-embedding index [2404.13950].

Stellar introduces a distinctly disk-native organization for multimodal late interaction. Documents are clustered by sparse lexical vectors using a balanced clustering algorithm, and all token embeddings of documents in each cluster are written contiguously into a disk block [2606.19960]. The in-memory metadata is limited to a document-level index mapping document IDs to block IDs, token counts, offsets, and lengths, and a block-level index mapping block IDs to document lists and total token counts [2606.19960]. Unlike ColBERTv2-style systems, Stellar stores original unquantized token embeddings on disk rather than quantized codes in memory [2606.19960].

## 4. Query-time execution and pruning strategies

In ColBERT re-ranking, query-time execution is simple but bandwidth-sensitive. The online pipeline is: encode the query once; gather the candidate documents’ embeddings into a 3D tensor; compute similarity matrices between the query embeddings and each document’s embeddings; max-pool over document tokens and sum over query tokens; and sort by the resulting scores [2004.12832]. Ignoring memory transfers, the interaction cost is $O(k \cdot |E_q| \cdot |E_d| \cdot m)$ dot-product FLOPs, but the paper notes that in practice latency is dominated by gathering and CPU→GPU transfer of document embeddings; query encoding and dot products can be only about 13 ms of the total [2004.12832].

For end-to-end retrieval, ColBERT builds a FAISS IVFPQ index over all document token embeddings. A representative configuration partitions the space into $P = 2000$ clusters, probes $p = 10$ partitions per query embedding, splits each vector into $s = 16$ sub-vectors stored at 1 byte each, and retrieves top-$k'$ nearest document embeddings per query token before mapping them back to document IDs and re-ranking the resulting candidate set exactly [2004.12832]. ColBERTv2 retains the two-stage design but changes the approximate stage: each query token probes its nearest centroids, candidate document tokens are recovered from centroid inverted lists, and a lower-bound MaxSim is accumulated before full per-candidate ranking [2112.01488]. Typical sweeps use $n_{\text{probe}} \in \{1,2,4\}$ and candidate counts of approximately $\text{probe} \times 2^{12}$ to $2^{14}$ depending on collection size [2112.01488].

PLAID decomposes online execution even further. It first computes the query-to-centroid score matrix
$$
S_{c,q} = C Q^\top,
$$
then uses centroid inverted lists for candidate generation, applies centroid pruning by keeping centroid $i$ only if
$$
\max_{j=1}^{|Q|} S_{c,q}[i, j] \ge t_{cs},
$$
runs centroid-only MaxSim on the remaining centroid IDs, and finally loads residuals only for the top survivors [2205.09707]. The recommended defaults are $(nprobe=1, t_{cs}=0.5, ndocs=256)$ for $k=10$, $(nprobe=2, t_{cs}=0.45, ndocs=1024)$ for $k=100$, and $(nprobe=4, t_{cs}=0.4, ndocs=4096)$ for $k=1000$, with Stage 3 typically outputting $ndocs/4$ candidates for exact refinement [2205.09707].

Sparse first stages alter the online profile by moving candidate generation to mature inverted-index engines. SPLATE computes sparse query vectors from the same frozen ColBERTv2 encoder used for the second-stage re-ranker, applies top-$k_q$ pooling, and retrieves candidates with WAND or Block-Max WAND before exact MaxSim re-ranking [2404.13950]. SLIM uses Lucene’s ImpactSearcher for the first stage and then loads only the top-$K$ candidates’ SciPy CSR matrices to compute exact sparsified late interaction on CPU [2302.06587].

Block-oriented DLI adds another level of scheduling. Stellar first performs lexical filtering to obtain top-$k_1$ candidates, then groups the candidates by disk block and chooses between Full Block Loading and Specific Vector Loading using the per-block cost model
$$
T_{\text{FBL}}(b_i) = \frac{N_{i,\text{total}} \cdot V_{\text{dim}} \cdot B_{\text{float}}}{R_{\text{seq}}},
\qquad
T_{\text{SVL}}(b_i) = \frac{N_{i,\text{req}} \cdot V_{\text{dim}} \cdot B_{\text{float}}}{R_{\text{rand}}},
$$
loading the cheaper alternative before running dense MaxSim and fusing sparse and dense scores [2606.19960]. In multimodal page retrieval with ColPali, a related though simpler pattern is to store each patch embedding as an individual item in an OpenSearch HNSW index, aggregate retrieved patch hits to page IDs via metadata, reconstruct only those candidate pages, and run late interaction on the reduced set [2507.12378].

## 5. Effectiveness, latency, and storage trade-offs

On MS MARCO passage re-ranking, ColBERT reports MRR@10 of 34.9/34.9 on Dev/Eval with cosine similarity, 61 ms latency for top-1000 re-ranking, and 7B FLOPs per query [2004.12832]. The reported baselines are 10,700 ms and 97T FLOPs for BERT base single-document scoring and 32,900 ms and 340T FLOPs for duoBERT, corresponding to a speedup of more than $170\times$ and a FLOPs reduction of about $13{,}900\times$ to $48{,}600\times$ depending on the baseline [2004.12832]. In end-to-end retrieval, ColBERT\_L2 reaches MRR@10 of 36.0 on Dev and 36.7 on Local Eval at 458 ms latency, with Recall values of 82.9@50, 92.3@200, and 96.8@1000 [2004.12832].

ColBERTv2 improves both quality and footprint. On MS MARCO Dev it reports MRR@10 = 39.7, R@50 = 86.8, and R@1k = 98.4; on Local Eval it reports MRR@10 = 40.8 [2112.01488]. The paper also reports end-to-end latencies of about 50–250 ms per query, mostly under about 150 ms, with the best quality typically near about 100 ms [2112.01488]. Out of domain, it reports nDCG@10 values such as 44.6 on DBPedia, 35.6 on FiQA, 56.2 on NQ, 66.7 on HotpotQA, 33.8 on NFCorpus, and 73.8 on TREC-COVID [2112.01488].

PLAID preserves ColBERTv2-quality retrieval while substantially accelerating search. The paper reports up to $7\times$ speedup on GPU and $45\times$ on CPU versus vanilla ColBERTv2, with tens of milliseconds latency on GPU and tens or just few hundreds of milliseconds on CPU, even at 140M passages [2205.09707]. On MS MARCO v1, PLAID at $k=1000$ matches vanilla quality with 38.4 ms versus 259.6 ms on GPU and 101.3 ms versus 4568.5 ms on CPU; at $k=10$ it reports 11.5 ms on GPU and 31.5 ms on CPU [2205.09707].

Sparse DLI variants show a different efficiency frontier. SPLATE’s abstract result is that it achieves the same effectiveness as the PLAID ColBERTv2 engine by re-ranking 50 documents that can be retrieved under 10 ms [2404.13950]. On MS MARCO Dev, SPLATE (e2e) reports MRR@10 = 40.0 versus ColBERTv2 39.7 and PLAID ColBERTv2 39.8; BEIR nDCG@10 is 49.6 for SPLATE (e2e) versus 49.7 for ColBERTv2 [2404.13950]. SLIM reports MRR@10 = 0.358 and BEIR = 0.451 with an 18.2 GB index, while SLIM++ reports MRR@10 = 0.404 and a 17.3 GB index; the paper states that SLIM++ achieves effectiveness comparable to ColBERT-v2 with an 83% decrease in CPU latency and 40% less disk storage [2302.06587].

Multimodal DLI results reinforce the same pattern. Stellar reports, on a 400K-document LargeDoc benchmark, 988 MB peak memory and 110 ms latency versus 147,355 MB and 42,115 ms for ColPali and 13,699 MB and 613 ms for QColPali, while improving effectiveness to R@1/R@10/MRR@10 of 75.53/87.23/79.47 [2606.19960]. Its paper summarizes the gains as memory reduction of up to $150\times$ and latency reduction of up to $382\times$ versus exact multi-vector baselines [2606.19960]. In a separate OpenSearch-based multimodal pipeline, ColPali 1.2 with OpenSearch + late interaction reports the same average Recall@1 of 74.56 on ViDoRe as in-memory late interaction, matching it on 8/10 datasets and remaining within about 1% on the other two [2507.12378].

For visual document retrieval at larger embedding dimensions, the storage challenge remains acute even when retrieval quality is high. Nemotron ColEmbed V2 reports ViDoRe V3 NDCG@10 of 63.42 for the 8B model, 61.54 for the 4B model, and 59.79 for the 3B model, but also reports fp16 storage footprints of about 5,897.5 GB, 3,686.0 GB, and 13,183.6 GB per 1M images, respectively [2602.03992]. Learned projection layers mitigate the footprint sharply: for the 8B model, projection to $d = 512$ reduces storage to 737.2 GB while retaining NDCG@10 = 59.81, and projection to $d = 128$ reduces storage to 184.3 GB while retaining NDCG@10 = 59.40 [2602.03992].

## 6. Variants, applications, and unresolved systems issues

The mature literature shows that DLI is not tied to one retrieval engine. PyLate exposes disk-backed indexes through `index_folder` and `index_name`, implements HNSW and PLAID-based retrieval, and adds post-hoc token pooling compression that cuts the index footprint roughly in half with negligible performance loss [2508.03555]. Because PyLate decouples modeling from indexation, the same embedding-level interfaces are intended to support ColBERT-family models and non-text modalities such as ColPali [2508.03555]. This suggests that DLI has become a systems pattern: precompute token embeddings, persist them, prune aggressively, and reserve exact late interaction for a manageable candidate set.

The same pattern now appears in multimodal retrieval and RAG. OpenSearch-based page retrieval with ColPali indexes 128-dimensional patch embeddings as separate HNSW items with `document id`, `page id`, and `patch id` metadata, aggregates candidate pages by metadata, reconstructs full page tensors of shape $(1030, 128)$, and then applies ColBERT-style late interaction before passing the top-ranked pages to an MLLM reader [2507.12378]. Stellar couples a sparse lexical filter with a balanced-cluster disk layout and a cost-aware loader, explicitly targeting CPU-only retrieval for multimodal pages [2606.19960]. Nemotron ColEmbed V2 emphasizes the opposite extreme: very strong late interaction accuracy with large token sequences and very large embedding dimensions, making disk-backed serving, blockwise MaxSim, and candidate prefiltering operationally necessary rather than optional [2602.03992].

Several limitations recur across the literature. Memory footprint remains substantial whenever many token vectors must be retained, even after compression or projection [2004.12832] [2112.01488] [2602.03992]. I/O bottlenecks often dominate re-ranking latency, especially CPU→GPU transfer of document embeddings or random access over residual streams and document blocks [2004.12832] [2205.09707]. Dynamic updates, incremental clustering, and caching policies are repeatedly identified as open systems problems rather than settled practice [2112.01488] [2606.19960]. Some frameworks also leave key engineering details unspecified: PyLate does not detail low-level file formats or OS-level memory mapping, and Nemotron ColEmbed V2 does not disclose a specific retrieval engine for production MaxSim serving [2508.03555] [2602.03992].

A common misconception is that late interaction necessarily implies full in-memory token storage or exhaustive scoring. The cited systems show otherwise. Candidate generation may be approximate, sparse, centroid-based, or block-aware, but the final stage is often still exact late interaction over a sharply reduced candidate set [2112.01488] [2205.09707] [2404.13950] [2606.19960]. A second misconception is that DLI is only a text-retrieval technique. The multimodal results with ColPali, Stellar, and Nemotron ColEmbed V2 show that the same architectural principle extends naturally to page patches and visual tokens, although the storage and serving constraints become more severe as token counts and embedding dimensions rise [2507.12378] [2606.19960] [2602.03992].

Source: https://www.emergentmind.com/topics/efficient-disk-backed-late-interaction-dli