Efficient Disk-Backed Late Interaction

• Efficient Disk-backed Late Interaction is a retrieval architecture that decouples query and document encoding by precomputing token embeddings offline for later token-level scoring.
• It employs techniques like centroid pruning, residual compression, and sparse candidate generation to substantially lower FLOPs, latency, and storage overhead.
• The approach is versatile, enabling scalable multimodal retrieval and efficient indexing by shifting heavy computations off the online path.

Efficient Disk-backed Late Interaction (DLI) denotes a retrieval architecture in which document-side multi-vector representations are precomputed offline, persisted on disk or in CPU-addressable storage, and loaded selectively at query time so that late interaction scoring is applied only where it is most useful. The paradigm originates in ColBERT’s “contextualized late interaction,” which decouples query and document encoding, preserves token-level evidence, and replaces full cross-encoding with a MaxSim aggregation over contextualized token embeddings (Khattab et al., 2020). Subsequent systems refined the same idea through residual compression, centroid pruning, sparse first-stage retrieval, disk-persisted indexes, and multimodal page-level token storage, making late interaction viable for large text and document collections rather than only for in-memory re-ranking (Santhanam et al., 2021, Santhanam et al., 2022, Guo et al., 18 Jun 2026).

1. Origins in contextualized late interaction

The foundational formulation appears in ColBERT, which independently encodes queries and documents with a shared BERT encoder and delays interaction until a lightweight token-level scoring stage (Khattab et al., 2020). Queries are prepended with [Q] after [CLS], documents with [D] after [CLS], query sequences are padded up to a fixed length $N_q$ with BERT [mask] tokens, and contextualized hidden states are projected to an $m$-dimensional space and L2-normalized so that dot products equal cosine similarity. Document embeddings filter punctuation tokens to reduce storage and compute. In the experiments described for ColBERT, a typical setting is $m = 128$, with query augmentation to $N_q = 32$ embeddings and WordPiece tokenization throughout (Khattab et al., 2020).

The key systems implication is that documents no longer need to be re-encoded per query. ColBERT explicitly exploits this by precomputing document token embeddings offline and storing them on disk or in CPU memory, while encoding the query once and reusing its embedding matrix across all candidate documents (Khattab et al., 2020). The paper reports that this architecture remains competitive with existing BERT-based ranking models while executing two orders-of-magnitude faster and requiring four orders-of-magnitude fewer FLOPs per query (Khattab et al., 2020). This is the central historical step from neural re-ranking toward DLI: token-level interaction is retained, but document-side transformer inference is removed from the online path.

The later literature extends the same decomposition rather than abandoning it. ColBERTv2 keeps the late interaction scorer but shrinks the space footprint through residual compression and denoised supervision (Santhanam et al., 2021). PLAID keeps ColBERTv2’s compressed representation and accelerates search through centroid interaction and centroid pruning (Santhanam et al., 2022). SLIM and SPLATE move first-stage candidate generation onto sparse inverted indexes while reserving exact late interaction for a second stage (Li et al., 2023, Formal et al., 2024). Stellar and later visual-document retrievers apply the same architectural principle to multimodal token sequences stored on disk (Guo et al., 18 Jun 2026, Moreira et al., 3 Feb 2026).

2. Scoring functions and interaction operators

The canonical late interaction score is ColBERT’s MaxSim:

$$s(Q, D) = \sum_{i=1}^{|E_q|} \max_{j \in [1, |E_d|]} E_{q_i}^\top E_{d_j}.$$

With L2-normalized token embeddings, the inner product is cosine similarity (Khattab et al., 2020). ColBERT also supports an end-to-end retrieval variant based on squared L2 distance,

$$s_{L2}(Q, D) = - \sum_i \min_j \lVert E_{q_i} - E_{d_j} \rVert_2^2,$$

which is equivalent to MaxSim over negative squared L2 and was used because FAISS was faster with L2 in that setup (Khattab et al., 2020). ColBERTv2 preserves the same MaxSim scorer and treats compression and indexing as orthogonal to the underlying late interaction function (Santhanam et al., 2021).

ColBERTv2’s main change is representational rather than algebraic. Each document token vector $v$ is approximated as “nearest centroid + quantized residual”,

$v \approx \tilde{v} = C_t + \tilde{r},$

with storage cost of 4 bytes for the centroid ID and 16 or 32 bytes for the residual when $b \in \{1,2\}$ bits are used across $d = 128$ dimensions (Santhanam et al., 2021). Query-time candidate generation uses an approximate lower-bound MaxSim obtained by probing centroids near each query token, while final ranking computes exact late interaction over the compressed per-candidate document representation (Santhanam et al., 2021).

A different line of work replaces dense token-space candidate generation with sparse lexical projections. SLIM maps each contextualized token vector $m$0 into a sparse vocabulary-aligned space through

$m$1

and then defines sparsified late interaction as

$m$2

Its first-stage retriever uses a linear interpolation between an upper and a lower bound of this score, reducing retrieval to a sparse dot product that is compatible with Lucene’s inverted indexes (Li et al., 2023). SPLATE performs a related decomposition by learning an MLM adapter on frozen ColBERTv2 token embeddings, aggregating token-to-vocabulary logits through a SPLADE-style transformation,

$m$3

and using the sparse dot product

$m$4

for candidate generation before exact ColBERTv2 MaxSim re-ranking (Formal et al., 2024).

Late interaction need not remain hand-crafted. LITE retains the similarity matrix $m$5 but replaces sum-max with a learnable scorer. In its separable variant, row-wise and column-wise MLP blocks transform the similarity matrix and the final score is

$m$6

The paper proves that LITE is a universal approximator of continuous scoring functions and reports that “small separable LITE” lowers latency and requires $m$7 storage compared to ColBERT while improving MS MARCO passage re-ranking quality (Ji et al., 2024). This suggests that DLI is compatible both with classical MaxSim and with more expressive late interaction operators, provided the document representation remains factorized and precomputable.

3. On-disk representation and index organization

The earliest DLI layout is straightforward dense persistence. ColBERT stores per-document token embedding matrices on disk, optionally in 16-bit or 32-bit format, and can batch them into 3D tensors for GPU re-ranking (Khattab et al., 2020). On MS MARCO, the reported footprint is 286 GiB for $m$8, 4 bytes/dim, cosine re-ranking with MRR@10 = 34.9; 154 GiB for $m$9, 2 bytes/dim, L2 end-to-end retrieval with MRR@10 = 36.0; 143 GiB for $m = 128$0, 2 bytes/dim, L2 re-ranking with MRR@10 = 34.8; 54 GiB for $m = 128$1, 4 bytes/dim, cosine with MRR@10 = 34.4; and 27 GiB for $m = 128$2, 2 bytes/dim, cosine with MRR@10 = 33.9 (Khattab et al., 2020). The quality degradation under smaller $m = 128$3 and lower-precision storage is modest relative to the space reduction.

ColBERTv2 changes the storage unit from full-precision dense token vectors to centroid IDs plus residual codes. Its on-disk structures comprise per-token records, per-document grouped compressed token codes, and inverted lists per centroid (Santhanam et al., 2021). On MS MARCO, the vanilla ColBERT index is 154 GiB, whereas ColBERTv2 reports 16 GiB for 1-bit residuals or 25 GiB for 2-bit residuals, plus about 4.5 GiB for inverted lists (Santhanam et al., 2021). Centroids are chosen by k-means, with $m = 128$4 proportional to $m = 128$5 and rounded to the nearest power of two; the centroid ID is stored in 4 bytes (Santhanam et al., 2021).

PLAID preserves ColBERTv2’s compressed representation but reorganizes the online structures around centroid interaction. Its practical DLI design consists of a global centroid codebook, centroid-to-passage inverted lists, a passage directory, a centroid-ID stream, and a packed residual stream (Santhanam et al., 2022). A particularly consequential engineering change is that PLAID stores passage IDs rather than embedding IDs in the centroid inverted lists; in MS MARCO v2 this reduced inverted-list space from 71 GB to 27 GB (Santhanam et al., 2022). Reported end-to-end index sizes are 24.6 GiB for vanilla versus 21.6 GiB for PLAID on MS MARCO v1, 105.2 GiB versus 92.0 GiB on Wikipedia, 14.0 GiB versus 12.3 GiB on LoTTE pooled, and 246.0 GiB versus 202.2 GiB on MS MARCO v2 (Santhanam et al., 2022).

Sparse DLI systems adopt different layouts. SLIM indexes the sequence-level vector

$m = 128$6

in Lucene as an impact-style inverted index, while storing the full token-level sparse matrices in SciPy CSR format for score refinement (Li et al., 2023). SPLATE adds a compact sparse index alongside the ColBERT index; on MS MARCO, the reported PISA index is approximately 2.2 GB, which the paper describes as negligible relative to ColBERT’s token-embedding index (Formal et al., 2024).

Stellar introduces a distinctly disk-native organization for multimodal late interaction. Documents are clustered by sparse lexical vectors using a balanced clustering algorithm, and all token embeddings of documents in each cluster are written contiguously into a disk block (Guo et al., 18 Jun 2026). The in-memory metadata is limited to a document-level index mapping document IDs to block IDs, token counts, offsets, and lengths, and a block-level index mapping block IDs to document lists and total token counts (Guo et al., 18 Jun 2026). Unlike ColBERTv2-style systems, Stellar stores original unquantized token embeddings on disk rather than quantized codes in memory (Guo et al., 18 Jun 2026).

4. Query-time execution and pruning strategies

In ColBERT re-ranking, query-time execution is simple but bandwidth-sensitive. The online pipeline is: encode the query once; gather the candidate documents’ embeddings into a 3D tensor; compute similarity matrices between the query embeddings and each document’s embeddings; max-pool over document tokens and sum over query tokens; and sort by the resulting scores (Khattab et al., 2020). Ignoring memory transfers, the interaction cost is $m = 128$7 dot-product FLOPs, but the paper notes that in practice latency is dominated by gathering and CPU→GPU transfer of document embeddings; query encoding and dot products can be only about 13 ms of the total (Khattab et al., 2020).

For end-to-end retrieval, ColBERT builds a FAISS IVFPQ index over all document token embeddings. A representative configuration partitions the space into $m = 128$8 clusters, probes $m = 128$9 partitions per query embedding, splits each vector into $N_q = 32$0 sub-vectors stored at 1 byte each, and retrieves top-$N_q = 32$1 nearest document embeddings per query token before mapping them back to document IDs and re-ranking the resulting candidate set exactly (Khattab et al., 2020). ColBERTv2 retains the two-stage design but changes the approximate stage: each query token probes its nearest centroids, candidate document tokens are recovered from centroid inverted lists, and a lower-bound MaxSim is accumulated before full per-candidate ranking (Santhanam et al., 2021). Typical sweeps use $N_q = 32$2 and candidate counts of approximately $N_q = 32$3 to $N_q = 32$4 depending on collection size (Santhanam et al., 2021).

PLAID decomposes online execution even further. It first computes the query-to-centroid score matrix

$N_q = 32$5

then uses centroid inverted lists for candidate generation, applies centroid pruning by keeping centroid $N_q = 32$6 only if

$N_q = 32$7

runs centroid-only MaxSim on the remaining centroid IDs, and finally loads residuals only for the top survivors (Santhanam et al., 2022). The recommended defaults are $N_q = 32$8 for $N_q = 32$9, $$s(Q, D) = \sum_{i=1}^{|E_q|} \max_{j \in [1, |E_d|]} E_{q_i}^\top E_{d_j}.$$0 for $$s(Q, D) = \sum_{i=1}^{|E_q|} \max_{j \in [1, |E_d|]} E_{q_i}^\top E_{d_j}.$$1, and $$s(Q, D) = \sum_{i=1}^{|E_q|} \max_{j \in [1, |E_d|]} E_{q_i}^\top E_{d_j}.$$2 for $$s(Q, D) = \sum_{i=1}^{|E_q|} \max_{j \in [1, |E_d|]} E_{q_i}^\top E_{d_j}.$$3, with Stage 3 typically outputting $$s(Q, D) = \sum_{i=1}^{|E_q|} \max_{j \in [1, |E_d|]} E_{q_i}^\top E_{d_j}.$$4 candidates for exact refinement (Santhanam et al., 2022).

Sparse first stages alter the online profile by moving candidate generation to mature inverted-index engines. SPLATE computes sparse query vectors from the same frozen ColBERTv2 encoder used for the second-stage re-ranker, applies top-$$s(Q, D) = \sum_{i=1}^{|E_q|} \max_{j \in [1, |E_d|]} E_{q_i}^\top E_{d_j}.$$5 pooling, and retrieves candidates with WAND or Block-Max WAND before exact MaxSim re-ranking (Formal et al., 2024). SLIM uses Lucene’s ImpactSearcher for the first stage and then loads only the top-$$s(Q, D) = \sum_{i=1}^{|E_q|} \max_{j \in [1, |E_d|]} E_{q_i}^\top E_{d_j}.$$6 candidates’ SciPy CSR matrices to compute exact sparsified late interaction on CPU (Li et al., 2023).

Block-oriented DLI adds another level of scheduling. Stellar first performs lexical filtering to obtain top-$$s(Q, D) = \sum_{i=1}^{|E_q|} \max_{j \in [1, |E_d|]} E_{q_i}^\top E_{d_j}.$$7 candidates, then groups the candidates by disk block and chooses between Full Block Loading and Specific Vector Loading using the per-block cost model

$$s(Q, D) = \sum_{i=1}^{|E_q|} \max_{j \in [1, |E_d|]} E_{q_i}^\top E_{d_j}.$$8

loading the cheaper alternative before running dense MaxSim and fusing sparse and dense scores (Guo et al., 18 Jun 2026). In multimodal page retrieval with ColPali, a related though simpler pattern is to store each patch embedding as an individual item in an OpenSearch HNSW index, aggregate retrieved patch hits to page IDs via metadata, reconstruct only those candidate pages, and run late interaction on the reduced set (Saxena et al., 16 Jul 2025).

5. Effectiveness, latency, and storage trade-offs

On MS MARCO passage re-ranking, ColBERT reports MRR@10 of 34.9/34.9 on Dev/Eval with cosine similarity, 61 ms latency for top-1000 re-ranking, and 7B FLOPs per query (Khattab et al., 2020). The reported baselines are 10,700 ms and 97T FLOPs for BERT base single-document scoring and 32,900 ms and 340T FLOPs for duoBERT, corresponding to a speedup of more than $$s(Q, D) = \sum_{i=1}^{|E_q|} \max_{j \in [1, |E_d|]} E_{q_i}^\top E_{d_j}.$$9 and a FLOPs reduction of about $$s_{L2}(Q, D) = - \sum_i \min_j \lVert E_{q_i} - E_{d_j} \rVert_2^2,$$0 to $$s_{L2}(Q, D) = - \sum_i \min_j \lVert E_{q_i} - E_{d_j} \rVert_2^2,$$1 depending on the baseline (Khattab et al., 2020). In end-to-end retrieval, ColBERT_L2 reaches MRR@10 of 36.0 on Dev and 36.7 on Local Eval at 458 ms latency, with Recall values of 82.9@50, 92.3@200, and 96.8@1000 (Khattab et al., 2020).

ColBERTv2 improves both quality and footprint. On MS MARCO Dev it reports MRR@10 = 39.7, R@50 = 86.8, and R@1k = 98.4; on Local Eval it reports MRR@10 = 40.8 (Santhanam et al., 2021). The paper also reports end-to-end latencies of about 50–250 ms per query, mostly under about 150 ms, with the best quality typically near about 100 ms (Santhanam et al., 2021). Out of domain, it reports nDCG@10 values such as 44.6 on DBPedia, 35.6 on FiQA, 56.2 on NQ, 66.7 on HotpotQA, 33.8 on NFCorpus, and 73.8 on TREC-COVID (Santhanam et al., 2021).

PLAID preserves ColBERTv2-quality retrieval while substantially accelerating search. The paper reports up to $$s_{L2}(Q, D) = - \sum_i \min_j \lVert E_{q_i} - E_{d_j} \rVert_2^2,$$2 speedup on GPU and $$s_{L2}(Q, D) = - \sum_i \min_j \lVert E_{q_i} - E_{d_j} \rVert_2^2,$$3 on CPU versus vanilla ColBERTv2, with tens of milliseconds latency on GPU and tens or just few hundreds of milliseconds on CPU, even at 140M passages (Santhanam et al., 2022). On MS MARCO v1, PLAID at $$s_{L2}(Q, D) = - \sum_i \min_j \lVert E_{q_i} - E_{d_j} \rVert_2^2,$$4 matches vanilla quality with 38.4 ms versus 259.6 ms on GPU and 101.3 ms versus 4568.5 ms on CPU; at $$s_{L2}(Q, D) = - \sum_i \min_j \lVert E_{q_i} - E_{d_j} \rVert_2^2,$$5 it reports 11.5 ms on GPU and 31.5 ms on CPU (Santhanam et al., 2022).

Sparse DLI variants show a different efficiency frontier. SPLATE’s abstract result is that it achieves the same effectiveness as the PLAID ColBERTv2 engine by re-ranking 50 documents that can be retrieved under 10 ms (Formal et al., 2024). On MS MARCO Dev, SPLATE (e2e) reports MRR@10 = 40.0 versus ColBERTv2 39.7 and PLAID ColBERTv2 39.8; BEIR nDCG@10 is 49.6 for SPLATE (e2e) versus 49.7 for ColBERTv2 (Formal et al., 2024). SLIM reports MRR@10 = 0.358 and BEIR = 0.451 with an 18.2 GB index, while SLIM++ reports MRR@10 = 0.404 and a 17.3 GB index; the paper states that SLIM++ achieves effectiveness comparable to ColBERT-v2 with an 83% decrease in CPU latency and 40% less disk storage (Li et al., 2023).

Multimodal DLI results reinforce the same pattern. Stellar reports, on a 400K-document LargeDoc benchmark, 988 MB peak memory and 110 ms latency versus 147,355 MB and 42,115 ms for ColPali and 13,699 MB and 613 ms for QColPali, while improving effectiveness to R@1/R@10/MRR@10 of 75.53/87.23/79.47 (Guo et al., 18 Jun 2026). Its paper summarizes the gains as memory reduction of up to $$s_{L2}(Q, D) = - \sum_i \min_j \lVert E_{q_i} - E_{d_j} \rVert_2^2,$$6 and latency reduction of up to $$s_{L2}(Q, D) = - \sum_i \min_j \lVert E_{q_i} - E_{d_j} \rVert_2^2,$$7 versus exact multi-vector baselines (Guo et al., 18 Jun 2026). In a separate OpenSearch-based multimodal pipeline, ColPali 1.2 with OpenSearch + late interaction reports the same average Recall@1 of 74.56 on ViDoRe as in-memory late interaction, matching it on 8/10 datasets and remaining within about 1% on the other two (Saxena et al., 16 Jul 2025).

For visual document retrieval at larger embedding dimensions, the storage challenge remains acute even when retrieval quality is high. Nemotron ColEmbed V2 reports ViDoRe V3 NDCG@10 of 63.42 for the 8B model, 61.54 for the 4B model, and 59.79 for the 3B model, but also reports fp16 storage footprints of about 5,897.5 GB, 3,686.0 GB, and 13,183.6 GB per 1M images, respectively (Moreira et al., 3 Feb 2026). Learned projection layers mitigate the footprint sharply: for the 8B model, projection to $$s_{L2}(Q, D) = - \sum_i \min_j \lVert E_{q_i} - E_{d_j} \rVert_2^2,$$8 reduces storage to 737.2 GB while retaining NDCG@10 = 59.81, and projection to $$s_{L2}(Q, D) = - \sum_i \min_j \lVert E_{q_i} - E_{d_j} \rVert_2^2,$$9 reduces storage to 184.3 GB while retaining NDCG@10 = 59.40 (Moreira et al., 3 Feb 2026).

6. Variants, applications, and unresolved systems issues

The mature literature shows that DLI is not tied to one retrieval engine. PyLate exposes disk-backed indexes through index_folder and index_name, implements HNSW and PLAID-based retrieval, and adds post-hoc token pooling compression that cuts the index footprint roughly in half with negligible performance loss (Chaffin et al., 5 Aug 2025). Because PyLate decouples modeling from indexation, the same embedding-level interfaces are intended to support ColBERT-family models and non-text modalities such as ColPali (Chaffin et al., 5 Aug 2025). This suggests that DLI has become a systems pattern: precompute token embeddings, persist them, prune aggressively, and reserve exact late interaction for a manageable candidate set.

The same pattern now appears in multimodal retrieval and RAG. OpenSearch-based page retrieval with ColPali indexes 128-dimensional patch embeddings as separate HNSW items with document id, page id, and patch id metadata, aggregates candidate pages by metadata, reconstructs full page tensors of shape $v$0, and then applies ColBERT-style late interaction before passing the top-ranked pages to an MLLM reader (Saxena et al., 16 Jul 2025). Stellar couples a sparse lexical filter with a balanced-cluster disk layout and a cost-aware loader, explicitly targeting CPU-only retrieval for multimodal pages (Guo et al., 18 Jun 2026). Nemotron ColEmbed V2 emphasizes the opposite extreme: very strong late interaction accuracy with large token sequences and very large embedding dimensions, making disk-backed serving, blockwise MaxSim, and candidate prefiltering operationally necessary rather than optional (Moreira et al., 3 Feb 2026).

Several limitations recur across the literature. Memory footprint remains substantial whenever many token vectors must be retained, even after compression or projection (Khattab et al., 2020, Santhanam et al., 2021, Moreira et al., 3 Feb 2026). I/O bottlenecks often dominate re-ranking latency, especially CPU→GPU transfer of document embeddings or random access over residual streams and document blocks (Khattab et al., 2020, Santhanam et al., 2022). Dynamic updates, incremental clustering, and caching policies are repeatedly identified as open systems problems rather than settled practice (Santhanam et al., 2021, Guo et al., 18 Jun 2026). Some frameworks also leave key engineering details unspecified: PyLate does not detail low-level file formats or OS-level memory mapping, and Nemotron ColEmbed V2 does not disclose a specific retrieval engine for production MaxSim serving (Chaffin et al., 5 Aug 2025, Moreira et al., 3 Feb 2026).

A common misconception is that late interaction necessarily implies full in-memory token storage or exhaustive scoring. The cited systems show otherwise. Candidate generation may be approximate, sparse, centroid-based, or block-aware, but the final stage is often still exact late interaction over a sharply reduced candidate set (Santhanam et al., 2021, Santhanam et al., 2022, Formal et al., 2024, Guo et al., 18 Jun 2026). A second misconception is that DLI is only a text-retrieval technique. The multimodal results with ColPali, Stellar, and Nemotron ColEmbed V2 show that the same architectural principle extends naturally to page patches and visual tokens, although the storage and serving constraints become more severe as token counts and embedding dimensions rise (Saxena et al., 16 Jul 2025, Guo et al., 18 Jun 2026, Moreira et al., 3 Feb 2026).