EigenLI: Spectral Approximations to Late Interaction

Abstract: Late-interaction models such as ColBERT achieve strong effectiveness by representing each document with many token-level vectors, but this expressivity leads to large indexing cost, storage footprints and expensive MaxSim scoring. We show that late-interaction representations exhibit an intrinsic low-rank structure: document token embeddings concentrate in a low-dimensional subspace that preserves most of the retrieval signal. Leveraging this observation, we introduce EigenLI, a spectral approximation framework that compresses late-interaction representations via document-specific low-dimensional subspaces. Unlike clustering or pooling methods, EigenLI identifies the dominant eigendirections of each document and uses them to construct reduced interaction representations. Empirically, $k$-EigenLI with $k \le 32$ outperforms k-means and Ward clustering based pooling methods on ColBERTv2 and AnswerAI-ColBERT-small; GTE-ModernColBERT exhibits a different tradeoff at $k=32$, where clustering methods perform better. The same spectral construction also yields EigenLI-SV, an ANN-compatible single-vector representation derived from the second-order summary of the reduced structure. Across multiple datasets and all three text models, EigenLI-SV consistently outperforms comparable single-vector surrogates such as MUVERA.

• The paper proposes EigenLI, a method that compresses document representations by approximating late-interaction retrieval through spectral properties, preserving retrieval quality without retaining full token sequences

Motivation and central claim

Late-interaction retrievers such as ColBERT represent each document with a sequence of token-level vectors and evaluate relevance through MaxSim. This representation is substantially more expressive than a single embedding, but its computational and systems costs scale with the number of document vectors: ColBERTv2 commonly uses 512 vectors per document, while visual document retrievers may use approximately 1,000–1,250 patch-level vectors. The resulting storage, indexing, and scoring overhead motivates compression methods that preserve token-level retrieval behavior without retaining the full sequence.

“EigenLI: Spectral Approximations to Late Interaction” (2609.07561) argues that document-side late-interaction representations possess exploitable low-rank structure. Rather than compressing a document into a smaller set of token representatives through pruning, clustering, or pooling, EigenLI replaces the document’s token-vector set with a document-specific subspace. The method is training-free and produces both a multi-vector-compatible representation, $k$-EigenLI, and an ANN-compatible single-vector representation, EigenLI-SV.

The empirical premise is that the document token vectors often concentrate in a low-dimensional subspace. Given document vectors $d_1,\ldots,d_m \in \mathbb{R}^d$, the method forms the uncentered second-moment matrix

$M_D = \sum_{i=1}^{m} d_i d_i^T.$

The dominant eigenvectors of $M_D$ identify directions containing most of the document’s token-level embedding mass.

Figure 1: Normalized spectra of document second-moment matrices for long and randomly selected documents across four datasets, illustrating the empirical eigenvalue decay motivating EigenLI.

The figure provides the geometric basis for the method, but it does not establish that spectral energy corresponds directly to retrieval relevance. EigenLI therefore introduces a new interaction function rather than claiming that ordinary MaxSim can be evaluated exactly on the compressed basis vectors.

The $k$-EigenLI representation

Let $w_1,\ldots,w_k$ be the top-$k$ orthonormal eigenvectors of $M_D$. The document is represented by their span, rather than by the vectors $w_j$ as semantic token surrogates. For a query with token vectors $q_1,\ldots,q_n$, EigenLI defines

$d_1,\ldots,d_m \in \mathbb{R}^d$0

Equivalently, if $d_1,\ldots,d_m \in \mathbb{R}^d$1 denotes the orthogonal projector onto the document-specific eigenspace,

$d_1,\ldots,d_m \in \mathbb{R}^d$2

Thus, each query token contributes according to the amount of its squared norm lying in the document’s retained subspace. This is materially different from MaxSim, which assigns each query token its largest signed similarity with any document token. EigenLI measures compatibility with a subspace and discards the eigenvalues from the scoring function.

The omission of eigenvalues is deliberate. Weighting each direction by its corresponding eigenvalue would produce a low-rank approximation to the document second-moment matrix and approximately compute

$d_1,\ldots,d_m \in \mathbb{R}^d$3

which aggregates similarities across all query–document token pairs. The authors report that this behaves poorly relative to MaxSim because it loses MaxSim’s selective, per-query-token matching behavior. The unweighted projector instead treats the retained eigenspace as a set of possible directions and preserves a closer analogue of token-level matching.

The paper establishes several equivalent interpretations of the score. For each query vector, the quantity $d_1,\ldots,d_m \in \mathbb{R}^d$4 is the maximum squared inner product between $d_1,\ldots,d_m \in \mathbb{R}^d$5 and any unit vector in the document subspace. In this sense, EigenLI replaces MaxSim’s maximization over a finite set of document vectors with maximization over a continuous subspace. This construction explains why the method can retain retrieval signal even though it no longer stores representative document tokens.

Relationship to MaxSim

The theoretical discussion connects spectral approximation to the reconstruction error of principal subspaces. Since the document vectors are unit normalized, the trace of $d_1,\ldots,d_m \in \mathbb{R}^d$6 equals the number of document vectors, $d_1,\ldots,d_m \in \mathbb{R}^d$7. The squared reconstruction error after projection onto the top-$d_1,\ldots,d_m \in \mathbb{R}^d$8 eigenspace is exactly the sum of the discarded eigenvalues:

$d_1,\ldots,d_m \in \mathbb{R}^d$9

Consequently, if the top-$M_D = \sum_{i=1}^{m} d_i d_i^T.$0 eigenvalues contain at least a $M_D = \sum_{i=1}^{m} d_i d_i^T.$1 fraction of the total spectral mass, the average squared projection error is at most $M_D = \sum_{i=1}^{m} d_i d_i^T.$2. This result supplies a formal justification for approximating document tokens by the retained subspace, but only in an average reconstruction sense.

The retrieval argument requires a stronger condition. To compare EigenLI with MaxSim on individual query-token matches, the paper assumes that every document token is close to the retained subspace, namely $M_D = \sum_{i=1}^{m} d_i d_i^T.$3 for all $M_D = \sum_{i=1}^{m} d_i d_i^T.$4. Under that assumption, a variant of the subspace score can lower-bound the MaxSim score up to an additive error proportional to the number of query tokens and $M_D = \sum_{i=1}^{m} d_i d_i^T.$5. The average spectral condition alone does not imply this uniform bound: a small number of document vectors may have large projection errors even when the total discarded spectral mass is small.

This qualification is important. The theory explains why concentrated spectra are favorable, but it does not provide a general retrieval-preservation guarantee for the actual squared projector score under only the empirically observed low-rank condition. The empirical results therefore remain central to the paper’s claim.

EigenLI-SV and exact single-vector scoring

A distinctive contribution is the conversion of the EigenLI score into an exact dot product between high-dimensional single-vector encodings. Using the tensor-product identity,

$M_D = \sum_{i=1}^{m} d_i d_i^T.$6

the query and document can each be encoded in a quadratic feature space. Applying the standard degree-two polynomial feature map reduces the dimension from $M_D = \sum_{i=1}^{m} d_i d_i^T.$7 to $M_D = \sum_{i=1}^{m} d_i d_i^T.$8. For ColBERTv2, where $M_D = \sum_{i=1}^{m} d_i d_i^T.$9, this yields an 8,256-dimensional representation, independent of $M_D$0.

The independence from $M_D$1 has two opposing consequences. The single-vector representation can use standard ANN indexes, vector quantization, and highly optimized dot-product kernels, but its raw dimensionality is larger than the $M_D$2-EigenLI multi-vector representation when $M_D$3 is small. At $M_D$4, the multi-vector representation contains $M_D$5 scalar dimensions, whereas EigenLI-SV contains 8,256. Its practical advantage therefore depends on ANN compatibility and lower scoring overhead rather than on raw dimensionality alone.

The paper reports numerical agreement between the direct EigenLI score and the quadratic single-vector dot product within $M_D$6. This makes EigenLI-SV an exact algebraic reformulation of the proposed score, not a separately trained approximation.

Experimental design

The evaluation covers three text late-interaction models on BEIR and one multimodal model on ViDoRe-v3:

• ColBERTv2;
• AnswerAI-ColBERT-small;
• GTE-ModernColBERT;
• ColQwen3 4B for visual document retrieval.

The primary multi-vector baselines are $M_D$7-means++ pooling and Ward hierarchical clustering, evaluated at $M_D$8. Full MaxSim is included as an uncompressed reference, although it is not a strict upper bound because EigenLI uses a different scoring function. The single-vector comparison is against MUVERA fixed-dimensional encodings.

The experiments use brute-force retrieval rather than an end-to-end ANN candidate-generation and reranking pipeline. This isolates representation effectiveness but leaves systems-level latency and index-construction conclusions partially unresolved.

Text retrieval results

For ColBERTv2, EigenLI-32 consistently improves over both clustering baselines in the aggregate BEIR evaluation. Relative improvements are computed per dataset and then averaged, which avoids allowing larger datasets to dominate the summary.

Comparison for ColBERTv2 at $M_D$9	Geometric mean	Arithmetic mean
nDCG@10 over $k$0-means++	$k$1	$k$2
nDCG@10 over Ward	$k$3	$k$4
MRR@10 over $k$5-means++	$k$6	$k$7
MRR@10 over Ward	$k$8	$$+7.6\%</td> </tr> <tr> <td>nDCG@10 relative to full MaxSim</td> <td style="text-align: right">$$k$9	$w_1,\ldots,w_k$0

The final comparison with full MaxSim shows that EigenLI-32 remains slightly below the uncompressed model, as expected from its lower representation budget and altered scoring function. However, it outperforms the two compressed token-pooling baselines despite using only 32 subspace directions. The result implies that compression quality depends not only on retaining token count but also on choosing a representation aligned with the geometry of the embedding set.

AnswerAI-ColBERT-small exhibits the same qualitative pattern. EigenLI-32 improves nDCG@10 over $w_1,\ldots,w_k$1-means++ by $w_1,\ldots,w_k$2 under the arithmetic mean and over Ward by $w_1,\ldots,w_k$3. Its comparison with full MaxSim is weaker than for ColBERTv2, with a $w_1,\ldots,w_k$4 arithmetic-mean decrease in nDCG@10. Thus, spectral compression is not uniformly close to the original model, even when it compares favorably with clustering.

The results are not model invariant. GTE-ModernColBERT reverses the ordering: at $w_1,\ldots,w_k$5, EigenLI trails both clustering baselines on nDCG and MRR. Relative to $w_1,\ldots,w_k$6-means++, EigenLI-32 decreases arithmetic-mean nDCG@10 by $w_1,\ldots,w_k$7; relative to Ward, it decreases it by $w_1,\ldots,w_k$8. The paper attributes this behavior to model-dependent embedding geometry and particularly degenerate low-rank structure. This is a substantive qualification to the central claim: low-rank structure is not by itself sufficient to make the dominant eigenspace the best retrieval-oriented compression.

The choice of $w_1,\ldots,w_k$9 is also non-monotonic. Increasing $k$0 can reduce discriminativeness when the retained subspace approaches the token-vector dimension. The authors observe degradation at $k$1 for ColBERTv2 with $k$2 and at $k$3 for ColQwen3 with $k$4. In addition, some datasets show substantial declines between $k$5 and $k$6 for GTE-ModernColBERT. The representation budget must therefore be selected in a model-aware manner; retaining more spectral directions does not guarantee improved ranking.

Visual document retrieval

On ViDoRe-v3, ColQwen3 4B represents each document with approximately 1,250 vectors of dimension 320, making token-count compression particularly consequential. At $k$7, EigenLI improves over $k$8-means++ by $k$9 in arithmetic-mean nDCG@10 and over Ward by $M_D$04.8\%$M_D$$13.1\%.</p> <p>These gains are smaller than those observed for ColBERTv2, but they are consistent across the eight ViDoRe-v3 datasets. EigenLI remains below full MaxSim: the arithmetic-mean nDCG@10 difference is$$M_D$2. The result supports the method’s applicability beyond text while also indicating that the magnitude of the benefit depends on the model and modality.

Comparison with MUVERA

EigenLI-SV produces the strongest aggregate results reported in the paper. For ColBERTv2 on 13 BEIR datasets, its arithmetic-mean relative improvement over MUVERA is $M_D$3 in nDCG@10, with a geometric-mean improvement of $M_D$4. The corresponding arithmetic-mean improvements for AnswerAI-ColBERT-small and GTE-ModernColBERT are $M_D$5 and $M_D$6, respectively.

Model	nDCG@10 improvement over MUVERA, geometric mean	nDCG@10 improvement over MUVERA, arithmetic mean
ColBERTv2	$M_D$7	$M_D$8
AnswerAI-ColBERT-small	$M_D$9	$w_j$0
GTE-ModernColBERT	$w_j$1	$w_j$2

For ColBERTv2, EigenLI-SV uses 8,256 dimensions versus MUVERA’s 10,240 dimensions. Its recall improvements are also broad: at Recall@10, the arithmetic-mean gain is $w_j$3, and at Recall@1000 it is $w_j$4. On TREC-COVID, for example, Recall@10 rises from $w_j$5 with MUVERA to $w_j$6 with EigenLI-SV; nDCG@10 rises from $w_j$7 to $w_j$8.

These large relative gains should be interpreted with caution because several MUVERA baselines have very low absolute scores. The paper also reports that its MUVERA results on MS MARCO are lower than those previously reported despite using the original implementation. Consequently, the relative improvements demonstrate a substantial empirical gap under the stated implementation and settings, but they do not establish universal superiority over all MUVERA configurations.

The comparison is particularly consequential for anisotropic models. MUVERA relies on SimHash-style random hyperplane projections, whose behavior can deteriorate when token embeddings occupy a narrow cone. Centering the embeddings without renormalization dramatically improves MUVERA. For AnswerAI-ColBERT-small, centered MUVERA Recall@10 increases from $w_j$9 to $q_1,\ldots,q_n$0 in the macro average; for GTE-ModernColBERT, it increases from $q_1,\ldots,q_n$1 to $q_1,\ldots,q_n$2. EigenLI is comparatively stable under centering: its corresponding values change from $q_1,\ldots,q_n$3 to $q_1,\ldots,q_n$4 and from $q_1,\ldots,q_n$5 to $q_1,\ldots,q_n$6, respectively.

The conclusion is not that EigenLI eliminates all geometric sensitivity. Its multi-vector form still underperforms clustering on GTE-ModernColBERT. Rather, the single-vector spectral construction appears less vulnerable than MUVERA to the particular anisotropy failure mode induced by random hyperplane hashing.

Quantization and compression cost

Because EigenLI-SV is an ordinary single vector, it is compatible with standard product quantization. In the reported ColBERTv2 experiments, 1-bit-per-dimension product quantization incurs mean relative losses against full MaxSim of $q_1,\ldots,q_n$7 for Recall@10 and $q_1,\ldots,q_n$8 for nDCG@10 across four datasets. At 8 bits per dimension, the losses decrease to $q_1,\ldots,q_n$9 and $d_1,\ldots,d_m \in \mathbb{R}^d$00, respectively.

The quantization study is limited to four datasets and one model, so it establishes compatibility rather than a comprehensive operating curve. Nevertheless, it indicates that the quadratic encoding can be combined with conventional ANN compression mechanisms without requiring a learned quantizer.

EigenLI also reduces offline compression time relative to clustering. Across 39 model–dataset pairs, $d_1,\ldots,d_m \in \mathbb{R}^d$01-means++ takes $d_1,\ldots,d_m \in \mathbb{R}^d$02 longer than EigenLI under the arithmetic mean of per-pair time ratios, while Ward takes $d_1,\ldots,d_m \in \mathbb{R}^d$03 longer. These measurements favor EigenLI’s spectral construction, but they do not include ANN index construction, query-time candidate generation, or full-MaxSim reranking. They therefore characterize offline representation compression rather than complete retrieval-system cost.

Limitations and open questions

The principal systems limitation is that the experiments evaluate brute-force retrieval. The practically relevant deployment architecture would likely encode documents as EigenLI-SV vectors, use an ANN index for candidate generation, and rerank candidates with full MaxSim or another late-interaction score. The paper does not compare this pipeline with optimized systems such as PLAID, WARP, or Ward-based ANN pipelines in terms of latency, index size, memory, recall, and reranking cost.

EigenLI-SV also scales quadratically with token-vector dimension. Its 8,256 dimensions are manageable for ColBERTv2, but the same construction would require 51,360 dimensions for ColQwen3, which the paper considers prohibitive and therefore does not evaluate for the visual model. Sketching, learned projections, or other dimensionality-reduction mechanisms are needed before the single-vector formulation can be applied broadly to high-dimensional multimodal models.

The method additionally lacks a monotonicity guarantee in $d_1,\ldots,d_m \in \mathbb{R}^d$04. More retained eigendirections can improve reconstruction while worsening retrieval ranking, as observed for Quora, Climate-FEVER, and GTE-ModernColBERT. The open technical problem is to identify which geometric statistics predict whether EigenLI or clustering will be preferable and whether a hybrid method can preserve the efficiency of spectral compression while avoiding its model-specific failures.

Finally, the empirical gap with MUVERA is in tension with worst-case results establishing strong approximation properties for MUVERA with respect to Chamfer-like objectives. The paper leaves open whether EigenLI benefits because actual late-interaction representations are low rank, because retrieval only requires preserving relative ordering among a small candidate set, or because the two methods optimize fundamentally different approximations. Resolving this discrepancy would require a retrieval-oriented theory that combines embedding geometry with ranking rather than uniform pairwise score approximation.

Conclusion

EigenLI proposes a principled alternative to token pruning and clustering for compressing late-interaction document representations. Its central operation is to replace each document’s token-vector set with the dominant eigenspace of its second-moment matrix and to score queries by projected energy in that space. At $d_1,\ldots,d_m \in \mathbb{R}^d$05, the method outperforms $d_1,\ldots,d_m \in \mathbb{R}^d$06-means++ and Ward on ColBERTv2 and on the evaluated visual retrieval model, while reducing compression cost substantially. Its exact quadratic-kernel reformulation, EigenLI-SV, performs markedly better than MUVERA in the reported BEIR experiments and remains compatible with ANN indexing and product quantization.

The results also establish clear boundaries. Performance is model dependent, the optimal $d_1,\ldots,d_m \in \mathbb{R}^d$07 is not monotonic, high-dimensional EigenLI-SV encodings are expensive, and end-to-end ANN evaluation is absent. The paper’s main contribution is therefore both an effective compression method for several late-interaction geometries and an empirical demonstration that document-specific spectral structure can be more retrieval-relevant than generic token pooling.