Generative Late-Interaction Embeddings For Visual Document Retrieval

Abstract: Late-interaction retrieval is the state-of-the-art for visual document search, but it pays for its accuracy in storage. Existing compression methods retain a subset or local average of the N~1,000 vectors per page. Under aggressive storage budgets, however, these methods degrade sharply, and alternatives require retraining the encoder. Investigating this degradation across three encoders, we found two consistent properties: the vectors lie exactly on the unit sphere and concentrate near a manifold of intrinsic dimension five to six. This geometry yields two insights. First, standard k-means centroids fall inside the sphere, causing systematic underestimation of MaxSim scores. Normalizing them to the surface is a free correction worth up to +0.093 nDCG@5 over raw centroids. Second, because the page manifold has few degrees of freedom, the full set of vectors can be regenerated from only a few. To this end, we introduce Generative Late-Interaction Embeddings (GLIE): k << N vectors per page learned from the normalized centroids to serve as both a lightweight index and a basis for regenerating the page's full embedding set. At query time, search runs exclusively on these k vectors, and a decoder expands only the top candidates back to all N vectors for exact rescoring. At four vectors per page on ViDoRe v1, GLIE retains nearly 80% of the uncompressed system's nDCG@5, against 70% for the best prior post-hoc method. These results use a 415K-parameter network fitted in under three GPU-minutes on just a thousand training pages. At a matched training budget, fine-tuning the encoder does not reach even the training-free stage of GLIE, and the full system beats it at every budget. These patterns hold across a second encoder and ViDoRe v2. By reconstructing evidence on demand rather than sampling it, GLIE opens a new axis for storage-efficient retrieval, with the decoder as its main design surface.

• The paper introduces a generative codec, GLIE, that compresses visual document embeddings for efficient retrieval by exploiting geometric properties of token embeddings
• The findings show that GLIE can maintain 91% of the retrieval quality nDCG@5 of the uncompressed version at only 16 vectors per page, reducing the storage requirement from 257.8 KB to 1.7 KB
• The generative read-out used by GLIE aims to preserve the essential token evidence while effectively exploiting data regularities, making it a promising approach for visual document retrieval in resource-limited environments.

Problem formulation and central thesis

“Generative Late-Interaction Embeddings For Visual Document Retrieval” (2609.11808) addresses the storage cost of late-interaction retrieval for visual documents. ColPali-style systems represent each page with approximately 1,031 patch-level vectors and evaluate a query–page pair using MaxSim: each query-token embedding selects its best matching page vector, and the resulting maxima are summed. This representation preserves localized evidence, including table cells, captions, and text fragments, but produces a substantial index footprint. In the paper’s ColPali configuration, the uncompressed representation requires approximately 257.8 KB per page in bfloat16, or about 258 GB for one million pages before additional index structures.

Existing compression methods generally reduce the stored set through pooling, clustering, pruning, merging, or quantization. These approaches either retain a subset of encoder vectors or replace local groups with averages. The paper argues that this extractive premise becomes inadequate at highly aggressive budgets, particularly below approximately 16 vectors per page. Alternative methods that operate at smaller budgets typically modify or retrain the encoder, requiring corpus re-encoding and making them unsuitable for already-indexed collections.

GLIE instead treats the page embedding as a low-dimensional geometric object that can be encoded and regenerated. It stores $k$ vectors per page, with $k \in \{2,4,8,16,32,64\}$, and uses a shared decoder to reconstruct a full-length representation only for candidates that survive first-stage retrieval. The method therefore separates inexpensive global candidate generation from expensive late-interaction rescoring.

The paper’s principal empirical claim is that page-level token embeddings exhibit two useful properties: they are exactly unit-normalized and have an intrinsic dimension of approximately five to six, despite ambient dimensions of 128 or 3,072. The proposed codec exploits both facts through spherical anchoring, learned code refinement, and generative read-out.

Geometric characterization of visual document embeddings

The geometric analysis is conducted over 6,729 pages from the evaluation corpora and three visual late-interaction encoders. For ColPali, whose token vectors have dimension 128, the median TwoNN intrinsic-dimension estimate is 4.9, with per-corpus medians ranging from 4.7 to 5.1. ColQwen2 yields a median of 5.1. Nemotron v2 operates in a 3,072-dimensional space but produces a median intrinsic dimension of 6.1. Thus, the ambient dimension changes by a factor of 24 across the encoders while the estimated intrinsic dimension changes by only approximately one dimension.

The authors use two controls to argue that this low-dimensionality estimate is not merely an artifact of the estimator or of covariance structure. A Gaussian fitted to each page’s covariance produces an estimated dimension of 32.2, while same-size uniform noise produces 61.4. The comparison suggests that the observed page token clouds are not adequately described by a linear Gaussian approximation and instead have concentrated, curved structure.

All three encoders L2-normalize their outputs. Consequently, every page token lies on the unit sphere $\mathbb{S}^{D-1}$. This observation has a direct consequence for standard $k$-means. The Euclidean centroid of a cluster of unit vectors generally lies inside the sphere, with its norm decreasing as the cluster becomes more dispersed. Since MaxSim uses dot products, an interior centroid systematically reduces inner products relative to unit-norm vectors in the same direction.

The paper formalizes this relationship through the identity

$$\frac{1}{n}\sum_i \lVert x_i-c\rVert^2 = 1-\lVert c\rVert^2,$$

for unit vectors $x_i$ with mean $c$. Cluster dispersion is therefore exactly encoded by the centroid norm. Normalizing each centroid back to the sphere is a cost-free correction to a geometric mismatch between the compression procedure and the encoder output space.

This correction has a substantial effect. Across the ViDoRe v1 benchmark, spherical anchoring improves nDCG@5 by 0.031–0.093 depending on the budget, with the largest improvement at small $k$. At $k=4$, normalized $k$-means reaches 0.605, compared with 0.512 for raw $k \in \{2,4,8,16,32,64\}$0-means. The improvement declines as $k \in \{2,4,8,16,32,64\}$1 increases, consistent with tighter clusters producing centroid norms closer to one.

The correction is not uniformly conservative. An unnormalized centroid can only underestimate the maximum similarity associated with its cluster in the relevant direction, whereas a normalized centroid may move beyond the original data and overestimate it. This distinction motivates the later use of one-sided overshoot penalties during decoder training.

GLIE architecture

GLIE consists of a per-page code, a shared refiner, and a shared generative decoder. The encoder remains frozen, and all learning operates on cached page embeddings. The complete refiner–decoder system contains 415K parameters, compared with the 3B-parameter backbone.

Spherical initialization and learned refinement

For each page, the method computes $k \in \{2,4,8,16,32,64\}$2-means centroids over the $k \in \{2,4,8,16,32,64\}$3 encoder vectors and normalizes them to obtain spherical anchors. These anchors are then refined using a shared cross-attention module. The anchors act as queries, while the full page token set supplies keys and values. A zero-initialized output projection ensures that the refiner initially returns the normalized centroids exactly:

$k \in \{2,4,8,16,32,64\}$4

where $k \in \{2,4,8,16,32,64\}$5 is initialized to zero.

This initialization is important methodologically. The learned code does not begin from an arbitrary representation that may be worse than a strong training-free baseline. It begins at normalized clustering and is optimized to encode MaxSim-relevant structure that the cluster means discard. The refinement module can therefore exploit the complete page token set while storing only $k \in \{2,4,8,16,32,64\}$6 vectors at inference time.

The paper’s ablation isolates the contribution of this learned code. Relative to normalized clustering, the learned code adds between 0.016 and 0.044 nDCG@5 through $k \in \{2,4,8,16,32,64\}$7, but provides essentially no additional gain at larger budgets on ViDoRe v1. This saturation is consistent with the interpretation that once the stored code has enough vectors to approximate the relevant support structure, further refinement has limited room to improve.

Anchored generative read-out

The decoder expands the $k \in \{2,4,8,16,32,64\}$8 stored vectors into $k \in \{2,4,8,16,32,64\}$9 unit vectors. It uses cluster counts to allocate output slots, preserving the empirical size of each cluster. The first slot associated with each cluster emits the refined anchor exactly. Hence, the regenerated set contains the stored code as a subset, which gives the pointwise guarantee

$\mathbb{S}^{D-1}$0

Regeneration can add candidate evidence but cannot remove the evidence already represented by the code.

The remaining child vectors are generated as bounded surface displacements around their cluster anchor. The maximum displacement is controlled by $\mathbb{S}^{D-1}$1, corresponding to an angular bound of approximately $\mathbb{S}^{D-1}$2. Positional sine and cosine features distinguish slots within a cluster without introducing cluster-specific parameters. This design imposes a localized decoder geometry: regenerated vectors remain within a bounded neighborhood of their anchors rather than being unconstrained points on the sphere.

The training objectives reflect the fact that the decoder is not intended to minimize ordinary reconstruction error. Pixel-independent vector reconstruction would encourage children to collapse toward cluster means, whereas MaxSim depends on directional extremes. GLIE therefore combines token-level MaxSim distillation, listwise ranking distillation, negative overshoot penalties, within-cluster Chamfer losses, and support-function matching over random directions. The support term explicitly targets the directional extent of the page token set, which is more relevant to MaxSim than pointwise Euclidean fidelity.

Figure 1: GLIE training freezes the encoder, initializes the projected code at normalized $\mathbb{S}^{D-1}$3-means centroids, and jointly optimizes the code and regenerated vectors with retrieval- and geometry-based losses.

Two-stage retrieval procedure

At inference time, GLIE uses an asymmetric cascade. First, every page is scored using MaxSim over its $\mathbb{S}^{D-1}$4 stored vectors. This stage operates exclusively on the compressed index. The top $\mathbb{S}^{D-1}$5 pages are then decoded to $\mathbb{S}^{D-1}$6 vectors and rescored using full MaxSim. Pages outside the shortlist retain their first-stage ordering.

Figure 2: Every page is ranked using its stored projected vectors, while only the top-$\mathbb{S}^{D-1}$7 candidates are decoded and rescored at full length.

The design addresses two distinct computational requirements. The first-stage code must preserve global page ranking over the entire corpus. The decoder need not be applied universally; it only needs to recover enough fine-grained evidence for a small candidate set. This separation allows GLIE to retain a generative representation without imposing full decoder cost on every indexed page.

At $\mathbb{S}^{D-1}$8, the storage footprint is approximately 1,040 bytes per page, compared with 257.8 KB for the uncompressed ColPali representation. For one million pages, the paper reports a reduction from approximately 258 GB to 1.0 GB. The method is also budget-elastic after indexing: changing $\mathbb{S}^{D-1}$9 affects the cached compressed representation rather than the encoder or the corpus embeddings.

Retrieval results

The main experiments use all ten ViDoRe v1 subsets, totaling 3,943 queries, and four ViDoRe v2 subsets. The codec is fitted on 5,000 pages from the public ColPali training collection and applied without access to test pages or queries. Learned results are averaged over three seeds, with reported seed standard errors no greater than 0.002 per cell.

ViDoRe v1

GLIE improves substantially over raw $k$0-means, token pooling, and cluster merging at aggressive budgets. The macro-averaged ViDoRe v1 results are:

Stored vectors per page	Raw $k$1-means	Token pooling	GLIE	Fraction of uncompressed quality
2	0.464	0.553	0.597	71%
4	0.512	0.584	0.657	79%
8	0.594	0.624	0.718	86%
16	0.662	0.657	0.759	91%
32	0.737	0.702	0.791	95%
64	0.779	0.747	0.811	97%
Uncompressed	—	—	0.836	100%

At four vectors per page, GLIE retains 79% of the uncompressed nDCG@5 while storing roughly 1/258 of the embedding bytes. At 16 vectors, it retains 91%. The strongest training-free baseline varies by subset and budget, but GLIE is reported to outperform every prior training-free baseline on every ViDoRe v1 and v2 subset at every evaluated budget.

The largest per-subset gains occur in difficult settings. On TAT-DQA, which contains 1,663 queries, GLIE’s margins over the strongest training-free baseline range from +0.039 to +0.054 for $k$2. This implies that the method is not benefiting only from high-ceiling subsets where modest compression preserves an already easy ranking; it also improves retrieval where the underlying task is more demanding.

Figure 3: GLIE’s margin over the strongest training-free baseline is largest at aggressive budgets and varies with benchmark saturation and available retrieval headroom.

The margin profile is nevertheless budget-dependent. On ViDoRe v1, the improvement is approximately 0.04 for $k$3, falls to about 0.02 at $k$4, and approaches noise by $k$5. At $k$6, the stored code alone becomes slightly worse than the strongest baseline, although the normalized-clustering baseline is already only 0.027 below the uncompressed ceiling. Four ViDoRe v1 subsets have ceilings between 0.94 and 0.98, so the high-budget decline partly reflects metric saturation rather than a general failure of the learned code.

ViDoRe v2 exhibits a different pattern. GLIE obtains nDCG@5 values of 0.267, 0.330, 0.388, 0.439, 0.472, and 0.490 for $k$7, respectively, against an uncompressed score of 0.517. These correspond to 52%, 64%, 75%, 85%, 91%, and 95% of uncompressed quality. Because v2 does not saturate in the same way as v1, the margin and the contribution of the decoder remain visible through $k$8.

Decoder and shortlist contributions

The ablation decomposition attributes the principal gains in ViDoRe v1 to spherical anchoring, not to the neural components. At $k$9, the progression is:

System component	nDCG@5
Raw $$\frac{1}{n}\sum_i \lVert x_i-c\rVert^2 = 1-\lVert c\rVert^2,$$0-means	0.512
Spherical anchoring	0.605
Learned code	0.641
Generative read-out	0.657

Thus, normalization contributes +0.093, learned refinement contributes a further +0.036, and generative read-out contributes +0.016. At $$\frac{1}{n}\sum_i \lVert x_i-c\rVert^2 = 1-\lVert c\rVert^2,$$1, the corresponding increments are +0.030, 0, and +0.002. This decomposition supports the paper’s more restrained interpretation: the neural codec is most valuable at aggressive budgets, while the simple spherical correction remains broadly useful.

The shortlist analysis distinguishes retrieval failure from decoding failure. At $$\frac{1}{n}\sum_i \lVert x_i-c\rVert^2 = 1-\lVert c\rVert^2,$$2, GLIE reaches 0.657, while an oracle that assumes perfect ranking within the selected shortlist reaches 0.782. The uncompressed ceiling is 0.836. Therefore, the observed residual gap decomposes into approximately 0.125 nDCG points attributable to decode fidelity and approximately 0.054 attributable to shortlist recall.

Figure 4: At aggressive budgets, the remaining gap to the uncompressed ceiling is dominated by decoder fidelity, while shortlist recall contributes a smaller but distinct component.

Increasing the shortlist from 5 to 100 raises the oracle from 0.705 to 0.822, but improves actual GLIE only from 0.647 to 0.660. Increasing $$\frac{1}{n}\sum_i \lVert x_i-c\rVert^2 = 1-\lVert c\rVert^2,$$3 from 20 to 100 adds just 0.002 to the actual system, despite a much larger oracle improvement. The implication is specific and important: the relevant candidates are generally entering the shortlist, but the current decoder fails to reconstruct enough of their MaxSim-relevant evidence. Decoder quality, rather than shortlist size, is the binding limitation under this configuration.

Transfer to ColQwen2

The same procedure transfers to ColQwen2, whose intrinsic dimension is estimated at 5.1. GLIE retains 82% of the uncompressed quality at $$\frac{1}{n}\sum_i \lVert x_i-c\rVert^2 = 1-\lVert c\rVert^2,$$4, with macro nDCG@5 of 0.727 against an uncompressed score of 0.883. It improves all ten subsets for $$\frac{1}{n}\sum_i \lVert x_i-c\rVert^2 = 1-\lVert c\rVert^2,$$5 and seven of ten at $$\frac{1}{n}\sum_i \lVert x_i-c\rVert^2 = 1-\lVert c\rVert^2,$$6.

The gains are smaller in absolute margin than on ColPali because ColQwen2’s stronger encoder and baseline clustering already approach the ceiling. This result is consistent with the paper’s claim that GLIE closes a representation-dependent compression gap rather than providing a fixed additive improvement. The available improvement depends on encoder quality, benchmark saturation, and the amount of evidence that can be represented by the selected budget.

Training efficiency and comparison with encoder fine-tuning

The paper makes a deliberately constrained comparison with Light-ColPali-style fine-tuning. Under a matched small training budget, a LoRA-adapted encoder with 13.3M trainable parameters is trained for approximately 1.5 GPU-hours on 4,000 query–page pairs. GLIE instead trains 415K parameters for under three GPU-minutes and leaves the backbone frozen.

The fine-tuned system underperforms even free normalized $$\frac{1}{n}\sum_i \lVert x_i-c\rVert^2 = 1-\lVert c\rVert^2,$$7-means at every evaluated budget:

Method	$$\frac{1}{n}\sum_i \lVert x_i-c\rVert^2 = 1-\lVert c\rVert^2,$$8	$$\frac{1}{n}\sum_i \lVert x_i-c\rVert^2 = 1-\lVert c\rVert^2,$$9	$x_i$0	$x_i$1	$x_i$2	$x_i$3
LoRA fine-tuning	0.523	0.544	0.586	0.632	0.661	0.701
Normalized $x_i$4-means	0.552	0.605	0.684	0.736	0.784	0.809
GLIE	0.597	0.657	0.718	0.759	0.791	0.811

GLIE’s advantage over the matched fine-tuning reproduction ranges from +0.074 to +0.132 nDCG@5. The result should not be interpreted as evidence that Light-ColPali is ineffective under its intended training regime. The paper explicitly notes that the published method uses approximately 130K queries and 72 GPU-hours per budget, whereas the reproduction uses only 4,000 query–page pairs and 1.5 GPU-hours. The comparison instead establishes that, under a constrained adaptation budget, directly optimizing a small post-hoc codec is more effective than weakly perturbing a multi-billion-parameter encoder.

The codec also shows strong data efficiency. Fitting on 1,250, 2,500, or 5,000 source pages yields margins of approximately +0.048, +0.052, and +0.049 at $x_i$5. Within this range, additional fitting pages do not produce a measurable improvement. The implication is that GLIE is exploiting regularities in the frozen encoder’s geometry rather than learning a corpus-specific document distribution.

Decoder capacity is similarly noncritical within the tested range. Models from 184K to 13M parameters differ by at most 0.009 nDCG@5, with no monotonic relationship between capacity and performance. This finding strengthens the paper’s claim that the principal bottleneck is the decoder’s inductive structure and training objective, not simply parameter count.

Limitations and open questions

The empirical scope is substantial but remains concentrated on ViDoRe v1 and v2 and two principal encoders, with the geometric analysis additionally including Nemotron v2. The claim that visual document token clouds generally have intrinsic dimension five to six therefore remains an empirical hypothesis about the evaluated encoder families, not a universal property of late-interaction embeddings. The paper itself identifies extension to additional encoders as an unresolved test.

The TwoNN estimates also summarize page-level geometry and do not establish that a single global low-dimensional coordinate system exists across pages. GLIE avoids requiring such a coordinate system by using per-page anchors and a shared decoder, but the relationship between intrinsic dimension, required budget, and retrieval difficulty is not fully characterized.

The decoder’s guarantee is limited. Including the refined anchors ensures that regenerated MaxSim cannot be lower than code-only MaxSim, but it does not prevent generated vectors from introducing false evidence. The method therefore requires one-sided negative overshoot penalties, and residual decoder errors remain the largest measured component of the gap to the oracle. At $x_i$6, perfect decoding of the existing shortlist would raise performance from 0.657 to 0.782, so the current results do not establish that the representational budget itself is the dominant limit.

The evaluation also does not report a complete end-to-end latency and memory analysis incorporating ANN indexing, decoder throughput, query batching, concurrent serving, and hardware-specific costs. The reported storage reduction is clear, but the operational trade-off between scoring all pages with $x_i$7 vectors and decoding the top 20 candidates requires system-level measurement.

Finally, GLIE is post hoc with respect to the frozen embedding model. If the encoder changes, the cached page embeddings and the fitted codec may no longer be valid. Conversely, quantization, dimensionality reduction, and candidate-generation methods operate on different axes and may alter the geometry on which GLIE depends. Their compatibility is proposed but not demonstrated in the reported experiments.

Conclusion

GLIE reframes visual late-interaction compression as a generative coding problem rather than subset selection. Its method rests on two measured properties of page embeddings: exact unit normalization and low intrinsic dimensionality. Spherical centroid normalization alone yields a large, training-free correction, while a zero-initialized refiner and anchored decoder recover additional MaxSim-relevant structure at very small budgets.

On ViDoRe v1, GLIE retains 79% of uncompressed nDCG@5 with four stored vectors per page and 91% with 16 vectors. It reduces the reported storage requirement from 257.8 KB to approximately 1.0 KB per page at $x_i$8, transfers to ViDoRe v2 and ColQwen2, and can be fitted in under three GPU-minutes on roughly one thousand pages. The experiments also identify the main remaining limitation: current decoding fails to exploit much of the evidence already captured by the shortlist. Consequently, the paper’s strongest technical conclusion is not merely that a small code can replace a large stored set, but that generative read-out introduces a distinct compression axis whose effectiveness is presently limited more by decoder fidelity than by candidate recall.