Generative Late-Interaction Embeddings (GLIE)

• Generative Late-Interaction Embeddings (GLIE) is a storage-efficient retrieval architecture that compresses a document's fine-grained embedding vectors into a learned code, which regenerates vectors when needed for scoring, optimizing storage and retrieval speed.
• GLIE employs spherical centroid anchoring and a learned code for retrieval, followed by generative reconstruction, preserving the document's detailed semantic structure and eliminating the need for permanent vector storage, optimizing retrieval efficiency.
• GLIE has demonstrated high performance, achieving 82% of the uncompressed retrieval quality at a significantly reduced storage size, with applicability across various visual document datasets and encoders but has limitations at very tight budgets (.2-vec/Page)

Generative Late-Interaction Embeddings (GLIE) are a storage-efficient retrieval architecture in which a document’s large set of fine-grained embedding vectors is represented by a much smaller learned code and regenerated when needed for late-interaction scoring. The term denotes a specific method for visual document retrieval introduced in “Generative Late-Interaction Embeddings For Visual Document Retrieval” (Eltahir et al., 10 Sep 2026). GLIE addresses the tension between the retrieval quality of multi-vector MaxSim representations and their storage cost by combining spherical centroid anchoring, a learned compact page code, generative reconstruction, and two-stage retrieval.

1. Conceptual foundations and terminology

Late-interaction retrieval encodes a query and document independently, retaining multiple contextual vectors rather than reducing either input to a single embedding. For a query-token matrix $Q=\{q_1,\ldots,q_m\}$ and document-vector set $X=\{x_1,\ldots,x_N\}$, relevance is computed using MaxSim:

$$\operatorname{MaxSim}(Q,X)=\sum_{j=1}^{m}\max_i \langle q_j,x_i\rangle .$$

Each query vector selects its best-matching document vector. In visual document retrieval, the document vectors may correspond to localized image patches, table cells, captions, diagram labels, or other page regions. This preserves fine-grained evidence that can be lost when a page is represented by one pooled vector.

The principal systems challenge is representation multiplicity. ColPali stores approximately $N=1{,}031$ vectors of dimension $D=128$ per page. In bfloat16, this corresponds to approximately $258$ KB per page, or approximately $258$ GB for one million pages before indexing overhead. Existing approaches reduce this cost through patch selection, local averaging, clustering, quantization, or encoder retraining. These methods either discard potentially relevant evidence, replace extrema with averages, or require modifying and re-encoding the underlying encoder.

GLIE uses a different principle: the stored vectors are a compact description of the page representation, not necessarily the complete evidence set used for final scoring. A small number of vectors is used for corpus-wide candidate generation; a decoder then reconstructs a larger set of vectors for candidate reranking. In this sense, GLIE replaces permanent sampling with on-demand reconstruction.

The method is related to the broader ColBERT family of late-interaction retrievers. ColBERTv2 demonstrates that residual compression, centroid-based inverted lists, and denoised distillation can reduce late-interaction index size by $6$–$10\times$ (Santhanam et al., 2021). Token-pruning analyses show that MaxSim contributions are nonuniform and that query-aware selection can outperform static pooling under tight budgets (Liu et al., 2024, Rishi et al., 7 Sep 2026). GLIE extends these directions by learning a generative representation of the original multi-vector set rather than merely retaining or pooling existing vectors.

2. Geometric basis of GLIE

GLIE is motivated by two empirical properties of visual-document embeddings. First, the evaluated encoders L2-normalize their outputs, so each page vector satisfies

$\|x_i\|_2=1, \qquad X\subseteq \mathbb{S}^{D-1}.$

Second, the vectors on a page concentrate near a low-dimensional manifold despite their high ambient dimension. Across $X=\{x_1,\ldots,x_N\}$0 pages, the TwoNN intrinsic-dimension estimator reports the following median values:

Encoder	Dimension	Median intrinsic dimension
ColPali	128	4.9
ColQwen2	128	5.1
Nemotron v2	3,072	6.1

The per-corpus ColPali medians range from $X=\{x_1,\ldots,x_N\}$1 to $X=\{x_1,\ldots,x_N\}$2. Synthetic controls have substantially larger estimates: $X=\{x_1,\ldots,x_N\}$3 for a Gaussian with the same covariance spectrum and $X=\{x_1,\ldots,x_N\}$4 for uniform noise. The reported interpretation is that roughly one thousand page vectors may contain only five or six important degrees of freedom.

This geometric concentration motivates representing the page by a few learned anchors. The intended representation is not a randomly chosen subset of patches, but a compact description of the manifold from which local evidence can later be regenerated.

Spherical anchoring

Suppose a cluster contains unit vectors $X=\{x_1,\ldots,x_N\}$5 with Euclidean mean

$X=\{x_1,\ldots,x_N\}$6

Unless all vectors coincide, the centroid lies inside the unit sphere:

$X=\{x_1,\ldots,x_N\}$7

The paper gives the identity

$X=\{x_1,\ldots,x_N\}$8

For a unit query vector $X=\{x_1,\ldots,x_N\}$9,

$$\operatorname{MaxSim}(Q,X)=\sum_{j=1}^{m}\max_i \langle q_j,x_i\rangle .$$0

Consequently, raw centroids shrink dot products relative to points on the sphere in the same direction. Since MaxSim takes maxima, this produces systematic underestimation of attainable scores. GLIE applies the normalization

$$\operatorname{MaxSim}(Q,X)=\sum_{j=1}^{m}\max_i \langle q_j,x_i\rangle .$$1

forming a spherical-anchoring representation $$\operatorname{MaxSim}(Q,X)=\sum_{j=1}^{m}\max_i \langle q_j,x_i\rangle .$$2.

On ViDoRe v1, spherical normalization improves nDCG@5 over raw $$\operatorname{MaxSim}(Q,X)=\sum_{j=1}^{m}\max_i \langle q_j,x_i\rangle .$$3-means by between $$\operatorname{MaxSim}(Q,X)=\sum_{j=1}^{m}\max_i \langle q_j,x_i\rangle .$$4 and $$\operatorname{MaxSim}(Q,X)=\sum_{j=1}^{m}\max_i \langle q_j,x_i\rangle .$$5, depending on the budget. At $$\operatorname{MaxSim}(Q,X)=\sum_{j=1}^{m}\max_i \langle q_j,x_i\rangle .$$6, raw $$\operatorname{MaxSim}(Q,X)=\sum_{j=1}^{m}\max_i \langle q_j,x_i\rangle .$$7-means obtains $$\operatorname{MaxSim}(Q,X)=\sum_{j=1}^{m}\max_i \langle q_j,x_i\rangle .$$8, whereas normalized centroids obtain $$\operatorname{MaxSim}(Q,X)=\sum_{j=1}^{m}\max_i \langle q_j,x_i\rangle .$$9. The improvement decreases as $N=1{,}031$0 increases because tighter clusters have centroids whose norms are closer to one.

Normalization does not preserve a lower-bound relationship to the original page. Raw centroids shrink scores, but normalized centroids move outward and can overestimate similarity in some query directions. GLIE therefore combines spherical anchoring with learned ranking, reconstruction, and overshoot objectives.

3. Compact page codes and generative reconstruction

For each page, GLIE clusters the frozen token set $N=1{,}031$1 into $N=1{,}031$2 groups, typically using two $N=1{,}031$3-means restarts, and normalizes the centroids. The evaluated budgets are

$N=1{,}031$4

compared with $N=1{,}031$5 original vectors. At $N=1{,}031$6, the representation uses approximately $N=1{,}031$7 fewer vectors than the uncompressed page.

Learned refinement

Normalized centroids are refined using a shared cross-attention module:

$N=1{,}031$8

Here, $N=1{,}031$9 is the normalized-centroid matrix, $D=128$0 is the frozen page representation, $D=128$1 supplies attention queries, $D=128$2 supplies keys and values, $D=128$3 is multi-head cross-attention, and $D=128$4 is a learned output projection. Every output row is L2-normalized.

The projection $D=128$5 is initialized to zero, so the initial learned code satisfies

$D=128$6

Normalized clustering is therefore the initialization floor for the learned representation. Training can modify the anchors using the original page vectors and retrieval objectives without changing the underlying visual-document encoder.

Generative read-out

A shared decoder expands the compact code $D=128$7 into $D=128$8 normalized vectors:

$D=128$9

Each anchor owns $258$0 output slots, where $258$1 is the number of original vectors assigned to cluster $258$2. This preserves the empirical cluster-count structure of the page. One slot per cluster emits the anchor exactly, giving

$258$3

Because MaxSim uses a maximum,

$258$4

at the per-query-token maximum level.

The remaining slots are generated as bounded movements around their cluster anchor on the sphere. The displacement magnitude is bounded by $258$5, corresponding to a maximum angular displacement of approximately

$258$6

Fixed sine and cosine positional features identify slots, allowing the decoder to handle clusters with different numbers of assigned vectors without learning a separate parameter for every slot.

The decoder is a two-layer MLP with width $258$7, while the refiner is one four-head cross-attention layer. Together they contain approximately $258$8K parameters. This is substantially smaller than the approximately $258$9B-parameter visual-document backbone.

4. Retrieval pipeline and optimization

GLIE uses an asymmetric two-stage retrieval procedure.

First, every page is scored using only its stored compact code:

$258$0

The top $258$1 pages are then decoded. Their compact codes are expanded into $258$2 vectors and rescored:

$258$3

Pages outside the shortlist retain their first-stage ordering. The expensive $258$4-vector interaction is therefore restricted to a small candidate set.

The compact code must retrieve relevant pages into the shortlist, while the decoder must reconstruct sufficient fine-grained evidence for accurate reranking. A page absent from the shortlist cannot be recovered by the decoder.

Training objectives

The encoder remains frozen. The refiner and decoder are trained using cached page embeddings and queries. For a representation containing vectors $258$5, GLIE distills the teacher’s per-query-token MaxSim values:

$258$6

The system also uses a listwise KL-divergence objective so that the ordering of candidate pages follows the frozen full-resolution teacher. For regenerated vectors, additional losses include:

• One-sided overshoot penalty: penalizes regenerated negative pages only when they score above the teacher.
• Cluster-wise Chamfer term: encourages generated children to occupy the same local regions as the original patches.
• Support-function loss: matches the maximum extent of real and regenerated point sets along $258$7 fixed random directions.

Ordinary pointwise reconstruction loss is deliberately omitted. Such a loss would tend to place generated vectors near cluster means, causing collapse toward averages and damaging the extreme points required by MaxSim.

The reported optimization uses AdamW, learning rate $258$8, weight decay $258$9, cosine decay over $6$0 epochs, gradient clipping at $6$1, batches of eight queries, and seven hard negatives per query. The loss weights are $6$2 for regenerated MaxSim and listwise losses, $6$3 for corresponding code losses, $6$4 for Chamfer and overshoot terms, and $6$5 for the support term. The listwise temperature is $6$6, and $6$7 sampled patches per cluster are used for Chamfer training.

Hard negatives are drawn from the fifty highest-scoring non-relevant pages under the frozen compact code. The codec is fitted on $6$8 pages from the public ColPali training collection, although data-efficiency experiments report statistically unchanged results when fitting on $6$9, $10\times$0, or $10\times$1 pages. One budget takes approximately $10\times$2 minutes, or under three GPU-minutes, on one NVIDIA A100 80GB. Three random seeds are used in the reported learned results.

Stored vectors are bfloat16, accompanied by one fp16 norm and one uint16 cluster count per cluster. At $10\times$3, storage is approximately $10\times$4 bytes per page, compared with $10\times$5 KB uncompressed; one million pages therefore require approximately $10\times$6 GB rather than $10\times$7 GB.

5. Empirical performance and comparisons

The principal evaluation uses ViDoRe v1 with all ten subsets, 3,943 queries, unique de-duplicated page corpora, query-less pages retained as distractors, and single-relevant nDCG@5. The main encoder is ColPali v1.3, producing $10\times$8 vectors of dimension $10\times$9 per page. Baselines are applied post hoc to the same frozen embeddings and matched storage budgets.

Method	$\|x_i\|_2=1, \qquad X\subseteq \mathbb{S}^{D-1}.$0	$\|x_i\|_2=1, \qquad X\subseteq \mathbb{S}^{D-1}.$1	$\|x_i\|_2=1, \qquad X\subseteq \mathbb{S}^{D-1}.$2	$\|x_i\|_2=1, \qquad X\subseteq \mathbb{S}^{D-1}.$3	$\|x_i\|_2=1, \qquad X\subseteq \mathbb{S}^{D-1}.$4	$\|x_i\|_2=1, \qquad X\subseteq \mathbb{S}^{D-1}.$5
Raw $\|x_i\|_2=1, \qquad X\subseteq \mathbb{S}^{D-1}.$6-means	.464	.512	.594	.662	.737	.779
Token pooling	.553	.584	.624	.657	.702	.747
Cluster merging	.447	.471	.531	.613	.690	.763
GLIE	.597	.657	.718	.759	.791	.811

The uncompressed reference is $\|x_i\|_2=1, \qquad X\subseteq \mathbb{S}^{D-1}.$7. GLIE retains approximately $\|x_i\|_2=1, \qquad X\subseteq \mathbb{S}^{D-1}.$8, $\|x_i\|_2=1, \qquad X\subseteq \mathbb{S}^{D-1}.$9, $X=\{x_1,\ldots,x_N\}$00, $X=\{x_1,\ldots,x_N\}$01, $X=\{x_1,\ldots,x_N\}$02, and $X=\{x_1,\ldots,x_N\}$03 of uncompressed nDCG@5 at $X=\{x_1,\ldots,x_N\}$04, $X=\{x_1,\ldots,x_N\}$05, $X=\{x_1,\ldots,x_N\}$06, $X=\{x_1,\ldots,x_N\}$07, $X=\{x_1,\ldots,x_N\}$08, and $X=\{x_1,\ldots,x_N\}$09, respectively. At four vectors per page, it retains nearly $X=\{x_1,\ldots,x_N\}$10 of uncompressed quality, whereas the strongest prior post-hoc method retains approximately $X=\{x_1,\ldots,x_N\}$11.

On ViDoRe v2, GLIE remains the strongest reported method among the evaluated alternatives:

$X=\{x_1,\ldots,x_N\}$12

for $X=\{x_1,\ldots,x_N\}$13. The uncompressed value is $X=\{x_1,\ldots,x_N\}$14, corresponding to $X=\{x_1,\ldots,x_N\}$15, $X=\{x_1,\ldots,x_N\}$16, $X=\{x_1,\ldots,x_N\}$17, $X=\{x_1,\ldots,x_N\}$18, $X=\{x_1,\ldots,x_N\}$19, and $X=\{x_1,\ldots,x_N\}$20 retention. On ColQwen2, the uncompressed value is $X=\{x_1,\ldots,x_N\}$21, and GLIE reaches $X=\{x_1,\ldots,x_N\}$22 at $X=\{x_1,\ldots,x_N\}$23, or $X=\{x_1,\ldots,x_N\}$24 retention. The paper reports improvements on all ten ColQwen2 subsets at $X=\{x_1,\ldots,x_N\}$25 and above.

Component contributions

The component ladder on ViDoRe v1 is:

Component	$X=\{x_1,\ldots,x_N\}$26	$X=\{x_1,\ldots,x_N\}$27	$X=\{x_1,\ldots,x_N\}$28	$X=\{x_1,\ldots,x_N\}$29	$X=\{x_1,\ldots,x_N\}$30	$X=\{x_1,\ldots,x_N\}$31
Raw $X=\{x_1,\ldots,x_N\}$32-means	.464	.512	.594	.662	.737	.779
+ spherical anchoring	.552	.605	.684	.736	.784	.809
+ learned code	.596	.641	.705	.752	.785	.809
+ generative read-out	.597	.657	.718	.759	.791	.811

Spherical normalization is the largest and cheapest contribution. The learned code is most useful at low budgets, adding $X=\{x_1,\ldots,x_N\}$33 at $X=\{x_1,\ldots,x_N\}$34 and $X=\{x_1,\ldots,x_N\}$35 at $X=\{x_1,\ldots,x_N\}$36. The decoder contributes most at aggressive budgets, adding approximately $X=\{x_1,\ldots,x_N\}$37 at $X=\{x_1,\ldots,x_N\}$38 and $X=\{x_1,\ldots,x_N\}$39 at $X=\{x_1,\ldots,x_N\}$40.

At $X=\{x_1,\ldots,x_N\}$41, increasing the shortlist from $X=\{x_1,\ldots,x_N\}$42 to $X=\{x_1,\ldots,x_N\}$43 raises the oracle from $X=\{x_1,\ldots,x_N\}$44 to $X=\{x_1,\ldots,x_N\}$45, while actual GLIE rises only from $X=\{x_1,\ldots,x_N\}$46 to $X=\{x_1,\ldots,x_N\}$47L=20$X=\{x_1,\ldots,x_N\}$480.657$X=\{x_1,\ldots,x_N\}$490.782. The reported bottleneck is therefore decoder fidelity rather than shortlist recall.

Decoder size is comparatively unimportant within the tested range. Configurations from $X=\{x_1,\ldots,x_N\}$50K to $X=\{x_1,\ldots,x_N\}$51M parameters change nDCG@5 by at most $X=\{x_1,\ldots,x_N\}$52, without a monotonic trend. The $X=\{x_1,\ldots,x_N\}$53K decoder is consequently a practical point in a relatively flat capacity region.

6. Relation to neighboring architectures

GLIE differs from several approaches that address multi-vector efficiency.

Static compression and pruning

Static pooling, token selection, and clustering decide which evidence remains available before the query is known. Query-aware token budgeting instead reopens original vectors for shortlisted pages and selects a query-specific subset (Rishi et al., 7 Sep 2026). GLIE goes further by generating new vectors from a compact code. It does not merely select existing patches; it reconstructs an expanded representation.

This distinction matters for MaxSim. A selected subset cannot recover evidence that was discarded. A decoder can synthesize vectors near missing local extrema, subject to the information contained in the compact code and the quality of the learned decoder.

Meta-token and compact multi-vector models

MetaEmbed uses a fixed number of learned Meta Tokens whose contextualized representations serve as compact multi-vector embeddings, with Matryoshka Multi-Vector Retrieval enabling test-time selection of nested budgets (Xiao et al., 22 Sep 2025). Unlike GLIE, MetaEmbed retrains the multimodal encoder and produces a compact representation directly. GLIE keeps the original encoder frozen, fits a small codec to its output geometry, and reconstructs the original-scale representation only for shortlisted pages.

Learnable late interaction

LITE replaces MaxSim with a learned function over the entire token-similarity matrix and studies reductions in document-token count and embedding dimension (Ji et al., 2024). GLIE retains the MaxSim operator but changes the representation supplied to it: a compact code is used for first-stage retrieval, and a decoded multi-vector set is used for reranking.

Query-conditioned document representations

Argus generates query-conditioned document embeddings through region-aware sparse mixture-of-experts routing before applying MaxSim (Abdallah et al., 3 Jun 2026). Its document representation is $X=\{x_1,\ldots,x_N\}$54 and therefore cannot generally be precomputed once for every future query. GLIE retains query-independent compact page codes and performs query-independent decoding in the described architecture; the query affects selection through retrieval and reranking rather than directly conditioning the decoder’s page representation.

Generative recoverability

The Generative Embedding Benchmark evaluates whether a frozen embedding retains information accessible to a downstream decoder (Li et al., 7 Aug 2026). It shows that discriminative embedding quality does not establish generative recoverability, particularly for scene text and visual documents. GLIE is relevant to this distinction because its decoder is explicitly trained to recover MaxSim-relevant evidence, but the reported GLIE evaluation measures retrieval rather than answer generation.

7. Limitations and future directions

GLIE changes, rather than eliminates, the cost structure of late-interaction retrieval. Its principal limitations are as follows.

• Compact-code bottleneck: if the relevant page structure cannot be inferred from a small number of anchors, reconstruction cannot recover it. This is especially apparent at $X=\{x_1,\ldots,x_N\}$55, where regeneration adds no improvement on ViDoRe v1 and only $X=\{x_1,\ldots,x_N\}$56 on ViDoRe v2.
• Shortlist dependence: pages not retrieved by compact-code MaxSim cannot be recovered during decoding. Increasing the shortlist improves the available oracle but increases decoding and reranking cost.
• Decoder fidelity: the gap between actual GLIE and the shortlist oracle shows that the decoder remains the principal source of headroom at small budgets.
• Potential hallucination or overshoot: generated vectors need not equal the original patches and may score above the full-resolution teacher. Overshoot, support, and Chamfer losses constrain but do not mathematically eliminate this behavior.
• Encoder dependence: the low-dimensional spherical geometry was observed for ColPali, ColQwen2, and Nemotron v2. It is not established that all future visual encoders will exhibit comparable intrinsic dimensions or that the same decoder will transfer across encoders.
• Storage accounting: the approximately $X=\{x_1,\ldots,x_N\}$57 GB figure for one million pages at $X=\{x_1,\ldots,x_N\}$58 applies to the compact code and associated metadata. It does not include the original full-token store if that store is retained for other purposes, nor indexing overhead.
• Frozen-encoder scope: GLIE does not adapt the visual-document encoder. This avoids corpus re-encoding and makes training inexpensive, but it limits the codec to information already present in the frozen representation.
• Evaluation scope: the reported retrieval results use ViDoRe v1 and v2, with a second-encoder evaluation on ColQwen2. Broader domains, languages, document types, and generative readout tasks remain open.

Future research directions include learned or query-conditioned decoders, larger and more diverse encoder families, improved shortlist generation, adaptive candidate budgets, approximate decoding, and integration with query-aware token allocation. A natural hybrid would combine a compact GLIE code for corpus-wide retrieval with query-aware selection or decoding for shortlisted pages. Another direction is to evaluate GLIE under generative readout protocols such as GEB, using multi-token decoder interfaces and matched byte, dimension, and computation budgets.

GLIE’s central contribution is the separation of stored representation, candidate-generation representation, and final evidence representation. A page can be indexed using a few spherical anchors, searched with compact MaxSim, and expanded into a richer multi-vector set only when its relevance warrants the additional computation. The resulting retrieval system preserves much of the fine-grained behavior of late interaction while shifting the primary compression problem from selecting which evidence to retain permanently toward learning how to regenerate evidence on demand.