Light-ColPali/ColQwen2 VDR Models

• Light-ColPali/ColQwen2 is a storage-efficient visual document retrieval model that reduces numerous patch embeddings through principled token merging strategies.
• It overcomes scalability issues by aggregating semantically similar patches into compact super-tokens, balancing accuracy with reduced memory usage.
• Empirical results demonstrate that fine-tuning merged representations retains up to 98.2% of baseline performance while significantly lowering storage requirements.

Light-ColPali/ColQwen2 refers to a class of storage-efficient visual document retrieval (VDR) models based on the ColPali/ColQwen2 architecture, which are designed for high-performance retrieval from visually rich documents while significantly reducing memory footprint. These models address the primary scalability limitation of earlier VLM-based retrievers—the excessive storage required for fine-grained patch-level embeddings—by introducing principled token (i.e., patch embedding) reduction strategies that preserve retrieval effectiveness at a fraction of the memory cost.

1. Background and Motivation

ColPali/ColQwen2 are VLM-based retrievers that use multi-vector (“late interaction”) representations analogous to ColBERT for text, but applied to images, especially document pages. Each page is encoded into a set of patch-level embeddings, typically producing hundreds of 128-dimensional vectors per page. This granularity enables state-of-the-art retrieval performance on visually complex documents but also multiplies the storage and compute cost relative to single-vector (dense) approaches. With real-world document corpora, the memory requirement quickly becomes the bottleneck for practical deployment.

The goal of Light-ColPali/ColQwen2 is to develop and empirically validate strategies for reducing the number of stored patch embeddings per page as much as possible, without access to queries at indexing time, while minimizing performance degradation. The result is a scalable retrieval paradigm balancing accuracy, latency, and storage cost.

2. Token Reduction Methodologies

Two main token reduction strategies are investigated: pruning and merging. Each has distinct algorithmic, mathematical, and practical implications.

2.1 Token Pruning

Token pruning reduces storage by selecting a subset $N_p' < N_p$ of the original patch embeddings to retain per page. Methods include:

• Random pruning: Discard a random subset of patch embeddings.
• Score-oriented pruning: Rank patches using a proxy “response potential” estimated via synthesized queries:

$$r_p^i(q) = \max_j {\mathbf{e}_p^i}^\top \mathbf{e}_q^j$$

Where $\mathbf{e}_p^i$ is patch $i$’s embedding, $\mathbf{e}_q^j$ is a query token embedding, and the maximum is taken over synthetic queries in lieu of true unknown queries at index time.

• Attention-oriented pruning: Retain patches that attract most cross-token attention in the final transformer layer during model processing.

Empirical findings demonstrate that all pruning schemes, including sophisticated score- and attention-guided variants, are fundamentally limited:

• Random pruning unexpectedly outperforms all heuristic methods in preserving retrieval at moderate compression rates.
• No approach supports aggressive reduction (order-of-magnitude, i.e., ≥90% patches pruned) without severe performance loss.
• The key limitation arises from high query-specific variability—pruning offline easily discards all relevant patches for particular queries.

2.2 Token Merging

Token merging compresses patch embeddings by aggregating groups of patches into single “super-tokens.” Unlike pruning, this preserves information by combining instead of discarding. The merging process is formulated as:

• Original embeddings: $$E_p = [e_p^1, \ldots, e_p^{N_p}] \in \mathbb{R}^{N_p \times d}$$
• After reduction: $E_p' \in \mathbb{R}^{N_p' \times d}$ with $N_p' \ll N_p$
• For cluster $C_k$, merged embedding:

$\bar{e}_p^k = \frac{1}{|C_k|}\sum_{i\in C_k} e_p^i$

Merging strategies considered:

• 1D/2D spatial pooling: Average pooling over sequential patches or spatial image neighborhoods.
• Semantic clustering: Hierarchical clustering by embedding similarity, grouping patches that are nearest neighbors in feature space.

After merging, retrieval computes: $$s_j = \max_i (\bar{e}_p^i)^\top e_q^j; \qquad s(q,p) = \sum_j s_j$$ where $e_q^j$ are query token embeddings.

Empirical analysis confirms that merging, particularly by semantic clustering, far surpasses pruning when aiming for aggressive embedding reductions.

3. Multi-Dimensional Optimization of Merging

Light-ColPali/ColQwen2 arrives at its optimal design by systematically exploring merging across three key axes:

1. Merging algorithm: Semantic (embedding-based) clustering is superior to naive spatial pooling, preserving information from non-contiguous but semantically related document regions.
2. Fine-tuning: Retraining the retriever end-to-end on merged (compressed) embeddings enables the model to adapt, recovering accuracy lost to compression. Merging-only (without fine-tuning) leaves substantial performance on the table.
3. Merging location: Merging in the model pipeline after the last projection layer (i.e., on low-dimensional, task-adapted vectors) maximizes information retention and computational efficiency, outperforming merging earlier (e.g., at raw patch or post-encoder representations).

This yields a methodology where semantic clustering is performed post-projection, merged embeddings are used for retrieval, and the entire model is fine-tuned to adjust for the downstream effect of token compression.

4. Empirical Results: Effectiveness and Efficiency

Experimental results across major VDR benchmarks (ViDoRE, VisRAG, MMLongBench-Doc) indicate:

• Memory efficiency: At a merging factor of 9 (i.e., $\approx 11.8\%$ of original tokens retained), Light-ColPali/ColQwen2 maintains 98.2% of the original NDCG@5 score.
• Further compression: With more aggressive merging (down to $2.8\%$ of original memory usage, “factor 49”), performance is still 94.6% of baseline and superior to DSE (single-vector) models at equivalent or lower memory.
• Overhead: The merging process introduces minimal computational cost relative to the substantial storage savings; the major resource savings are at index build and storage, enabling scaling to large document corpora.

A summary from the main results table:

Model/Variant	Relative Memory	NDCG@5 (relative)
ColQwen2 (orig)	64.4× DSE	81.4 (100%)
Light-ColQwen2 (9×)	7.6×	79.9 (98.2%)
Light-ColQwen2 (49×)	1.8×	77.0 (94.6%)
DSE-Qwen2	1.0×	74.1 (91.0%)

5. Significance for Visual Document Retrieval

The findings demonstrate that:

• Query-agnostic token pruning is fundamentally unsuitable for VDR, because patch relevance is highly query-dependent and unpredictable pre-query.
• Semantic clustering-based merging with fine-tuning achieves an effective compression/accuracy trade-off and is practical for production.
• Scaling up VDR: The Light-ColPali/ColQwen2 approach removes the main storage bottleneck, making VLM-based document retrieval feasible for million-document corpora and resource-constrained environments.
• Future work is expected to refine adaptive per-document merging, integrate with further compression techniques (e.g., quantization), and ultimately explore query-conditioned reduction methods.

6. Mathematical and Implementation Details

The Light-ColPali/ColQwen2 system is characterized by:

• Patch-to-query late interaction:

$s_j = \max_i \left( e_p^i \right)^\top e_q^j$

where retrieval score is aggregated as $s(q, p) = \sum_j s_j$.

• Semantic-clustering merge:

$$\bar{e}_p^k = \frac{1}{|C_k|} \sum_{i \in C_k} e_p^i$$

applied after the last projection layer.

• End-to-end retriever fine-tuning: All model parameters are updated after merging is introduced, restoring lost representational and retrieval ability.

7. Broader Implications and Baseline Status

Light-ColPali/ColQwen2 sets a new practical baseline for storage-efficient VDR methods. The empirical demonstration that merging (not pruning) is the viable pathway for scalable visual retrieval architectures is a key contribution. The approach is deployable across domains requiring fast, scalable, and storage-conscious page-level search—enabling new applications in enterprise, scientific, and industrial search contexts.

A plausible implication is that similar merging and fine-tuning strategies can be generalized as a standard compression paradigm in multi-vector representations across vision-language retrieval domains, beyond just ColPali/ColQwen2 style models.