ColPali: Visual Document Retrieval Model

• ColPali is a multi-vector vision-language retriever that represents document pages as images using patch-level embeddings instead of OCR-derived text.
• It leverages a ColBERT-style late interaction mechanism, allowing query tokens to align with specific visual evidence to improve retrieval precision.
• Benchmark results demonstrate significant gains in retrieval performance while also highlighting challenges such as high storage demands and vulnerability to adversarial attacks.

ColPali is a vision-language document retriever that indexes document pages as images rather than OCR-derived text, represents each page as a set of patch-level embeddings, and ranks pages with ColBERT-style late interaction. Introduced together with the ViDoRe benchmark, it was built on PaliGemma-3B to support page-level retrieval over visually rich documents such as PDFs, tables, figures, charts, and slides while avoiding the lengthy OCR, layout, and captioning pipelines typical of text-centric document retrieval (Faysse et al., 2024).

1. Historical positioning and problem formulation

ColPali emerged from the observation that document retrieval pipelines were being bottlenecked less by embedding models than by ingestion: PDF parsing, OCR, layout recovery, table extraction, and figure captioning are complex, brittle, and often discard exactly the visual cues that matter for retrieval. In the formulation associated with ViDoRe, the retrieval unit is the page image itself, not a parsed text chunk, and the query remains natural-language text (Faysse et al., 2024).

In that sense, ColPali extends the ColBERT late-interaction paradigm from text-only retrieval to visual document retrieval. Later analyses describe it as a canonical multi-vector vision-language retriever: instead of compressing a page to one dense vector, it stores many patch-level vectors so that different query tokens can align with different localized evidence on the page (Park et al., 6 Jul 2026). The ViDoRe benchmark later used to study ColPali spans multiple page-level retrieval settings and domains, including arXiVQA, DocQ, InfoQ, TabFQuAD, TATDQA, Shift, AI, Energy, Gov, and Health, which made ColPali a reference point for subsequent reproducibility, scaling, and systems studies (Qiao et al., 12 May 2025).

This positioning also explains why later work repeatedly uses ColPali as the baseline or backbone for visual retrieval in slide search, multi-page DocVQA, historic map search, and multimodal RAG. A plausible implication is that ColPali is best understood not merely as a single checkpoint, but as the defining retrieval pattern for page-image indexing with late interaction.

2. Representation, indexing, and MaxSim scoring

The core representational idea is that a page image is converted into a grid of contextualized visual tokens and then projected into a shared retrieval space with the query tokens. Later technical treatments write the document-side representation as

$V(I)=\{v_i\}_{i=1}^{N}, \quad v_i \in \mathbb{R}^D$

and the query-side representation as

$$Q(q)=\{q_j\}_{j=1}^{M}, \quad q_j \in \mathbb{R}^D,$$

with both sides sharing the same retrieval dimensionality $D$ (Park et al., 6 Jul 2026). In ColPali deployments and analyses, $D=128$ is the standard retrieval head dimension (Giouroukis et al., 18 Sep 2025).

Across the literature, ColPali-style configurations are described with closely related token counts. A fixed $32\times 32$ patch grid yields $1024$ visual tokens per page in one common description, while later operational studies report roughly $1030$ to $1031$ patch-level embeddings per page depending on token handling and checkpoint details (Yeroyan, 13 Feb 2026). This token budget is what gives ColPali its fine-grained evidence retention, but it is also the source of its storage and scoring cost.

A common scoring form in the ColPali literature is

$$S(q,I) = \frac{1}{M}\sum_{j=1}^{M}\max_{i\in[N]} q_j^\top v_i,$$

while other descriptions present the canonical ColBERT score as a sum over query-token maxima; in both formulations, each query token selects its best-scoring page token (Park et al., 6 Jul 2026). Operationally, this means that a query such as one asking about a table entry, figure legend, or small region does not require the entire page to be summarized into one vector. Instead, each query token can latch onto a specific patch, preserving object-, phrase-, and region-level evidence.

3. Empirical retrieval behavior and benchmark performance

A central result in the reproducibility literature is that ColPali’s retrieval gains are not accidental. One study reports average ViDoRe nDCG@5 of 84.8 for ColPali versus 58.8 for BiPali, and its own retraining reproduces the same gap at 83.0 versus 55.9 (Qiao et al., 12 May 2025). With a Qwen2-VL backbone, the same study reports 87.7 for ColQwen2 versus 62.5 for BiQwen2, which reinforces the conclusion that late interaction, rather than merely a stronger backbone, is the decisive factor (Qiao et al., 12 May 2025).

In slide retrieval, ColPali is treated as a state-of-the-art visual late-interaction embedding model. On SlideVQA, ColPali (Visual) reports NDCG@10 82.7 and Recall@10 89.9; on LPM it reports NDCG@10 66.7 and Recall@10 80.6 (Giouroukis et al., 18 Sep 2025). The same study also shows that ColPali can be repurposed textually: ColPali (Textual) over Gemma3-generated captions plus a BGE reranker reaches NDCG@10 84.0 on SlideVQA, essentially matching strong caption-based dense baselines while using less embedding storage (Giouroukis et al., 18 Sep 2025).

In multi-page and open-domain document QA, ColPali functions as the retrieval stage in M3DocRAG. With ColPali plus Qwen2-VL 7B, the reported numbers are F1 36.5 and EM 31.4 on M3DocVQA with top-4 pages, ACC 21.0 and F1 22.6 on MMLongBench-Doc, and ANLS 0.8444 on MP-DocVQA, which is reported as state of the art in that setting (Cho et al., 2024). These results are especially notable because the same pipeline retrieves from raw page images and can answer questions whose evidence lies in images, tables, or across multiple pages.

Taken together, these studies suggest that ColPali’s main empirical advantage appears when retrieval must preserve local evidence that would otherwise be blurred by single-vector pooling or lost in OCR-centric preprocessing.

4. Storage, latency, and the compression/acceleration literature

The same token granularity that makes ColPali effective also makes it expensive. One analysis reports approximately 1030 visual tokens per image, retrieval dimension 128, and 263.7 GB of storage per million images; at full token count, MaxSim evaluation reaches 36.87\times10^{12} operations on Flickr30K and 845.51\times10^{12} on MSCOCO (Park et al., 6 Jul 2026). Another systems paper observes that for visual ColPali at 10K documents, the materialized query-token × document-token similarity tensor alone is about 21 GB in FP16, enough to exhaust a 40 GB GPU (Pony et al., 28 May 2026).

Several later papers therefore treat ColPali as the canonical target for retrieval compression. Flash-MaxSim replaces the naive einsum + max + sum implementation with an IO-aware fused kernel that preserves exact MaxSim semantics while never materializing the full similarity tensor; it reports up to 3.9x speedup on an A100, 4.7x on an H100, up to 16x less inference memory, about 28x less training memory, and 100% top-20 agreement with an FP32 reference (Pony et al., 28 May 2026). This is an operator-level acceleration: ColPali’s scoring rule is unchanged, but its systems cost becomes tractable.

Training-free compression work takes a different approach by reducing the number of page tokens. The Visual RAG Toolkit applies model-aware pooling and multi-stage search to ColPali-style retrievers and reports approximately 4x QPS improvement with minimal degradation at common cutoffs such as $k \leq 10$ (Yeroyan, 13 Feb 2026). Structural Anchor Pruning argues that important structural patches persist in ColPali’s middle layers and reports over 90% index-vector reduction with robust retrieval fidelity on ViDoRe (Liu et al., 27 Jan 2026). HPC-ColPali combines K-Means quantization, attention-guided dynamic pruning, and optional binary encoding; it reports up to 32x storage reduction, up to 60% reduction in late-interaction computation with less than 2% nDCG@10 loss, and 30–50% lower query latency under HNSW indexing (Bach, 19 Jun 2025).

SaMer pushes compression further by changing how image-side tokens are merged rather than simply pruned. Applied to ColPali with $$Q(q)=\{q_j\}_{j=1}^{M}, \quad q_j \in \mathbb{R}^D,$$0, it removes more than 93% of image-side tokens, reduces storage by 16.09x, and improves R@1 on both Flickr30K and MSCOCO, which is presented as evidence that preserving query-selectable object evidence matters more than feature-only pooling or heuristic pruning (Park et al., 6 Jul 2026). Across these papers, a recurring theme is that ColPali’s bottleneck is not only vector count, but the need to preserve the right evidence for future, unknown queries.

5. Applications, derivatives, and ecosystem extensions

Because ColPali operates on page images and exposes patch-level evidence, it has been adopted well beyond the original ViDoRe setting. In historic map retrieval, map-RAS uses the ColQwen variant of the ColPali framework to search 101,233 Library of Congress map images, storing 768 patch embeddings per page in a shared 128-dimensional space and supporting text search, reverse image search, and inter-collection search (Mahowald et al., 29 Oct 2025). In that system, ColPali-style late interaction is the engine that makes non-metadata visual motifs retrievable.

A second line of work turns ColPali’s internal spatial signals into explicit region retrieval. Patch-to-region relevance propagation uses ColPali’s patch-level similarity scores as spatial relevance filters over OCR-extracted bounding boxes, formalizes the coordinate mapping from the vision-transformer patch grid to OCR regions, and performs relevance propagation entirely at inference time without additional training (Georgiou, 2 Dec 2025). This shifts ColPali from page retrieval toward region-grounded RAG.

ColPali also serves as the conceptual template for broader multimodal retrievers. Omni-Embed-Nemotron explicitly positions ColPali as the exemplar of layout-aware image-based document retrieval and reports 85.7 average nDCG@5 on ViDoRe, above 84.7 for ColPali v1.3, while extending retrieval to audio and video (Xu et al., 3 Oct 2025). Argus, in turn, keeps ColPali-style MaxSim but makes the document representation query-conditioned, replacing the fixed page embedding $$Q(q)=\{q_j\}_{j=1}^{M}, \quad q_j \in \mathbb{R}^D,$$1 with $$Q(q)=\{q_j\}_{j=1}^{M}, \quad q_j \in \mathbb{R}^D,$$2; its Argus-9B model reports 86.0 NDCG@5 on the combined ViDoRe V1+V2 leaderboard and frames itself as a direct evolution of ColPali-style retrieval (Abdallah et al., 3 Jun 2026).

This broader ecosystem indicates that ColPali’s enduring contribution is architectural: late-interaction visual retrieval over page images has become a reusable primitive for document search, multimodal RAG, and even cross-modal extensions outside classic PDF retrieval.

6. Limitations, controversies, and current research directions

Despite strong benchmark results, ColPali has not uniformly dominated all real-world settings. On the DocDeg benchmark of noisy and unseen documents, a VLM-based ColPali-style pipeline reports overall MRR 0.2471, Recall@5 0.3554, and NDCG@5 0.2740, whereas an OCR-based pipeline using Llama 3.2 OCR plus text embeddings reports MRR 0.4852, Recall@5 0.6359, and NDCG@5 0.5229 (Most et al., 8 May 2025). Yet on the fine-tuned DocVQA subset of ViDoRe, the same paper reports NDCG@5 0.6027 and Recall@5 0.6674 for ColQwen2, versus 0.3373 and 0.4058 for OCR-based baselines (Most et al., 8 May 2025). This contrast suggests that ColPali’s performance is highly sensitive to domain shift, visual quality, and whether the retriever has been adapted to the target distribution.

A related capacity-dependent effect appears in multimodal biomedical QA. In glycobiology MM-RAG, with Gemma-3-27B-IT, Text and Multi-modal augmentation reach 0.722 and 0.740 average accuracy, while ColPali reaches 0.510; with GPT-4o, the gap narrows, with Multi-modal at 0.808, Text at 0.782, and ColPali at 0.745; with the GPT-5 family, the best ColPali and ColFlor configurations reach 0.828, and ColPali, ColQwen, and ColFlor are reported as statistically indistinguishable (Kocbek et al., 18 Dec 2025). This indicates that OCR-free visual retrieval can shift burden from the retriever to the generator: when the reader model is strong enough, ColPali becomes competitive; when it is not, explicit conversion of visuals into text is often more reliable.

Security is another identified weakness. Pixel poisoning attacks against document screenshot retrievers show that injecting a single adversarial screenshot can poison the top-10 results for 26.4% of queries in ColPali, with even higher success in targeted settings (Zhuang et al., 28 Jan 2025). Since ColPali operates directly on pixels, this attack surface differs from, and in some scenarios exceeds, that of text-only dense retrievers.

Compression has also been a point of debate. Some prior work, summarized by later pruning papers, argued that visual token importance is inherently query-dependent, casting doubt on training-free pruning. Structural Anchor Pruning challenges that claim by locating persistent middle-layer structural anchors, while Argus goes further and makes the document representation explicitly query-conditioned through a region-aware Mixture-of-Experts mechanism (Liu et al., 27 Jan 2026). This suggests two active research directions: improving static compression without losing future-query evidence, and relaxing ColPali’s original assumption that a page should have one fixed multi-vector representation for every query.

ColPali therefore occupies a dual role in current research. It is both a strong retrieval model in its own right and the baseline against which newer visual retrievers define progress—whether by improving late-interaction efficiency, introducing spatial grounding, or making the page representation itself adaptive to the query.