ColParse: Layout-Aware Document Retrieval

• ColParse is a layout-informed system that parses document pages into semantically meaningful sub-images and fuses them with a global vector.
• It replaces dense grid-based representations with a compact multi-vector index, significantly reducing storage (often by over 95%).
• Empirical evaluations show that ColParse improves retrieval accuracy and interpretability across diverse benchmarks and models.

ColParse is a layout-informed multi-vector retrieval framework for Visual Document Retrieval (VDR) that replaces dense grid-based page representations with a small set of parsed, semantically meaningful sub-image embeddings fused with a global page-level vector. In the formulation introduced in "Beyond the Grid: Layout-Informed Multi-Vector Retrieval with Parsed Visual Document Representations," ColParse treats document pages as structured visual objects containing text, tables, figures, and layout relations, and uses document parsing to produce a compact multi-vector index that is both structurally aware and storage-efficient (Yan et al., 2 Mar 2026).

1. Problem setting and motivation

VDR asks: given a text query $q$, retrieve relevant document pages from a large corpus $\mathcal{C}$. In this setting, a document page $d$ is not merely an image; it contains text, tables, figures, layout structure, reading order, and visual-semantic relations between page components. The core difficulty is that query relevance is often localized to specific regions rather than to an entire page-level embedding (Yan et al., 2 Mar 2026).

Modern VDR systems therefore often use a multi-vector late-interaction setup inspired by ColBERT-style retrieval. In the grid-based formulation, a page image is split into $N_p$ patches $\{p_j\}_{j=1}^{N_p}$, which are embedded as

$$\mathbf{D}_{\text{grid}} = \{\mathbf{d}_j\}_{j=1}^{N_p}, \quad \mathbf{d}_j \in \mathbb{R}^D$$

while the query is encoded into token vectors

$$\mathbf{Q} = \{\mathbf{q}_i\}_{i=1}^{N_q}, \quad \mathbf{q}_i \in \mathbb{R}^D.$$

Relevance is then computed with MaxSim: $$s(q, d) = \sum_{i=1}^{N_q} \max_{j=1}^{N_p} \left( \mathbf{q}_i^\top \mathbf{d}_j \right).$$ This permits each query token to align with its best-matching page region, but it also creates the central systems problem addressed by ColParse: if a page has hundreds or thousands of patches, storage scales as $O(N_p \times D)$, which becomes prohibitive at corpus scale (Yan et al., 2 Mar 2026).

The paper identifies three common compression strategies in prior multi-vector retrieval. Merging or clustering patch vectors can reduce vector count but may blur fine-grained information; pruning can improve efficiency but may degrade performance under aggressive compression; and abstract learnable tokens can be compact but are not explicitly grounded in document layout. ColParse is proposed as an alternative built on the premise that documents already possess structure, so the relevant units for indexing should be parsed layout regions rather than arbitrary grid patches (Yan et al., 2 Mar 2026).

2. Representation model and retrieval formulation

The core representational move in ColParse is to construct a multi-vector document representation from layout-informed sub-images and a global page-level vector. Instead of retaining hundreds of patch embeddings, the method stores only $k$ vectors, where $\mathcal{C}$0 is dynamic, depends on page complexity, and is typically less than 10 (Yan et al., 2 Mar 2026).

The first stage applies a document parser

$\mathcal{C}$1

to detect semantic regions in page image $\mathcal{C}$2. Here $\mathcal{C}$3 is the bounding box of region $\mathcal{C}$4, and $\mathcal{C}$5 is a content-type label such as title, table, figure, or paragraph. The page is then cropped into sub-images

$\mathcal{C}$6

Each $\mathcal{C}$7 preserves the original visual integrity of a layout component (Yan et al., 2 Mar 2026).

A single-vector retrieval encoder $\mathcal{C}$8 is used for both local regions and the full page. Local encoding yields

$\mathcal{C}$9

and thus

$d$0

Global encoding yields

$d$1

The global vector is intended to capture page-level topic and context, while each local vector captures region-specific evidence (Yan et al., 2 Mar 2026).

ColParse then performs global-local fusion by weighted element-wise addition: $d$2 The final stored representation is

$d$3

At query time, ColParse retains late interaction but applies it over the fused parsed vectors: $d$4 The operational consequence is that MaxSim is computed over a small set of layout-aware vectors rather than a large patch grid (Yan et al., 2 Mar 2026).

3. Indexing pipeline and parser dependency

ColParse is described as an offline indexing pipeline with three stages: layout-informed document parsing, dual-stream encoding, and global-local fusion. The first stage is structurally decisive because it determines the units over which the rest of the retrieval stack operates (Yan et al., 2 Mar 2026).

For practical parsing, the paper uses MinerU2.5, described as first performing coarse layout analysis and then high-resolution region recognition. The unified parser selected for the experiments is MinerU2.5, chosen for a strong accuracy/efficiency trade-off. The reported OmniDocBench result is MinerU2.5: 90.67 overall, and the paper states that this was best among the compared parsers (Yan et al., 2 Mar 2026).

The dependence on a document parser is not incidental. ColParse is explicitly not a post hoc compression mechanism applied to already-produced patch vectors. Its representation is generated from parser-produced semantic regions, and its indexing structure therefore depends on the parser’s ability to detect coherent units such as tables, figures, titles, and paragraphs. This suggests that ColParse belongs to a class of retrieval systems in which document understanding is front-loaded into the indexing stage rather than deferred entirely to the encoder or scorer (Yan et al., 2 Mar 2026).

The method is also described as plug-and-play and training-free. In the reported instantiation, the same single-vector multimodal retrieval encoder can be applied to cropped regions and whole pages, with no additional training stage introduced between parsing and retrieval. A plausible implication is that ColParse was designed to retrofit existing single-vector encoders into compact multi-vector retrieval systems without requiring retraining of the base backbone (Yan et al., 2 Mar 2026).

4. Theoretical motivation and interpretability

The paper frames VDR compression through an Information Bottleneck perspective. Let $d$5 denote the document, $d$6 the query, $d$7 the relevance variable, and $d$8 the compressed representation. The objective is written as

$d$9

The paper argues that this objective is intractable at indexing time because the query distribution is unknown, and motivates a region-wise decomposition of the document representation instead of monolithic compression (Yan et al., 2 Mar 2026).

The parsing stage decomposes the page into semantic regions $N_p$0. The paper introduces a “Semantic Concentration” assumption: for any query, relevance is usually determined by one primary region $N_p$1. Under that assumption, multiple region-wise channels provide a more natural compression strategy than compressing the entire page into a single channel or compressing a patch grid without regard to structure (Yan et al., 2 Mar 2026).

For fused vectors $N_p$2, the paper writes

$N_p$3

and defines

$N_p$4

Fusion is argued to be useful when $N_p$5, meaning that the global vector contributes additional relevance-relevant information beyond the local region alone. This is the formal rationale for not simply indexing parsed regions independently. In ColParse, local evidence is retained, but it is explicitly conditioned by page-level context before storage and scoring (Yan et al., 2 Mar 2026).

Interpretability follows from the same design. Because each stored vector corresponds to a parsed region, a retrieval score can be traced back to a specific table, paragraph, figure, or other page component. The paper presents this as a major practical advantage over grid-based or abstract-token approaches. This suggests a retrieval explanation model in which the highest-scoring MaxSim matches can be localized to semantically meaningful evidence regions rather than to uninterpretable latent patches (Yan et al., 2 Mar 2026).

5. Empirical evaluation, baselines, and ablations

ColParse is evaluated on 24 datasets across 5 benchmark suites: ViDoRe-V1, ViDoRe-V2, VisRAG, ViDoSeek, and MMLongBench. These benchmarks cover academic papers, scientific documents, tables, figures, charts, long documents, cross-page reasoning, and multilingual or enterprise-style retrieval (Yan et al., 2 Mar 2026).

The method is applied as a plug-and-play module to 10 single-vector multimodal retrieval models, including VLM2Vec-V1-2B / 7B, VLM2Vec-V2-2B, LamRA-Ret, GME-2B / 7B, UniME-V2-2B / 7B, and B3-2B / 7B. The main reported metric is nDCG@5, and experiments were run on NVIDIA A100 80G GPUs (Yan et al., 2 Mar 2026).

The comparison includes several baseline families: the original single-vector base model; Multi-img, which feeds all parsed sub-images simultaneously into the base model to produce one vector; parser-guided chunking-layout variants such as type-cluster, type-mean, subimg-cluster, and subimg-mean; chunking-semantic baselines that cluster token embeddings without using layout structure; and Single2multi variants such as scoring-add and scoring-multiply that parse sub-images and encode them separately but use alternative scoring rather than fusion (Yan et al., 2 Mar 2026).

The paper reports that ColParse consistently outperforms the original single-vector bases and all compression or chunking baselines. In ViDoRe-V1, for VLM2Vec-V1-2B, the average score improves from 20.73 to 52.37; for VLM2Vec-V1-7B, from 20.16 to 62.85; for VLM2Vec-V2-2B, from 74.16 to 78.41; and for GME-7B, from 89.36 to 89.56. On MMLongBench, the paper reports improvements such as 25.93 → 32.07 for VLM2Vec-V1-2B and 29.31 → 44.21 for UniME-V2-2B (Yan et al., 2 Mar 2026).

The ablations isolate the contribution of the main design choices. Token-level clustering or mean-pooling often harms performance, whereas ColParse preserves performance by keeping raw visual-semantic layout units intact. The s2m-g-i variant, which adds the global page vector to local region vectors without deeper fusion, shows that simple inclusion helps, but weighted fusion helps more. The s2m-t-c variant, which groups vectors by semantic type, usually underperforms, which the paper attributes to the fact that two regions of the same type may differ substantially in meaning depending on spatial context. The balancing factor $N_p$6 is reported to be robust over $N_p$7, with performance staying above the baseline across that range and best values often around 0.6–0.8 (Yan et al., 2 Mar 2026).

6. Efficiency, storage compression, and scope of the term

A central claim of ColParse is that it sharply reduces the storage burden of multi-vector retrieval. Relative to standard grid-based multi-vector approaches, the paper reports storage reduction by over 95%, and in some comparisons by more than 99%. A concrete example compares ColQwen* at 768 vectors/page with ColParse at 5.9 vectors/page, which the paper characterizes as a >99% reduction in stored vectors (Yan et al., 2 Mar 2026).

The asymptotic distinction is straightforward. Grid methods store $N_p$8, while ColParse stores $N_p$9 with $\{p_j\}_{j=1}^{N_p}$0. On the best-performing model GME-7B, the paper reports Performance: 80.61 on MMEB-visdoc, Storage: 5.9 vectors/page, and Latency: 0.81 s/page. The appendix further reports that the parser usually produces only a few vectors per page, often around 2–6 vectors on many datasets and sometimes up to approximately 9 in harder cases (Yan et al., 2 Mar 2026).

The term “ColParse” may invite confusion because several other papers in the supplied literature address parsing, parser generation, or context-sensitive parsing, but do not define this retrieval framework. "A Model-Driven Probabilistic Parser Generator" introduces a model-driven probabilistic parser generator built on top of ModelCC rather than a system named ColParse (Quesada et al., 2012). "A Constraint-Satisfaction Parser for Context-Free Grammars" introduces Fence, a bottom-up chart parser for context-free grammars with ambiguity and constraints, and explicitly does not introduce a parser named ColParse (Quesada et al., 2011). A related ambiguity appears in discussions of file-format parsing and secure binary parsing, where the relevant systems are IPG-based parser generation and PulseParse rather than ColParse (Zhang et al., 2023); (Ramananandro et al., 22 May 2025).

In current usage within the supplied corpus, ColParse refers specifically to the 2026 VDR framework rather than to a general parser generator or a grammar formalism. Its scope is multimodal document retrieval, not language parsing, file-format parsing, or verified serialization (Yan et al., 2 Mar 2026).