Pyramid Indexing Framework

• Pyramid Indexing Framework is a multi-level semantic retrieval method that extracts global summaries, section headers, and fact-level cues to represent documents efficiently.
• It leverages a three-pass methodology in systems like VisionRAG to fuse textual and visual signals, reducing storage needs and enhancing recall compared to patch-level approaches.
• The framework extends to distributed similarity search with meta-HNSW routing, achieving high throughput, low latency, and robust performance on massive datasets.

The Pyramid Indexing Framework refers to a structured, multi-level approach for efficient semantic vector retrieval, especially in large-scale vision or multimodal document systems and distributed similarity search applications. Recent developments demonstrate two principal instantiations: the three-pass pyramid approach in the VisionRAG architecture for vision-enhanced document retrieval (Roy et al., 26 Nov 2025), and the meta-HNSW-based distributed similarity search framework in large-scale databases (Deng et al., 2019). Both leverage coarse-to-fine semantic representation and hierarchical index routing for high recall, robust ranking, and scalable query performance.

1. Three-Pass Pyramid Indexing: Semantics for Document Retrieval

The VisionRAG system operationalizes “pyramid indexing” as a three-tier vectorization of document pages. Each pass extracts different levels of semantic abstraction:

• Global Page Summaries: Pass 1 generates a short textual summary per page to anchor topic-level relevance.
• Section Headers & Visual Hotspots: Pass 2 extracts hierarchical headers (section titles, captions) and concise textual summaries of visually salient cues (tables, charts, highlighted values).
• Fact-Level Cues: Pass 3 isolates atomic entities and numbers, supporting precise query intents.

This coarse-to-fine construction yields a compact, semantically rich surrogate for exhaustive patch-level representation. The artifact-specific vectors capture thematic, structural, visual, and lexical signals, supporting robust retrieval for visually complex documents without reliance on brittle OCR or patch aggregation (Roy et al., 26 Nov 2025).

2. Embedding Architecture and Artifact Vectorization

The VisionRAG pyramid framework remains model-agnostic, supporting any VLM capable of artifact extraction and text encoder φ for vectorization. In the primary setup:

• Page images (160–200 dpi) are processed via VLM prompts, generating four artifact types: summary, sections, facts, hotspots.
• Each artifact text is embedded using φ, typically producing 1536-dimensional float16 vectors (also supporting 1024/3072 dimensions).
• No explicit OCR, patch cropping, or structural parsing precedes extraction.

Typical per-page vector breakdown is:

Artifact Type	Vectors per Page	Description
Global Fused Page	1	φ(summary + hotspot summary)
Section Headers	2–4	φ(header_hs), s=1...S
Fact Vectors	5–8	φ(fact_fi), i=1...F
Hotspot Vectors	2–4	φ(hotspot_sj), j=1...H

Median total: ~12 vectors/page (B = 1+S+F+H ≈ 11–17), dramatically reducing storage versus patch-level grids, which can require 341–1024 vectors per page (Roy et al., 26 Nov 2025).

3. Mathematical Formulation and Retrieval Logic

Mapping functions define the vectorization process:

• $$f_{global}(I) = \varphi(\text{summary} + \text{hotspot summary})$$
• $f_{section}(H) = \varphi(\text{header } H)$
• $f_{fact}(C) = \varphi(\text{fact } C)$
• $f_{hotspot}(S) = \varphi(\text{hotspot } S)$

Similarity scoring is typically via dot product or cosine similarity:

$$\mathrm{score}(q, v) = \langle \varphi(q), v \rangle$$

During query-time retrieval, VisionRAG employs reciprocal rank fusion (RRF) to robustly combine scores from each artifact type and query variant:

$$S_{\mathrm{RRF}}(d,p) = \sum_{i \in \mathcal{I}} \sum_{j=0}^2 \frac{w_i}{\alpha + r_{i,j}(d,p)}$$

where $\alpha=60$, $w_i$ are uniform weights, $r_{i,j}(d,p)$ is the rank for artifact index $i$ and query variant $j$, and $K_{\text{pre}}=200$ candidates are considered per index-query pair (Roy et al., 26 Nov 2025).

4. Index Construction and Query Pipeline

VisionRAG maintains four independent vector indices—fused-page, section, fact, and hotspot—using ANN structures such as HNSW for rapid retrieval. Each vector is linked to (document ID, page ID, artifact ID), supporting efficient backtracking and downstream retrieval.

At query-time:

1. The user question $q$ is expanded to three variants ($q^{(0)}$, $q^{(1)}$, $q^{(2)}$).
2. Each is embedded and ANN-searched in all four indices, yielding ranked candidates.
3. Scores are fused via RRF, and top-$K$ (e.g., $K=10$ for FinanceBench, $K=100$ for TAT-DQA) pages are selected.
4. Base64-encoded page images are fed to a multimodal LLM (e.g., GPT-4o, GPT-5, or instructBLIP) for final answer synthesis under a deterministic prompt.

This pipeline decouples retrieval from vision backbone or patch geometry, enabling rapid, model-independent updates and future-proofing vector storage (Roy et al., 26 Nov 2025).

5. Distributed Pyramid Indexing for Similarity Search

In large-scale, distributed similarity search, “pyramid” refers to a meta-graph-based hierarchical partitioning and routing formulation (Deng et al., 2019). The system consists of:

• Coordinators: Manage queries, perform meta-HNSW routing, aggregate results.
• Executors: Serve as replicas for sub-HNSW indices, process partitioned datasets.
• Brokers (Kafka): Reliable message buses.
• Zookeeper: Oversees fault-tolerance and service orchestration.

Index construction entails:

1. Sampling a subset $X'\subset X$ for $m$-center (spherical) $k$-means.
2. Building a meta-HNSW $G_m$ on centers; partitioning the bottom layer graph into $w$ balanced clusters.
3. Assigning all points in $X$ to their nearest meta-center.
4. Constructing an independent HNSW index locally per partition.

Query routing activates only those clusters corresponding to the $K$ nearest meta-centers to the query, ensuring low latency and high throughput even for datasets of hundreds of millions of vectors. This meta/partitioned strategy allows >90% recall with only 10–20% of partitions visited per query, multi-node throughput exceeding 100,000 qps, and robust straggler/failure handling via Kafka and Zookeeper infrastructure (Deng et al., 2019).

6. Efficiency, Scalability, and Empirical Performance

A salient feature of pyramid indexing is its compactness and speed relative to classical patch or distributed solutions. In VisionRAG, the vector budget is typically $\sim$14 vectors/page—yielding 42 KB/page at 1536D float16, about 2$\times$ smaller than the lowest-pooling patch methods (e.g., ColPali pooled: 85.2 KB/page). For a 1M-page corpus, total memory is 27–41 GB (VisionRAG) versus >80 GB (patch-based) (Roy et al., 26 Nov 2025).

On financial benchmarks:

Benchmark	K	Recall	Accuracy	nDCG@K	EM@K
FinanceBench	10	0.7352	0.8051	—	—
TAT-DQA	100	0.9629	—	0.6121	0.8340

Query latency (for 1M pages): VisionRAG with local CPU embedding and ANN achieves $\sim$14 ms per query, versus ColPali’s 65 ms GPU-based pipeline (Roy et al., 26 Nov 2025). At distributed scale, meta-HNSW routing attains 2–3 ms 90th percentile latency and $>$100,000 qps throughput with strong recall—scaling empirically near-linearly with additional hardware (Deng et al., 2019).

VisionRAG and distributed Pyramid indexing frameworks demonstrate substantial engineering benefits in vector storage, downstream compute requirements, and system maintainability by forgoing patch grids, late-interaction vision backbones, and complex recovery/coordination machinery.

7. Robustness, Flexibility, and Future Directions

The pyramid formulation’s modularity enables:

• Flexible replacement of VLMs and text encoders for embedding, supporting future modalities and architectures.
• Easy scaling to million-page corpora or $10^9$-scale vector repositories under distributed regimes.
• Compatibility with generic ANN search backends, meta-graph routing, and replica/partition management for straggler+failure mitigation.

A plausible implication is that pyramid-style indexing, by making explicit use of semantic structure (summaries, hierarchical anchors, salient visual/text cues), may remain robust to shifts in document layout, multi-language content, and model-lineage changes, compared to both purely text-first and patch-based pipelines. Future work may further explore hybridization with meta-graph partitioning, dense/fine-grained vector fusion schemes, and contextually-adaptive artifact extraction for highly heterogeneous or semi-structured data repositories (Roy et al., 26 Nov 2025, Deng et al., 2019).