Stellar: Scalable Multimodal Document Retrieval for Natural Language Queries

Abstract: Multimodal document retrieval--selecting the most relevant multimodal document from a large corpus to answer a natural language query--plays an essential role in Retrieval-Augmented Generation (RAG) systems. State-of-the-art methods represent each document and query with multiple token-level embeddings and use late interaction to achieve high effectiveness. However, such multi-vector representations incur substantial memory overhead during retrieval, leading to poor scalability and hindering real-world deployment. In this paper, we present Stellar, a scalable multimodal document retrieval framework that stores token-level document embeddings on disk and loads only a small set of candidate embeddings into memory for late interaction. Stellar comprises two key components: (i) Lexical Representation-based Filtering (LRF), which fine-tunes a Multimodal LLM (MLLM) as a sparse encoder to produce high-quality lexical representations, enabling efficient and effective document filtering to significantly reduce the candidate set; (ii) Efficient Disk-backed Late Interaction (DLI), which designs an on-disk token embedding storage layout guided by a balanced clustering algorithm, and dynamically loads only the necessary token embeddings into memory using a simple yet effective cost model. Extensive experiments on four real-world benchmarks and a newly presented large-scale dataset demonstrate that Stellar reduces memory overhead and query latency by 1-2 orders of magnitude compared to existing methods without compromising retrieval effectiveness.

• The paper introduces a disk-backed, scalable retrieval framework that combines sparse lexical filtering with efficient late interaction to optimize memory and latency.
• It demonstrates a significant reduction in memory usage by 1–2 orders of magnitude and maintains sub-150 ms query latency on large-scale corpora.
• Empirical evaluations on multiple benchmarks confirm Stellar’s competitive recall and retrieval effectiveness compared to state-of-the-art multi-vector methods.

Stellar: A Scalable Framework for Multimodal Document Retrieval

Motivation and Problem Formulation

Retrieval-Augmented Generation (RAG) systems require effective retrieval from multimodal document corpora in response to natural language queries. Existing state-of-the-art methods employ multi-vector representations and late interaction mechanisms for high retrieval effectiveness, but incur significant memory and latency costs—especially at scale—due to the necessity of keeping extensive token-level embeddings in memory. Stellar is introduced to address these deficiencies by enabling scalable, efficient, and effective multimodal document retrieval for large-scale corpora, leveraging disk-backed storage and sparse lexical filtering.

Figure 1: Multimodal document retrieval for NL queries.

Prior Representational Paradigms

Conventional approaches either rely on single-vector representations, which lack semantic granularity, or multi-vector late interaction models, which are memory-intensive. The latter, as exemplified by ColPali, require all token embeddings resident in memory, resulting in unsustainable costs for large corpora. Attempts to mitigate these drawbacks with quantization-based retrieval trade off accuracy and still exhibit scaling inefficiencies. Stellar proposes a paradigm shift: embeddings are stored on disk and selectively loaded into memory, fundamentally optimizing both retrieval and infrastructure constraints.

Figure 2: Single-vector vs. multi-vector (late interaction) methods; multi-vector models capture fine-grained semantics but are memory-intensive.

Stellar Framework Architecture

Stellar consists of two synergistic components:

1. Lexical Representation-based Filtering (LRF): Documents and queries are projected into a unified sparse lexical space using a fine-tuned MLLM. The sparse encoder leverages the model's pre-trained language head, and achieves high sparsity through contrastive learning and FLOPs-based regularization, supporting efficient in-memory inverted index-based filtering.
2. Efficient Disk-backed Late Interaction (DLI): Token-level embeddings are organized on disk via balanced clustering in semantic space, maximizing sequential I/O and load efficiency. A cost-aware loading strategy dynamically chooses between full block loading and selective vector access based on corpus, query, and candidate distributions. Sparse-dense score fusion integrates retrieval signals for final ranking.

   Figure 3: Overview of Stellar showing components for lexical filtering and disk-backed late interaction.

Lexical Representation-based Filtering

The lexical encoder maps documents and queries to high-sparsity vectors in the vocabulary space of the MLLM, enabling effective filtering in milliseconds via compact inverted indexes. Max pooling extracts salient lexical features; in-batch contrastive learning aligns representations for retrieval. Empirical results indicate typical sparsity rates above 99.8% (see Figure 4), enabling large-scale filtering with minimal memory footprint.

Figure 5: Runtime of LRF and DLI shows consistently low latency, even at scale.

Figure 4: Average non-zero dimensions (~200/151,936) in sparse lexical representations; high sparsity facilitates efficient filtering.

Disk-backed Late Interaction and Storage Optimization

Stellar organizes token embeddings on disk using balanced clustering, ensuring semantic locality and uniform block sizes. The cost-aware data loading model, leveraging on-disk index structures, chooses optimal I/O strategies per query and candidate subset. This enables low-latency retrieval and scalable token-level late interaction without memory blowup.

Empirical Evaluation and Scalability

Stellar was evaluated on four established multimodal document benchmarks and the LargeDoc benchmark, encompassing 400,000 documents. Across Recall@1, Recall@10, and MRR@10 metrics, Stellar consistently matches or exceeds prior state-of-the-art multi-vector models, even while reducing peak memory usage by 1–2 orders of magnitude. Query latency is similarly improved, achieving sub-150 ms even at 400K scale.

Figure 6: Scalability study—Stellar exhibits sublinear growth in memory and latency with corpus size; contrasting sharply with prior multi-vector methods.

Filtering Efficiency and Score Fusion

Stellar's lexical filtering outperforms dense filtering baselines in recall while using less memory and runtime, as seen in experimental comparisons. Sparse-dense score fusion is validated by ablation studies: removing fusion reduces accuracy substantially, affirming the complementary nature of lexical and dense token signals.

Figure 7: Filtering methods—lexical filtering achieves higher recall and lower resource usage compared to dense baselines.

Figure 8: Sensitivity analysis of fusion weight $\alpha$; optimal retrieval balances sparse and dense scores.

Practical and Theoretical Implications

Stellar demonstrates that disk-backed retrieval with sparse lexical filtering enables practical deployment of late interaction multi-vector models at unprecedented scale, obviating memory constraints and reducing infrastructure costs for large RAG systems. The representational and storage co-design is transferable to other multimodal retrieval tasks and can be extended for multi-document retrieval, hardware-aware optimizations, and real-time inference scenarios.

Conclusion

Stellar provides a scalable solution for multimodal document retrieval, combining sparse lexical filtering and efficient disk-backed late interaction. The empirical results confirm its superiority in scalability and competitiveness in retrieval effectiveness. Future research may focus on further optimizing disk-to-memory transfer, adaptive caching strategies, and expansion to multi-document and cross-modal retrieval settings.

---

Reference: "Stellar: Scalable Multimodal Document Retrieval for Natural Language Queries" (2606.19960)