LightSTAR: Efficient Visual Document Retrieval via Lightweight Selection with Vision-Adaptive Refinement

Abstract: Visual document retrieval requires rapidly locating relevant pages from large multi-modal corpora in response to user queries. While recent methods powered by Multi-modal LLMs (MLLMs) show competitive accuracy, they suffer from prohibitive computational costs by applying intensive MLLM encoding to every single page. Meanwhile, we observe that user queries are typically keyword-anchored, containing semantically rich words that are expected to appear directly in the visible text of relevant pages, offering an efficient cue for quickly narrowing down candidate pages. Building on this insight, we propose LightSTAR, an efficient framework that decomposes visual document retrieval into: 1) LLM-free Visual Selection, which utilizes content-grounded query encoding to focus on informative words and employs LLM-free visual embeddings to produce a high-recall candidate set; and 2) Vision-adaptive Semantic Refinement, which further performs fine-grained semantic matching exclusively on these top candidates via adaptive region-wise feature fusion to effectively combine textual and layout cues, optimized through a hardness-aware contrastive objective. Experimental results demonstrate that LightSTAR achieves state-of-the-art retrieval accuracy while reducing end-to-end latency by several-fold, offering a highly practical solution to the accuracy-efficiency trade-off in visual document retrieval. Code is available at https://github.com/bokufa/LightSTAR.

• The paper introduces a two-stage process that decouples retrieval into an LLM-free visual selection and a vision-adaptive semantic refinement module.
• It achieves high-recall candidate filtering and near state-of-the-art NDCG@5 accuracy while significantly reducing computational latency.
• Empirical analysis demonstrates robust scalability and efficiency, outperforming OCR and MLLM-based baselines on large, visually-rich document corpora.

Efficient Visual Document Retrieval via Lightweight Visual Selection and Vision-Adaptive Refinement: A Technical Perspective on LightSTAR

Introduction

The expansion of visually rich, multi-modal document corpora in domains such as enterprise search, RAG, and scientific analysis has exposed the inefficacy of conventional retrieval systems predicated on unimodal or OCR-dependent approaches. These methods fail to exploit non-textual document attributes such as layout, figures, and tables, leading to substantial semantic loss. Multi-modal LLMs have recently improved the semantic fidelity of retrieval but at the cost of prohibitive computational overhead, especially for long documents where relevant content is sparsely distributed. "LightSTAR: Efficient Visual Document Retrieval via Lightweight Selection with Vision-Adaptive Refinement" (2606.23539) addresses these challenges through an explicit decoupling of retrieval into two computationally optimized stages: (1) an LLM-free Visual Selection module for high-recall candidate filtration, and (2) a Vision-Adaptive Semantic Refinement module for fine-grained ranking using MLLM-level descriptors.

Figure 1: Architecture schematic—LightSTAR decomposes retrieval into LLM-free Visual Selection and Vision-Adaptive Semantic Refinement modules, enabling state-of-the-art accuracy with the lowest measured end-to-end latency.

LightSTAR Architecture

LLM-Free Visual Selection

This first-stage module eschews heavyweight LLMs, employing a lightweight, text-aware ViT initialized with strong multi-modal priors. The key idea builds on the empirical observation that document queries are usually anchored by distinctive keywords—that is, content words visually present in relevant pages. Natural language queries are filtered via POS tagging to isolate semantically informative tokens, discarding function words that are ubiquitous and visually uninformative. Filtered tokens are directly matched to document patch embeddings using a scale-adaptive late interaction mechanism: for each query token, the average over top-$k$ matches across document patches is computed (where $k$ adapts with document complexity). This robustifies similarity estimation against noisy or spurious local maxima and increases recall, especially in visually dense or noisy documents.

Vision-Adaptive Semantic Refinement

After high-recall candidate selection, the second module performs late-interaction cross-modal matching using an MLLM, but only on a small candidate pool. Both query and document are projected into normalized embedding spaces, where late interaction is applied for final ranking. To maximize synergy and avoid redundant computation, LightSTAR shares lower layers of the visual encoder across both stages. A parallel dual-branch ViT design in the final encoder layer extracts both text-sensitive features (highly aligned to textual content) and layout-sensitive features (responsive to spatial, non-textual structure). An adaptive region-wise feature fusion, modulated by similarity to a whitespace embedding, allocates each spatial region to the appropriate feature stream, preserving foreground (textual) and background (layout) cues as appropriate.

Hardness-aware contrastive learning is used during refinement optimization: hard negatives (confusable candidate pages) are weighted more heavily, improving the discriminability of the learned metric in challenging retrieval cases.

Figure 2: Token-level visual-text alignment heatmaps—query tokens are accurately localized within document images, illustrating the fine-grained grounding of the lightweight visual encoder.

Empirical Analysis

Retrieval Quality

LightSTAR consistently outperforms OCR and conventional vision-language contrastive baselines in both recall-oriented candidate selection and final NDCG@5. On the ViDoRe benchmark, LightSTAR achieves an overall NDCG@5 of 89.1, surpassing prior state-of-the-art MLLM-based methods despite using significantly fewer parameters (2B vs. 3B+ in ColQwen2.5, ColPali, and VisRAG-Ret).

Efficiency and Scalability

A primary claim is state-of-the-art accuracy with reduced latency. LightSTAR achieves a 2.3–10.4x speedup over the fastest established MLLM-based retrievers, with the gap widening as corpus size increases. As corpus size grows from 500 to 7,000 document pages, LightSTAR’s linear latency scaling exhibits the flattest slope, validating the computational savings from restricting expensive MLLM operations to a small, high-quality candidate set.

Figure 3: Latency comparison—LightSTAR demonstrates sublinear latency scaling relative to leading MLLM retrievers as corpus size increases.

Module Effectiveness

The LLM-Free Visual Selection module alone—using only 542M parameters—achieves an average Recall@100 of 97.8 on ViDoRe, validating the hypothesis that content-word-focused filtering recovers nearly all ground-truth relevant pages with negligible parameter and computation cost.

Ablation studies show that the complete pipeline achieves near-refinement-only accuracy (89.1 vs. 89.3 NDCG@5) at less than one-third the latency, validating the correctness and sufficiency of the candidate filtration strategy.

Key Hyperparameter Sensitivity

Ablations on the scale-adaptive interaction parameter $\gamma$ support the robustness of LightSTAR: small values ($\gamma \to 0$) yield unstable matching; excessive smoothing ($\gamma \to 1$) sacrifices discriminability. Optimal fusion threshold $\tau$ is found around 0.65, balancing the contribution of textual and background layout cues through adaptive masking.

Figure 4: Ablation—retrieval performance sensitivity to the scale-adaptive parameter $\gamma$ controlling patch aggregation granularity.

Theoretical and Practical Implications

LightSTAR effectively demonstrates that high-recall, content-aware candidate filtration can be achieved efficiently with lightweight, non-LLM models while ensuring almost all relevant semantic candidates survive for heavy MLLM refinement. This yields a new paradigm for multi-stage multi-modal information retrieval systems: computational cost is non-uniformly allocated based on both query semantics and region-specific document structure. High computational precision (i.e., accuracy per FLOP) is thus achieved without sacrificing semantic richness.

Further, the architecture can generalize to other sequence retrieval scenarios where visually grounded semantic entities are sparsely distributed, including long-form video, scientific chart, or spatial-cue-anchored document collections.

Future Directions

LightSTAR offers several immediately actionable avenues:

• Memory Efficiency: Further reduction of system memory usage via parameter sharing and knowledge distillation.
• Cross-modal Retrieval Generalization: Application of the candidate-filtration-plus-adaptive-refinement paradigm to more diverse multi-modal scenarios, including audio-text or video-text retrieval.
• Active Cascade Depth: Automatic candidate set size/adaptive cascade depth selection based on real-time query difficulty estimation.
• Fine-Grained Supervision: More adaptive contrastive schemes, utilizing external knowledge, for hard negative mining.
• Robustness: Investigation into out-of-distribution and adversarial query robustness in the presence of ambiguous or visually non-anchored queries.

Conclusion

LightSTAR (2606.23539) introduces a two-stage framework that addresses the inherent efficiency-accuracy trade-off in large-scale visual document retrieval. By leveraging keyword-anchored queries for LLM-free content-grounded candidate selection and restricting heavy MLLM-based refinement to a compact pool, it achieves state-of-the-art retrieval fidelity at minimized latency and computational overhead. The explicit architectural decoupling with joint weight sharing and region-adaptive fusion sets a template for future work on scalable, efficient multi-modal information retrieval.