LightSTAR: Efficient Two-Stage Visual Document Retrieval

• LightSTAR is a two-stage framework that efficiently ranks document pages using content-grounded, text-aware visual embeddings and adaptive fusion techniques.
• Its LLM-free Visual Selection stage employs top-k aggregation and in-batch contrastive learning to achieve high-recall filtering at low computational cost.
• The Vision-adaptive Semantic Refinement stage utilizes fusion of textual and layout cues with a hardness-aware objective to precisely rerank candidate pages.

Searching arXiv for LightSTAR and closely related visual document retrieval papers for up-to-date grounding. LightSTAR is a two-stage visual document retrieval framework introduced for large multi-modal corpora, where the objective is to rank document pages against a natural-language query without incurring the cost of applying a heavy Multi-modal LLM (MLLM) to every page. Its central claim is that visual document retrieval can be decomposed into a lightweight, high-recall filtering stage and a higher-capacity reranking stage: LLM-free Visual Selection first narrows the corpus using content-grounded query tokens and text-aware visual embeddings, and Vision-adaptive Semantic Refinement then reranks only the retained candidates through adaptive fusion of textual and layout cues (Guan et al., 22 Jun 2026).

1. Retrieval task and motivating bottleneck

LightSTAR formulates visual document retrieval as page ranking over a collection $\mathcal{D}=\{d_1,d_2,\ldots,d_N\}$ for a query $q$:

$$[d_{\pi_1}, d_{\pi_2}, \ldots, d_{\pi_N}] = \operatorname*{argsort}_{d_i \in \mathcal{D}} \mathcal{S}(q, d_i).$$

The practical bottleneck identified by the paper is that prior MLLM-based retrievers process the full corpus with a large multimodal encoder, leading to cost that scales as

$\mathcal{O}(N \cdot C_{\text{MLLM}}).$

This scaling is presented as prohibitive for long documents and large corpora (Guan et al., 22 Jun 2026).

The framework is motivated by an empirical observation about query structure: user queries are often keyword-anchored, with semantically rich words that are expected to appear directly in the visible text of relevant pages. On that basis, LightSTAR replaces corpus-wide heavy inference with a staged computation:

$$\mathcal{O}(N \cdot C_{\text{VE}} + |\mathcal{D}_c| \cdot C_{\text{MLLM}}),$$

where $C_{\text{VE}} \ll C_{\text{MLLM}}$ and $|\mathcal{D}_c| \ll N$. This suggests that the system is designed not merely as a retrieval model but as a computational allocation strategy, reserving expensive semantic processing for a small candidate pool.

2. Two-stage system design

The architecture consists of two explicitly separated stages.

Stage 1, LLM-free Visual Selection, is designed to produce a high-recall candidate set $\mathcal{D}_c$ over the entire corpus. Its stated components are content-grounded query embedding, text-aware visual embedding, and scale-adaptive late interaction. The stage is trained with in-batch contrastive learning.

Stage 2, Vision-adaptive Semantic Refinement, reranks only the candidates retained by Stage 1. Its stated components are a query encoder built from text and LLM modules, a document encoder using shared ViT, MLP, and LLM components, and adaptive region-wise feature fusion that combines text-aware and layout-aware signals. The stage is trained with a hardness-aware contrastive loss (Guan et al., 22 Jun 2026).

The division of labor is structurally important. Selection is optimized for cheap corpus-wide pruning; refinement is optimized for disambiguating near-matches among already plausible pages. The paper’s ablations indicate that neither stage alone matches the joint accuracy-efficiency profile of the complete pipeline.

3. LLM-free Visual Selection

The first stage begins with content-grounded query encoding, which removes function words and punctuation-like tokens through POS filtering:

$$q' = \{\, t_i \in q \mid \mathrm{POS}(t_i) \notin \{\mathrm{CC}, \mathrm{IN}, \mathrm{DT}\} \land t_i \notin \{\mathcal{B}, \mathcal{P}\} \,\}.$$

Here, $\mathrm{CC}$ denotes coordinating conjunctions, $q$0 prepositions or subordinating conjunctions, $q$1 determiners, $q$2 be-verbs, and $q$3 punctuation. The retained tokens are embedded as

$q$4

The rationale given is that function words such as “the”, “of”, “and”, and “to” create noisy matches because they appear ubiquitously, whereas entities, technical terms, numbers, and other content-bearing words provide stronger visual retrieval cues.

For a document page image $q$5, a ViT backbone produces patch-level features:

$q$6

In the reported experiments, the backbone is InternViT-300M, initialized from a pretrained multimodal model, further pretrained with a visual-text alignment objective using token-mask pairs, and fine-tuned for retrieval with LoRA on the last ViT layer and projector. These features are described as text-aware visual embeddings because they align image patches with textual tokens in document images.

Instead of ColBERT-style max pooling, LightSTAR uses scale-adaptive late interaction with top-$q$7 averaging:

$q$8

with

$q$9

The paper argues that max pooling can be brittle because a single accidental patch match can dominate the score; top-$$[d_{\pi_1}, d_{\pi_2}, \ldots, d_{\pi_N}] = \operatorname*{argsort}_{d_i \in \mathcal{D}} \mathcal{S}(q, d_i).$$0 aggregation is used to stabilize evidence accumulation as document density varies.

The selection module is trained with in-batch contrastive learning:

$$[d_{\pi_1}, d_{\pi_2}, \ldots, d_{\pi_N}] = \operatorname*{argsort}_{d_i \in \mathcal{D}} \mathcal{S}(q, d_i).$$1

This objective explicitly encourages each positive page to score above the hardest in-batch negative. A plausible implication is that Stage 1 is optimized less for precise ordering than for robust candidate retention under aggressive computational constraints.

4. Vision-adaptive Semantic Refinement

After candidate generation, LightSTAR applies a stronger MLLM-based reranker only to $$[d_{\pi_1}, d_{\pi_2}, \ldots, d_{\pi_N}] = \operatorname*{argsort}_{d_i \in \mathcal{D}} \mathcal{S}(q, d_i).$$2. The query encoder in this stage is

$$[d_{\pi_1}, d_{\pi_2}, \ldots, d_{\pi_N}] = \operatorname*{argsort}_{d_i \in \mathcal{D}} \mathcal{S}(q, d_i).$$3

with

$$[d_{\pi_1}, d_{\pi_2}, \ldots, d_{\pi_N}] = \operatorname*{argsort}_{d_i \in \mathcal{D}} \mathcal{S}(q, d_i).$$4

The document encoder is

$$[d_{\pi_1}, d_{\pi_2}, \ldots, d_{\pi_N}] = \operatorname*{argsort}_{d_i \in \mathcal{D}} \mathcal{S}(q, d_i).$$5

with

$$[d_{\pi_1}, d_{\pi_2}, \ldots, d_{\pi_N}] = \operatorname*{argsort}_{d_i \in \mathcal{D}} \mathcal{S}(q, d_i).$$6

Scoring uses standard late interaction:

$$[d_{\pi_1}, d_{\pi_2}, \ldots, d_{\pi_N}] = \operatorname*{argsort}_{d_i \in \mathcal{D}} \mathcal{S}(q, d_i).$$7

The distinguishing mechanism in this stage is vision-adaptive fusion. The paper states that the text-aware features inherited from Stage 1 are effective for text regions but may miss layout, tables, figures, whitespace structure, and visual organization. To address this, LightSTAR introduces two complementary visual streams. Shared lower ViT layers are reused:

$$[d_{\pi_1}, d_{\pi_2}, \ldots, d_{\pi_N}] = \operatorname*{argsort}_{d_i \in \mathcal{D}} \mathcal{S}(q, d_i).$$8

At layer $$[d_{\pi_1}, d_{\pi_2}, \ldots, d_{\pi_N}] = \operatorname*{argsort}_{d_i \in \mathcal{D}} \mathcal{S}(q, d_i).$$9, two branches are formed:

$\mathcal{O}(N \cdot C_{\text{MLLM}}).$0

$\mathcal{O}(N \cdot C_{\text{MLLM}}).$1

Background-like regions are detected by comparing text-aware features with a whitespace token embedding:

$\mathcal{O}(N \cdot C_{\text{MLLM}}).$2

followed by min-max normalization and binary masking:

$\mathcal{O}(N \cdot C_{\text{MLLM}}).$3

Fusion is then performed patchwise:

$\mathcal{O}(N \cdot C_{\text{MLLM}}).$4

In the paper’s interpretation, $\mathcal{O}(N \cdot C_{\text{MLLM}}).$5 denotes a background-dominant patch and $\mathcal{O}(N \cdot C_{\text{MLLM}}).$6 a text-dominant patch. Text regions therefore retain text-aware features, while layout-heavy regions use layout-aware features. This is “vision-adaptive” because the selection of feature source is spatially conditioned on the page content rather than fixed globally.

Training in this stage uses a hardness-aware contrastive objective:

$\mathcal{O}(N \cdot C_{\text{MLLM}}).$7

with

$\mathcal{O}(N \cdot C_{\text{MLLM}}).$8

The stated purpose is to emphasize hard negatives, which become especially important after Stage 1 has already removed easy mismatches.

5. Training protocol, benchmark setting, and empirical results

The reported training configuration is parameter-efficient. In Stage 1, the backbone is InternViT-300M, with batch size 32, 10 epochs, learning rate $\mathcal{O}(N \cdot C_{\text{MLLM}}).$9, LoRA rank $$\mathcal{O}(N \cdot C_{\text{VE}} + |\mathcal{D}_c| \cdot C_{\text{MLLM}}),$$0, scaling $$\mathcal{O}(N \cdot C_{\text{VE}} + |\mathcal{D}_c| \cdot C_{\text{MLLM}}),$$1, and LoRA applied to the final ViT layer and projector while all other parameters remain frozen. In Stage 2, the backbone is InternVL3, with batch size 80, 10 epochs, learning rate $$\mathcal{O}(N \cdot C_{\text{VE}} + |\mathcal{D}_c| \cdot C_{\text{MLLM}}),$$2, the same LoRA configuration, frozen shared lower ViT layers, and updates applied to the new parallel layout-aware ViT branch, the MLP connector, the LLM, and the projection layer via LoRA (Guan et al., 22 Jun 2026).

Evaluation is conducted on ViDoRe, described as a large-scale visual document retrieval benchmark spanning scientific papers, technical reports, government documents, healthcare documents, AI-related documents, and energy documents. For scaling analysis, the paper uses corpus subsets of 500, 1000, 2000, 5000, and 7000 pages. The principal metrics are Recall@100 for the selection module and NDCG@5 for final retrieval quality. End-to-end latency includes query and document encoding, similarity computation, and candidate reranking, and is measured on 1 NVIDIA A800 80GB GPU with batch size 16 under matched settings.

Measurement	LightSTAR	Comparison or note
Overall final retrieval	NDCG@5 = 89.1%	ColQwen2.5 88.8%; Nomic-Embed-Multimodal 86.3%; ColPali 81.9%; VisRAG-Ret 78.0%
Selection quality	Recall@100 = 97.8% average	BM25 90.1; BGE-M3 91.5; SigLIP 91.2; CLIP 74.9
End-to-end latency, 5000 pages	123.9 s	ColPali 257.3 s; VisRAG-Ret 1219.9 s
End-to-end latency, 7000 pages	161.1 s	VisRAG-Ret 1678.9 s; ColQwen2.5 656.7 s; ColPali 362.6 s
Reported speedups at 7000 pages	10.4× / 4.1× / 2.3×	Faster than VisRAG-Ret / ColQwen2.5 / ColPali

The paper also reports subdataset results including DocQ: 63.5, InfoQ: 91.7, TATQ: 81.0, AI: 99.5, and Health: 99.8. The selection-stage assumption is further supported by an appendix result stating that 97.8% of queries across datasets contain at least one keyword appearing in the ground-truth page.

Ablation results provide the clearest account of how the full system is intended to function. Three pipeline settings are reported: selection only achieves NDCG@5 = 80.8 with 86.8 s latency; refinement only achieves 89.3 with 465.1 s latency; and the full system achieves 89.1 with 123.9 s latency. In the selection module, removing content-grounded query embedding reduces Recall@100 by 0.8, and replacing scale-adaptive late interaction with standard max pooling reduces Recall@100 by 1.0. In the refinement module, removing adaptive feature fusion reduces NDCG@5 from 89.1 to 88.3, while removing the hardness-aware objective reduces it from 89.1 to 87.9. The paper also states that the best fusion threshold is around 0.65, while noting that too small a $$\mathcal{O}(N \cdot C_{\text{VE}} + |\mathcal{D}_c| \cdot C_{\text{MLLM}}),$$3 is unstable and too large a $$\mathcal{O}(N \cdot C_{\text{VE}} + |\mathcal{D}_c| \cdot C_{\text{MLLM}}),$$4 over-smooths, and that larger candidate size $$\mathcal{O}(N \cdot C_{\text{VE}} + |\mathcal{D}_c| \cdot C_{\text{MLLM}}),$$5 improves accuracy at the cost of latency.

6. Interpretation, adjacent methods, and name ambiguity

A recurring misunderstanding would be to treat LightSTAR as merely a generic cascade in which any cheap retriever feeds any expensive reranker. The paper explicitly reports that independent cascades such as CLIP + ColQwen2.5 and SigLIP + ColQwen2.5 perform substantially worse than LightSTAR. This suggests that the gain is not attributed simply to staged inference, but to a unified design combining content-grounded query filtering, query-patch alignment, shared visual representations, and region-wise fusion within the reranker (Guan et al., 22 Jun 2026).

Another potential misunderstanding concerns nomenclature. In arXiv literature, names that resemble “LightSTAR” also appear in astronomy and astronomical instrumentation, but they denote unrelated systems. STARI is a CubeSat pathfinder for STarlight Acquisition and Reflection toward Interferometry, concerned with formation control, beam transfer, and single-mode-fiber injection in space, rather than document retrieval (Monnier et al., 2024). STARRED is a wavelet-based two-channel method for PSF reconstruction, deconvolution, and light-curve extraction in imaging time series (Millon et al., 2024). Separate astronomy papers also discuss “LightSTAR-like” ideas in the context of starlight coupling, telescope pointing by star reconstruction, and the Flashlights survey for highly magnified lensed stars (Diab et al., 2020, Foffano et al., 2021, Meena et al., 2022). The 2026 LightSTAR paper, by contrast, belongs to multimodal information retrieval and large-scale document search.

Within that domain, the system’s stated practical significance is its suitability for RAG, enterprise search, and document-heavy QA systems. This suggests a deployment-oriented interpretation: LightSTAR is designed for settings in which retrieval quality must remain close to high-capacity MLLM systems, but end-to-end latency and corpus-scale compute make full-corpus MLLM encoding operationally unattractive. Code availability is reported at https://github.com/bokufa/LightSTAR (Guan et al., 22 Jun 2026).