Argus-Retriever: Vision-LLM Late-Interaction Retrieval with Region-Aware Query-Conditioned MoE for Visual Document Retrieval

Abstract: Late-interaction vision-language retrievers represent each document page as many visual token embeddings and score queries with MaxSim. In systems such as ColPali, ColQwen, ColNomic, and Nemotron ColEmbed, the document embeddings are produced without seeing the query, so the same page is represented identically for a table lookup, a chart question, and a layout-sensitive evidence request. We introduce \textbf{Argus}, a family of query-conditioned late-interaction retrievers built on Qwen3.5-VL. Argus adds a region-aware Mixture-of-Experts module: the query encoder produces both retrieval embeddings and a compact context vector, the document page is pooled into spatial regions, and a query-aware router selects latent experts per region before MaxSim. The output remains a multi-vector index compatible with ColPali-style retrieval, but the document representation is now dependent on the query (i.e., $\mathbf{D}(q)$). All Argus models use a 1024-dimensional retrieval head, compared with the 2560-dimensional and 4096-dimensional heads of recent state-of-the-art systems, and are trained on roughly 9\% of the available public supervision rather than the full pool. The 9B model reaches \textbf{92.67} NDCG@5 on ViDoRe V1 and \textbf{86.0} NDCG@5 on the combined V1+V2 leaderboard, the highest reported value for an open late-interaction model on the combined leaderboard. Wrapped in a Qwen3.6-27B agentic retrieval pipeline on ViDoRe V3, Argus-9B further improves its NDCG@10 from 60.28 to \textbf{64.80} over public tasks, showing that the same retriever serves both as a strong standalone system and as a search primitive for iterative LLM agents.

• The paper introduces a query-conditioned, region-aware MoE module that dynamically adapts document representations based on the query.
• It combines a specialized region router and dense fallback to enable efficient multi-vector retrieval with minimal storage overhead.
• Empirical results show that Argus-Retriever outperforms existing models on ViDoRe and MIRACL-Vision benchmarks, especially under out-of-domain conditions.

Query-Conditioned MoE Late-Interaction Architectures for Visual Document Retrieval: An Analysis of Argus-Retriever

Introduction

Recent advances in retrieval-augmented generation (RAG) and visual document retrieval (VDR) have increasingly leveraged vision-LLMs (VLMs) for high-fidelity retrieval from complex, multi-modal documents such as PDFs and scientific articles. However, predominant late-interaction retrievers represent each document with fixed embeddings, independent of the query, limiting adaptivity and domain transfer. The Argus family of retrievers—introduced in "Argus-Retriever: Vision-LLM Late-Interaction Retrieval with Region-Aware Query-Conditioned MoE for Visual Document Retrieval" (2606.04300)—addresses this by incorporating a query-conditioned, region-aware Mixture-of-Experts (MoE) module into a late-interaction VLM retriever, enabling dynamic, query-dependent document representations while retaining compatibility with efficient multi-vector retrieval.

Figure 1: Where does the query interact with the document? Argus augments MaxSim late interaction with a query-conditioned, region-level router and MoE expert fusion to achieve adaptive document encoding.

Core Architecture and Methodology

Query-Conditioned Region Routing

Traditional late-interaction models, such as ColPali and ColQwen, generate multi-vector document representations agnostic to the query. Argus departs from this regime by introducing a region-aware router into the document encoding path. This router is explicitly conditioned on a pooled query context vector, per-region content, and normalized region positions. As a result, each spatial region of a document page may be routed through different latent experts depending on the query.

Figure 2: Argus architecture. The query branch yields retrieval embeddings $\mathbf{Q}$ and a pooled query context $\mathbf{z}_q$; the document branch is augmented with a query-aware region router and a latent expert bank.

This design recovers some adaptivity characteristic of cross-encoders, while maintaining efficient MaxSim-based scoring and ColPali-style indexing.

MoE-Driven Representation Fusion

The router selects top-$k$ latent experts per region, each responsible for modulating a subset of the overall representation. The output of these experts is fused with a learned, always-active dense expert via gated residual connections. Gating scalars are learned jointly with the rest of the model, balancing expert specialization and dense fallback. The late-interaction retrieval head maintains a comparatively narrow width (1024 dimensions at every scale), improving storage efficiency.

Training Objective and Data

Argus is trained with a standard contrastive InfoNCE objective using MaxSim similarity, and an additional load-balance loss to ensure all experts are utilized. Notably, all models use at most 9.3% of the available public supervision and maintain moderate embedding width, contrasting with prior approaches that rely on massive training sets, wide projections, or post-training model merging.

Empirical Results

Retrieval Performance Across ViDoRe Benchmarks

Argus achieves state-of-the-art performance on MTEB ViDoRe V1+V2, V3, and MIRACL-Vision, outperforming all published open late-interaction models.

• ViDoRe V1+V2: Argus-9B attains 92.67 NDCG@5 (V1) and 86.0 (V1+V2 average), outperforming Nemotron-colembed-8b-v2 by +1.3 and Ops-Colqwen3-4B by +1.4 absolute on average. The out-of-domain V2 tasks see the largest margin, confirming the value of query-adaptive representations.
• ViDoRe V3: Argus-9B matches or surpasses baselines using wider retrieval heads and larger parameter counts, validating the efficiency of the MoE approach.
• MIRACL-Vision: Argus-9B achieves top macro-average NDCG@10 (0.7552) and leads in 5 out of 10 languages, including low-resource splits such as Yoruba.

  Figure 3: Standalone versus agentic Argus retrieval on the public ViDoRe V3 tasks, demonstrating the gains from agentic refinement using Argus as the retrieval primitive.

Expert Utilization and Query Sensitivity

The router shows non-trivial, query-sensitive routing over the expert bank, with specialization emerging dynamically depending on both query and document content. Ablation studies confirm that all three routing inputs—query context, region position, and the shared expert—are individually necessary, with the loss of any resulting in a 3-4 point NDCG@5 drop (especially pronounced on out-of-domain evaluation).

Figure 4: Expert utilization on ViDoRe V2: average region-level top-2 expert shares confirm balanced expert activation and effective specialization.

Visualization of per-query routing further demonstrates that substantial region reassignment occurs for the same page across different queries.

Figure 5: Same page, different query contexts: Top-1 expert assignment per region evidences query-sensitive dynamic routing.

Inference and Storage Efficiency

A critical design constraint for large-scale deployment is per-page index size and query-time latency. Argus maintains 1024-dimensional embeddings, resulting in a 4.2 MB/page index, which is $\approx$4.5$\times$ smaller than Nemotron-colembed-8b-v2 while achieving higher retrieval accuracy. Query conditioning is implemented entirely in the cached region feature space, ensuring that inference cost remains manageable.

Figure 6: Argus inference pipeline: offline precompute phase caches all query-independent document features, enabling efficient online query-conditioned routing and retrieval.

Figure 7: Storage and accuracy trade-off: Argus-9B advances the Pareto frontier with superior NDCG@5 at substantially reduced index size.

Scaling Analysis

Scaling studies reveal that the primary benefit of larger Argus variants (e.g., 9B) materializes on out-of-domain (ViDoRe V2) tasks, whereas in-domain accuracy saturates by the 4B scale. This result substantiates that increased VLM capacity predominantly aids generalization under distributional shift, mediated via the MoE module.

Figure 8: Scaling of the Argus family on ViDoRe V1+V2: out-of-domain (V2) performance increases with model size while in-domain (V1) plateaus.

Theoretical and Practical Implications

Theoretical Implications

Argus demonstrates that late-interaction retrieval models can regain substantial cross-encoder-style adaptivity via query-conditioned, region-level MoE design, without forgoing their efficiency and multi-vector index structure. This architecture presents a flexible avenue for future research into joint vision-language representations where sparse expert specialization is dynamically determined by query context and spatial content.

The modular design—decoupling query-dependent routing from core visual feature extraction—highlights the potential value of post-hoc adaptivity in contexts where first-phase indexing must remain amortized.

Practical Implications

In deployment settings requiring tight storage budgets and low latency (e.g., financial or scientific document retrieval at scale), Argus provides a clear accuracy-storage trade-off improvement over prior models. The compatibility with both standalone and agentic retrieval settings extends the usage spectrum, from batch offline retrieval to interactive, multi-step reasoning pipelines. Notably, Argus validates that the same retriever can fuel both first-pass ranking and iterative agentic workflows effectively.

Limitations and Future Directions

While Argus advances late-interaction retrieval adaptivity, it retains the inherent storage and per-query latency overhead relative to single-vector bi-encoders. Query-conditioned routing precludes universal index caching, necessitating retrieve-then-rerank architectures for deployments scaling to extremely large corpora. The learned latent experts are not mapped to semantically interpretable classes, which may complicate error analysis and system debugging.

Future developments may include hierarchical, multi-stage routing, semantic supervision of expert specialization, integration with retrieval-augmented generation for cross-modal reasoning, and adaptation for ultra-large-scale distributed retrieval settings.

Conclusion

Argus-Retriever establishes a new high-water mark for open late-interaction retrieval on visual documents through its query-conditioned, region-aware MoE architecture. By introducing query sensitivity at the document encoding stage while preserving efficient MaxSim compatibility and restrained index size, Argus demonstrates superior performance—most notably under distributional shift and multilingual scenarios—while holding practical deployment cost constraints in check. This work paves the way for further research into sparse, adaptive, and query-contingent architectures for robust, next-generation RAG systems.