WikiSeeker: Rethinking the Role of Vision-Language Models in Knowledge-Based Visual Question Answering

Abstract: Multi-modal Retrieval-Augmented Generation (RAG) has emerged as a highly effective paradigm for Knowledge-Based Visual Question Answering (KB-VQA). Despite recent advancements, prevailing methods still primarily depend on images as the retrieval key, and often overlook or misplace the role of Vision-LLMs (VLMs), thereby failing to leverage their potential fully. In this paper, we introduce WikiSeeker, a novel multi-modal RAG framework that bridges these gaps by proposing a multi-modal retriever and redefining the role of VLMs. Rather than serving merely as answer generators, we assign VLMs two specialized agents: a Refiner and an Inspector. The Refiner utilizes the capability of VLMs to rewrite the textual query according to the input image, significantly improving the performance of the multimodal retriever. The Inspector facilitates a decoupled generation strategy by selectively routing reliable retrieved context to another LLM for answer generation, while relying on the VLM's internal knowledge when retrieval is unreliable. Extensive experiments on EVQA, InfoSeek, and M2KR demonstrate that WikiSeeker achieves state-of-the-art performance, with substantial improvements in both retrieval accuracy and answer quality. Our code will be released on https://github.com/zhuyjan/WikiSeeker.

• The paper introduces a dual-agent VLM architecture—comprising a Refiner for visual-grounded query expansion and an Inspector for evidence validation—to enhance KB-VQA performance.
• The methodology employs reinforcement learning and parameterized multi-modal reranking, achieving a Recall@1 improvement of up to 11.7 points over visual-only baselines.
• The Inspector module optimizes response routing between LLMs and VLMs, yielding SOTA accuracy and robust handling of noisy visual data in KB-VQA tasks.

WikiSeeker: Redefining the Role of Vision-LLMs in Knowledge-Based Visual Question Answering

Introduction

Knowledge-Based Visual Question Answering (KB-VQA) is distinguished by its requirement for answering visual questions that necessitate not only image understanding but also the integration of external factual knowledge. While Retrieval-Augmented Generation (RAG) frameworks have shown effectiveness in this domain, existing approaches predominantly rely on visual-only retrieval pipelines, often marginalizing the nuanced contributions of Vision-LLMs (VLMs). Typical systems utilize VLMs merely as answer generators and fail to exploit their multi-modal reasoning capabilities during both retrieval and generation. The WikiSeeker framework advances KB-VQA by implementing a dual-agent VLM architecture and introducing multi-modal retrieval strategies that comprehensively leverage both visual and textual signals.

Figure 1: The architectural distinction between prior visual-only retrieval methods and WikiSeeker's multi-modal pipeline, where the VLM is repositioned as both Refiner and Inspector.

Methodology

Multi-Modal Knowledge Retrieval Architecture

WikiSeeker proposes a modular pipeline consisting of three critical stages: Retrieval, Rerank, and Generation. The core innovation lies in augmenting the retrieval process to be truly multi-modal—by concatenating visual and textual embeddings, indexed at the fine-grained level of <image, section> pairs within the KB. The weighting between modalities is controlled via $\alpha$, optimizing the semantic contribution from image and text. Empirical search identifies optimal weighting for specific datasets, consistently showing that hybrid (multi-modal) retrieval surpasses unimodal baselines.

Figure 2: WikiSeeker pipeline, outlining the Refiner-assisted retrieval, multi-modal dense retrieval and reranking, and Inspector-based decoupled answer generation strategy.

VLM as Refiner: Reinforcement Learning for Query Expansion

To address the inherent ambiguity in user queries, a VLM-driven Refiner module rewrites the input question, enriching it with visual semantics. This is framed as a reinforcement learning problem: the Refiner generates structured queries via, e.g., chain-of-thought prompting, and is trained with Group Relative Policy Optimization (GRPO) using a composite reward function that integrates retrieval success and syntactic correctness. The reward mapping is calibrated to the retrieval rank of the ground-truth entity, strongly incentivizing high-precision expansion strategies.

Figure 3: RL-based pipeline for training the Refiner, optimizing query expansion with joint retrieval and format rewards using GRPO.

Multi-Stage Ranking: Parameterized Modal and Reranking Weighting

Retrieval efficacy is sensitive to the balance between visual and textual signals. Systematic parameter searches demonstrate that optimal recall is achieved with $\alpha \approx 0.6$, emphasizing a slight preference for visual input in this domain. This is further refined with a two-stage reranker: first, a multi-modal reranker fuses global (retrieval) and local (rerank) scores ($\beta_1$), and then a textual reranker ($\beta_2$) focuses on intra-article relevance. Both stages are critical for maximizing top-$k$ retrieval metrics.

Figure 4: Retrieval accuracy versus multi-modal rerank weight $\beta_1$, demonstrating that a hybrid scoring function yields maximal recall.

Figure 5: Recall as a function of modality weighting $\alpha$, substantiating the superiority of multi-modal fusion over unimodal approaches.

Figure 6: Impact of textual reranker weight $\beta_2$ on retrieval accuracy, with performance peaking at moderate, not exclusive, reliance on the textual reranker.

Inspector: Decoupled Generation and Evidence Validation

Empirical results show that VLMs, when fed external textual evidence, often underperform compared to LLMs in extracting and reasoning over text, mainly due to noise introduced by visual tokens. WikiSeeker's Inspector module addresses this by operating as a routing agent: for each query, the Inspector assesses whether the retrieved evidence is both sufficient and consistent with the visual input. If so, the context is passed to a textual LLM generator; otherwise, the VLM’s internal knowledge is invoked.

This decoupled strategy is validated both qualitatively and quantitatively: the Inspector reliably maximizes accuracy by selecting the optimal reasoning path on a per-instance basis.

Experimental Results

Extensive quantitative evaluation is performed on challenging KB-VQA benchmarks EVQA, InfoSeek, and M2KR. The core findings include:

• Retrieval: Refined queries, augmented by the VLM Refiner, yield substantial absolute improvements over baseline visual-only retrieval (e.g., Recall@1 gain of 6.9–11.7 points across datasets).
• QA Accuracy: WikiSeeker with Inspector + LLM generator outperforms all existing multi-modal RAG methods—achieving SOTA on EVQA and InfoSeek, with accuracy increases up to 5.45 percentage points over previous methods.
• Ablation: Removal of either the Refiner or Inspector consistently degrades performance, confirming their orthogonality and necessity.
• Inspector Analysis: Binary routing accuracy exceeds 82%, and the Inspector demonstrates a conservative bias, minimizing catastrophic failure cases that would propagate noise to the LLM.

Qualitative Analysis

Case studies further highlight two key mechanisms:

• The Refiner disambiguates vague queries by extracting salient visual entities via chain-of-thought reasoning, directly impacting retrieval quality.
• The Inspector successfully differentiates scenarios where LLMs or VLMs are optimal, automatically bypassing misaligned retrievals and preventing model hallucinations.

  Figure 7: Refiner case study—original vague query fails to retrieve correct entity, whereas refined query generated through visual reasoning enables successful entity retrieval.

  

  Figure 8: Inspector-mediated routing—accurate retrieval triggers LLM generation via context validation (left), and insufficient retrieval invokes VLM’s parametric answer (right).

Practical and Theoretical Implications

WikiSeeker demonstrates that rethinking the architectural roles of VLMs in KB-VQA is crucial for closing the performance gap caused by unimodal or monolithic pipelines. By decomposing the VLM into functionally specialized agents (Refiner for visual-grounded query expansion, Inspector for evidence validation), the framework not only increases accuracy but also provides interpretability into decision pathways. The reinforcement learning-based optimization of query rewriting is broadly applicable to other multi-hop reasoning and open-domain retrieval settings.

Practically, these innovations establish a paradigm where interactions between retrieval, reasoning, and generation agents can be dynamically orchestrated based on empirical sufficiency, rather than static hardwired roles or assumptions about the informativeness of visual input.

Future Directions

Key open avenues include:

• Optimizing the Inspector’s routing through more adaptive, perhaps soft, decision mechanisms and integrating collaborative LLM–VLM generation.
• Extending the Refiner and Inspector agents to support multi-hop or iterative retrieval and reasoning over longer QA pipelines.
• Scaling KB indexing and training to even larger, more diverse multi-modal documents, probing transfer and generalization.

Conclusion

WikiSeeker systematically redefines the role of VLMs within retrieval-augmented KB-VQA, introducing RL-trained multi-modal query refinement and context validation agents that yield significant empirical gains over both classical and recent multi-modal RAG baselines. The modularity and extensibility of this framework will inform the design of next-generation KB-VQA and multi-modal agent systems.

---

Reference:

"WikiSeeker: Rethinking the Role of Vision-LLMs in Knowledge-Based Visual Question Answering" (2604.05818)