Learning to Search: A Decision-Based Agent for Knowledge-Based Visual Question Answering

Abstract: Knowledge-based visual question answering (KB-VQA) requires vision-LLMs to understand images and use external knowledge, especially for rare entities and long-tail facts. Most existing retrieval-augmented generation (RAG) methods adopt a fixed pipeline that sequentially retrieves information, filters it, and then produces an answer. Such a design makes it difficult to adapt to diverse question types. Moreover, it separates retrieval from reasoning, making it hard for the model to decide when to search, how to refine queries, or when to stop. As a result, the retrieved evidence is often poorly aligned with the question. To address these limitations, we reformulate KB-VQA as a search-agent problem and model the solving process as a multi-step decision-making procedure. At each step, the agent selects one of four actions-Answer, Image Retrieval, Text Retrieval, and Caption-based on its current information state. We further design an automated pipeline to collect multi-step trajectories that record the agent's reasoning process, tool usage, and intermediate decisions. These trajectories are then used as supervision for fine-tuning. Experiments on InfoSeek and E-VQA demonstrate that our method achieves state-of-the-art performance, consistently outperforming prior baselines and confirming the effectiveness of our framework.

• The paper reframes KB-VQA as a multi-step decision process that dynamically selects retrieval actions based on contextual cues.
• It introduces DBAgent, which uses a think-act paradigm and trajectory-based supervised fine-tuning to guide precise evidence acquisition.
• Experimental results demonstrate that DBAgent achieves higher accuracy and robustness compared to static RAG pipelines across diverse datasets.

Learning to Search: A Decision-Based Agent for Knowledge-Based Visual Question Answering

Problem Statement and Motivation

Knowledge-Based Visual Question Answering (KB-VQA) tasks require multimodal LLMs (MLLMs) to answer questions about images by leveraging external knowledge sources, particularly for long-tail entities and facts not typically encoded in model parameters. Traditional pipeline-based Retrieval-Augmented Generation (RAG) approaches exhibit rigid and static behavior: retrieval and reasoning phases are segregated, impeding adaptation to heterogeneous question types and limiting the model's ability to dynamically determine when and how to retrieve knowledge. This inflexibility leads to suboptimal evidence selection and poor alignment between retrieved resources and the question semantics.

Formulation: KB-VQA as Sequential Decision-Making

The paper "Learning to Search: A Decision-Based Agent for Knowledge-Based Visual Question Answering" (2604.07146) proposes reframing KB-VQA as a multi-step decision process. Instead of passively consuming evidence via a fixed pipeline, the model becomes an agent that interacts iteratively with an external environment, choosing among four atomic actions at each step: direct answering, image retrieval, text retrieval, or image captioning, based on the current information state accumulated during the reasoning trajectory.

Figure 1: Conceptual comparison between classical MLLMs with/without RAG and the dynamic search-agent paradigm.

This adaptive framework enables fine-grained and context-dependent evidence acquisition, facilitating structured accumulation of supporting knowledge and adaptive termination of the search sequence.

The DBAgent Framework

Trajectory-Based Supervised Fine-Tuning

To induce effective multi-stage decision behavior in the model, the authors develop an automated data pipeline for constructing multi-step reasoning trajectories annotated with tool choices, intermediate observations, and explicit reasoning. These trajectories cover a wide variety of question complexities and evidence patterns and are used for supervised fine-tuning (SFT), where supervision is applied only to the decision tokens (reasoning, action selection) and not the evidence strings, preventing content memorization.

Figure 2: Overview of DBAgent and the trajectory-based SFT pipeline, including multi-turn reasoning, action selection, and structured trajectory curation.

DBAgent operates under a think-act paradigm: after each reasoning turn, only one action tag is produced, making the decision process explicit and easily auditable. Captions serve as semantic bridges when visual disambiguation is needed before reliable textual queries can be formed.

Experimental Analysis

Benchmarks, Baselines, and Main Results

DBAgent is benchmarked on InfoSeek and Encyclopedic-VQA (E-VQA)—datasets with significant long-tail entity coverage and diverse evidence requirements. Comparisons include zero-shot MLLMs (e.g., BLIP-2, LLaVA, GPT-4V), classical RAG pipelines (DPR, Wiki-LLaVA, EchoSight), and recent reflection/reasoning-augmented retrieval agents (ReflectiVA, VLM-PRF).

DBAgent demonstrates consistent and strong numerical superiority. On E-VQA (All), it delivers 45.2% accuracy—outpacing the strongest prior by ~6 points. On InfoSeek (All), the method reaches 48.7%, uniformly outperforming all baselines—including those employing RL or explicit reasoning.

Trajectory and Difficulty Stratification

A trajectory-level analysis reveals a distribution over action sequences with varying depths: approximately 5% of questions are answered without retrieval, while ~68% require either a single-step image or text retrieval. However, as sequence length (refined queries and evidence steps) increases, both evidence recall and answer accuracy fluctuate—longer sequences correspond to more difficult queries and a higher error rate, validating the need for a dynamic rather than fixed pipeline.

Robustness and Ablations

DBAgent's design results in significant robustness to knowledge base scale: as the corpus grows from 10k to 100k entities, accuracy degrades smoothly (from 63.3% to 48.7%), whereas fixed strategies exhibit sharp performance drops.

Figure 3: Ablation: DBAgent maintains accuracy under varying knowledge base scale, outperforming all static retrieval strategies.

Additional ablations confirm that naive retrieval for all samples (either image or text) confers negligible benefit, and collective gains arise from learning when and how to retrieve, not from unconditional augmentation.

Qualitative Case Studies

The authors present several trajectory case patterns:

• Direct answering without retrieval (minimal state dependence; parametric knowledge sufficient).
• Single-step image or text retrieval based on initial information sufficiency.
• Chained retrieval: visual grounding followed by caption-guided text retrieval for fine-grained attributes.
• Multi-step text query refinement when initial retrieval is insufficient.
• Failure modes: (1) correct retrieval but flawed reasoning; (2) incorrect retrieval leading to answer errors.

  Figure 4: The agent answers directly from the image, skipping external tool invocation.

  

  Figure 5: After failing at visual identification, single-step image retrieval provides necessary evidence for the correct answer.

  

  Figure 6: The agent utilizes image grounding, generates a caption, then refines the query with targeted text retrieval, resolving ambiguity for hard questions.

  

  Figure 7: Failure: despite retrieving supporting evidence, reasoning misalignment produces an incorrect answer.

Implications and Theoretical Relevance

Reconceptualizing KB-VQA as a multi-turn agent decision process yields several implications:

• Expressivity: The framework supports variable-depth, multimodal policies, allowing for adaptive tool invocation and termination, which are crucial under realistic, heterogeneous query distributions.
• Supervisability/Auditability: Explicit recording of decisions and states renders the agent inherently interpretable, revealing the contribution of retrieval versus parametric knowledge.
• Robustness/Generalization: DBAgent shows superior adaptation to unseen questions/entities and maintains stability amid noise and scaling in the knowledge source.
• Utility for Broader Multimodal Reasoning: The paradigm extends naturally to open-world, tool-augmented, and modular agent architectures.

Limitations and Future Directions

Identified limitations include:

• Dependence on high-quality trajectory generation and prompt engineering.
• Restriction to a fixed action set (answer, image/text retrieval, caption); extension to arbitrary tools (e.g., knowledge graphs, API access) remains open.
• Inference efficiency: iterative multi-step interaction incurs higher latency compared to single-pass baselines.
• Evaluation limited to well-curated KB-VQA datasets—further benchmarking in unrestricted, real-world settings is necessary.

Anticipated directions include integrating richer tool-sets, automated tool acquisition, curriculum learning on trajectory difficulty strata, and hybrid SFT/RL objectives for decision-focused policy optimization.

Conclusion

The work demonstrates that knowledge-intensive multimodal VQA fundamentally requires explicit, flexible decision processes rather than static pipelines. DBAgent approaches the problem as a sequential search and reasoning task, showing strong empirical improvements, robust behavior under scale, and interpretable multimodal trajectories. The formulation enables adaptive, context-aware tool use and lays the groundwork for future multimodal AI agents requiring compositional knowledge acquisition and dynamic planning.

Reference: "Learning to Search: A Decision-Based Agent for Knowledge-Based Visual Question Answering" (2604.07146).