CogPlanner: Adaptive MRAG Planning

• CogPlanner is a planning-augmented MRAG framework that models multimodal retrieval as a sequential decision process with iterative query reformulation and action selection.
• It integrates planning agents into the MRAG pipeline to optimize retrieval strategies under computational constraints using MLLMs like GPT-4o and Qwen2-VL.
• Empirical results on CogBench demonstrate substantial improvements in answer quality and efficiency, with up to +18pp claim F1 gains on complex queries.

CogPlanner is a planning-augmented framework tailored for Multimodal Retrieval Augmented Generation (MRAG) systems, specifically designed to bridge the gap between one-step retrieval protocols and the need for dynamic, adaptive information acquisition in complex multimodal tasks. Drawing inspiration from human cognitive strategies, CogPlanner formulates the entire MRAG pipeline as a sequential decision process and introduces explicit planning agents to optimize both query refinement and retrieval strategy selection under computational constraints. Empirical studies on large-scale benchmarks demonstrate substantial improvements in both answer quality and efficiency, with negligible computational overhead, across mainstream large multimodal LLMs (MLLMs) (Yu et al., 26 Jan 2025).

1. Formal MRAG Planning Problem

CogPlanner reframes the standard MRAG system—where MLLMs answer user queries through retrieval from a multimodal external knowledge base—as a constrained planning problem. At each iteration, the system maintains an information state

$$\mathcal{S}_t = (\mathcal{Q}_0, \{\mathcal{Q}_i\}_{i\le t}, \bigcup_{i \le t} \mathcal{D}^i),$$

where $\mathcal{Q}_0 = (q, v)$ is the initial multimodal query comprised of text $q$ and image $v$, and $\mathcal{D}^i$ represents documents retrieved at iteration $i$.

The planning agent sequentially selects actions $(\mathcal{A}_t, \mathcal{Q}_t)$ from the joint space of retrieval modalities ($$\mathcal{A}_t \in \{\mathrm{none}, \mathrm{text}, \mathrm{image}\}$$) and query reformulations. The transition function $\mathcal{F}$ updates $\mathcal{S}_t$, and the process iterates until an explicit stopping criterion is met (e.g., $\mathcal{Q}_0 = (q, v)$0 or exceeding a cost budget $\mathcal{Q}_0 = (q, v)$1). The principal objective is to maximize answer quality

$\mathcal{Q}_0 = (q, v)$2

subject to $\mathcal{Q}_0 = (q, v)$3, where $\mathcal{Q}_0 = (q, v)$4 is a task-specific evaluation metric (Yu et al., 26 Jan 2025).

2. CogPlanner System Architecture

CogPlanner is architected around three primary modules orchestrated in a closed-loop system:

1. Planning Expert: This agent comprises two subcomponents—(i) a query reformulator that suggests query variants or decomposition; (ii) a retrieval action selector that chooses among no retrieval, text retrieval, or image retrieval. The Planning Expert is instantiated using MLLMs such as GPT-4o, Qwen2-VL, or Pixtral, utilizing in-context prompting to drive decision-making.
2. Retriever: Upon receiving a reformulated query–action pair, the retriever calls an external multimodal search API (e.g., Google Web or Google Image) to collect relevant evidence snippets.
3. Response Synthesizer: The culminating module synthesizes all collected context (original query, refined queries, accumulated evidence) as a prompt for the generation-capable MLLM to produce the final answer.

The system can be deployed in either a sequential or parallel planning regime. In sequential mode, the agent makes the query reformulation first, then selects the retrieval action conditioned on the newly formulated query. In parallel mode, both decisions are taken independently in each iteration. The process continues until a pre-specified halting condition is satisfied.

3. Core Algorithms and Planner Dynamics

At each planning iteration $\mathcal{Q}_0 = (q, v)$5, the agent evaluates the state $\mathcal{Q}_0 = (q, v)$6 and undertakes:

• Query Reformulation: $\mathcal{Q}_0 = (q, v)$7, where $\mathcal{Q}_0 = (q, v)$8 may return a set of sub-queries to target distinct modalities or facets.
• Action Selection: $\mathcal{Q}_0 = (q, v)$9, where $q$0 is a utility scoring function estimated by the underlying MLLM.
• Retrieval and State Update: The selected action and reformulated query are executed, new evidence is appended, and planning proceeds with the updated state.

The planner can efficiently operate under token or API call constraints by maintaining a running cost and terminating when the budget is exhausted. This dual optimization regime explicitly trades retrieval thoroughness against computational expense.

Pseudocode (Sequential Variant)

$q$1 (Yu et al., 26 Jan 2025)

4. CogBench: Benchmarking Planning-augmented MRAG

CogBench is a dedicated evaluation suite developed to measure MRAG planning agents. The dataset contains over 5,700 samples (train/test split: 5,307/401), covering nine heterogeneous domains (e.g., academic posters, product UIs, scanned news; information acquisition requiring both textual and visual retrieval). Tasks include varying degrees of multi-hop reasoning (20% 1-hop, 33% 2-hop, 47% >2-hop), and the data encompasses a spectrum from closed-ended to open-ended queries (76% versus 24%).

Evaluation protocols assess both answer accuracy (token-level F1; claim-level precision/recall) and planning quality (BLEU, ROUGE, F1 on reformulated queries), as well as planning cost (token-overhead ratio during planning relative to final answer length). Annotation chains are derived via GPT-4o and human curation, providing ground-truth planning trajectories for supervised or comparative evaluation (Yu et al., 26 Jan 2025).

5. Empirical Results and Comparative Performance

Experiments conducted over CogBench show that CogPlanner outperforms both retrieval-agnostic and single-step retrieval baselines across all tested LLMs (e.g., GPT-4o):

Model	Token-F1	Claim-Precision	Claim-Recall
GPT-4o (no retrieval)	0.212	0.291	0.389
GPT-4o + fixed image retrieval	0.193	0.270	0.337
GPT-4o + fixed text retrieval	0.183	0.248	0.233
CogPlanner (parallel, GPT-4o)	0.227	0.342	0.396
CogPlanner (sequential, GPT-4o)	0.235	0.362	0.405

CogPlanner achieves +12–15 percentage points (pp) improvement in claim-level F1 and +10 pp in token-level F1 over the best fixed-retrieval pipeline. The largest gains manifest on queries involving >2-hop reasoning (+18 pp claim F1). Moreover, the lightweight CogPlanner expert, fine-tuned from Qwen2-7b-VL, evaluates with <10% planning token overhead, ensuring practical integrability (Yu et al., 26 Jan 2025).

6. Ablation Studies, Limitations, and Future Directions

Ablation studies confirm that:

• Iterative, multi-step planning and query reformulation deliver 10–20% BLEU/F1 improvement in planning quality over non-planning prompt-based approaches.
• Parallel and sequential modeling are approximately equivalent in answer F1, with the parallel approach being ~20% more cost-efficient with respect to token overhead.

Limitations include:

• Over-search: Large MLLMs tend to overplan, invoking redundant retrievals.
• Output sensitivity: Downstream answer quality is bounded by retriever fidelity; noisy or off-target retrieved snippets degrade response accuracy.
• Planner capacity: Quality of planning is ultimately limited by the MLLM used to instantiate the Planning Expert.

Key future directions proposed:

• Learning lightweight, specialized policy networks via reinforcement learning to replace MLLM-based planners and further reduce computational cost.
• Joint optimization of reformulation and retrieval embedding functions for better retrieval accuracy.
• Broadening to richer modality retrieval (e.g., video, tables), and integrating cost–accuracy trade-offs via end-to-end reward design (Yu et al., 26 Jan 2025).

7. Context and Significance

CogPlanner operationalizes MRAG Planning by tightly coupling multimodal query reformulation with explicit, iterative retrieval-action selection, leveraging powerful MLLMs as both reasoning and control agents. This architecture departs from rigid, static retrieval paradigms by modeling the information acquisition process as a Markovian control problem under resource budgets, thus enabling both adaptive and efficient evidence gathering.

CogBench establishes a new empirical standard for evaluating planning-aware MRAG agents. The demonstrated gains in multi-hop benchmarking tasks and the scalability of the approach across strong base MLLMs indicate CogPlanner’s robust applicability to real-world deployment scenarios where retrieval adaptivity and efficiency are of critical importance (Yu et al., 26 Jan 2025).