VimRAG: Multimodal Retrieval & Reasoning

• VimRAG is a multimodal framework that uses a dynamic memory graph to organize and integrate evidence from text, images, and videos.
• It employs graph-modulated visual memory encoding with adaptive token allocation to prioritize and preserve high-value visual evidence.
• It integrates graph-guided policy optimization for precise credit assignment, achieving state-of-the-art performance on multimodal RAG benchmarks.

VimRAG is a multimodal retrieval-augmented reasoning framework for long-horizon agents that must retrieve, reason over, and understand text, images, and especially videos under severe token constraints. It was introduced to address a failure mode of standard RAG and memory-augmented agent pipelines: linear interaction histories break down when evidence is visually heavy, semantically sparse, and accumulated over many iterative steps. In place of flat trajectory concatenation, VimRAG models reasoning as a dynamic multimodal memory graph, couples that graph with adaptive visual token allocation, and optimizes the policy with graph-aware credit assignment. Reported experiments show state-of-the-art performance across multimodal RAG benchmarks, with overall accuracy rising from 40.6 to 45.2 on Qwen3-VL-4B-Instruct and from 43.6 to 50.1 on Qwen3-VL-8B-Instruct (Wang et al., 13 Feb 2026).

1. Problem setting and motivation

VimRAG is designed for multimodal retrieval-augmented reasoning settings in which an agent repeatedly searches, inspects, and verifies evidence across text, images, and video. The paper identifies a central systems problem: naïvely appending all retrieved observations to a linear context causes context explosion, repeated searches, and “state blindness,” while aggressive compression of visual evidence into text can erase fine-grained details that are required for verification (Wang et al., 13 Feb 2026).

This design target is particularly relevant when the evidence is token-heavy but information-sparse. Video is the clearest case in the paper’s framing: long sequences consume substantial context budget, yet only a small subset of frames may matter for the answer. A linear memory therefore fails in two opposite ways. It either stores too much, making the agent lose track of what it already did, or stores too little, producing a modality gap between the original visual evidence and its textual surrogate.

The paper’s response is structural rather than purely compressive. Instead of asking the model to remember a chronological transcript, VimRAG asks it to maintain an explicit reasoning state in which retrieval actions, summaries, and multimodal evidence are organized by causal and temporal relations. This makes the core problem one of multimodal working memory, not merely prompt packing.

2. Memory-graph formalism and inference workflow

The reasoning process is modeled as a dynamic directed acyclic graph (DAG) $\mathcal{G}_t=(\mathcal{V}_t,\mathcal{E}_t)$ that evolves step by step (Wang et al., 13 Feb 2026). Each node is an epistemic state,

$v_i \triangleq (p_i, q_i, s_i, m_i),$

where $p_i$ encodes parent indices, $q_i$ is the decomposed sub-query, $s_i$ is a concise textual summary, and $m_i$ is the multimodal episodic memory bank containing retrieved visual tokens or other evidence. Edges $\mathcal{E}_t=\{(v_j,v_i)\mid j<i\}$ encode the causal or temporal reasoning flow.

Inference is framed as a POMDP. The policy $\pi_\theta$ operates over the current graph state and selects among retrieval, memory-populating, and answer actions. In operational terms, the paper describes three phases. First, the agent performs exploratory expansion: if evidence is insufficient, it creates a search node and retrieves raw multimodal observations $\mathcal{O}_t$. Second, it performs multimodal perception and memory populating: retrieved items are distilled into a node summary $s_t$ and memory $v_i \triangleq (p_i, q_i, s_i, m_i),$0 with coarse-to-fine filtering. For video, this includes temporally grounded frame selection so that relevant frames are associated with timestamps. Third, once the graph contains enough evidence, the agent emits an answer node, and the critical path from root to answer becomes the solution trace.

The graph is explicitly not treated as passive storage. The paper characterizes it as the agent’s working memory and reasoning scaffold. In that role, it supports branch-sensitive reasoning: the agent can distinguish dead-end branches from promising lines of inquiry and can avoid redundant loops. This topology is also the substrate for later memory allocation and policy optimization.

3. Graph-Modulated Visual Memory Encoding

The paper presents Graph-Modulated Visual Memory Encoding as the solution to the “visual memory resolution dilemma”: important evidence should be preserved at high resolution, while trivial or redundant clues should be compressed or discarded (Wang et al., 13 Feb 2026). This is formulated as a resource-allocation problem over visual tokens.

Each retrieved visual item $v_i \triangleq (p_i, q_i, s_i, m_i),$1 in node $v_i \triangleq (p_i, q_i, s_i, m_i),$2 is assigned an energy score. Its intrinsic energy depends on three factors named explicitly in the paper: normalized fine-grained semantic priority $v_i \triangleq (p_i, q_i, s_i, m_i),$3, the out-degree $v_i \triangleq (p_i, q_i, s_i, m_i),$4 of the node that contains it, and a temporal decay term $v_i \triangleq (p_i, q_i, s_i, m_i),$5. In effect, semantically salient evidence in structurally central nodes is prioritized, but recency still matters. The reported hyperparameter for temporal decay is $v_i \triangleq (p_i, q_i, s_i, m_i),$6.

VimRAG then augments intrinsic energy with recursive reinforcement from successors:

$v_i \triangleq (p_i, q_i, s_i, m_i),$7

where $v_i \triangleq (p_i, q_i, s_i, m_i),$8 controls feedback strength and $v_i \triangleq (p_i, q_i, s_i, m_i),$9 aggregates child energies. The reported setting is $p_i$0. Functionally, this propagates value backward through the reasoning graph: an apparently modest clue can be preserved more aggressively if it supports a high-value downstream chain.

Token allocation uses the energy ranking to determine which visual items to retain and at what resolution. High-energy evidence receives more vision tokens, whereas low-energy items are compressed more heavily or removed. The paper states that the resulting effect is twofold: token usage becomes more efficient, and retrieval quality improves because the model preserves the exact visual details needed for verification instead of converting everything into lossy text summaries.

A pilot study reported in the paper compares three memory regimes. Purely textual memory is token-efficient but suffers from a modality gap; raw visual memory is too expensive and noisy; semantically related selective visual memory provides the best trade-off. This supports the broader claim that multimodal memory should be selective without becoming text-only.

4. Graph-Guided Policy Optimization and training regime

VimRAG addresses credit assignment with Graph-Guided Policy Optimization (GGPO), motivated by the observation that a single trajectory-level reward is too coarse for agentic RAG (Wang et al., 13 Feb 2026). A correct final answer may include redundant retrieval steps, and an incorrect final answer may still contain useful retrievals. The graph structure is used to disentangle these cases.

Rollouts are segmented into atomic reasoning cycles,

$p_i$1

with $p_i$2. Before optimization, nodes are pruned with the mask

$p_i$3

where $p_i$4 is the critical path from root to answer in positive trajectories, and $p_i$5 are valuable retrieval steps inside negative trajectories. For correct trajectories, dead-end branches are masked so they do not receive positive gradient; for incorrect trajectories, useful retrievals are masked so they are not unfairly penalized.

The optimization objective is a clipped PPO-like loss with the pruning mask. The paper also describes the training narrative as Graph-Guided Rejection Sampling, but states that the implementation is essentially graph-pruned PPO or GSPO-style optimization. The intended effect is finer credit assignment, with gradient flowing only through valid and relevant steps, which the paper associates with faster convergence and improved stability.

Training proceeds in two stages. Supervised fine-tuning is performed with LoRA on the Qwen3-VL base model family. Reinforcement learning uses rLLM with a GSPO-style loss and a small actor learning rate. Reported training settings include LoRA rank 32, SFT cutoff length 16384, RL max prompt length 20240, max response length 512, learning rate $p_i$6 for SFT and $p_i$7 for RL, batch size 64 via gradient accumulation, and experiments on NVIDIA H20-3e 141G GPUs. Dynamic visual allocation is enabled at inference; during training, the system averages pixels in the multimodal memory bank.

5. Evaluation protocol and empirical findings

The evaluation is intentionally unified across modalities and spans text, image-text reasoning, visually rich documents, long video understanding, video-corpus retrieval and generation, and cross-video reasoning (Wang et al., 13 Feb 2026).

Modality emphasis	Benchmarks
General text	HotpotQA, SQuAD
Image-text and visual documents	WebQA, SlideVQA, MMLongBench
Video and cross-video reasoning	LVBench, WikiHowQA, SyntheticQA, XVBench

The search corpus merges about 200k multimodal items, and retrieval uses GVE-7B embeddings to support text, image, and video retrieval. The reward model is Qwen3-Max, which outputs binary correctness labels. Baselines include Vanilla RAG, ReAct, UniversalRAG, VideoRAG, MemAgent, and Mem1.

Against those baselines, VimRAG is reported as the best performer overall on both tested model sizes. The paper highlights overall accuracy improvements from 40.6 to 45.2 on Qwen3-VL-4B-Instruct and from 43.6 to 50.1 on Qwen3-VL-8B-Instruct. It also reports the strongest results on many individual benchmarks, including HotpotQA, SQuAD, SlideVQA, MMLongBench, LVBench, WikiHowQA, SyntheticQA, and XVBench.

The ablations are organized to isolate the three main design decisions. Replacing iterative memory with graph memory produces a substantial boost, which the paper interprets as evidence that explicit topology addresses state blindness. Adding multimodal selective memory further improves performance, indicating that preserving critical visual tokens is preferable to flattening everything into text or storing all raw tokens. Adding energy-based allocation yields the full gain, supporting the claim that dynamic resolution scaling is essential. A separate GGPO ablation shows faster and more stable training than a no-pruning baseline.

6. Relation to adjacent multimodal RAG systems, applications, and limitations

VimRAG belongs to a broader multimodal RAG family, but it is not interchangeable with other systems that use related names. “Visual RAG” is a training-free retrieval-plus-prompting framework for image classification that uses CLIP, FAISS, and Gemini with retrieved $p_i$8 demonstrations; its purpose is adaptation to new visual domains without fine-tuning, not long-horizon multimodal reasoning over a structured memory graph (Bonomo et al., 18 Jan 2025). “Visual-RAG” is a benchmark for text-to-image retrieval augmented generation on visual knowledge intensive questions, centered on whether MLLMs can retrieve and use clue images as evidence (Wu et al., 23 Feb 2025). “Visual RAG Toolkit” is a systems layer for scaling multi-vector visual retrieval with training-free pooling and multi-stage search; it is positioned as an efficiency-enabling component for ColPali-style or VimRAG-style visual retrieval, rather than as a replacement for VimRAG’s memory-graph reasoning mechanism (Yeroyan, 13 Feb 2026).

The paper describes VimRAG as best suited to multimodal tasks where iterative search and verification matter, such as document QA, video understanding, digital assistants, and deep research agents (Wang et al., 13 Feb 2026). This suggests a deployment niche in which the cost of maintaining explicit graph state is justified by the need to preserve branch structure and selectively retain visual detail.

Its limitations are also explicit. VimRAG still depends on a strong base model and a relatively good retriever. Real-time applications may struggle because the system is multi-turn and graph-maintaining. Retrieval errors can still propagate. These caveats delimit the method’s scope: the paper does not present the framework as a universal replacement for simpler RAG pipelines, but as a specialized architecture for long-horizon multimodal reasoning under tight token budgets.

The central insight can be stated in the paper’s own terms: in long-horizon multimodal reasoning, the agent should remember not just facts, but which evidence mattered, where it sits in the reasoning graph, and how much visual detail that evidence deserves (Wang et al., 13 Feb 2026).