Visual Memory Retriever (VMR): Mechanisms & Applications
- Visual Memory Retriever (VMR) is a family of methods that constructs, stores, and retrieves visual or multimodal observations to support downstream tasks.
- It integrates retrieval from diverse memory substrates, ranging from explicit image‐text banks to latent feature maps, to guide generation and navigation modules.
- Empirical outcomes reveal that VMR techniques improve performance in visual question answering, video understanding, and navigation while reducing computational costs.
Visual Memory Retriever (VMR) denotes a family of mechanisms that construct a memory from visual or multimodal observations and retrieve evidence judged relevant to a downstream task. In REVEAL, VMR appears within an end-to-end Retrieval-Augmented Visual LLM whose memory stores image-text pairs, question answering pairs, and knowledge graph triplets (Hu et al., 2022). Later work uses the same label for event-fragment selection in video understanding (Wu et al., 9 Oct 2025), continual multimodal experience recall in vision-and-language navigation (Li et al., 9 Mar 2026), and training-free streaming-memory retrieval under a fixed token budget (Wu et al., 8 May 2026). Across these uses, the recurring pattern is retrieval from a visual or multimodal memory substrate rather than a single fixed architecture.
1. Scope, nomenclature, and problem settings
The term is used across several problem classes. In retrieval-augmented vision-language pre-training, the memory contains world knowledge and is queried to answer knowledge-intensive queries (Hu et al., 2022). In efficient video understanding, the retriever is the front end of a “retrieve-then-compress” pipeline that selects semantically coherent clips before token compression (Wu et al., 9 Oct 2025). In vision-and-language navigation, it retrieves prior experiences indexed by panoramic observations and landmarks, then injects the retrieved result as a navigation rule into a language-model prompt (Li et al., 9 Mar 2026). In streaming video understanding, it mediates between query-agnostic memory construction and query-aware frame selection under a bounded token budget (Wu et al., 8 May 2026).
A distinct but related line frames VMR as a formal memory-retrieval subproblem in embodied settings. The Visually Grounded Memory Assistant formulation defines visual observations , semantic features , and an indexed memory with , where encodes visual-semantic information and stores location (Hahn et al., 2022). The same source separates memory retrieval from answer prediction and explicitly notes that its simplified baseline does not build an end-to-end retrieval-plus-answering pipeline.
The label is also broadened beyond explicit external memories. In “Memory Disentangling,” the discussion refers to recovering current and past semantic content from a single fMRI volume by splitting the representation into “now” and “before” components (Xia et al., 2024). In MemVR and Vision Remember, the retrieved object is not a database entry but condensed visual evidence reinjected into a multimodal decoder during inference (Zou et al., 2024), or multi-level feature maps resampled between decoder layers (Feng et al., 4 Jun 2025). This suggests that the literature uses VMR both for explicit indexed memory retrieval and for latent visual re-access inside a model’s computation.
The acronym is not semantically unique. “Mamba-VMR: Multimodal Query Augmentation via Generated Videos for Precise Temporal Grounding” uses VMR to mean Video Moment Retrieval, with the goal of predicting a temporal span in an untrimmed video rather than retrieving a memory entry (Sun et al., 23 Mar 2026). Any technical reading of “VMR” therefore depends on the paper-specific definition.
2. Memory substrates and indexing schemes
The memory substrate varies from unified dense vectors to structured experience stores and rendered optical archives.
| System | Memory contents | Indexing form |
|---|---|---|
| REVEAL (Hu et al., 2022) | image-text pairs, QA pairs, KG triplets | large memory bank of |
| MARC (Wu et al., 9 Oct 2025) | event fragments from 1 fps video | vector-search library, e.g. Faiss |
| CMMR-VLN (Li et al., 9 Mar 2026) | viewpoint panoramas, landmark texts, trajectories/errors | FAISS over CLIP image-plus-text embeddings |
| VisualMem (Nguyen et al., 27 May 2026) | entity slots, fact slots, verbalized facts | Visual Reference Table, EntityTable, FactTable, graph |
| SAVEMem (Wu et al., 8 May 2026) | short-, mid-, and long-term streaming buffers | three-tier FIFO memory under token budget |
| OCR-Memory (Li et al., 29 Apr 2026) | rendered trajectory chunks with segment IDs | memory bank |
REVEAL gives the clearest external-memory specification (Hu et al., 2022). For an image-text pair 0, it computes 1 and 2, then stores
3
For QA pairs it encodes 4 and 5 and projects them with 6, while KG triplets are embedded as 7 and projected with 8. The paper specifies 9 for ViT-Base, 0, unified memory dimension 1, tied biases 2, and a memory bank of size 3 with “e.g. 4 entries.”
MARC uses event segmentation as memory construction rather than storing every frame (Wu et al., 9 Oct 2025). A pre-trained TransNet V2 detector scans the video sampled at 1 fps and predicts temporal boundaries 5; each interval 6 becomes an “event fragment.” Each clip 7 is embedded as 8 by a visual encoder 9, the query is embedded as 0 by a text encoder 1, and all 2 are pre-indexed with a vector-search library. The paper sets 3, matching CLIP’s default.
CMMR-VLN organizes memory as a collection of memory-units, one per discrete viewpoint in the Matterport3D environment (Li et al., 9 Mar 2026). Each unit stores a viewpoint ID, a panoramic SkyBox image, and Detic-derived salient-landmark texts. Indexing uses a joint CLIP embedding
4
with FAISS entries pointing back to the underlying memory units. Memory update is selective: successful episodes store full trajectories and reasoning sequences, while failures store the first wrong decision with an error tag MRD, FGR, or PGC.
VisualMem makes the structured-memory design explicit (Nguyen et al., 27 May 2026). Its visual store separates explicit evidence, represented as recurring entities with embeddings and metadata, from implicit evidence, represented as inferred latent facts with confidence. The paper proposes an EntityTable, a FactTable, and a graph 5 over entities, facts, and sessions. Unlike caption-only storage, it aims to preserve identity, ownership, and durable user facts.
OCR-Memory replaces vector memory with optical archival memory (Li et al., 29 Apr 2026). Agent trajectories are chunked into text segments, rendered into images, and annotated with Set-of-Mark bounding boxes carrying unique integer IDs. The stored item is
6
where 7 stores timestamps and episode identifiers. The images may be stored at an “aging” resolution 8.
A separate branch of work internalizes the memory substrate. VisMem adds invocation tokens and two latent memory formers, one short-term and one long-term, that generate memory tokens on demand from current multimodal hidden states (Yu et al., 14 Nov 2025). Vision Remember caches multi-level feature maps from the frozen vision encoder, such as layers 9 of CLIP-ViT-L/14, and treats them as a feature memory that can be resampled later in the decoder (Feng et al., 4 Jun 2025).
3. Retrieval operators and scoring rules
Despite the diversity of memory substrates, the scoring mechanisms fall into a small number of patterns: dot-product retrieval, cosine retrieval, hybrid multimodal scoring, late interaction, region classification, and uncertainty-triggered lookup.
REVEAL forms a joint query vector
0
and scores each memory slot by scaled dot-product,
1
with a contrastive-softmax retrieval loss over one positive and in-batch plus random memory-bank negatives (Hu et al., 2022).
MARC and CMMR-VLN both use cosine similarity in a shared embedding space, but at different granularities. MARC computes
2
for event fragments and keeps 3 with 4 as the reported best balance between pruning and coverage (Wu et al., 9 Oct 2025). CMMR-VLN first fuses the candidate next-view panoramas by instruction-aware attention,
5
then retrieves the top-6 memory unit by cosine similarity 7; the paper states that in practice 8 is sufficient (Li et al., 9 Mar 2026).
VisualMem defines an explicitly multimodal score (Nguyen et al., 27 May 2026):
9
and also gives the equivalent form 0. This formulation treats retrieval as fusion over text and image embeddings rather than purely visual nearest-neighbor search.
SAVEMem uses token-level semantic salience and late interaction rather than whole-item similarity (Wu et al., 8 May 2026). For a visual token 1 in frame 2, semantic importance is defined by a fixed pseudo-question bank 3:
4
At query time the short-term buffer is tested by
5
and the anchor-conditioned recency gate is
6
If the gate is closed, candidate frames in 7 are ranked by ColBERT-style late interaction,
8
OCR-Memory converts retrieval into binary region selection (Li et al., 29 Apr 2026). For each annotated segment 9, the retriever outputs logits 0 and 1, which are turned into a calibrated relevance probability
2
Selected regions satisfy thresholding and TopK fallback,
3
Retrieval therefore identifies pointers to stored text rather than generating a textual summary.
MemVR uses uncertainty itself as the retrieval trigger (Zou et al., 2024). At intermediate layer 4, it computes
5
and when 6, reinjects visual key-value memory at layer 7. The lookup term
8
retrieves condensed visual evidence from projected keys and values.
Vision Remember performs local cross-attention into multi-level feature memory (Feng et al., 4 Jun 2025). Each compressed vision token 9 is projected to 0, assigned a local window 1 in the feature tensor 2, and updated by scaled dot-product attention with saliency-weighted values:
3
The paper characterizes this as saliency-enhancing local attention.
4. Integration with generation, compression, navigation, and decoding
In most systems, retrieval is not the endpoint; it is a control signal for another module.
REVEAL couples retrieval directly to a Transformer decoder (Hu et al., 2022). At generation step 4, the decoder cross-attends to the query representation 5 and the top-6 retrieved memory vectors 7, with keys and values formed by 8. The decoder is trained with teacher forcing to maximize
9
The retrieved memory is thus fused into token generation rather than appended as plain text.
MARC uses retrieval as the first half of a retrieve-then-compress pipeline (Wu et al., 9 Oct 2025). The top-0 event fragments are concatenated in temporal order to form the retrieved video memory, then passed to the memory-aware temporal compressor and finally into the Student LLM policy trained via C-GRPO. The paper states that no further attention or weighting is done at the retrieval stage; downstream modules treat all 1 clips equally.
CMMR-VLN embeds retrieval in a Retrieval-Augmented Generation Pipeline (RAGP) (Li et al., 9 Mar 2026). At each navigation step, the agent collects the instruction text 2, candidate next-view panoramas 3, history 4, and current semantic topological map 5; retrieves the best-matching memory unit 6; transforms it into a natural-language navigation rule 7; inserts 8 into a Prompt Manager 9 as a high-priority constraint; and calls the LLM to produce Analysis, Planning 0, and action 1. The retrieved memory is therefore operationalized as an explicit rule rather than a dense vector.
OCR-Memory also avoids free-form text generation at the retrieval stage (Li et al., 29 Apr 2026). Its locate-and-transcribe pipeline retrieves region indices, then fetches the exact stored text
2
Because “no free-form decoding happens,” the retrieved evidence remains verbatim. This design is specifically motivated as a way to reduce hallucination.
VisMem internalizes retrieval as latent-token insertion (Yu et al., 14 Nov 2025). When the model emits 3 or 4, a Query Builder 5 turns the current multimodal hidden states into a query 6, and the corresponding memory former 7 or 8 generates 9 or 00 memory tokens that are spliced into the decoder context between the invocation and end tokens. The base VLM remains frozen; the inserted memories enrich subsequent generation.
MemVR and Vision Remember take yet another integration route. MemVR reinjects condensed visual key-value memory into the feed-forward sublayer of the triggered decoder layer, producing a “look-twice” effect when uncertainty rises (Zou et al., 2024). Vision Remember inserts an “in-between” module after decoder layer 1 and again before decoder layer 4, so that compressed vision tokens can re-memorize high-resolution vision features multiple times during decoding (Feng et al., 4 Jun 2025). In both cases the integration point is the decoder’s hidden-state path rather than an external prompt or retrieved text block.
5. Learning objectives, update policies, and maintenance regimes
The training regimes range from fully end-to-end optimization to frozen retrieval modules and entirely training-free deployment.
REVEAL jointly optimizes memory encoding, retrieval, and generation (Hu et al., 2022). Its total loss is
01
with all 02 typically set to 03. The retriever uses a standard contrastive-softmax InfoNCE loss, and the paper describes the system as pre-trained end-to-end on a massive amount of data.
MARC trains the embedding space offline, not during RL compression (Wu et al., 9 Oct 2025). Given a mini-batch of 04 query-positive-fragment pairs, it minimizes an InfoNCE objective with temperature 05, and reports that 06 works well, outperforming 07 on downstream recall. Once trained, 08 and 09 are frozen during C-GRPO; there is no back-propagation from C-GRPO into VMR.
CMMR-VLN is deployed in a zero-shot, training-free regime (Li et al., 9 Mar 2026). CLIP, Detic, and GPT-4o are pre-trained, there is no additional back-propagation or loss term for retrieval or reflection during navigation, and the only learned parameter in retrieval is the projection matrix 10 used in attention fusion, whose training lies outside the VLN loop. Its distinctive contribution is not an optimization objective but a reflection-based memory update strategy: success stores complete successful paths and reasoning sequences, filtered to keep shorter trajectories; failure stores the first wrong decision together with an error-type tag.
SAVEMem is also explicitly training-free (Wu et al., 8 May 2026). Its maintenance policy is algorithmic: new frames enter the short-term buffer, old short-term frames are moved to mid-term memory via Temporal Semantic Pruning, old mid-term frames move to long-term memory via Spatial Semantic Selection, and Selective Forgetting removes the least semantically salient long-term tokens until the global token budget 11 is satisfied. The architecture is query-agnostic at memory-construction time and query-adaptive only at retrieval time.
Memory Disentangling uses a hybrid objective to separate current from past semantic content in fMRI (Xia et al., 2024). It combines
12
with 13 and best performance reported at 14. The encoder 15 produces two branches, 16 and 17, and the positive pair in contrastive learning is 18.
VisMem uses a two-stage GRPO recipe (Yu et al., 14 Nov 2025). Stage I optimizes memory formation by training the Query Builder and the short- and long-term memory formers to maximize task reward relative to a no-memory baseline. Stage II freezes the formers and trains only the policy parameters governing invocation tokens, penalizing wrong memory choice and harmful invocations. This decomposition separates memory content creation from memory invocation.
The Visually Grounded Memory Assistant baseline demonstrates a different learning target: predicting whether a human participant will answer immediately, self-navigate, request assistance, or answer correctly (Hahn et al., 2022). It uses hand-selected visual and semantic features, binary cross-entropy for the MLP, and an 80/20 train-test split with no question template repetition across the split. The paper’s summary concludes that future systems should jointly learn retrieval and assistance policies end-to-end.
6. Empirical outcomes, recurring findings, and open limitations
Empirical results show that VMR-style mechanisms improve a wide range of downstream tasks, but the gains depend strongly on the memory substrate and the retrieval target.
REVEAL reports state-of-the-art results on visual question answering and image captioning (Hu et al., 2022). MARC shows that VMR alone, without RL-based compression, improves Qwen2.5-VL-3B from 42.21 to 45.56 mean accuracy when uniform frame sampling is replaced with VMR retrieval, and on MVBench from 44.77 to 57.24, a +12.47 absolute gain (Wu et al., 9 Oct 2025). The same paper reports that full MARC achieves near-baseline accuracy using only one frame’s tokens, reducing visual tokens by 95%, GPU memory by 72%, and latency by 23.9%.
CMMR-VLN reports large navigation gains in both simulation and real-world testing (Li et al., 9 Mar 2026). On the R2R “Unseen” validation split in zero-shot mode, it improves success rate over NavGPT from 34 to 52, over MapGPT from 43 to 52, and over DiscussNav from 43 to 52, with SPL increasing from 29 to 51 against NavGPT. In real-world tests on TurtleBot 4 Lite with 20 instructions, success rate rises from 10% for NavGPT and 20% for MapGPT and DiscussNav to 30%.
VisualMem argues that personal visual memory is distinct from text memory and backs that claim with benchmark results (Nguyen et al., 27 May 2026). On its Personal Visual Memory benchmark, Full Context gives 95.1% with 325K tokens, Oracle gives 98.6% with 1.9K tokens, the best RAG-based system is approximately 27%, MemOS reaches 56.0% with 1.2K tokens, and VisualMem reaches 84.1% with 1.98K tokens. On LOCOMO and PersonaMem, it remains competitive with its text-memory backend, improving MemOS from 56.8% to 58.1% and from 45.5% to 46.3%, respectively.
SAVEMem shows that training-free streaming memory management can still yield substantial gains (Wu et al., 8 May 2026). On OVO-Bench with Qwen2.5-VL-7B, overall score improves from 52.27 to 62.69; Real-Time Visual Perception rises from 59.90 to 74.93, and Backward Tracing from 44.65 to 50.44. Peak GPU memory at 128 frames drops from 35.8 GB to 18.5 GB, a 48% reduction. Its ablations show Stage 1 only at 57.79, Stage 2 only at 59.95, and the full two-stage design at 62.69; random vectors as pseudo-question prior score 56.05, a single prompt 59.73, and the five-prompt bank 62.69.
MemVR addresses hallucination rather than retrieval accuracy in the usual sense (Zou et al., 2024). On POPE with LLaVA-7B it reports gains of +5.7 Acc and +6.2 F1 on MSCOCO, +7.0/+7.5 on A-OKVQA, and +6.2/+6.4 on GQA. On CHAIR it reduces CHAIR_S by 3.4 and CHAIR_I by 2.4 while increasing Recall by 3.7. Its efficiency table shows token-generation latency on an NVIDIA A40 of 1880 ms for baseline LLaVA and 1861 ms for MemVR on 20 tokens, compared with 4537 ms for VCD and 6243 ms for OPERA.
Vision Remember targets visual forgetting induced by aggressive token compression (Feng et al., 4 Jun 2025). With an adaptive-average-pooling projector, the average over 13 benchmarks rises from 44.8 to 47.1, with especially large gains on ChartQA, 40.7 to 48.7; DocVQA, 45.4 to 50.7; TextVQA, 42.1 to 48.6; OCRBench, 30.5 to 34.8; and RealWorldQA, 45.4 to 49.2. Time-to-first-token on an A100 is 48.5 ms for the compressed 144-token baseline and 52.1 ms with Vision Remember, only +3.6 ms overhead.
OCR-Memory shows that visual archival memory can exceed text-only baselines under strict context limits (Li et al., 29 Apr 2026). Under a 4k-token context limit on Mind2Web, OCR-Memory reaches 53.8% element accuracy, 59.2% F1, 46.1% step success rate, and 4.8% task success rate, outperforming Zero-Shot, Retrieval, MemoryBank, and AWM. Its ablations show that removing Set-of-Mark and using text generation drops element accuracy to 46.5% and raises latency from 1.7 s to 5.3 s. In Needle-in-Haystack evaluation, Recall@1 is 98.5% at 4k context length and 94.1% at 32k, with compression around 10.3× to 10.6×.
Memory Disentangling provides a different empirical perspective: retrieval of recent semantic content from a single fMRI snapshot (Xia et al., 2024). It reports best current-image CIDEr of approximately 39.9% versus 34.3% for the straightforward MSE-only baseline, while past-image decoding remains challenging at below 14% CIDEr. The Visually Grounded Memory Assistant baseline likewise shows partial predictability rather than full retrieval competence: on test-set AUC-ROC, the MLP achieves 0.57 for “No navigation,” 0.59 for “Assistance requested,” and 0.67 for “Answered correctly,” while all models remain near chance for “Self navigation” (Hahn et al., 2022).
Several limitations recur across the literature. Generated or low-quality visual priors can mislead later stages: Mamba-VMR notes dependence on the quality of generated videos and possible LLM bias in subtitle selection (Sun et al., 23 Mar 2026). Memory Disentangling notes that decoding of true past images remains weak and often collapses to generic labels (Xia et al., 2024). Vision Remember leaves scaling to 7B or 72B models, dynamic window sizing, and video or audio modalities for future work (Feng et al., 4 Jun 2025). OCR-Memory notes higher disk or storage cost, slightly increased retrieval latency, and pipeline complexity (Li et al., 29 Apr 2026). These results suggest that VMR is most mature when retrieval targets are semantically coherent, storage is structured, and the downstream interface preserves retrieved evidence rather than forcing it through lossy summarization.