MemVerse: Hybrid Memory Framework

Updated 4 July 2026

MemVerse is a hybrid memory framework that integrates retrieval-based long-term memory with fast parametric memory to combat catastrophic forgetting and support lifelong multimodal interaction.
It employs a structured architecture with short-term sliding windows, hierarchical knowledge graphs for durable storage, and a rule-based orchestrator for memory consolidation.
Empirical evaluations on benchmarks like ScienceQA and MSR-VTT reveal notable accuracy improvements and reduced latency compared to conventional memory retrieval methods.

Searching arXiv for the MemVerse paper and closely related context papers to ground the article. MemVerse is a memory framework for multimodal AI agents that combines explicit retrieval-based long-term memory with a lightweight parametric memory model, with the stated aim of addressing catastrophic forgetting, weak long-horizon reasoning, incoherent multimodal interaction, inefficient retrieval, and the lack of interpretability and control in purely parametric memory. It is presented as a model-agnostic, plug-and-play system in which recent context is handled by short-term memory, durable knowledge is organized as hierarchical knowledge graphs, and selected long-term knowledge is periodically distilled back into model parameters for fast recall (Liu et al., 3 Dec 2025).

1. Problem setting and design thesis

MemVerse is motivated by the claim that current LLMs, vision-LLMs, and multimodal agents are largely stateless. Even when such systems appear to remember, that memory is either embedded implicitly in pretrained parameters or approximated through retrieval over raw logs. In the MemVerse formulation, neither strategy is sufficient for lifelong multimodal interaction.

The paper characterizes purely parametric memory as fixed-capacity, expensive to update, vulnerable to interference between old and new knowledge, and opaque. It characterizes purely retrieval-based memory as decoupled from model parameters but typically reliant on raw text chunks, logs, or documents that become noisy, redundant, and expensive as the store grows. Raw logs are also described as weak for multi-hop reasoning, relational inference, compact summarization, and cross-modal grounding.

The central thesis is therefore a hybrid “fast and slow” memory design. Slow, structured retrieval provides durable and controllable memory; fast parametric recall provides low-latency access to compressed knowledge. This hybridization is the framework’s defining claim: lifelong multimodal agents require explicit external memory and periodic neural distillation rather than only more parameters or larger context windows.

2. Architectural organization

MemVerse has three principal memory components plus a controller: Short-Term Memory (STM), Long-Term Memory (LTM), Parametric Memory, and a Memory Orchestrator. The orchestrator is explicitly described as rule-based and as introducing no additional trainable parameters (Liu et al., 3 Dec 2025).

Component	Role	Stated organization
STM	Recent conversational or interaction context	Sliding window
LTM	Explicit retrieval-based memory	Core, episodic, semantic knowledge graphs
Parametric Memory	Distilled fast recall	Lightweight neural model
Memory Orchestrator	Coordinates memory operations	Rule-based, no additional trainable parameters

STM stores the recent interaction history in a sliding window. Given dialogue turns $\{q_1, q_2, \dots, q_t\}$ , it is defined as

$\mathcal{M}_{\text{STM}} = \{ q_{t-K+1}, q_{t-K+2}, \dots, q_t \}.$

The design intention is to avoid writing every turn into durable storage and to avoid unnecessary long-term retrieval when the answer is still available in local context.

LTM is defined as

$\mathcal{M} = \Big(\{\mathcal{G}_k\}, \mathcal{C}\Big), \quad k \in \{\text{core, episodic, semantic}\},$

where each submemory graph is

$\mathcal{G}_k = (\mathcal{V}_k, \mathcal{R}_k).$

The three-way partition is functional. Core memory stores durable, user-specific facts and preferences. Episodic memory stores time-ordered event-based interactions as detailed entries. Semantic memory stores abstract, generalizable knowledge about entities, concepts, and objects.

The paper uses model-agnostic to mean that the framework can be instantiated from any pretrained multimodal or LLM. It uses plug-and-play to mean that MemVerse can be attached to an existing model as an external memory service or interface rather than by redesigning transformer internals or retraining the backbone from scratch.

3. Representation, consolidation, and retrieval

A central design choice in MemVerse is to convert arbitrary multimodal experiences into a unified textual substrate and then organize abstractions of that text into structured graphs. In the implementation described in the paper, GPT-4o-mini is used to caption images, Whisper is used for audio transcription, and videos are handled through frame sampling plus a VLM captioning step. The resulting text chunk becomes the canonical symbolic description of the original multimodal input.

The long-term store does not remain a flat collection of chunks. Instead, the chunk set $\mathcal{C} = \{c_i\}$ is compressed by an LLM into salient memory descriptions and then structured into a Multimodal Knowledge Graph (MMKG) with nodes for entities, concepts, and events and typed edges for relations. The graph remains grounded through support mappings:

$\ell_v : \mathcal{V} \to \mathcal{P}(\mathcal{C}), \quad \ell_v(v)=\{\text{Chunks supporting } v\},$

$\ell_r : \mathcal{R} \to \mathcal{P}(\mathcal{C}), \quad \ell_r(r)=\{\text{Chunks supporting } r\}.$

This means that symbolic graph items remain traceable to textual evidence and, through that, to the original multimodal source.

The orchestrator governs addition, update, deletion, and retrieval, along with consolidation and summarization. Consolidation occurs periodically or when enough new information has accumulated. The stated write path is: recent chunks are compressed into salient memory descriptions, entities and relations are extracted by an LLM, the relevant memory type is identified, the relevant subgraph is updated, and support links are stored. Read access can then proceed through STM, explicit LTM retrieval, or parametric memory.

Retrieval is described as graph-based and hierarchical, grounded by chunk links and used in a retrieval-augmented generation style. The retrieval pipeline is: receive query $q$ , consult STM if applicable, query the appropriate LTM graph or graphs, activate relevant entities and relations, recover support chunks and associated multimodal data, and feed the retrieved memory back into the backbone model for answer generation or action. The paper does not specify a low-level retrieval score such as BM25, cosine similarity, random walk, or a graph-neural objective.

The framework repeatedly claims adaptive forgetting and bounded memory growth, but the visible text does not provide a forgetting score, retention threshold, merge rule, or decay schedule. What is explicit is that bounded growth is supported by STM-mediated avoidance of redundant writes, periodic consolidation, compression into salient descriptions, graph abstraction, and orchestrated deletion.

4. Periodic distillation and learning regime

One of MemVerse’s distinctive claims is its periodic distillation mechanism, which compresses essential knowledge from long-term explicit memory into a lightweight parametric model. The parametric memory is initialized from pretrained weights,

$\mathcal{M}_{\text{parametric}}^{0} = \Theta_{\text{pretrained}},$

and updated incrementally as

$\mathcal{M}_{\text{parametric}}^{t+1} = \mathcal{M}_{\text{parametric}}^{t} + \Delta \Theta_t.$

The supervision for this module is not taken directly from raw environment data. Instead, it is generated from explicit memory retrieval: for each instance, the input is a question $\mathcal{M}_{\text{STM}} = \{ q_{t-K+1}, q_{t-K+2}, \dots, q_t \}.$ 0 and the target is retrieved memory content $\mathcal{M}_{\text{STM}} = \{ q_{t-K+1}, q_{t-K+2}, \dots, q_t \}.$ 1. The training objective is token-level cross-entropy,

$\mathcal{M}_{\text{STM}} = \{ q_{t-K+1}, q_{t-K+2}, \dots, q_t \}.$ 2

with the parametric model learning to reproduce retrieved content. The paper presents this as teaching the model to emulate retrieval behavior.

Distillation is described as periodic, not continuous. The stated rationale is reduced compute overhead and more stable updates. The visible text does not provide a fully formal update schedule, although it notes that update interval studies were performed on ScienceQA.

The reported training setup uses Qwen2.5-7B for the parametric memory experiments, AdamW, learning rate $\mathcal{M}_{\text{STM}} = \{ q_{t-K+1}, q_{t-K+2}, \dots, q_t \}.$ 3, gradient clipping with max norm 1.0, sequence length 2048, bfloat16 / mixed precision, gradient checkpointing, and at most one A100 80G GPU (Liu et al., 3 Dec 2025). The paper contains a textual inconsistency on the scheduler, describing it once as cosine and elsewhere as linear with 10% warm-up.

5. Empirical evaluation

MemVerse is evaluated on ScienceQA, LoCoMo, and MSR-VTT, which the paper uses to cover multimodal reasoning, long-context conversational memory, and video-text retrieval respectively (Liu et al., 3 Dec 2025).

On ScienceQA, which contains 21,208 multimodal science questions with about 48.7% with images, the strongest reported result is GPT-4o-mini + MemVerse: 85.48% average accuracy. This exceeds GPT-4: 82.69, CoT (GPT-4): 83.99, HoT-T5-Large: 83.38, Qwen2.5-72B: 78.37, and the GPT-4o-mini baseline: 76.82. For Qwen models, the gains are smaller: Qwen2.5-7B baseline: 74.72 versus Qwen2.5-7B + MemVerse: 75.62, and Qwen2.5-72B baseline: 78.37 versus Qwen2.5-72B + MemVerse: 80.25. The paper interprets this as evidence that memory quality alone is insufficient; the backbone must also be able to integrate retrieved knowledge into downstream reasoning.

On the same benchmark, the latency comparison is used to motivate periodic distillation. The reported times are RAG: 20.17 s/question, compressed long-term memory retrieval: 8.26 s/question, and parametric memory: 2.28 s/question. The paper summarizes this as about 89% faster than RAG and 72% faster than long-term retrieval, while claiming similar accuracy. It also states that short-term memory contributes relatively little on ScienceQA, because the questions are mostly independent and non-sequential.

On LoCoMo, the benchmark is described as 10 very long-term multimodal dialogues, each around 600 turns and 16,618 tokens on average, spread over up to 32 sessions. It is included as the framework’s long-term conversational-memory evaluation. However, the visible text states that full quantitative and qualitative results are in Appendix C and does not provide the actual numbers.

On MSR-VTT, which contains 10,000 YouTube video clips in 20 categories with 20 human captions per video, MemVerse is evaluated in a test-time memory-augmented retrieval setup in which query text retrieves relevant memory entries from the knowledge graph and the query is rewritten by concatenating retrieved information before final matching. The reported results are text-to-video: R@1 90.4, R@5 95.6, R@10 98.1 and video-to-text: R@1 89.2, R@5 92.7, R@10 99.0, compared with a CLIP baseline of 29.7 / 48.9 / 58.8 for text-to-video and 21.4 / 38.6 / 44.3 for video-to-text. The paper presents these numbers as evidence for the power of memory-based query rewriting and semantic linking, while also noting that a careful reader would likely want more detail on protocol comparability.

6. Interpretation, later comparative context, and limitations

The paper’s main interpretation is that MemVerse succeeds by turning raw experience into structured memory, separating recent context from persistent knowledge, and combining explicit external memory with distilled parametric memory. It also stresses that good memory does not guarantee good reasoning: GPT-4o-mini benefited much more than Qwen in the reported experiments, which the paper attributes to stronger integration of retrieved evidence into reasoning chains.

The trade-offs are stated explicitly. External memory is interpretable, controllable, expandable, and grounded, but slower. Parametric memory is fast and differentiable, but compressed and less transparent. Compression into graph abstractions improves scalability and retrieval efficiency, but may lose detail. The framework is modular and model-agnostic, but it adds orchestration, graph maintenance, multimodal captioning or transcription, and periodic fine-tuning overhead.

Several limitations are either stated directly or made explicit by omission. The paper says it plans to explore more adaptive memory-control strategies and to deploy MemVerse in open-world environments across a variety of domains. It also makes clear that the visible text does not specify a rigorous forgetting policy, does not provide a detailed component ablation for every claimed module, and leaves some details of the update schedule and retrieval scoring mechanism unspecified. The multimodal pipeline is text-first, which the paper itself implies may lose fine-grained perceptual information.

Later work places MemVerse in a broader comparative landscape. In M $\mathcal{M}_{\text{STM}} = \{ q_{t-K+1}, q_{t-K+2}, \dots, q_t \}.$ 4Exam, MemVerse appears as a multimodal memory baseline with Overall: 0.4481, including FM EM: 0.5474, MR LLM-J: 0.5613, and II LLM-J: 0.6048, which indicates competitive performance on some multimodal memory categories but persistent weakness on the hardest interpretive tasks (Huang et al., 5 Jun 2026). In Omni-SimpleMem, MemVerse is characterized as using hierarchical episodic-semantic memory and a multimodal knowledge graph, with three LLM calls per ingested item and an ingestion rate of 0.22 items/sec, and is positioned as a more structured but heavier ingestion pipeline than retrieval-centric alternatives (Liu et al., 1 Apr 2026).

Taken together, these later comparisons suggest a stable interpretation of MemVerse. It is best understood as a hybrid explicit-memory architecture for lifelong multimodal agents: recent context is maintained in a sliding window, durable knowledge is organized into core, episodic, and semantic graphs, and selected long-term knowledge is periodically distilled into a lightweight neural memory. A plausible implication is that its enduring significance lies less in any single benchmark number than in the design pattern it crystallizes: external structured memory plus periodic parametric compression as a middle path between flat retrieval stores and fully parametric continual learning.