Graph-Modulated Visual Memory Encoding

• The paper introduces a graph-governed memory paradigm that decomposes visual input into graph-addressable units, resulting in adaptive, efficient encoding.
• Graph attention and query-conditioned read/write controls are used to modulate memory formation, selectively retaining relevant visual information while reducing redundancy.
• Applications span video adaptation, robotic navigation, and multimodal reasoning, with empirical benchmarks showing significant performance improvements.

Graph-modulated visual memory encoding denotes a family of architectures in which a graph, or a graph-interpretable dependency structure, governs how visual evidence is stored, weighted, updated, compressed, and retrieved. Across the literature, the memory carrier may be a frame graph for video adaptation, an object/frame graph for temporal sentence localization, a semantic-spatial attribute graph for robotics, a topological map for navigation, a directed acyclic graph of agent states for multimodal retrieval-augmented reasoning, a cross-layer recurrent state inside a vision encoder, or a graph-conditioned reservoir for cognitive connectomics (Hu et al., 2022, Liu et al., 2023, Riva et al., 20 Apr 2026, Li et al., 2024, Wang et al., 13 Feb 2026, Liu et al., 28 Feb 2026, Soussia et al., 13 Sep 2025). This suggests that the phrase refers less to a single canonical algorithm than to a recurrent design pattern: graph structure acts as the control surface for visual memory formation.

1. Structural primitives and representational scope

A common substrate across these systems is the decomposition of visual input into graph-addressable units. In unsupervised video domain adaptation, each video is encoded as a single graph whose nodes are frame features and whose edges capture temporal adjacency and top-$K$ visual similarity, with $K=5$ in the reported implementation (Hu et al., 2022). In temporal sentence localization, HVSARN constructs fully connected object-level and frame-level graphs in both visual and semantic spaces, with query-conditioned graph memories operating hierarchically from objects to frames (Liu et al., 2023). In embodied robotics, EmbodiedLGR stores only atomic object observations as nodes with attributes $\{EOD,P,T\}$, while explicit pairwise edges are minimal or unused; spatial and temporal relations are instead recovered from pose and timestamp attributes during retrieval (Riva et al., 20 Apr 2026).

Topological navigation systems make the graph itself the memory backbone. MemoNav maintains a sparse topological map whose node features form short-term memory, augments it with a global long-term memory node, and derives working memory from graph attention over retained nodes and the global node (Li et al., 2024). Graph Attention Memory similarly builds an explicit topological graph from exploration priors and uses recurrent graph attention to propagate goal information across nodes (Li et al., 2019). By contrast, VimRAG models the reasoning process as a dynamic DAG in which each node is an epistemic state $(p_i,q_i,s_i,m_i)$, and memory items inside nodes can be text, images, or temporally grounded video segments (Wang et al., 13 Feb 2026).

Other instantiations broaden the meaning of “graph” while retaining the same organizing principle. SeCG uses fully connected object graphs with memory slots appended to key/value banks, so that global language semantics modulate relation-oriented visual encoding (Xiao et al., 2024). DMMGR combines a dynamic key-value memory with a spatial-aware image graph, using knowledge-derived query states to modulate node and edge attention in the visual graph (Li et al., 2022). SCVM does not instantiate an explicit adjacency matrix inside the vision tower; instead, it introduces a persistent cross-layer memory state and then interprets the resulting token–summary–memory dependencies as a graph perspective over representation evolution (Liu et al., 28 Feb 2026). Earlier antecedents in layered visual memory treated local parts, lateral associations, and identity units as lower- and higher-layer graph structures formed through experience-driven self-organization (0905.2125).

Setting	Memory units	Relational structure
Video UDA (Hu et al., 2022)	Frame nodes	Temporal adjacency + top-$K$ visual similarity
TSLV (Liu et al., 2023)	Object and frame nodes	Fully connected, attention-defined edges
Embodied robotics (Riva et al., 20 Apr 2026)	Object-instance nodes $\{EOD,P,T\}$	Attribute graph; relations inferred by distance/time
Navigation (Li et al., 2024, Li et al., 2019)	Viewpoint nodes + global/LTM node	Topological connectivity
Multimodal RAG (Wang et al., 13 Feb 2026)	Epistemic state nodes	Directed acyclic dependency graph
3D grounding / KVQA (Xiao et al., 2024, Li et al., 2022)	Object proposals, memory slots, image nodes	Fully connected or sparse spatial graphs

2. Modulation mechanisms

The most common mechanism is graph attention. In the video UDA model, single-head attention over neighbors $\mathcal{N}(i)$ is defined by

$$e_{ij}^{(h)}=\mathrm{LeakyReLU}\left(a^{(h)\top}[W^{(h)}x_i \parallel W^{(h)}x_j]\right), \qquad \alpha_{ij}^{(h)}=\frac{\exp(e_{ij}^{(h)})}{\sum_{k\in\mathcal{N}(i)}\exp(e_{ik}^{(h)})},$$

so attention coefficients determine which frames and relations are propagated and therefore what the model effectively “remembers” across time and appearance (Hu et al., 2022). MemoNav uses GATv2 on retained short-term memory nodes plus a global node, so working memory is an attention-weighted encoding of only the goal-relevant subgraph (Li et al., 2024). SeCG likewise uses multi-head graph attention, but augments keys and values with a text-conditioned memory bank and multi-view relative position encoding, turning global semantics into a modifier of relational aggregation (Xiao et al., 2024).

A second mechanism is query-conditioned read/write control. HVSARN treats graph memory explicitly as an iterative controller that reads graph content into a query state and writes the updated state back into nodes. Its controller update takes the form

$Q^l = G^l \odot Q^{l-1} + (1-G^l)\odot (Q^{l-1})',$

while node states are updated by gated fusion of prior node state, updated query, and neighbor-aggregated context (Liu et al., 2023). DMMGR follows a related pattern at the knowledge–vision interface: a dynamic key-value memory produces a knowledge-aware question representation $R^t$, and $K=5$0 directly modulates node and edge attentions in a spatial-aware image graph, coupling explicit knowledge reasoning to implicit visual reasoning (Li et al., 2022).

A third mechanism is stateful cross-layer modulation inside a vision encoder. SCVM updates a persistent cross-layer memory with

$K=5$1

and then refreshes token representations via a Token-Adaptive Gate,

$K=5$2

so that accumulated memory regulates representation evolution across depth rather than only post hoc feature fusion (Liu et al., 28 Feb 2026). Gramformer pushes this logic into transformer attention itself by multiplying post-softmax attention with an anti-similarity graph $K=5$3 and by adding centrality embeddings to node features, thereby diversifying memory retrieval away from redundant similar patches (Lin et al., 2024).

A fourth mechanism is graph-governed admission, compression, and retrieval. EmbodiedLGR uses thresholds $K=5$4 and $K=5$5 to decide whether an object observation updates an existing node or creates a new node, and then exposes semantic, spatial, and temporal graph tools $K=5$6, $K=5$7, and $K=5$8 for retrieval (Riva et al., 20 Apr 2026). VimRAG evaluates each memory item by a topology-aware reinforced energy

$K=5$9

and allocates visual token budget proportionally,

$\{EOD,P,T\}$0

so graph position determines whether an image region or video segment is retained at high resolution, compressed, or discarded (Wang et al., 13 Feb 2026). In CogGNN, the generated graph becomes the recurrent matrix $\{EOD,P,T\}$1 of an echo state network, making the graph itself the dynamical substrate of visual memory recall (Soussia et al., 13 Sep 2025).

3. Major application families

In video representation learning, graph-modulated visual memory encoding is used to reconcile temporal structure with cross-domain invariance. The video UDA model replaces sub-video storage and source–target pairing with one sparsified graph per video, aligns source and target at both frame and video levels, and uses graph attention to emphasize salient frames while suppressing redundant or noisy ones (Hu et al., 2022). HVSARN extends the same logic to language-conditioned temporal grounding, where visual graph memory and semantic graph memory jointly reason over objects and then frames, and where bidirectional visual-to-semantic and semantic-to-visual enhancement exposes query-relevant entities and temporal structure (Liu et al., 2023).

In embodied agents and navigation, the graph functions as an explicit operating memory. MemoNav partitions memory into short-term memory, long-term memory, and working memory, with a forgetting module removing the low-attention fraction of the topological map before GATv2 encoding (Li et al., 2024). Graph Attention Memory uses exploration-built topological graphs and recurrent graph attention to produce a goal-conditioned feature $\{EOD,P,T\}$2 for control (Li et al., 2019). EmbodiedLGR uses a lightweight semantic-spatial memory graph for fast low-level queries and a vector database for semantically richer caption retrieval, with the LLM deciding whether to stay on-graph or fall back to retrieval-augmented generation (Riva et al., 20 Apr 2026). This suggests that, in embodied settings, graph modulation is often as much about selective retention and query latency as about representational expressiveness.

In multimodal reasoning and grounding, graph modulation is primarily a mechanism for reducing distractors and concentrating alignment. SeCG injects text-conditioned memory into a fully connected object graph so that cross-modal encoding becomes relation-oriented rather than language-independent (Xiao et al., 2024). DMMGR uses a dynamic knowledge memory to guide graph attention over noisy visual relations (Li et al., 2022). VimRAG structures agent states and evidence in a DAG, then uses topological importance to regulate visual resolution under token budgets (Wang et al., 13 Feb 2026). SCVM adapts this idea to MLLMs by making a recurrent memory state regulate the formation of visual features across layers while staying compatible with a pretrained cross-attention interface (Liu et al., 28 Feb 2026).

A broader interpretive lineage links these methods to cognitive and neuroscientific models. The 2009 layered visual memory model encodes local parts, their lateral relations, and higher-level identities through self-organized sparse associative structure (0905.2125). CogGNN makes a generated connectome preserve visual-memory traits by constraining reservoir dynamics with the graph (Soussia et al., 13 Sep 2025). Modular Dirichlet Energy, while an analysis method rather than a generative encoder, uses a connectivity-derived graph to localize the temporal dynamics of visual short-term memory binding in occipital and frontal modules with millisecond precision (Smith et al., 2016).

4. Efficiency, scalability, and resource allocation

A persistent motivation for graph-modulated encoding is memory efficiency. In video UDA, sub-video storage baselines incur $\{EOD,P,T\}$3 space and source–target pairing can require $\{EOD,P,T\}$4 space, whereas the single-graph formulation stores $\{EOD,P,T\}$5 node features and $\{EOD,P,T\}$6 adjacency, with overall storage $\{EOD,P,T\}$7 per batch; on an RTX 3090, GPU memory was reduced by approximately $\{EOD,P,T\}$8 versus TA$\{EOD,P,T\}$9N at $(p_i,q_i,s_i,m_i)$0 frames (Hu et al., 2022). MemoNav pursues the same objective by forgetting a percentage $(p_i,q_i,s_i,m_i)$1 of the lowest-score nodes, so computation scales with $(p_i,q_i,s_i,m_i)$2 rather than the full map; the retained subgraph is then fused with long-term memory to form working memory (Li et al., 2024).

Other systems expose the opposite trade-off: richer relational structure at higher cost. HVSARN uses fully connected object- and frame-level graphs, with object-level reasoning approximately $(p_i,q_i,s_i,m_i)$3 and frame-level reasoning $(p_i,q_i,s_i,m_i)$4, so scalability depends directly on the number of object proposals and the downsampled frame count (Liu et al., 2023). Gramformer retains the quadratic cost of dense attention and kNN graph construction, reporting $(p_i,q_i,s_i,m_i)$5M parameters, $(p_i,q_i,s_i,m_i)$6 GFLOPs on $(p_i,q_i,s_i,m_i)$7 inputs, and inference time $(p_i,q_i,s_i,m_i)$8 units for $(p_i,q_i,s_i,m_i)$9 images on a single RTX 3080 (Lin et al., 2024). VimRAG adds graph-level bookkeeping overhead—$K$0 for recursive reinforcement and $K$1 for item ranking—but offsets this by compressing low-significance evidence and reserving high-resolution tokens for pivotal nodes (Wang et al., 13 Feb 2026).

EmbodiedLGR demonstrates that minimal graph structure can be an efficiency strategy in its own right. Its graph tools perform linear scans over $K$2, yet graph-only latency is much lower than caption-based retrieval: with Florence-2-base, graph only averages $K$3 s versus $K$4 s for vector DB only; with Florence-2-large, graph only averages $K$5 s versus $K$6 s for vector DB only (Riva et al., 20 Apr 2026). SCVM similarly frames its contribution as efficiency-compatible modulation: it reports consistent gains without expanding visual tokens, introducing additional vision encoders, or modifying or fine-tuning the LLM (Liu et al., 28 Feb 2026).

5. Empirical performance across tasks

The strongest quantitative evidence comes from domain-specific benchmarks rather than a single cross-domain evaluation protocol. In unsupervised video domain adaptation, the single-graph GAT model reports on UCF-HMDB_small that Ours-CDAN+I3D achieves $K$7 and $K$8; on UCF-HMDB_full it achieves $K$9 and $\{EOD,P,T\}$0; and on UCF-Olympic it reaches $\{EOD,P,T\}$1 and $\{EOD,P,T\}$2 (Hu et al., 2022). HVSARN reports new state-of-the-art temporal sentence localization performance, including ActivityNet $\{EOD,P,T\}$3 IoU$\{EOD,P,T\}$4, TACoS $\{EOD,P,T\}$5 IoU$\{EOD,P,T\}$6, and Charades-STA $\{EOD,P,T\}$7 IoU$\{EOD,P,T\}$8 (Liu et al., 2023).

In embodied and navigation settings, MemoNav reports substantial improvements over prior map-based baselines. On Gibson 1-goal tasks it reaches SR $\{EOD,P,T\}$9 and SPL $\mathcal{N}(i)$0 versus VGM at $\mathcal{N}(i)$1, and on Gibson 2-goal tasks it reaches PR/PPL $\mathcal{N}(i)$2 versus $\mathcal{N}(i)$3 for VGM (Li et al., 2024). Graph Attention Memory reports that, on Maze2, baseline success rates are $\mathcal{N}(i)$4 while GAM reaches $\mathcal{N}(i)$5, with scores of $\mathcal{N}(i)$6 versus negative scores for feed-forward and LSTM baselines (Li et al., 2019). EmbodiedLGR on NaVQA shows that Florence-2-large with Graph + ReMEmbR reaches positional accuracy $\mathcal{N}(i)$7, temporal accuracy $\mathcal{N}(i)$8, and average accuracy $\mathcal{N}(i)$9, while also improving positional accuracy over vector-only retrieval (Riva et al., 20 Apr 2026).

In multimodal reasoning and grounding, the performance gains are likewise concrete. SCVM improves LLaVA-v1.5-7B from $$e_{ij}^{(h)}=\mathrm{LeakyReLU}\left(a^{(h)\top}[W^{(h)}x_i \parallel W^{(h)}x_j]\right), \qquad \alpha_{ij}^{(h)}=\frac{\exp(e_{ij}^{(h)})}{\sum_{k\in\mathcal{N}(i)}\exp(e_{ik}^{(h)})},$$0 to $$e_{ij}^{(h)}=\mathrm{LeakyReLU}\left(a^{(h)\top}[W^{(h)}x_i \parallel W^{(h)}x_j]\right), \qquad \alpha_{ij}^{(h)}=\frac{\exp(e_{ij}^{(h)})}{\sum_{k\in\mathcal{N}(i)}\exp(e_{ik}^{(h)})},$$1 on DocVQA, from $$e_{ij}^{(h)}=\mathrm{LeakyReLU}\left(a^{(h)\top}[W^{(h)}x_i \parallel W^{(h)}x_j]\right), \qquad \alpha_{ij}^{(h)}=\frac{\exp(e_{ij}^{(h)})}{\sum_{k\in\mathcal{N}(i)}\exp(e_{ik}^{(h)})},$$2 to $$e_{ij}^{(h)}=\mathrm{LeakyReLU}\left(a^{(h)\top}[W^{(h)}x_i \parallel W^{(h)}x_j]\right), \qquad \alpha_{ij}^{(h)}=\frac{\exp(e_{ij}^{(h)})}{\sum_{k\in\mathcal{N}(i)}\exp(e_{ik}^{(h)})},$$3 on MME, from $$e_{ij}^{(h)}=\mathrm{LeakyReLU}\left(a^{(h)\top}[W^{(h)}x_i \parallel W^{(h)}x_j]\right), \qquad \alpha_{ij}^{(h)}=\frac{\exp(e_{ij}^{(h)})}{\sum_{k\in\mathcal{N}(i)}\exp(e_{ik}^{(h)})},$$4 to $$e_{ij}^{(h)}=\mathrm{LeakyReLU}\left(a^{(h)\top}[W^{(h)}x_i \parallel W^{(h)}x_j]\right), \qquad \alpha_{ij}^{(h)}=\frac{\exp(e_{ij}^{(h)})}{\sum_{k\in\mathcal{N}(i)}\exp(e_{ik}^{(h)})},$$5 on SQA, and from $$e_{ij}^{(h)}=\mathrm{LeakyReLU}\left(a^{(h)\top}[W^{(h)}x_i \parallel W^{(h)}x_j]\right), \qquad \alpha_{ij}^{(h)}=\frac{\exp(e_{ij}^{(h)})}{\sum_{k\in\mathcal{N}(i)}\exp(e_{ik}^{(h)})},$$6 to $$e_{ij}^{(h)}=\mathrm{LeakyReLU}\left(a^{(h)\top}[W^{(h)}x_i \parallel W^{(h)}x_j]\right), \qquad \alpha_{ij}^{(h)}=\frac{\exp(e_{ij}^{(h)})}{\sum_{k\in\mathcal{N}(i)}\exp(e_{ik}^{(h)})},$$7 on POPE (Liu et al., 28 Feb 2026). VimRAG raises Qwen3-VL-8B-Instruct overall accuracy from $$e_{ij}^{(h)}=\mathrm{LeakyReLU}\left(a^{(h)\top}[W^{(h)}x_i \parallel W^{(h)}x_j]\right), \qquad \alpha_{ij}^{(h)}=\frac{\exp(e_{ij}^{(h)})}{\sum_{k\in\mathcal{N}(i)}\exp(e_{ik}^{(h)})},$$8 for Mem1 to $$e_{ij}^{(h)}=\mathrm{LeakyReLU}\left(a^{(h)\top}[W^{(h)}x_i \parallel W^{(h)}x_j]\right), \qquad \alpha_{ij}^{(h)}=\frac{\exp(e_{ij}^{(h)})}{\sum_{k\in\mathcal{N}(i)}\exp(e_{ik}^{(h)})},$$9, with an intermediate improvement to $Q^l = G^l \odot Q^{l-1} + (1-G^l)\odot (Q^{l-1})',$0 when graph topology and multimodal memory are introduced before energy-based allocation (Wang et al., 13 Feb 2026). SeCG reports $Q^l = G^l \odot Q^{l-1} + (1-G^l)\odot (Q^{l-1})',$1 on Nr3D and $Q^l = G^l \odot Q^{l-1} + (1-G^l)\odot (Q^{l-1})',$2 on Sr3D, with especially large gains on multi-relation settings; on ScanRefer validation it reports overall [email protected] $Q^l = G^l \odot Q^{l-1} + (1-G^l)\odot (Q^{l-1})',$3 and [email protected] $Q^l = G^l \odot Q^{l-1} + (1-G^l)\odot (Q^{l-1})',$4 (Xiao et al., 2024). DMMGR reports $Q^l = G^l \odot Q^{l-1} + (1-G^l)\odot (Q^{l-1})',$5 top-1 on KRVQR and $Q^l = G^l \odot Q^{l-1} + (1-G^l)\odot (Q^{l-1})',$6 top-1 on FVQA, rising to $Q^l = G^l \odot Q^{l-1} + (1-G^l)\odot (Q^{l-1})',$7 with dense captioning (Li et al., 2022). Gramformer reports MAE/MSE $Q^l = G^l \odot Q^{l-1} + (1-G^l)\odot (Q^{l-1})',$8 on UCF-QNRF, with attention diversity ANVar rising from $Q^l = G^l \odot Q^{l-1} + (1-G^l)\odot (Q^{l-1})',$9 for a vanilla transformer to $R^t$0 (Lin et al., 2024).

CogGNN adds a distinct evaluation axis: preservation of cognitive traits in generated brain graphs. Vis-CogGNN significantly outperforms DGN on visual memory capacity in $R^t$1 hemisphere-dataset settings and improves downstream discriminative performance, including AD/LMCI RH ACC $R^t$2 versus $R^t$3 and AUC $R^t$4 versus $R^t$5 (Soussia et al., 13 Sep 2025). In neuroscience analysis rather than generative modeling, Modular Dirichlet Energy identifies significant shape-versus-binding differences in occipital MDE at $R^t$6–$R^t$7 ms ($R^t$8) and $R^t$9–$K=5$00 ms ($K=5$01), followed by frontal effects at $K=5$02–$K=5$03 ms ($K=5$04) and $K=5$05–$K=5$06 ms ($K=5$07), revealing a temporally ordered occipital-to-frontal sequence during visual short-term memory encoding (Smith et al., 2016).

6. Conceptual ambiguities, limitations, and likely directions

The literature makes clear that “graph-modulated visual memory encoding” is not a single settled architectural category. In some systems the graph is explicit and heavily relational, as in fully connected object/frame graphs or DAG-structured reasoning states; in others it is deliberately minimal, as in EmbodiedLGR’s attribute graph with minimal or unused pairwise edges; and in still others the graph view is partly interpretive, as in SCVM’s cross-layer memory state (Liu et al., 2023, Riva et al., 20 Apr 2026, Liu et al., 28 Feb 2026). This suggests that the central commitment is not any specific graph topology, but the use of graph-governed structure to decide what visual information survives abstraction.

Several recurring limitations appear across implementations. Video UDA is sensitive to backbone feature quality and to the choice of $K=5$08 in top-$K=5$09 similarity edges, and extremely long videos may still require hierarchical graphs or multi-scale pooling (Hu et al., 2022). HVSARN inherits the quadratic cost of fully connected object and frame graphs, and its semantic embeddings may be noisy because they depend on detected object classes and attributes (Liu et al., 2023). SeCG notes failure modes on rare attributes, negation, and subtle compositional relations, and does not include an explicit contrastive alignment loss (Xiao et al., 2024). DMMGR is sensitive to detection quality and limited fact retrieval, while over-stepping degrades performance when the dataset requires only shallow reasoning (Li et al., 2022).

Embodied and agentic systems expose a different failure surface. EmbodiedLGR depends on reliable LLM tool use and on the semantic granularity of small edge-deployable VLMs; it can fail on complex attribute queries such as “Find me a bench with a red wooden tint” when motion degrades observation quality (Riva et al., 20 Apr 2026). VimRAG is sensitive to graph construction errors and to the hyperparameters $K=5$10 and $K=5$11 in topology-aware energy reinforcement, so noisy child nodes can inflate the significance of suboptimal evidence (Wang et al., 13 Feb 2026). MemoNav’s forgetting module is test-time only and uses a fixed threshold $K=5$12, which the authors explicitly note as a limitation (Li et al., 2024).

Future work in these papers converges on a few themes. One is adaptive sparsification: adaptive $K=5$13, learned edge weights, dynamic graph construction, and phrase- or query-conditioned routing recur across video adaptation, sentence localization, 3D grounding, and long-context reasoning (Hu et al., 2022, Liu et al., 2023, Xiao et al., 2024, Wang et al., 13 Feb 2026). A second is richer multimodal integration, such as transformer-based memory controllers, motion features inside graph memories, or hierarchical semantic-spatial abstractions (Liu et al., 2023, Riva et al., 20 Apr 2026). A third is stronger cognitive grounding: CogGNN shows one route by letting a generated graph define the dynamics of a memory system, while the older layered visual memory model indicates that sparsity, competition, and structured part relations remain relevant principles for durable visual recall (Soussia et al., 13 Sep 2025, 0905.2125).