Memory-Conditioned Scene Persistence

• Memory-conditioned scene persistence is the technique that retains coherent scene states by integrating semantic appearance and geometric structures over time.
• It underpins applications such as video generation, dynamic reconstruction, and SLAM by ensuring consistent object identities and spatial relations across edits and revisits.
• The approach leverages diverse memory representations, retrieval strategies, and update policies to prevent issues like identity drift and silent change in dynamic systems.

Searching arXiv for the papers on arXiv and closely related work to ground the article in current literature. Memory-conditioned scene persistence denotes the ability of a system to maintain a coherent world state by conditioning current prediction on persistent memory rather than on local context alone. In video generation, this encompasses semantic appearance persistence and geometric or structural persistence across time, revisited viewpoints, and editing operations; in action-conditioned world models, it is framed as preventing “silent change” after leave-and-return motions; in mapping and SLAM, it appears as the persistence of dynamic identity or static structure inside the representation itself (Yang et al., 15 Jun 2026, King et al., 8 Jun 2026, Xiao, 28 Jun 2026). The concept therefore spans multiple regimes—scene editing, long-form narrative generation, interactive camera control, dynamic reconstruction, robot manipulation, and long-video understanding—but a common objective recurs: previously established entities, layouts, and scene states should remain recoverable when the system later needs them (Zhou et al., 7 Jan 2026, Chen et al., 28 Mar 2026, Garcin et al., 3 Mar 2026).

1. Conceptual scope and canonical failure modes

The literature defines persistence at several granularities. PermaVid formalizes a scene at time $t$ by semantic appearance $A_t$—object identity, texture, color, illumination, and style—and geometric structure $G_t$—layout, shape, and spatial relations. Persistence after edits requires that an edited appearance $A_t'$ become the canonical state for later times and revisited viewpoints, while geometry remain stable unless the edit explicitly changes structure (Yang et al., 15 Jun 2026). VideoMemory formulates the same problem at the entity level: characters, props, and backgrounds should remain recognizable across distant shots despite viewpoint changes, lighting changes, and scene transitions (Zhou et al., 7 Jan 2026). Echo-Memory casts the problem for action world models as preserving “the same world (geometry, salient objects, identities)” after the camera leaves and later returns, and identifies the characteristic failure as “silent change,” in which locally plausible frames do not preserve the previously seen world (King et al., 8 Jun 2026).

A related formulation appears in representation-centric systems. MoPe argues that dynamic-ness is not an instantaneous appearance property but “a temporal property defined by motion history,” so a region that once exhibited motion should not immediately revert to static status when it momentarily appears still (Xiao, 28 Jun 2026). SceneBench extends the same idea to long-video understanding: a “scene” is a contiguous span with coherent visual setting and semantic continuity, and failures emerge as a sharp drop from local clip-level accuracy to scene-level reasoning accuracy (Chen et al., 28 Mar 2026). Across these formulations, memory-conditioned scene persistence targets several recurrent pathologies: identity drift, stale appearance reuse after edits, forgetting after long temporal gaps, camera-consistent geometry failure on revisits, and ghosting or afterimages when dynamic content is mistakenly absorbed into a supposedly stable map (Yang et al., 15 Jun 2026, Xiao, 28 Jun 2026).

2. Memory representations and state design

A major axis of variation is the representation of memory itself. PermaVid decouples spatial context into two complementary banks: an RGB context memory,

$$\mathcal{M}^{rgb} = \{(I_i^{rgb}, p_i, t_i, g_i)\}_{i=1}^N,$$

which is appearance-aware and stores a semantic version identifier $g_i$, and a depth context memory,

$$\mathcal{M}^{dep} = \{(I_j^{dep}, p_j, t_j)\}_{j=1}^M,$$

which preserves geometry-only structure and is invariant to purely semantic edits (Yang et al., 15 Jun 2026). VideoMemory instead uses three specialized banks—$M^{char}$, $M^{prop}$, and $M^{bg}$—whose entries are entity frames $A_t$0, with a string identifier, attribute text, and a disk-stored PNG reference image (Zhou et al., 7 Jan 2026). This makes persistence explicitly entity-centric rather than view-centric.

Other systems emphasize memory as latent context. Echo-Memory isolates four families—raw Context, Compression, Spatial Memory, and State-Space—and argues that capacity, compression, read-out, and recurrence should be treated as separate axes rather than conflated design choices (King et al., 8 Jun 2026). MemLearner augments a latent video diffusion model with context tokens $A_t$1, query tokens $A_t$2, and predicted tokens $A_t$3, so that memory remains historical latent frames but retrieval becomes learned and token-level rather than rule-based (Yu et al., 30 Jun 2026). MAG stores history directly as compressed key-value cache entries, retaining only a block summary $A_t$4 per block and updating memory by

$A_t$5

thereby holding approximately $A_t$6 retained KV frames and achieving about $A_t$7 compression when $A_t$8 (Zhu et al., 21 Dec 2025).

A second lineage uses explicit spatial memory. Spatia maintains a persistent 3D scene point cloud and updates it iteratively through visual SLAM, while the earlier long-term spatial-memory world model uses a TSDF voxel volume plus a derived static point map, supplemented by a sparse episodic memory of keyframes (Zhao et al., 17 Dec 2025, Wu et al., 5 Jun 2025). PERSIST goes further by making a latent 3D world-frame $A_t$9 the primary memory, paired with camera state $G_t$0 and queried through a differentiable renderer (Garcin et al., 3 Mar 2026). MosaicMem stores localized patches

$G_t$1

that are lifted into 3D for targeted retrieval and composition (Yu et al., 17 Mar 2026). VMem and Mem-World use surfel-indexed memories; in VMem, a surfel is $G_t$2, whereas Mem-World adds timestamps and task relevance, $G_t$3, to anchor manipulation-specific observations to the wrist camera over time (Li et al., 23 Jun 2025, Zheng et al., 17 Jun 2026). MoPe represents memory as a persistent per-pixel dynamic posterior $G_t$4 and Gaussian-level dynamic ratios $G_t$5, thereby embedding temporal state directly into the mapping representation (Xiao, 28 Jun 2026).

This variety suggests that “memory” in this literature is not a single data structure. It may be a bank of views, a bank of named entities, a compressed KV cache, a recurrent state, a point cloud, a TSDF, a surfel set, or a persistent probabilistic field. What unifies these forms is that the stored state is intended to outlive the local generation window and to be selectively reused when scene persistence becomes load-bearing.

3. Retrieval, addressing, invalidation, and write policies

A central technical problem is deciding what to read, what to preserve, and what to forget. PermaVid makes edit-awareness explicit. For a target path $G_t$6, each memory unit receives a spatial overlap score

$G_t$7

and RGB retrieval is version-gated,

$G_t$8

while depth retrieval is not gated by the semantic version (Yang et al., 15 Jun 2026). A global edit triggers

$G_t$9

whereas a local edit invalidates only footprints overlapping $A_t'$0. This is one of the clearest formalizations of memory invalidation in the recent literature.

Learned retrieval replaces hard heuristics in MemLearner. Query tokens first attend to both predicted and context tokens,

$A_t'$1

and predicted tokens then attend to $A_t'$2, so attention weights themselves function as adaptive retrieval scores. The mechanism is designed to overcome the failure of FOV-overlap heuristics behind occluding surfaces and the brittleness of geometry-based retrieval under moving objects (Yu et al., 30 Jun 2026). By contrast, Context-as-Memory and MemCam use geometry-based heuristics deliberately: the former retrieves co-visible frames via FOV overlap between camera poses, while the latter computes a Monte Carlo co-visibility score

$A_t'$3

with $A_t'$4, and then selects the top-1 historical frame per target view (Yu et al., 3 Jun 2025, Gao et al., 27 Mar 2026).

I3DM uses target-conditioned features from a feed-forward novel view synthesis model to score candidates with an uncertainty map $A_t'$5, converts uncertainty to confidence by $A_t'$6, and selects frames by greedy maximum coverage over a global confidence canvas (Li et al., 24 Mar 2026). VMem renders visible surfels from the queried pose and scores each stored view index by how often it appears among visible surfels,

$A_t'$7

then conditions the generator on the top-$A_t'$8 views; Mem-World performs an analogous operation but renders timestamped surfels from an average future wrist pose derived from an action chunk, making retrieval action-conditioned rather than purely viewpoint-conditioned (Li et al., 23 Jun 2025, Zheng et al., 17 Jun 2026).

At longer temporal scales, retrieval becomes scene- or shot-level. VideoMemory uses an LLM-based matcher over semantic attributes and applies temporal logic to decide reuse versus insertion (Zhou et al., 7 Jan 2026). Scene-RAG embeds scenes and queries, computes cosine similarity

$A_t'$9

forms $$\mathcal{M}^{rgb} = \{(I_i^{rgb}, p_i, t_i, g_i)\}_{i=1}^N,$$0, and aggregates retrieved scene memory at question time (Chen et al., 28 Mar 2026). Echo-Forcing turns retrieval and forgetting into cache management: Scene Recall Frames compress historical scenes into spatially structured KV representations, while Difference-aware Memory Decay scales old keys and values by $$\mathcal{M}^{rgb} = \{(I_i^{rgb}, p_i, t_i, g_i)\}_{i=1}^N,$$1 according to their discrepancy from the new scene (Wu et al., 15 May 2026).

Write policies are equally consequential. PermaVid appends the last generated RGB frame and its estimated depth after each chunk (Yang et al., 15 Jun 2026). VMem updates surfels by merging or inserting new geometric observations (Li et al., 23 Jun 2025). MoPe propagates and fuses a dynamic posterior with bounded log-odds updates instead of re-deciding from scratch (Xiao, 28 Jun 2026). PERSIST updates a latent 3D world-frame at every step, so the camera acts as a spatial lookup key into a persistent environment state rather than a retrieval key into past pixels (Garcin et al., 3 Mar 2026). The literature repeatedly shows that persistence is not only a matter of storage capacity; it depends on whether the write and invalidation rules respect the semantic, geometric, and temporal structure of the task.

4. Conditioning and generation mechanisms

Memory must ultimately influence prediction. The dominant backbone is a diffusion or flow-matching Transformer, but the conditioning pathways vary sharply. PermaVid conditions a DiT backbone by a dedicated memory branch and cross-attention inside the diffusion Transformer, with denoising written as

$$\mathcal{M}^{rgb} = \{(I_i^{rgb}, p_i, t_i, g_i)\}_{i=1}^N,$$2

where retrieved RGB and depth frames are encoded via a shared 3D VAE and fused under mixed-modality memory (Yang et al., 15 Jun 2026). Context-as-Memory takes a simpler path: noisy predicted latents $$\mathcal{M}^{rgb} = \{(I_i^{rgb}, p_i, t_i, g_i)\}_{i=1}^N,$$3 and clean context latents $$\mathcal{M}^{rgb} = \{(I_i^{rgb}, p_i, t_i, g_i)\}_{i=1}^N,$$4 are concatenated along the frame dimension, and only the predicted latents are denoised (Yu et al., 3 Jun 2025). MemCam likewise concatenates compressed context tokens with prediction tokens so that memory participates directly as additional keys and values in self-attention (Gao et al., 27 Mar 2026).

Several systems inject memory through auxiliary branches. Spatia renders scene projection videos from its point-cloud memory and feeds their tokens through parallel ControlNet blocks; the outputs are projected by an MLP and fused additively into the main generator features (Zhao et al., 17 Dec 2025). The long-term spatial-memory world model renders static geometry from TSDF memory and injects it through ControlNet-style conditioning, while episodic keyframes enter via cross-attention (Wu et al., 5 Jun 2025). I3DM first uses a feed-forward NVS model to warp selected historical frames into the target view, encodes the aligned guidance to $$\mathcal{M}^{rgb} = \{(I_i^{rgb}, p_i, t_i, g_i)\}_{i=1}^N,$$5, and concatenates it with the noisy diffusion latent; camera features from a camera adapter are added to latent channels (Li et al., 24 Mar 2026). MosaicMem composes a persistence prior,

$$\mathcal{M}^{rgb} = \{(I_i^{rgb}, p_i, t_i, g_i)\}_{i=1}^N,$$6

from 3D-lifted patches and then lets a DiT refine or inpaint what should evolve (Yu et al., 17 Mar 2026).

Other formulations make the memory itself the state being evolved. PERSIST models the proxy state $$\mathcal{M}^{rgb} = \{(I_i^{rgb}, p_i, t_i, g_i)\}_{i=1}^N,$$7, renders

$$\mathcal{M}^{rgb} = \{(I_i^{rgb}, p_i, t_i, g_i)\}_{i=1}^N,$$8

and conditions a pixel denoiser on the rendered 3D memory rather than on retrieved image context (Garcin et al., 3 Mar 2026). MoPe uses a persistent posterior to modulate tracking weights, mapping weights, insertion masks, and Gaussian cleanup, so memory influences every stage of the SLAM pipeline instead of only a final decoder (Xiao, 28 Jun 2026). Make-A-Story reads a visual-linguistic memory by sentence-conditioned soft attention,

$$\mathcal{M}^{rgb} = \{(I_i^{rgb}, p_i, t_i, g_i)\}_{i=1}^N,$$9

and injects the resulting context at multiple U-Net scales (Rahman et al., 2022). VideoMemory and Scene-RAG are structurally lighter: they condition keyframe or answer generation on retrieved entity or scene references without training new generative backbones (Zhou et al., 7 Jan 2026, Chen et al., 28 Mar 2026).

The loss design is strikingly conservative across many of these systems. PermaVid introduces no additional handcrafted temporal or geometric losses (Yang et al., 15 Jun 2026). VideoMemory does not train new generators at all (Zhou et al., 7 Jan 2026). MemLearner uses a standard diffusion loss on predicted tokens only (Yu et al., 30 Jun 2026). PERSIST trains world and pixel denoisers with rectified flow matching (Garcin et al., 3 Mar 2026). This suggests that the main research emphasis has shifted from devising new consistency losses toward engineering memory state, retrieval, and injection so that standard generative objectives can exploit them.

5. Evaluation protocols and empirical findings

Evaluation protocols now probe persistence more directly than short-horizon reconstruction alone. Echo-Memory separates replay quality, in-domain loop revisit, and open-domain return probes, and reports that “the branches routinely disagree,” showing that replay fidelity is not a sufficient proxy for remembering a world (King et al., 8 Jun 2026). PermaVid evaluates global edits with depth PSNR, SSIM, LPIPS, CLIP-Vid, and VBench-Avg, and local edits with revisit PSNR, SSIM, LPIPS, and VBench-Avg (Yang et al., 15 Jun 2026). VideoMemory introduces a 54-case multi-shot consistency benchmark over characters, props, and backgrounds (Zhou et al., 7 Jan 2026). MemLearner uses both ground-truth comparison and revisit comparison on long videos with occlusions and dynamic objects (Yu et al., 30 Jun 2026). MoPe evaluates tracking robustness, rendering quality, and artifact-focused background metrics in dynamic SLAM sequences (Xiao, 28 Jun 2026). PERSIST uses FVD and a user study emphasizing 3D spatial consistency and temporal consistency (Garcin et al., 3 Mar 2026).

System	Setting	Representative result
PermaVid (Yang et al., 15 Jun 2026)	Global edits	Structure: PSNR 22.84, SSIM 0.8703, LPIPS 0.2102; CLIP-Vid 27.87
VideoMemory (Zhou et al., 7 Jan 2026)	54-case multi-shot benchmark	Character avg 0.63; Prop avg 0.58; Background avg 0.72
MemLearner (Yu et al., 30 Jun 2026)	Occlusion/dynamic dataset	GT Comp. PSNR 21.23; Revisit Comp. PSNR 18.57
Echo-Memory (King et al., 8 Jun 2026)	Open-domain return	Block-wise recurrence: Open-domain VLM 69.00
MoPe (Xiao, 28 Jun 2026)	Wild-SLAM / Bonn / TUM	ATE RMSE 0.46 → 0.38; 2.31 → 1.95; 1.51 → 1.28
VMem (Li et al., 23 Jun 2025)	RealEstate10K cycle	LPIPS 0.371; PSNR 14.82; SSIM 0.275
Mem-World (Zheng et al., 17 Jun 2026)	Wrist view	PSNR 19.21; SSIM 0.691; LPIPS 0.236; object-consistency 0.524
PERSIST (Garcin et al., 3 Mar 2026)	Luanti world models	FVD 706 (Oasis), 596 (WorldMem), 116 (PERSIST-XL+W_0)

Several results are notable because they isolate different aspects of persistence. In PermaVid, HY-WorldPlay preserves some structure under global edits, with depth PSNR 22.34, but fails to propagate updated semantics, with CLIP-Vid 19.96, illustrating the specific problem of reusing stale appearance when geometry remains valid (Yang et al., 15 Jun 2026). In VideoMemory, removing the Dynamic Memory Bank lowers the 8-shot background score from 0.72 to 0.28, indicating that background persistence is especially dependent on explicit memory (Zhou et al., 7 Jan 2026). MemLearner shows its largest gains on the occlusion/dynamic dataset, with GT Comp. PSNR 21.23 versus 19.85 for CaM and 19.59 for VMem, and Revisit Comp. PSNR 18.57 versus 17.61 for CaM (Yu et al., 30 Jun 2026). Echo-Memory shows that raw context improves open-domain return far more than replay metrics, and that block-wise state-space recurrence yields the strongest open-domain return mechanism in the matched matrix (King et al., 8 Jun 2026). Scene-RAG, in a non-generative but closely related setting, improves average performance from 66.6 to 70.9 and raises Scene from 53.8 to 57.6 and SceneQA from 50.4 to 56.3, while the paper characterizes the gain as a +2.50% improvement in long-video reasoning (Chen et al., 28 Mar 2026).

These metrics are not directly comparable across papers, because the tasks range from edited revisits to manipulation rollouts to SLAM tracking. A plausible implication is that the field increasingly relies on setting-specific protocols—loop closure, revisit comparison, scene-level retrieval, or dynamic-artifact suppression—rather than a single unified benchmark.

6. Trade-offs, limitations, and open problems

A recurrent controversy concerns whether memory should be explicit, implicit, compressed, geometric, or learned end-to-end. Echo-Memory argues that “compactness is not a free substitute for capacity”: aggressive spatial and hybrid-compression memories lose return-critical evidence, and dedicated cross-attention helps but remains insufficient if the stored summary is not identity-aware (King et al., 8 Jun 2026). MemLearner reaches a similar conclusion from the opposite direction: a separate query module trained from scratch fails badly, producing near-zero query-context similarity, whereas query tokens embedded inside the pre-trained generator exploit the model’s visual priors effectively (Yu et al., 30 Jun 2026). PermaVid shows that unified entangled memories fail after edits because systems either reuse stale appearance or discard useful geometry (Yang et al., 15 Jun 2026). Mem4D frames an analogous “Memory Demand Dilemma” for dynamic reconstruction: a single memory cannot simultaneously provide long-term stability for static structure and high-frequency responsiveness for dynamic motion (Cai et al., 11 Aug 2025).

Another controversy concerns geometry. Explicit 3D memory improves revisit consistency and camera control, but many papers enumerate geometry-dependent failure modes. PermaVid notes that inaccurate $g_i$0 or mis-specified affects_geometry flags can leave geometric inconsistencies after local edits (Yang et al., 15 Jun 2026). MoPe depends on depth $g_i$1 for posterior warping, so monocular depth errors can degrade propagation; its conservative insertion may also reduce Gaussian density and fine detail (Xiao, 28 Jun 2026). VMem acknowledges that dynamic objects and thin occlusion boundaries remain difficult for surfel-based indexing (Li et al., 23 Jun 2025). Context-as-Memory reports that simple FOV heuristics can fail in multi-room indoor environments, under occlusion, or with low overlap (Yu et al., 3 Jun 2025). I3DM is explicitly motivated by those failures and by the error accumulation and scale ambiguity of explicit reconstruction (Li et al., 24 Mar 2026).

Scaling and evaluation remain unresolved. MemLearner states that the current paradigm still stores full context and that practical memory should not scale linearly with time (Yu et al., 30 Jun 2026). Echo-Forcing keeps a bounded active cache but relies on heuristic quota allocation, prompt routing thresholds, and scene-recall construction rules (Wu et al., 15 May 2026). PERSIST fixes memory footprint by maintaining an agent-centered cuboid, but distant regions are dropped and exposure bias can still accumulate over thousands of steps (Garcin et al., 3 Mar 2026). Mem-World improves policy evaluation and synthetic data generation, yet it does not explicitly enforce physical fidelity and can still produce plausible but physically inconsistent outcomes (Zheng et al., 17 Jun 2026). SceneBench shows that even strong VLMs exhibit a sharp drop on scene-level questions, and Scene-RAG only partially alleviates the problem (Chen et al., 28 Mar 2026).

The main research directions stated in the literature are correspondingly diverse: modality-agnostic memory banks that incorporate normals, flow, or scene graphs; learned retrieval modules with stronger identity encoders; 3D-consistent or object-centric memories for dynamic entities; return-aware or object-retention losses; adaptive update and forgetting policies; multi-hop scene retrieval; and hybrid systems that combine persistent geometry with flexible latent generation (Yang et al., 15 Jun 2026, Zhou et al., 7 Jan 2026, King et al., 8 Jun 2026, Cai et al., 11 Aug 2025). What the recent work collectively establishes is not a single preferred solution, but a stable principle: persistence improves when memory state, addressing, and invalidation are aligned with the structure of the world the model is supposed to remember.