Papers
Topics
Authors
Recent
Search
2000 character limit reached

PermaVid: Consistent Video Generation Across Edits via Disentangled Context Memory

Published 15 Jun 2026 in cs.CV | (2606.16449v2)

Abstract: Consistent video generation under editing operations requires persistence: when edits modify scene appearance or layout, subsequent generations should remain coherent across time and viewpoints. However, existing memory designs struggle to maintain long-term consistency after such modifications, as stored contexts may become outdated or invalid. To address this, we propose PermaVid, a novel framework built upon a multi-modal context memory that disentangles spatial context into semantic appearance and geometric structure, together with an edit-aware memory update and retrieval strategy that keeps memory evolution aligned with subsequent observations. Specifically, we develop two complementary memory banks: an RGB context memory that captures appearance-aware observations while implicitly encoding geometry, and a depth context memory that preserves geometry-only structure disentangled from semantics. Building on this design, we introduce a memory-guided video generation model that performs multi-modal feature fusion under reference conditions drawn from mixed-modality memory contexts. Experiments demonstrate that our method maintains strong long-term semantic and structural consistency after edits, significantly outperforming state-of-the-art methods.

Summary

  • The paper introduces a novel multi-modal memory architecture that disentangles semantic appearance and geometric structure to ensure edit-consistency.
  • It employs selective invalidation and fusion mechanisms, achieving superior PSNR, SSIM, LPIPS, and CLIP-Vid metrics compared to state-of-the-art methods.
  • The approach demonstrates practical viability for interactive, long-form video editing with minimal overhead and robust qualitative performance.

PermaVid: Disentangled Memory for Edit-Consistent Video Generation

Introduction

The proliferation of diffusion-based and transformer-based video generative models has markedly advanced the controllability and realism of camera-conditioned video synthesis. Despite this progress, persistent problems plague the field: long-horizon geometric and semantic consistency, especially under user-driven editing operations (global or local), remains insufficiently addressed. "PermaVid: Consistent Video Generation Across Edits via Disentangled Context Memory" (2606.16449) systematically targets this deficiency. The paper introduces a multi-modal, edit-aware memory architecture that explicitly disentangles semantic appearance and geometric structure, achieving robust video consistency in challenging editing scenarios.

Motivation and Background

Prior approaches typically employ unified memory representations (e.g., RGB/frame caches, pose-conditioned retrieval) to stabilize generation across time and viewpoint [sun2025worldplay, li2025vmem, wu2025video]. These designs, while enhancing spatial/temporal coherence in static scenarios, entangle appearance and geometry. When editing actions alter semantic appearance (e.g., style, lighting) or introduce local changes (object insertion/replacement), historical contextual memory becomes partially or wholly invalid, yet naively persists in the retrieval process, introducing semantic drift, inconsistencies, and failure to propagate edits across views. This fundamental limitation motivates the PermaVid architecture.

Disentangled Context Memory

PermaVid's core innovation is the decomposition of spatial context into complementary appearance (RGB) and structure (depth) modalities, maintained as separate, spatially-indexed memory banks:

  • RGB Memory: Captures edited visual semantics (object identity, texture, illumination) along with pose metadata and an explicit semantic version index.
  • Depth Memory: Stores geometry-only scene structure (layout, shape relationships), robustly invariant to global appearance modifications.

Edit operations are categorized following Ditto’s taxonomy [bai2025ditto]:

  • Global Edits (e.g., style/season/illumination changes): Invalidate all past RGB memories, advance the semantic version, but retain depth since geometry is unchanged.
  • Local Edits (object-level): Selectively invalidate the RGB and, if necessary, depth memory units overlapping the edited region, preserving unaffected context elsewhere.

This selective invalidation and ongoing update ensure only spatially and semantically valid memory influences subsequent generation.

Edit-Aware Retrieval and Fusion Mechanism

At inference time, references are retrieved from both memory banks based on geometric relevance (pose overlap with the camera trajectory) and semantic version alignment. Retrieval uses a coverage-maximizing, redundancy-avoiding greedy selection to construct a diverse reference set. Only appearance-aligned, version-coherent RGB contexts are included; depth context retrieval is based solely on spatial alignment unless geometry has been edited.

The generation backbone is built on a DiT (Diffusion Transformer) architecture, conditioned not only on text and camera pose but also a memory-guided branch. RGB and depth references, encoded via a 3D VAE, are fused through context blocks at multiple layers, with relative rather than absolute positional encoding to ensure generalization to variable-length, nonuniformly edited sequences.

Dataset and Training

To robustly learn edit-aware long-term consistency, the authors introduce UE-Mem—a synthetic dataset generated with Unreal Engine 5. An embodied navigation agent produces revisiting camera paths spanning thousands of frames within diverse photorealistic scenes. These sequences are crucial: publicly available datasets typically lack pose annotations and revisitation statistics necessary for such memory-driven video generation research.

Two-stage training is deployed: initial pretraining on SpatialVid for camera-guided generation, followed by long-form memory-guided generation training on UE-Mem with mixed RGB/depth context exposure. The memory branch is held fixed to preserve multi-modal perception (initialized from VACE), while the main DiT and camera encoder are tunable.

Experimental Results

Extensive benchmark evaluation across both global and local edit scenarios is conducted against recent SOTA methods: HY-Worldplay [sun2025worldplay], HY-Gamecraft [li2025hygamecraft], Matrix-Game-2.0 [he2025matrix], and VMem [li2025vmem]. Evaluation includes paired-repeat camera trajectories and edit timings, assessing view recall consistency (PSNR, SSIM, LPIPS), semantic alignment (CLIP-Vid), and visual quality (VBench).

Numerical superiority is consistently established:

  • Global Edits: PermaVid achieves the highest PSNR (22.84), SSIM (0.8703), and lowest LPIPS (0.2102) for structural recall, and highest CLIP-Vid (27.87) for semantic propagation. Competing models either fail to maintain updated appearance (HY-Worldplay) or degrade both properties due to entangled, non-edit-aware memory (Matrix-Game-2.0, VMem).
  • Local Edits: Similarly, PermaVid surpasses baselines in accurate recall of post-edit local content upon viewpoint revisitation, as quantified by all image similarity metrics and VBench-Avg (0.8544).

Qualitative analysis confirms that only PermaVid reliably reflects global style edits across revisiting views while preserving geometry, and precisely recalls local modifications in both the correct spatial region and the correct temporal context post-edit.

An ablation study on memory disentanglement demonstrates that eliminating modality separation causes progressive semantic inconsistency after edits, substantiating the architectural claim.

Memory overhead is quantitatively negligible: Memory retrieval and maintenance increase only marginally with sequence length, accounting for a minute fraction of overall inference time, making the approach viable for long-form, interactive deployment.

Implications and Future Outlook

This work sets a new standard for the treatment of historical context in generative video pipelines, especially in interactive and editing-based applications. The explicit disentanglement of appearance and structure, coupled with spatially and semantically aware memory evolution, enables consistent propagation of edits over arbitrary time horizons and camera paths. Practically, this opens the door for robust real-time editing workflows, interactive storyboarding tools, and deployment in VR/AR contexts demanding repeatable, high-fidelity scene revisitation after iterative edits.

Theoretically, PermaVid highlights that entangling all scene factors into monolithic latent or frame-based memories is fundamentally limiting in nonstationary, user-steered environments. Future directions may extend the modality set (e.g., semantics/instances, physics), develop continually learning or hierarchically compositional memories, or integrate scene understanding and generation even more tightly.

Conclusion

PermaVid (2606.16449) demonstrates that disentangling semantic and geometric scene representations within an edit-aware, multi-modal context memory is essential for consistent long-form video generation in editable worlds. The architecture exhibits strong quantitative and qualitative benefits, modest overhead, and establishes a principled framework for memory management under dynamic, user-driven modifications. This approach is likely to serve as a foundation for next-generation world modeling and video editing systems within both research and application ecosystems.

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Explain it Like I'm 14

What is this paper about?

This paper introduces PermaVid, an AI system that makes and edits videos while keeping everything consistent over time. Imagine you’re filming a scene while walking around. If you change the scene’s style (like turning “day” into “night”) or add a new object (like placing a chair), PermaVid makes sure those changes stay the same when the camera moves, turns around, and later revisits the same places.

What questions did the researchers ask?

They focused on a simple but tricky problem:

  • How can a video-making AI remember edits so the video stays consistent across time and different camera angles?
  • How can it keep the scene’s “shape” (where things are) steady, even when the “look” (colors, textures, style) changes?
  • How can it update its memory after edits so it doesn’t reuse outdated information?

How did they do it?

To explain their approach, think of a scene as having two parts:

  • Appearance (semantics): how things look — colors, textures, style (e.g., “cartoon,” “nighttime”).
  • Geometry (structure): where things are and their shape — the layout of the room, the size and position of objects.

PermaVid keeps these two parts separate in memory, like two folders.

Two kinds of memory

  • RGB memory (appearance folder): Stores normal color images. This helps the system remember style, textures, and details.
  • Depth memory (geometry folder): Stores a “distance map,” like a 3D blueprint of the scene. This captures where objects are, independent of color or style.

Analogy: RGB is a photo scrapbook; depth is the architect’s floor plan.

Edit-aware updating (handling changes smartly)

There are two types of edits:

  • Global edits: Change the whole scene’s look (e.g., day → night, realistic → watercolor). PermaVid clears the appearance folder (because those old styles are now outdated) but keeps the geometry folder (the floor plan is still valid).
  • Local edits: Change just one part (e.g., add/remove/replace an object). PermaVid only clears the parts of the appearance folder that touch the edited area. If the edit also changes shape (like adding a new statue), it also updates the geometry folder for that area.

This way, the system doesn’t throw away useful geometric knowledge when only the style changes, and it doesn’t accidentally bring back pre-edit content.

Using the memory to make videos

As the camera moves, the system:

  • Picks reference memories that “overlap” with where the camera is looking (like a helpful librarian finding the most relevant pages).
  • Combines both folders — the appearance scrapbook and the geometry blueprint — to guide the new frames it generates.
  • Uses a modern video-generation model (a diffusion model: it starts from noisy images and “polishes” them into a clean video) that can read these memory hints to keep the video coherent over time and across views.

Teaching the system (training data)

The team built a training dataset using a game engine (Unreal Engine 5). They let a virtual camera wander, explore, and loop back to the same spots, capturing many scenes from different angles. This gave the AI lots of practice with revisiting places — exactly what it needs to learn stable memory and consistency.

What did they find, and why does it matter?

In tests against other top methods, PermaVid:

  • Kept shapes and layouts steady after big style changes (strong geometry consistency).
  • Spread new styles consistently across time and different viewpoints (strong appearance consistency).
  • Remembered local edits (like a replaced object) when revisiting the same location later.
  • Maintained good overall visual quality.
  • Added very little extra computing time for memory handling.

Why this matters:

  • Videos look less “glitchy” when the camera turns back to a place it saw before.
  • Edits don’t randomly disappear or revert to the old look.
  • Creators can confidently apply edits mid-video and trust they’ll persist.

What could this be used for?

  • Video editing tools: Make a style change once; it stays consistent across the whole sequence.
  • Filmmaking and animation: Reliable scene continuity as cameras move.
  • Games, VR/AR, and virtual tours: Stable worlds where changes stick as you explore.
  • Creative workflows: Faster iteration — no need to fix the same inconsistency again and again.
  • Building better “world models”: Steps toward AIs that understand and maintain a coherent 3D world over time.

In short, PermaVid shows that splitting “how things look” from “where things are,” and updating memory based on the kind of edit, makes video generation much more consistent and dependable.

View Paper Prompt  View All Prompts 

Knowledge Gaps

Knowledge gaps, limitations, and open questions

Below is a concise, actionable list of what remains missing, uncertain, or unexplored in the paper.

  • Automatic edit understanding: The method assumes known edit type (global vs. local) and edited region Ω_e, but does not provide an algorithm to detect edit regions, classify edit types, or decide whether a local edit changes geometry; develop learned detectors and uncertainty-aware edit classifiers.
  • Geometry-change detection: For local edits, the rule “invalidate depth memory only if geometry changes” lacks a mechanism to infer geometry change; investigate depth/normal residual tests or learned predictors to decide structural updates.
  • Reliance on accurate poses: The approach assumes accurate 6-DoF camera poses; evaluate robustness to pose noise/estimation errors and explore joint pose refinement or SLAM-in-the-loop alternatives.
  • Monocular depth reliability: Depth contexts are predicted from generated frames at inference; study failure modes under stylized/global edits, low light, motion blur, and domain shifts, and incorporate depth uncertainty into retrieval/fusion.
  • Feedback drift: Using predicted depth from generated frames can accumulate errors over long horizons; quantify drift, and evaluate correction mechanisms (e.g., periodic re-anchoring, bundle adjustment, or consistency losses).
  • Memory management at scale: Memory banks grow unbounded aside from invalidations; design and evaluate eviction, compression, deduplication, and summarization policies for hour-scale or session-scale generation.
  • Learned vs. heuristic retrieval: Retrieval is based on a hand-crafted frustum-overlap score and threshold τ; explore learning-to-retrieve with content-, uncertainty-, and edit-aware scoring, and analyze τ/B sensitivity.
  • Per-region versioning: Semantic versioning g* is global, potentially over-invalidate/under-invalidate in partially global edits; develop spatially varying version maps or layered version control.
  • Frequent/overlapping edits: Behavior under rapid sequences of mixed global/local edits, overlapping edits, or re-edits of the same region is not analyzed; study stability, priority resolution, and rollback mechanisms.
  • Dynamic/non-rigid scenes: The method largely assumes static geometry except edited regions; extend to moving objects, deformable actors, and scene dynamics while preserving edit persistence.
  • Beyond depth-only geometry: Geometry memory is depth-only; investigate richer geometric representations (normals, meshes, point clouds, NeRF/SDF) and their edit/update rules.
  • Memory fusion design: The memory-context branch is frozen (from VACE) and uses shared 3D VAE encoders for RGB/depth; compare against trained fusion, modality-specific encoders, cross-attention variants, and gating based on uncertainty.
  • Edit persistence vs. identity: For object-level edits, long-range identity preservation and interactions with unedited objects are not measured; add identity/track consistency metrics and mechanisms (e.g., instance-level memory).
  • Physical consistency of global edits: Global style/lighting edits are not constrained by illumination/physics; evaluate and enforce photometric coherence (e.g., shadows/specularities) and 3D-consistent relighting.
  • Real-world generalization: Training relies heavily on synthetic UE-Mem; validate on real captured videos with estimated poses, real edits, and domain-shift scenarios, including failure-case analysis.
  • Benchmark coverage: The 200-case benchmark mixes web and AI images and scripted edits; expand to diverse real scenes, moving actors, longer loops, varied edit tools, and multi-step edit protocols.
  • Evaluation breadth: Semantic consistency via CLIP-Vid and structure via depth PSNR/SSIM/LPIPS may be insufficient; add human studies, edit-localized metrics (mask-aware), temporal flicker measures, and geometry accuracy w.r.t. GT scans.
  • Occlusion/visibility modeling: Retrieval and fusion do not explicitly reason about occlusion changes between memory views and current view; integrate visibility prediction and occlusion-aware blending.
  • Confidence-weighted fusion: The method treats memory tokens uniformly; incorporate confidence maps from depth/pose/edit detectors to weight references and suppress stale/noisy tokens.
  • Partial/global edit granularity: Some “global” edits affect subsets of materials or lighting clusters; support material-/region-conditioned global updates to avoid unnecessary invalidations.
  • Resource profiling: Overhead analysis focuses on runtime; report GPU memory footprint vs. video length, memory bank size, and reference budget B, and characterize hardware scaling.
  • Ablation completeness: Provide ablations on B (reference budget), τ (overlap threshold), ratios of RGB/depth/mixed contexts during training, and depth estimator choice to guide practitioners.
  • Transparency and reproducibility: Release UE-Mem, evaluation scripts, and edit protocols (including how Ω_e is specified) to facilitate standardized comparisons and further research.
View Paper Prompt  View All Prompts 

Practical Applications

Immediate Applications

The following applications can be deployed with current generative video pipelines and standard GPU/cloud setups, leveraging PermaVid’s disentangled RGB/depth memory and edit-aware update/retrieval.

  • Consistent VFX and post-production across long takes (Media/Entertainment)
    • Use case: Apply global style transformations (e.g., day→night, film LUTs) or local object edits (prop replacement) that persist across revisits and camera switches within a scene.
    • Tools/workflows: Nuke/After Effects/DaVinci plugins; pipeline: camera solve → depth prediction → PermaVid memory banks → edit-aware generation → compositing.
    • Assumptions/dependencies: Accurate camera poses (from trackers/solvers), reliable depth for real footage, access to a capable video diffusion backbone and compute.
  • Multi-view product videos with brand-consistent styling (Advertising/E-commerce)
    • Use case: Generate product clips with consistent textures/branding after global style updates; recall local packaging edits across different angles and shots.
    • Tools/workflows: E-commerce video generator API; batch style experiments; asset management that stores “semantic version” metadata for each edit.
    • Assumptions/dependencies: Stable geometry captures (turntable/structured light), licensing for brand assets, consistent lighting or robust depth estimation.
  • Level previsualization and walkthroughs with persistent edits (Game Dev/Previz)
    • Use case: Designers insert/replace objects in blockouts and swap art styles; when replaying camera paths or revisiting rooms, edits remain stable.
    • Tools/workflows: Unreal/Unity plugin harnessing in-engine camera poses; use UE-Mem-like trajectories for testing.
    • Assumptions/dependencies: Engine integration for pose streaming, stylized domains may need fine-tuning, GPU budget.
  • Virtual tours and staging with recallable local changes (Real Estate/AEC)
    • Use case: Stage interiors (e.g., swap couch, recolor walls) and perform seasonal/day-night edits that remain consistent across long camera tours.
    • Tools/workflows: Matterport/RealityCapture export → camera trajectories → PermaVid service for consistent edited tours.
    • Assumptions/dependencies: Pose-aligned captures (SfM/SLAM), depth quality in glossy/textureless areas, privacy compliance for residential content.
  • Persistent AR content across viewpoint changes (AR/Marketing/Events)
    • Use case: Insert virtual signage or objects that remain stable despite global style filters (e.g., retro look) and during user movement.
    • Tools/workflows: ARKit/ARCore camera pose feed → online PermaVid memory → on-device/cloud rendering.
    • Assumptions/dependencies: Low-latency depth/pose signals, edge compute for near-real-time performance, battery limits on mobile.
  • Creator tools for long-form video edits that “stick” (Social/Creator Economy)
    • Use case: Apply a global aesthetic or local sticker/logo once; maintain it across long vlogs with revisits or re-cuts without re-editing each segment.
    • Tools/workflows: Mobile/desktop editor plugin; “semantic version” tagging to manage global/local edit history; background depth precompute.
    • Assumptions/dependencies: Cloud or on-device acceleration, watermarking/provenance for transparency policies, user-friendly UI.
  • Synthetic data generation with controlled semantics and stable geometry (CV/ML)
    • Use case: Produce long-horizon training videos where texture/style changes (domain randomization) while geometry remains constant, improving view-consistency training.
    • Tools/workflows: Automated batch generation; track semantic versions and local edit masks for labels.
    • Assumptions/dependencies: Licensing of base models, dataset governance; for real-world resemblance, careful prompt/camera design.
  • Benchmarks and protocols for long-term edit consistency (Academia)
    • Use case: Use UE-Mem-like revisiting trajectories to evaluate structural vs semantic consistency post-edit; compare memory strategies.
    • Tools/workflows: Public benchmark scripts; evaluation metrics for RGB (PSNR/SSIM/LPIPS), depth, and CLIP-Vid semantics.
    • Assumptions/dependencies: Availability of released code/data; standardized edit taxonomies and metrics.
  • Training material with consistent visuals across camera moves (Education)
    • Use case: Produce lab demos (e.g., optics, mechanics) where style overlays (labels/highlights) and local insertions (tools) persist through revisits.
    • Tools/workflows: Lesson generator with camera paths; template library for global styles and local overlays.
    • Assumptions/dependencies: Pedagogical review, accessible compute for schools, IP clearance for assets.

Long-Term Applications

These require further research for real-time performance, broader domain robustness, or ecosystem standardization.

  • Interactive world simulators with persistent, user-driven edits (Gaming/UGC/Metaverse)
    • Use case: Players or creators can restyle worlds or add objects at runtime; the system maintains cross-session, cross-view consistency.
    • Tools/products: In-engine “edit-aware” generative layer; persistent memory stores with versioning per region.
    • Assumptions/dependencies: Real-time diffusion/transformer acceleration, multi-user conflict resolution, content moderation.
  • Live broadcast with stable virtual overlays and ads (Media/Sports)
    • Use case: Field markings/ads remain consistent after camera changes and global style adjustments during live production.
    • Tools/products: Broadcast graphics engine integration; low-latency memory retrieval and pose tracking.
    • Assumptions/dependencies: Sub-100ms latency targets, robust tracking under occlusions, regulatory guidelines for disclosures.
  • City-scale AR navigation and digital signage (Smart Cities)
    • Use case: Persistent virtual wayfinding/signage edited globally (theme) or locally (storefront update) and recalled across user devices/views.
    • Tools/products: Cloud-hosted memory per geo-region; SLAM/visual localization linking to memory retrieval.
    • Assumptions/dependencies: Accurate geo-poses, privacy/security, standardization across AR platforms.
  • Collaborative, versioned video editing with provenance (Creative Suites/Policy)
    • Use case: Multi-user editing where global “semantic versions” and local edit masks are tracked, enabling reversible, auditable edit history.
    • Tools/products: Edit-aware metadata schema; integration with C2PA-like provenance standards for video.
    • Assumptions/dependencies: Industry-wide metadata standards, UI/UX for conflict resolution, legal alignment on disclosures.
  • Simulation for autonomy with controlled semantics (Autonomous Driving/Robotics)
    • Use case: Test environments where lighting/weather vary (global edits) while geometry stays fixed; safeness of local roadwork edits is evaluated across revisits.
    • Tools/products: Sensor-faithful video generation (multi-camera, depth/LiDAR emulation) tied to agent benchmarks.
    • Assumptions/dependencies: Physics- and sensor-accurate rendering, evaluation frameworks, validation against real-world data.
  • Medical/industrial training videos with layered, persistent edits (Healthcare/Manufacturing)
    • Use case: Insert anomalies/instruments or change appearance (e.g., staining) while preserving anatomy/machine geometry across camera motions.
    • Tools/products: Domain-specialized backbones, approval pipelines, audit trails.
    • Assumptions/dependencies: High-fidelity, validated content; strict compliance and ethics review; domain fine-tuning.
  • Generalist agents with disentangled spatial memory (AI Agents/Robotics)
    • Use case: Agents that separate appearance from geometry for robust perception and planning; edits simulate environment changes without invalidating maps.
    • Tools/products: Memory modules combining RGB/depth banks with edit-aware updates; integration with SSMs/transformer agents.
    • Assumptions/dependencies: Cross-modal learning at scale, online SLAM integration, long-horizon credit assignment.
  • Compliance auditing and deepfake forensics via edit-aware metadata (Policy/Safety)
    • Use case: Regulators/tools identify where/when global/local edits occurred via semantic version counters and region masks; ensure disclosures persist.
    • Tools/products: Forensic APIs consuming edit lineage; conformance tests using UE-Mem-like scenarios.
    • Assumptions/dependencies: Adoption by platforms, tamper-resistant provenance, standardized reporting.

Cross-cutting assumptions and dependencies

  • Accurate camera poses and robust depth estimation are critical for real footage; in-engine/SLAM sources are preferred.
  • Performance: current backbones (e.g., transformer-based diffusion) are compute-heavy; real-time use cases need model compression, distillation, or specialized hardware.
  • Content rights and safety: ensure licensing for assets and responsible-use guardrails (watermarking, provenance).
  • Domain robustness: dynamic scenes with true geometry changes, heavy occlusions, or reflective surfaces may degrade performance unless the depth memory and update rules are adapted.
  • Ecosystem integration: success depends on plugins/SDKs for major DCC tools (Adobe, Autodesk, Blackmagic), game engines (Unreal/Unity), and AR stacks (ARKit/ARCore).
View Paper Prompt  View All Prompts 

Glossary

  • 3D VAE: A variational autoencoder operating on spatiotemporal volumes to encode video frames into latent tokens. "each RGB or depth frame is independently encoded using a shared 3D VAE."
  • 4-DoF: Four degrees of freedom used to control planar motion and camera pitch. "The agent uses a continuous 4-DoF control vector u=[vx,vy,ω,g]\mathbf{u}=[v_x,v_y,\omega,g]"
  • 6-DoF: Six degrees of freedom specifying a full 3D camera pose (position and orientation). "with accurate 6-DoF camera poses."
  • Autoregressive approaches: Methods that generate videos step-wise, conditioning on previously generated content. "Recently, autoregressive approaches~\cite{chen2024diffusion, henschel2025streamingt2v} have been explored in video diffusion"
  • Autoregressive iteration: One step in an autoregressive generation loop where new frames are produced using past outputs. "At each autoregressive iteration, the last frame of the previously generated video chunk is used as the input image."
  • Camera-controlled video generation: Video synthesis guided by explicit camera poses or trajectories. "Camera-controlled video generation has emerged as an important direction toward controllable and interactive video synthesis."
  • Cascaded Context Blocks: Stacked conditioning modules that inject external context into selected transformer layers. "a dedicated memory context branch composed of distributed and cascaded Context Blocks duplicated from selected DiT layers."
  • CLIP-Vid similarity: A semantic similarity metric based on CLIP-style video-text alignment. "semantic consistency, evaluated by CLIP-Vid similarity~\cite{clip}"
  • Depth context memory: A memory bank storing depth observations to preserve geometry independent of appearance. "and a depth context memory that preserves geometry-only structure disentangled from semantics."
  • Depth predictor: A model that estimates per-frame depth used as geometric context. "The generated video is then fed into a depth predictor~\cite{chen2025vda}"
  • Diffusion Transformer (DiT): A transformer backbone used within diffusion models for video generation. "PermaVid uses a memory-guided video generation model built upon a diffusion Transformer (DiT)."
  • Edit-aware memory update and retrieval: A strategy that invalidates or selects memory entries based on the type and scope of edits. "together with an edit-aware memory update and retrieval strategy that keeps memory evolution aligned with subsequent observations."
  • First-person camera: A camera attached to the agent to capture egocentric video. "and a first-person camera rigidly attached to it captures long RGB and depth sequences"
  • Global edits: Edits that change appearance across the entire scene while largely preserving geometry. "Global edits, such as style transfer, seasonal change, or lighting transformation, modify the semantic appearance of the whole scene while usually preserving its geometry."
  • Global semantic version: A version tag that tracks the current global appearance state to invalidate outdated RGB memory. "and gig_i records the global semantic version when an RGB memory unit is inserted."
  • Goal-driven navigator: A policy that drives exploration toward distant goals to expand scene coverage. "a goal-driven navigator that explores distant regions"
  • KV cache: Cached key-value attention states used to extend context across chunks in transformers. "their limited short-term context (e.g., chunk-level KV cache) is insufficient to support reliable view recall over time."
  • Local edits: Edits confined to specific regions or objects, potentially altering local semantics or geometry. "Local edits, such as object insertion, removal, or replacement, affect only a bounded region"
  • LPIPS: A perceptual distance metric used to assess visual similarity. "measured using PSNR, SSIM, and LPIPS on paired RGB frames"
  • Memory bank: A structured cache of past observations used to guide future generation. "we develop two complementary memory banks"
  • Multi-modal context memory: A memory design storing different modalities (e.g., RGB and depth) for disentangled context. "a multi-modal context memory that disentangles spatial context into semantic appearance and geometric structure"
  • Multi-modal feature fusion: Combining features from multiple modalities to condition generation. "performs multi-modal feature fusion under reference conditions drawn from mixed-modality memory contexts."
  • Novel-view synthesis: Generating frames from new camera viewpoints consistent with scene structure. "provides appearance and geometry references for consistent novel-view synthesis under edits."
  • Pose-conditioned feature retrieval: Selecting past features based on camera pose similarity to maintain cross-view coherence. "leverage pose-conditioned feature retrieval, enabling the model to reuse previously observed visual information under similar viewpoints."
  • Pose-conditioned generation: Conditioning the generator on camera pose inputs to control viewpoint. "By leveraging pose-conditioned generation or spatially-aware representations, these approaches partially alleviate geometric inconsistency"
  • PSNR: Peak Signal-to-Noise Ratio, a fidelity metric for image/video reconstruction. "measured using PSNR, SSIM, and LPIPS"
  • Relative positional encoding: Positional embeddings defined relative to reference order, not absolute time indices. "with relative positional encoding within the memory set"
  • RGB context memory: A memory bank storing RGB observations capturing appearance (and implicit geometry). "an RGB context memory that captures appearance-aware observations while implicitly encoding geometry"
  • Semantic appearance: The visual attributes of a scene such as texture, color, identity, and style. "Semantic appearance includes visual attributes such as object identity, texture, color, illumination, and style"
  • Spatiotemporal context representations: Features encoding space and time jointly for video memory. "or spatiotemporal context representations~\cite{yang2025cambrian,zhang2025pretraining}"
  • SSIM: Structural Similarity Index, measuring perceptual structural similarity. "measured using PSNR, SSIM, and LPIPS"
  • U-Net–based models: Diffusion architectures using U-Net backbones for generative modeling. "Video generation has progressed from U-Net–based models~\cite{wang2023modelscope,guo2023animatediff} to Transformer-based diffusion frameworks"
  • VACE context-branch: A pretrained conditioning branch architecture used to encode external context. "the memory branch follows the VACE~\cite{jiang2025vace} context-branch design"
  • VBench: A benchmark suite for evaluating video generation quality. "Visual quality is measured using VBench~\cite{huang2024vbench}."
  • View footprint: The image-space region of the scene covered by a memory observation from a given pose. "invalidates only memory units whose view footprint overlaps the edited region."
  • View-frustum overlap: A measure of overlap between camera frustums used for spatial relevance. "and L\mathcal{L} measures normalized view-frustum overlap."
  • View recall consistency: The ability to reproduce the same content when revisiting a viewpoint after edits. "View recall consistency is measured using PSNR, SSIM, and LPIPS on paired RGB frames"
  • Viewpoint revisiting: Returning to previously seen viewpoints during a trajectory. "with viewpoint revisiting, making it suitable for testing long-term cross-view consistency after edits."
  • Yaw rate: Angular velocity around the vertical axis controlling camera heading. "ω\omega is yaw rate"
  • Pitch angle: The camera’s tilt angle controlling up/down orientation. "gg is pitch angle."
  • Pose tracker: A policy that creates local loops by following target poses to induce revisits. "and a pose tracker that induces local loops by following targets."
  • Revisiting trajectories: Camera paths that deliberately return to earlier viewpoints. "long revisiting trajectories"
View Paper Prompt  View All Prompts 

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up

Tweets

Sign up for free to view the 1 tweet with 52 likes about this paper.

Sign Up for Free