L3G-World: Interactive Video Model

• L3G-World is an interactive video world model generating explorable 3D environments from a single image, camera trajectory, or optional text, characterized by its Latent 3D Gaussian Memory (L3G-Mem) system and real-time generation capabilities (FPS:9.02),
• Key components include L3G-Mem for persistent, efficient 3D memory and L3G-Gen, a latent video diffusion transformer, generating and refining video chunks with selective memories updates,
• It excels in scene consistency, long-horizon, and camera control, even though memory expansion is not linear with audio length, and can fail unpredictively.

L3G-World is an interactive video world model that generates spatially consistent, explorable environments from a single image, a camera or action trajectory, and optionally a text prompt. Its defining component is Latent 3D Gaussian Memory (L3G-Mem), a persistent scene representation in which each Gaussian stores geometric, appearance, opacity, and VAE-latent information. A memory updater alternates with a latent video diffusion transformer, L3G-Gen, to generate new video chunks, extend memory into newly observed regions, and refine previously reconstructed regions. By rendering stored VAE features directly into the diffusion model’s latent space, L3G-World avoids repeated RGB rendering followed by VAE encoding and reports real-time generation at 9.02 FPS (Choi et al., 29 Oct 2026).

1. Problem formulation and motivation

Interactive world models must satisfy three simultaneous requirements: long-horizon spatial consistency, real-time generation, and unbounded exploration. A model should preserve the geometry and appearance of previously visited locations, respond to new camera trajectories, and extend its representation as exploration proceeds.

Prior approaches rely primarily on either temporal context or explicit 3D memory. Temporal-context systems retain previous frames, compressed histories, or attention key-value caches. Their memory and attention cost grows approximately with the number of generated frames $T$ and spatial resolution $R$, expressed as $O(TR)$. Such representations are temporally indexed rather than inherently spatial, and their computational cost increases with the generated stream.

Explicit 3D-memory systems reconstruct frames into point clouds or 3D Gaussian primitives. Feed-forward reconstruction is efficient but may add dense primitives from every frame, repeatedly representing the same surface. Optimization-based memories can be more compact and scale with scene extent, but repeated scene optimization is too slow for interactive use. A further bottleneck arises when a latent video diffusion model is conditioned through an explicit RGB memory: the system must render RGB and then re-encode it with a VAE.

L3G-World addresses these limitations through two design choices:

1. Selective 3D memory growth: L3G-Mem spawns new Gaussians only in uncovered or poorly reconstructed regions and refines visible existing Gaussians.
2. Direct latent rendering: every Gaussian stores a VAE latent feature that can be splatted directly into the latent space of L3G-Gen.

The intended memory complexity therefore scales primarily with explored spatial extent $S$, approximately $O(S)$, rather than with the number of generated frames.

2. Autoregressive system architecture

L3G-World alternates between L3G-Mem and L3G-Gen. If $\mathcal{W}_k$ denotes the current generated or observed video window and $\mathcal{M}_{k-1}$ the preceding memory, the memory update is

$$\mathcal{M}_{k}=\operatorname{Update}(\mathcal{M}_{k-1},\mathcal{W}_{k}).$$

The resulting memory is rendered under requested future cameras and supplied to the video generator. The inference cycle is:

1. Initialize memory from a single input image and camera.
2. Render the current memory under target camera views.
3. Generate the next video chunk with L3G-Gen.
4. Decode the generated latents to RGB for display.
5. Feed the generated chunk, cameras, and VAE latents to L3G-Mem.
6. Spawn Gaussians in uncovered or inaccurate regions.
7. Refine visible existing Gaussians.
8. Repeat as exploration continues.

L3G-Mem is the persistent spatial component, whereas L3G-Gen is the visual-temporal generation component. The memory is not a frame buffer: it retains a camera-independent 3D representation that can be revisited and updated.

L3G-Mem representation

After window $k$, memory is represented as

$$\mathcal{M}_{k}=\left\{(\mathrm{G}_{j},\mathrm{T}_{j})\right\}_{j=1}^{N_k},$$

where $R$0 is a learned memory token and $R$1 is its Gaussian representation. In the implementation, a Gaussian contains

$R$2

where $R$3 is the 3D center, $R$4 is an anisotropic covariance, $R$5 is opacity, $R$6 is RGB or view-dependent appearance, and $R$7 is a stored VAE latent feature. The covariance is parameterized using scale $R$8 and quaternion $R$9:

$O(TR)$0

The representation follows Gaussian splatting. Projected Gaussians are depth ordered and alpha composited:

$O(TR)$1

The same projected geometry and opacity are used to composite the stored latent features, producing target-view latent maps without VAE re-encoding.

3. Adaptive spawning and Gaussian refinement

At initialization, the memory is empty:

$O(TR)$2

The first input image, camera, and VAE latent are processed using Depth Anything 3, geometry feature layers, and a convolutional VAE-latent encoder. Geometry features are clustered with K-means to create spawning queries. In the reported implementation, each spawn query decodes 32 Gaussian primitives.

At subsequent steps, L3G-Mem renders the existing memory under the current camera views. It computes RGB reconstruction error and accumulated opacity. A patch is selected for spawning when it is insufficiently covered or poorly reconstructed:

$O(TR)$3

where $O(TR)$4 is patch-level RGB MSE and $O(TR)$5 is accumulated alpha.

Spawning therefore occurs in two situations:

• Uncovered regions: areas with low accumulated opacity.
• Inaccurate regions: areas with high reconstruction error despite existing coverage.

Only geometry features within the spawning mask are clustered. The number of clusters varies with masked area, with reported default values of $O(TR)$6 and $O(TR)$7 during memory training and inference. Unchanged regions retain their existing Gaussians.

Refinement of existing memory

Previously created Gaussians may contain errors in position, scale, opacity, appearance, or latent feature. L3G-Mem retrieves visible memory tokens and predicts residual updates

$O(TR)$8

A Gaussian is considered visible when one of its projected probe points lies within the image and satisfies depth and local-opacity conditions. The default thresholds are $O(TR)$9 and $S$0.

The refinement head separately gates geometric/RGB residuals and latent-feature residuals. Visible entries can therefore be corrected in place, while non-visible or non-retrieved entries remain unchanged. This selective updating is intended to prevent newly generated observations from indiscriminately rewriting the entire memory.

Joint spawn/refinement processing

Spawn queries and retrieved refinement tokens are processed jointly. The transformer performs:

• cross-attention between spawn or refinement streams and geometry features;
• self-attention within each stream;
• global self-attention over the concatenated streams.

Global attention allows newly spawned and previously reconstructed content to exchange information. Ablations show that removing this interaction degrades both reconstruction quality and memory efficiency.

4. Direct latent conditioning of video generation

The central efficiency mechanism of L3G-World is latent lifting. Conventional explicit-memory conditioning follows

$S$1

L3G-World instead follows

$S$2

The memory renderer produces target-view latent features $S$3 and an opacity or validity map $S$4. These are concatenated with noisy target video latents:

$S$5

Only target latents are denoised; reference latents remain fixed conditioning tokens. The generator can additionally retrieve up to three previously observed latents using camera-pose proximity. Camera conditioning and RayRoPE, a projective ray-based rotary positional encoding, tie spatial attention to camera rays and target poses.

VAE-latent storage

The latent-lifting branch embeds input VAE latents into 128-dimensional VAE tokens. Spatial indices used for geometry spawning also initialize VAE queries. These queries cross-attend to the complete VAE-token set, and the resulting latent tokens are concatenated with geometric spawn tokens:

$S$6

The spawn head predicts Gaussian geometry, RGB parameters, and a 16-dimensional VAE latent feature for each Gaussian. The latent rasterizer uses the same Gaussian projection and alpha compositing as RGB rendering, while geometry is detached for the VAE loss so that latent supervision does not inadvertently move Gaussian centers.

L3G-Gen

L3G-Gen is based on the 2B-parameter Cosmos-Predict2.5 DiT. The backbone is frozen while LoRA adapters, camera-conditioning modules, and the widened input projection are trained. The teacher uses rectified flow. For clean target latents $S$7, Gaussian noise $S$8, and $S$9,

$O(S)$0

The velocity model is trained with a flow-matching objective. It is conditioned on memory-rendered latent features, opacity, reference latents, camera information, and target poses.

For real-time inference, the teacher is distilled using AnyFlow-style flow-map and on-policy distillation. The student learns transitions between arbitrary source and destination noise levels and is trained on its own autoregressive rollouts. The reported student configuration generates three latent frames per step using four denoising steps and no classifier-free guidance. The teacher generates 81-frame clips with 35 denoising steps.

5. Training, consistency, and evaluation

Memory training

L3G-Mem is trained autoregressively over $O(S)$1 consecutive non-overlapping chunks, with $O(S)$2 images per chunk. The memory begins empty, and gradients are stopped through previous memory states to enforce causal online behavior.

The total memory objective is

$O(S)$3

The RGB, depth, and VAE terms include

$O(S)$4

$O(S)$5

and

$O(S)$6

The visibility loss encourages each Gaussian to lie within at least one input-view frustum. Depth supervision comes from Depth Anything 3. RGB supervision remains important even though the generator is conditioned on VAE features: removing RGB supervision substantially worsens both RGB and VAE reconstruction.

Long-horizon consistency

L3G-World maintains consistency through:

1. 3D anchoring: Gaussians are tied to spatial coordinates rather than frame order.
2. Selective updates: revisited regions refine existing memory rather than accumulating duplicate reconstructions.
3. Latent conditioning: future video chunks receive target-view latent features rendered from persistent memory.

The loop-closure benchmark uses 481-frame trajectories. The camera moves away from the initial location and then returns along the reverse trajectory, testing whether the original scene remains consistent after exploration.

The method does not claim exact reconstruction of genuinely unseen geometry. New regions are generated from learned video priors and sparse evidence, while persistent memory primarily anchors previously observed space.

Experimental configuration

Training uses DL3DV for L3G-Mem and L3G-Gen, and RealEstate10K for L3G-Gen training and out-of-domain evaluation. Training uses four NVIDIA A100 GPUs. L3G-Mem has approximately 1.7B total parameters and 1.6B trainable parameters. L3G-Gen has approximately 2B total parameters, with 75.8M trainable parameters in the teacher configuration.

The video resolution is $O(S)$7, corresponding to a $O(S)$8 VAE latent resolution. Efficiency measurements are made on a single NVIDIA A100. Baselines include GEN3C, Lyra 2.0, Spatia, SPMem, 3DScenePrompt, NeoVerse, Yume 1.5, and Matrix-Game 3.0.

Evaluation measures:

• self-consistency using PSNR, SSIM, and LPIPS;
• generation quality using LPIPS, FID, VBench IQ, AQ, subject consistency, and background consistency;
• camera control using translation and rotation error;
• geometric consistency using MEt3R;
• memory efficiency using primitive count, memory footprint, reconstruction FPS, RGB-rendering FPS, and VAE-conditioning FPS.

6. Results, limitations, and significance

Quantitative performance

On DL3DV, the main L3G-World model reports:

• 25K stored primitives;
• 11.88 MiB memory;
• 48.0 reconstruction FPS;
• 460.73 RGB-rendering FPS;
• 1,086.54 VAE-rendering FPS;
• self-consistency PSNR of 16.80, SSIM of 0.441, and LPIPS of 0.366;
• generation LPIPS of 0.547 and FID of 48.57;
• translation error of 0.093 and rotation error of 0.142;
• MEt3R error of 0.078.

On RealEstate10K, it reports:

• 25K primitives;
• 8.30 MiB memory;
• 16.86 reconstruction FPS;
• 577.0 RGB-rendering FPS;
• 1,101.93 VAE-rendering FPS;
• self-consistency PSNR of 18.36, SSIM of 0.604, and LPIPS of 0.269;
• generation LPIPS of 0.518 and FID of 36.09;
• translation error of 0.023 and rotation error of 0.216;
• MEt3R error of 0.029.

The distilled L3G-World Fast model uses 70K primitives on DL3DV and 42K on RealEstate10K in the reported table, trading some consistency and reconstruction quality for speed.

The paper reports 9.02 FPS end-to-end in a 400-frame memory analysis on one H100 GPU, with 7.4 GiB peak GPU memory and $O(S)$9 memory scaling. The headline speedups are reported relative to the most efficient prior explicit 3D-memory baseline for each metric:

• up to $\mathcal{W}_k$0 fewer 3D primitives;
• approximately $\mathcal{W}_k$1 faster memory reconstruction;
• up to $\mathcal{W}_k$2 faster VAE-feature rendering.

These ratios are metric- and baseline-dependent. Direct comparisons with NeoVerse yield substantially different primitive-count ratios, while VAE-conditioning speedups reflect the avoidance of RGB rendering followed by VAE encoding.

Ablation findings

The ablations support the principal components:

• Without adaptive spawning: approximately 1.8 times more Gaussians and worse RGB and VAE novel-view synthesis.
• Without refinement: reconstruction quality decreases because previous geometry cannot be corrected.
• Without global self-attention: spawn and refinement streams coordinate less effectively.
• Without RGB loss: VAE reconstruction also degrades, demonstrating the role of RGB supervision in learning accurate geometry.
• Without VAE lifting: VAE rendering falls to approximately 2.97 FPS because RGB must be rendered and re-encoded.

Removing memory refinement from the complete world model worsens self-consistency, generation quality, pose accuracy, and 3D consistency.

Limitations

L3G-World is primarily designed for static scenes. It does not explicitly disentangle independently moving objects from persistent geometry, so dynamic objects may be fused into memory or generate inconsistent updates. Memory does not grow linearly with frame count, but it still grows with explored area and may become large during extensive exploration.

Autoregressive errors remain another limitation. An incorrect generated chunk may be written into memory, and later refinement cannot guarantee recovery. Unseen-space generation is also uncertain because it relies on learned video priors rather than ground-truth geometry.

The method depends on Depth Anything 3 for geometric features, initialization, spawning, and depth supervision. Monocular-depth failures may therefore affect camera alignment and memory construction. Its large models require substantial computational resources, and real-time performance on smaller hardware is not established.

Physical interpretation is outside the primary scope of L3G-World. Unlike code-defined or simulator-grounded systems, its memory is a learned visual representation rather than an executable physical state. Consequently, spatial consistency and visual plausibility do not by themselves establish correct object dynamics, collision behavior, mass, or interaction affordances.

L3G-World’s principal contribution is the integration of selective, revisable 3D Gaussian memory with directly renderable VAE latent features. L3G-Mem makes persistent memory spatially anchored and selectively extensible, while latent Gaussian splatting removes the RGB-to-VAE bottleneck in diffusion conditioning. Combined with few-step DiT distillation, these mechanisms provide a framework for real-time, long-horizon interactive video generation at approximately 9 FPS while preserving previously explored scene content more effectively than temporal-context and explicit RGB-memory baselines.