L3G-World: Real-Time Infinite World Generation with Latent 3D Gaussian Memory

Abstract: Interactive world models generate visual content conditioned on user actions, yet maintaining spatial consistency over long-horizon exploration in real time remains challenging. Existing approaches rely on either temporal context or explicit 3D memory, both of which scale poorly because memory grows with the length of the generated stream. Moreover, explicit 3D memories condition latent video generators by re-encoding RGB renderings, which precludes real-time generation. In this paper, we introduce a Latent 3D Gaussian Memory (L3G) that combines the efficiency of feed-forward reconstruction with a compact, rapidly renderable scene representation. Building on this memory, we propose L3G-World, in which a memory updater and a video diffusion transformer (DiT) alternate autoregressively to generate spatially consistent worlds. The memory updater spawns Gaussians in newly observed regions and refines existing ones, so that the memory grows with scene extent rather than stream length. Each Gaussian further stores a VAE latent feature, allowing the memory to be rendered directly into the DiT's latent space without VAE re-encoding. Experiments show that L3G-World achieves state-of-the-art generation quality and long-horizon spatial consistency, with up to $38.6\times$ fewer 3D primitives, $3.2\times$ faster memory updates, and $359\times$ faster memory rendering relative to the most efficient prior 3D-memory baseline for each metric, enabling real-time world generation at 9 FPS.

• The paper introduces L3G-World, a method using Latent 3D Gaussian Memory (L3G) for real-time, efficient 3D world generation, improving long-horizon consistency.
• L3G-World achieves significant efficiency gains, using 266-417 times fewer primitives and 42-67 times less memory compared to existing baselines
• The system demonstrates high performance with 9.02 FPS on a single GPU, making it suitable for real-time applications with strong self-consistency in generated video.

Problem setting and central contribution

L3G-World addresses the computational and geometric difficulty of long-horizon interactive video generation. A world model must synthesize successive video chunks under a camera trajectory while preserving scene identity when the trajectory revisits previously generated regions. Existing approaches typically use either temporal memory—such as retrieved frames or attention key–value caches—or explicit 3D memory reconstructed from generated observations. Temporal memory grows with the number of generated frames and therefore scales with stream length. Explicit 3D memory provides stronger spatial anchoring, but feed-forward methods often accumulate redundant per-frame primitives, whereas optimization-based methods incur prohibitive reconstruction latency.

The paper’s central contribution is a Latent 3D Gaussian Memory (L3G): a persistent collection of 3D Gaussian primitives that stores both geometric and appearance information and an associated VAE latent feature. L3G-World alternates two modules autoregressively. L3G-Mem updates the persistent scene representation by spawning Gaussians in insufficiently represented regions and refining visible existing Gaussians. L3G-Gen renders the stored latent features directly under the next camera trajectory and uses them as conditioning for a latent video diffusion transformer. This avoids the conventional pipeline of RGB rendering followed by VAE re-encoding.

The resulting representation is intended to scale with explored spatial extent rather than with the number of generated frames. This distinction is operationally important: a revisited region is revised in place rather than represented by another accumulated set of primitives. The paper reports real-time generation at 9.02 FPS on a single H100 GPU, while retaining long-horizon loop-closure consistency (2610.40614).

Latent 3D Gaussian memory

L3G-Mem represents the memory after processing window $k$ as paired Gaussian primitives and learned memory tokens. Each token is decoded into a compact group of Gaussians, and each spawned Gaussian additionally stores a VAE latent feature. The updater processes windows of frames, camera parameters, and VAE latents, using a geometry foundation model to extract image features.

The memory update is governed by two complementary mechanisms. First, adaptive spawning identifies image regions where the current memory either has insufficient opacity coverage or produces a high RGB reconstruction error. Geometry features from these regions are clustered with K-means to produce a bounded set of spawning queries. Because clustering is restricted to missing or poorly reconstructed regions, the updater does not allocate new primitives uniformly over the entire input window. Each query produces a group of Gaussians, allowing the memory to remain substantially more compact than dense per-pixel reconstruction.

Second, the refinement stream retrieves memory tokens corresponding to Gaussians visible from the current cameras. A transformer processes spawning and refinement tokens jointly, allowing newly observed geometry and previously reconstructed geometry to exchange information. The refinement head predicts residual updates to Gaussian position, scale, orientation, opacity, color, and latent feature. In-place refinement is a substantive component of the method: the memory is not merely an append-only cache, but a mutable scene representation that can correct errors propagated from earlier generated chunks.

The memory updater is trained autoregressively over consecutive windows. Its objective combines RGB reconstruction, LPIPS, depth, VAE latent, and visibility losses. Supervision is applied not only to the current window but also to target views from earlier windows, encouraging the updated memory to remain globally consistent. Gradients are stopped through previous memory states to preserve the causal update structure.

The VAE-lifting branch is particularly important for the system-level efficiency claim. It aligns VAE latent tokens with spawning queries and decodes them jointly with Gaussian parameters. The resulting latent features are rasterized using the same geometry and alpha compositing as the RGB attributes. Consequently, the memory can supply the diffusion model with target-view latent features without running a VAE encoder during inference.

Memory-conditioned video generation

L3G-Gen is built on a video DiT derived from Cosmos-Predict2.5. At each generation step, the system renders the previous memory from the target cameras, obtaining a spatially aligned VAE latent condition and opacity mask. These are concatenated with the noisy target latents. Previously generated reference latents are also provided as fixed conditioning tokens, while only the target chunk is denoised.

Camera conditioning uses camera-ray positional encoding, and temporal rotary embeddings place reference latents immediately before the current target chunk. This design combines persistent spatial conditioning with short-term temporal context. The 3D memory determines what should appear at a camera pose, while retrieved reference latents provide local appearance and motion context.

The teacher model is trained with rectified flow matching. To achieve interactive latency, the paper distills it into a few-step student using flow-map distillation and on-policy distribution matching. The student is trained on its own autoregressive trajectories rather than exclusively on ground-truth-conditioned trajectories, directly addressing train–test mismatch in long autoregressive rollouts. At inference, the student generates short latent chunks with four denoising steps and periodically updates the L3G memory.

The complete loop is therefore:

1. Render the current latent Gaussian memory under the target cameras.
2. Generate a new video chunk with memory and reference-latent conditioning.
3. Decode RGB only when necessary for observation and presentation.
4. Feed the generated latent and RGB observations to L3G-Mem.
5. Spawn new Gaussians and refine visible existing Gaussians.
6. Use the updated memory for the next camera trajectory segment.

This coupling makes the 3D memory both the persistent spatial state and the interface between world reconstruction and video generation.

Experimental methodology

The evaluation uses DL3DV and RealEstate10K, with a single-image initialization setting. The authors construct 481-frame loop-closure trajectories: the camera explores away from the initial view and then retraces the trajectory in reverse. Self-consistency is measured by comparing generated frames at corresponding poses on the outgoing and returning paths using PSNR, SSIM, and LPIPS.

The evaluation also measures generation quality with LPIPS, FID, and VBench metrics; camera controllability with translation and rotation error; geometric consistency with MEt3R; and memory efficiency through primitive count, memory footprint, reconstruction FPS, RGB rendering FPS, and VAE-space rendering FPS. Explicit-memory baselines are aligned using metric depth from Depth Anything 3 to reduce scale discrepancies in the single-image setting.

This evaluation is well matched to the claimed contribution. A short forward trajectory could conceal accumulation and loop-closure failures, whereas the reverse traversal directly tests whether the persistent memory remains spatially anchored after extended autoregressive generation.

Quantitative results

The efficiency results are the strongest aspect of the empirical study. Relative to NeoVerse, the Gaussian-based baseline, L3G-World uses approximately 266–417 times fewer primitives and 42–67 times less memory across the two datasets. It achieves 8.3–10.4 times faster RGB rendering and 349–359 times higher VAE-conditioning throughput. Against Lyra 2.0, identified as an especially efficient point-cloud baseline, L3G-World reconstructs memory 8.9–9.5 times faster, uses 7.8–10.4 times less memory, and provides 643–648 times higher VAE-rendering throughput. The magnitude of the VAE-space speedup follows directly from eliminating RGB-to-VAE re-encoding rather than from rasterization alone.

The computational comparison in the paper’s broader memory analysis places L3G-World at 7.4 GiB peak GPU memory and 9.02 FPS for 400-frame generation on one H100, whereas temporal-context and dense feed-forward memory baselines either scale as $O(TR)$, reach substantially higher memory use, or fail to achieve real-time throughput. Optimization-based Gaussian memory has favorable scene-extent scaling but remains too slow because of iterative reconstruction.

On DL3DV, L3G-World achieves 16.80 PSNR and 0.366 LPIPS for the reported self-consistency measures, compared with 13.90 PSNR and 0.573 LPIPS for the strongest listed point-cloud alternatives in the relevant group. It also obtains FID 48.57, imaging quality 0.658, aesthetic quality 0.497, and background consistency 0.820. The distilled L3G-World Fast variant obtains higher imaging and aesthetic quality in this setting—0.672 and 0.515, respectively—although its self-consistency is weaker than the full model.

On RealEstate10K, L3G-World obtains 18.36 PSNR, 0.269 self-consistency LPIPS, FID 36.09, and background consistency 0.841. These results are stronger than those of the listed explicit-memory baselines on the principal appearance and consistency metrics. The model does not dominate every geometric measure: some baselines achieve lower pose or MEt3R errors on RealEstate10K. The paper therefore supports a more specific conclusion than universal superiority: L3G-World offers a favorable joint trade-off among long-horizon consistency, generation quality, and memory efficiency, while remaining comparatively competitive rather than uniformly best on camera and geometric estimation metrics.

The online reconstruction comparison reinforces this trade-off. On DL3DV, L3G-Mem uses 39K primitives, 14.11 MiB, and reconstructs at 28.66 FPS, while ReCoSplat at the same $480 \times 832$ resolution uses 11,550K primitives, 1,101.48 MiB, and reconstructs at 2.15 FPS. L3G-Mem also reaches 735.3 VAE-rendering FPS compared with 2.7 FPS for ReCoSplat at that resolution. On RealEstate10K, L3G-Mem uses 25K primitives and 8.94 MiB, with 29.03 FPS reconstruction and 1,333.3 VAE-rendering FPS. These comparisons show that the representation is deliberately optimized for repeated online updates and latent conditioning rather than for maximum standalone novel-view fidelity.

Ablation evidence

The ablations establish that the main components are functionally coupled rather than merely implementation choices. Removing adaptive spawning increases the number of Gaussians from 41K to 75K—approximately 1.8 times more—while reducing VAE latent quality and RGB NVS performance. This supports the claim that selective allocation reduces redundancy without simply imposing an efficiency–quality trade-off.

Removing refinement reduces the number of Gaussians but degrades VAE MSE from 0.177 to 0.213, VAE cosine similarity from 0.782 to 0.743, PSNR from 16.52 to 15.52, and LPIPS from 0.534 to 0.571. The implication is that compactness alone is insufficient: without residual correction, errors in previously spawned geometry remain fixed and contaminate later conditioning.

Removing global self-attention between spawning and refinement streams also weakens NVS quality. This indicates that new geometry and existing geometry must be coordinated during an update; independent processing is inadequate when newly observed regions are spatially related to already stored content.

RGB supervision is unexpectedly important even though L3G-Gen consumes rendered VAE features. Without RGB loss, PSNR falls sharply to 10.26, SSIM to 0.272, and LPIPS worsens to 0.751. Thus, latent supervision does not by itself enforce sufficiently accurate geometric and appearance reconstruction. The RGB rendering objective provides a necessary constraint on the shared Gaussian representation.

The world-generation ablation yields a similar result. Removing memory refinement reduces DL3DV self-consistency PSNR from 15.11 to 14.82, worsens generation LPIPS from 0.583 to 0.592, increases FID from 48.09 to 50.63, and increases translation error from 0.278 to 0.314. Replacing direct VAE latent rendering with RGB rendering followed by VAE encoding also lowers generation quality and introduces the large conditioning overhead documented in the efficiency experiments. These results link the local memory-update design to the system-level long-horizon behavior.

Limitations and open questions

The paper explicitly targets predominantly static scenes and does not model independently moving objects. Its memory associates persistent Gaussian geometry and latent appearance with spatial locations, so dynamic objects would require temporal separation, object-level state, or a mechanism for distinguishing motion from reconstruction error.

Although memory growth is tied to explored spatial extent rather than stream length, it is not bounded under arbitrarily large exploration. The memory can still expand as new regions are observed. The paper identifies pruning and hierarchical memory as unresolved mechanisms, but does not evaluate them.

Autoregressive error propagation also remains a limitation. Refinement can correct inaccurate Gaussians when later observations expose them, but generated video chunks are themselves used as observations for subsequent memory updates. Errors that are not revisited, or that are reinforced by later generations, can therefore persist. The experiments demonstrate minute-long generation and 481-frame loop closures, but they do not establish indefinite stability.

The comparison also depends on several assumptions that constrain interpretation. Camera trajectories and camera parameters are provided to the system; explicit-memory baselines receive metric-depth alignment from a common foundation model; and training uses datasets with ground-truth or externally estimated geometry and camera information. The evaluation consequently measures image-conditioned generation under controlled camera trajectories rather than fully autonomous scene discovery. Finally, the method’s strongest throughput claims concern latent conditioning and memory operations; end-to-end interactive latency also depends on the distilled DiT, decoding, camera processing, and the frequency of memory updates.

Conclusion

L3G-World presents a coherent solution to the tension between persistent 3D scene memory and real-time latent video generation. Its principal technical contributions are adaptive Gaussian spawning, residual refinement of existing primitives, and direct storage and rendering of VAE latent features. The resulting memory scales primarily with explored scene extent, avoids repeated VAE encoding, and provides spatially aligned conditioning for an autoregressive video DiT.

The reported results support the paper’s main claim: L3G-World substantially reduces primitive count, memory use, reconstruction cost, and latent-conditioning overhead while achieving strong long-horizon self-consistency and 9.02 FPS generation on one H100. Its remaining technical questions concern dynamic-scene memory, bounded scaling under very large exploration, and robustness against unrevisited autoregressive errors.