Latent 3D Gaussian Memory (L3G)

• Latent 3D Gaussian Memory (L3G) is a spatial memory architecture that uses 3D Gaussian primitives for explicit geometric addressing and compact latent features for encoding various task-relevant information.
• L3G supports several applications, including semantic querying, persistent scene reconstruction, robotic manipulation, and autoregressive world generation with examples like 'Robust Dreamer' and 'L3G-World',
• The effectiveness of L3G is demonstrated in tasks such as visual world generation, streaming novel-view reconstruction, open-vocabulary semantic mapping, and long-horizon robotic manipulation.
• Follow_up_question_1
• What are the primary advantages of L3G over conventional 3D Gaussian Splatting (3DGS) systems?
• Follow_up_question_2
• How does L3G address the challenges of memory growth and spatial addressing in extensive environments?
• Follow_up_question_3
• What impact has streamSplat's voxel-based addressing mechanism had on the efficiency and performance?
• Follow_up_question_4
• How does the use of 3D Gaussian primitives improve the recall process in memory systems?
• Follow_up_question_5,
• Find recent papers about memory-centric Gaussian optimization in spatial data structures.

Latent 3D Gaussian Memory (L3G) denotes a class of spatial memory architectures in which 3D Gaussian primitives provide explicit geometric addressing while compact latent features encode appearance, semantics, diffusion states, or other task-relevant information. Unlike conventional 3D Gaussian Splatting (3DGS), which stores geometry and RGB or spherical-harmonic appearance primarily for rendering, an L3G system stores or generates Gaussian-associated latent variables and recalls them by projection and Gaussian splatting. The representation can therefore support view-aligned retrieval, novel-view synthesis, semantic querying, persistent scene reconstruction, robotic manipulation, and autoregressive world generation. The term is used explicitly for the memory mechanism in “Robust Dreamer” (Chen et al., 29 May 2026) and “L3G-World” (Choi et al., 29 Oct 2026), while several other works provide related latent, compressed, streaming, or task-driven Gaussian representations.

1. Conceptual foundations and terminology

An L3G memory combines an explicit spatial substrate with a latent payload. A Gaussian primitive generally has a center $\boldsymbol{\mu}$, covariance $\boldsymbol{\Sigma}$, opacity $\alpha$, and appearance-related attributes. Its covariance is commonly parameterized as

$$\boldsymbol{\Sigma} = \mathbf{R}\operatorname{diag}(\mathbf{s}^2)\mathbf{R}^{\top},$$

where $\mathbf{s}$ is an anisotropic scale vector and $\mathbf{R}$ is derived from a rotation quaternion. An L3G primitive additionally stores a latent feature $\mathbf{z}$ or $\mathbf{x}$:

$$\mathcal{G}_i = (\boldsymbol{\mu}_i,\boldsymbol{\Sigma}_i,\alpha_i,\mathbf{c}_i,\mathbf{z}_i).$$

The Gaussian geometry determines where the memory entry is located, its projected footprint, visibility, depth ordering, and spatial support. The latent feature determines what information is recalled at that location. In diffusion-oriented systems, this feature can be a VAE latent rather than RGB; in semantic mapping, it can be a compact query vector that reconstructs a high-dimensional VLM embedding; in robotic memory, it can be a learned feature consumed by a VLA policy.

The central operation is latent Gaussian splatting. For a target pixel $u$, depth-ordered Gaussian contributions are alpha-composited:

$\boldsymbol{\Sigma}$0

The same visibility weights can render RGB, depth, semantic features, or diffusion latents. This permits a memory query to be expressed as a camera-conditioned projection rather than as retrieval from a purely temporal token sequence.

L3G should be distinguished from several neighboring concepts. A compressed Gaussian model may reduce the number of explicit primitives without introducing latent variables, as in “Distilled-3DGS” (Xiang et al., 19 Aug 2025). A latent Gaussian representation may provide generative compression without persistent updates or addressing, as in “Sampling 3D Gaussian Scenes in Seconds with Latent Diffusion Models” (Henderson et al., 2024). A structured latent codec may store a scene code from which Gaussians are regenerated, as in “CodecSplat” (Yu et al., 25 May 2026). By contrast, a persistent L3G system additionally requires mechanisms for spatial addressing, memory writes, updates, fusion, pruning, and—where relevant—temporal consistency.

2. Representation and memory addressing

Spatially explicit Gaussian state

The geometric component of L3G is typically a set of anisotropic Gaussians in a common coordinate system. In “Robust Dreamer,” each memory element is

$\boldsymbol{\Sigma}$1

where $\boldsymbol{\Sigma}$2 is a diffusion-space latent feature. In “LatentAM,” the corresponding feature-augmented primitive is

$\boldsymbol{\Sigma}$3

with a default compact semantic feature dimension $\boldsymbol{\Sigma}$4, compared with a CLIP feature dimension $\boldsymbol{\Sigma}$5 (Lee et al., 12 Feb 2026). In “StreamSplat,” the persistent state is instead a set of latent 3D tokens containing world-space position, latent feature, confidence, and a high-confidence seed feature; these tokens are decoded into Gaussian primitives after cache updates (Song et al., 3 Aug 2026).

This separation allows the memory to preserve explicit metric geometry while avoiding the cost of storing a high-dimensional feature independently at every primitive. “LatentAM” reconstructs VLM embeddings using a shared dictionary:

$\boldsymbol{\Sigma}$6

where $\boldsymbol{\Sigma}$7 contains compact per-Gaussian queries, $\boldsymbol{\Sigma}$8 is a learned projection, and $\boldsymbol{\Sigma}$9 is a dictionary of representative VLM features. The Gaussian stores the compact query, while the dictionary stores shared high-dimensional semantic information (Lee et al., 12 Feb 2026).

Spatial indexing

Persistent memory requires an addressing mechanism. Several systems use voxel-based addressing. “StreamSplat” stores at most $\alpha$0 latent pivots per voxel, with voxel side length $\alpha$1 and confidence threshold $\alpha$2. When a voxel is full, the two most similar tokens are merged using confidence-weighted position and feature averages (Ren et al., 16 Aug 2026).

“LatentAM” uses a voxel hash with default voxel size $\alpha$3 m. The global map and hash are stored on the CPU, while only an active local region is transferred to the GPU. The resulting GPU memory is governed by the active spatial neighborhood rather than the total trajectory length (Lee et al., 12 Feb 2026).

“L3G-World” uses memory tokens that decode into groups of 32 Gaussian primitives. Its updater identifies uncovered or poorly reconstructed image patches, spawns new memory tokens in those regions, and refines existing visible Gaussians rather than appending duplicates (Choi et al., 29 Oct 2026). The intended scaling is with explored spatial extent rather than the number of generated frames.

An alternative is structured latent addressing. “Fusion-Aware Direct 3D Gaussian Generation with Structured Patch Latent Flows” (Wang et al., 18 Aug 2026) decomposes a Gaussian object into canonical local patches, each with a patch index, local frame, local latent, density, visibility, and global semantic latent. “TriFold” (Wang et al., 16 Oct 2026) organizes Gaussian information into a fixed-layout triplane with four latent slots per plane cell and uses occupancy prediction to identify regions that should be decoded. These representations offer regular latent addresses but do not themselves implement persistent memory updates.

World coordinates, local coordinates, and canonicalization

World-coordinate addressing is essential for sequential observations and camera motion. “StreamSplat” back-projects depth estimates into world-space tokens before writing them into the Voxel-Aligned Causal Cache. “Robust Dreamer” predicts globally aligned geometry for each frame and attaches the frame’s diffusion latent directly to the corresponding Gaussian primitives. “GaussMemory” augments Gaussian entries with object identifiers, creation time, modification time, sub-task indices, and existence probabilities, thereby extending spatial memory with temporal and episodic metadata (Hu et al., 15 Aug 2026).

For object-level storage, canonical local coordinates can improve comparability. “TriFold” transforms each scene into a canonical domain and encodes Gaussian information into triplane cells. “Fusion-Aware Direct 3D Gaussian Generation” uses canonical patch anchors, local coordinate frames, PCA-based orientations, and Morton ordering within patches (Wang et al., 18 Aug 2026). These operations are suitable for object-centric retrieval and local patch replacement, but persistent scene memory additionally requires transformations between local frames and a stable global frame.

3. Writing, updating, and consolidating memory

Observation-to-memory fusion

An L3G update generally consists of extracting Gaussian or latent candidates from a new observation, associating them with existing memory entries, and either refining, merging, or inserting them. The update must account for geometry, appearance, visibility, confidence, and temporal relevance.

“StreamSplat” uses a causal update cycle. Current calibrated views are encoded into image features and a cost volume; depth hypotheses are back-projected into world-space tokens; historical geometry is projected into the current views; and the resulting tokens are inserted into voxel cells. Low-confidence tokens are rejected. When a voxel reaches its pivot capacity, the most similar latent tokens are merged using confidence-weighted position and feature updates (Ren et al., 16 Aug 2026).

Its History-Projected Depth Anchoring (HPDA) provides geometric retrieval. Cached geometry is projected into the current camera and converted into a confidence-dependent depth distribution. The historical distribution is combined softly with the current cost-volume distribution, allowing existing geometry to guide but not hard-constrain new estimates. Cache-Guided Feature Injection (CGFI) performs the analogous operation for latent appearance features, injecting more historical evidence where current depth confidence is low.

“L3G-World” uses a complementary strategy based on rendered residuals. The previous memory is rendered from the current context cameras. A spawn mask marks image patches with high RGB error or insufficient opacity. Depth Anything 3 features are extracted only in those regions, clustered into spawning queries, and decoded into new Gaussian groups. Existing Gaussians visible in the current views are retrieved and updated through residual predictions for position, scale, rotation, opacity, RGB appearance, and latent feature (Choi et al., 29 Oct 2026).

Learned update policies

In task-driven memory, uniform fusion is insufficient because not all scene elements have equal importance. “GaussMemory” uses Unified Memory Attention (UMA) to couple memory readout and memory writing. Current Gaussian observations are pooled into object-level tokens, while task-conditioned queries are generated from the language instruction and sub-task index. Both streams attend to persistent Gaussian memory.

The observation-side attention matrix provides soft correspondences between current object tokens and stored memory entries. For memory entry $\alpha$4, the evidence is

$\alpha$5

A learned update gate determines how strongly the stored position is moved toward the observed position:

$\alpha$6

The system also maintains an existence probability and inserts observations that are poorly matched to existing entries. Entries with existence probability below $\alpha$7 are pruned. The reported average update coefficient is approximately $\alpha$8 for manipulated objects and approximately $\alpha$9 for background Gaussians (Hu et al., 15 Aug 2026).

This establishes a distinction between geometric persistence and task-conditioned precision. A plausible implication is that an L3G memory can allocate higher update bandwidth to task-relevant regions while retaining low-frequency or low-confidence background structure in a compressed state.

Pruning, compression, and memory budgets

Memory consolidation can be based on rendering contribution, spatial redundancy, task relevance, confidence, or latent similarity. “A Refined 3D Gaussian Representation for High-Quality Dynamic Scene Reconstruction” uses a learnable denoising mask that removes redundant or noisy Gaussians and applies static and motion-consistency constraints (Zhang et al., 2024). Its ablation reduced point count from $$\boldsymbol{\Sigma} = \mathbf{R}\operatorname{diag}(\mathbf{s}^2)\mathbf{R}^{\top},$$0k to $$\boldsymbol{\Sigma} = \mathbf{R}\operatorname{diag}(\mathbf{s}^2)\mathbf{R}^{\top},$$1k and storage from $$\boldsymbol{\Sigma} = \mathbf{R}\operatorname{diag}(\mathbf{s}^2)\mathbf{R}^{\top},$$2 MB to $$\boldsymbol{\Sigma} = \mathbf{R}\operatorname{diag}(\mathbf{s}^2)\mathbf{R}^{\top},$$3 MB.

“Gaussians on a Diet” enforces a target Gaussian budget throughout training rather than pruning only after uncontrolled densification. It combines hybrid position-and-color gradients, error-guided Gaussian compensation, adaptive position adjustment, and rendering-contribution pruning. On Jetson AGX Xavier, reported peak memory decreased from $$\boldsymbol{\Sigma} = \mathbf{R}\operatorname{diag}(\mathbf{s}^2)\mathbf{R}^{\top},$$4 GB for 3DGS to $$\boldsymbol{\Sigma} = \mathbf{R}\operatorname{diag}(\mathbf{s}^2)\mathbf{R}^{\top},$$5 GB, or $$\boldsymbol{\Sigma} = \mathbf{R}\operatorname{diag}(\mathbf{s}^2)\mathbf{R}^{\top},$$6 GB with a dynamic data loader (Zhang et al., 21 Apr 2026).

“StreamingGS” addresses deployment traffic rather than persistent semantic memory. It stores Gaussians by voxel, uses hierarchical filtering, vector quantization, and memory-centric streaming, reporting up to $$\boldsymbol{\Sigma} = \mathbf{R}\operatorname{diag}(\mathbf{s}^2)\mathbf{R}^{\top},$$7 speedup and $$\boldsymbol{\Sigma} = \mathbf{R}\operatorname{diag}(\mathbf{s}^2)\mathbf{R}^{\top},$$8 energy reduction relative to a mobile Nvidia Ampere GPU (Zhang et al., 9 Jun 2025). Its main L3G-relevant principle is to retain lightweight geometric metadata and decode expensive attributes only after relevance has been established.

“Distilled-3DGS” consolidates multiple teacher Gaussian representations into a smaller explicit student using rendered supervision and voxel-histogram structural distillation. It reports Gaussian-count reductions of approximately $$\boldsymbol{\Sigma} = \mathbf{R}\operatorname{diag}(\mathbf{s}^2)\mathbf{R}^{\top},$$9–$\mathbf{s}$0 relative to vanilla 3DGS on the evaluated datasets, but it does not introduce a latent code, retrieval policy, or continual update mechanism (Xiang et al., 19 Aug 2025).

4. Memory recall and downstream use

View-conditioned latent recall

The defining L3G read operation is projection from persistent world-space memory into a target camera. “Robust Dreamer” stores diffusion latents directly on Gaussian primitives and recalls them through latent-space Gaussian splatting. Its condition for the diffusion model is

$\mathbf{s}$1

where $\mathbf{s}$2 is the target-view latent rendered from memory and $\mathbf{s}$3 is the action or camera control (Chen et al., 29 May 2026).

This avoids Latent–RGB Cycling: generated latent features are not decoded to RGB and re-encoded before being written to memory. The memory therefore remains in the representation space used by the video diffusion model. The paper reports approximately $\mathbf{s}$4 memory reduction and $\mathbf{s}$5 runtime reduction relative to a naïve 3D-aware memory configuration for an 81-frame sequence at $\mathbf{s}$6 resolution.

“L3G-World” follows the same direct-latent principle. Each Gaussian stores a VAE latent feature, and memory rendering produces a target-view latent and opacity map that are concatenated directly with the noisy target latent of the video DiT. The system alternates between L3G-Mem, which spawns and refines Gaussian memory, and L3G-Gen, which generates subsequent video chunks (Choi et al., 29 Oct 2026). It reports real-time world generation at 9 FPS, up to $\mathbf{s}$7 fewer 3D primitives, $\mathbf{s}$8 faster memory updates, and $\mathbf{s}$9 faster memory rendering relative to the relevant prior 3D-memory baselines.

Semantic recall

In semantic mapping, the latent payload is not necessarily a diffusion feature. “LatentAM” renders compact Gaussian query vectors into a spatial feature map and reconstructs approximate CLIP, DINOv3, or LSeg embeddings using a learned dictionary. Text embeddings can then be compared with reconstructed spatial features to produce open-vocabulary scores over pixels or Gaussians (Lee et al., 12 Feb 2026).

The system combines streaming K-means dictionary initialization, online dictionary optimization, trust-region regularization, historical replay, and CPU-global/GPU-local map management. It reports approximately $\mathbf{R}$0–$\mathbf{R}$1 FPS, a default dictionary size of $\mathbf{R}$2, and successful mapping over a custom trajectory of approximately $\mathbf{R}$3 m, where competing methods experienced GPU memory exhaustion.

Robotic task recall

“GaussMemory” uses Gaussian memory as a tokenized spatial substrate for a VLA policy. Persistent Gaussians are encoded into memory tokens, retrieved through task-conditioned attention, and passed to a Llama-2-7B-based multimodal policy together with current visual tokens, language tokens, and proprioceptive state tokens. Memory readout and update share UMA layers, allowing action loss to influence which observations are retained and how aggressively they are updated (Hu et al., 15 Aug 2026).

On LIBERO, GaussMemory reports success rates of $\mathbf{R}$4 on Goal and $\mathbf{R}$5 on Long-10, compared with $\mathbf{R}$6 and $\mathbf{R}$7 for MemoryVLA. On VLABench, it reports $\mathbf{R}$8 success on Track 1 and $\mathbf{R}$9 on Track 6, compared with $\mathbf{z}$0 and $\mathbf{z}$1 for $\mathbf{z}$2-FAST-ft. Its memory cap is $\mathbf{z}$3K Gaussians, with approximately $\mathbf{z}$4 live Gaussians grouped into roughly 20 object-level observation tokens.

Generative completion and scene reconstruction

Several systems use latent Gaussian representations without persistent memory. A 3D-aware VAE and latent diffusion model can encode multi-view images into compact tensors, decode those tensors into Gaussian splats, and render novel views after denoising (Henderson et al., 2024). The reported latent representation gives a $\mathbf{z}$5 compression relative to treating splat parameters directly as diffusion variables, and generation can take approximately $\mathbf{z}$6 seconds in the reported experiments.

“Prometheus” uses an RGB-D latent space, cross-view transformer, Plücker-ray conditioning, and a Gaussian decoder to generate object- and scene-level Gaussian representations from text, images, or camera-conditioned multi-view inputs (Yang et al., 2024). Its reported text-to-3D generation time is approximately 8 seconds. These systems demonstrate latent-to-Gaussian decoding and uncertainty-aware completion, but their latent codes are generative compression rather than persistent, addressable episodic memories.

5. Training objectives and consistency mechanisms

L3G training typically combines rendering fidelity, latent reconstruction, geometric consistency, memory sparsity, and task or temporal objectives.

For dynamic Gaussian reconstruction, “A Refined 3D Gaussian Representation” uses a standard rendering objective based on $\mathbf{z}$7 and D-SSIM, augmented with denoising, sparsity, static-motion, and motion-consistency losses. The deformable MLP predicts time-dependent offsets to Gaussian position, rotation, scale, and color. The static and motion-consistency constraints suppress spurious motion and reduce temporal variation among similarly moving points (Zhang et al., 2024).

For semantic latent memory, “LatentAM” combines RGB, depth, and feature reconstruction losses:

$\mathbf{z}$8

The feature objective contains cosine error, Euclidean feature error, and a trust-region penalty on dictionary movement. Historical replay and trust-region regularization reduce semantic drift during online dictionary adaptation (Lee et al., 12 Feb 2026).

For autoregressive latent world generation, “Robust Dreamer” uses flow matching and a Dynamic Deviation Archive. The archive stores rollout-induced latent deviations indexed by autoregressive stage and denoising timestamp. During training, deviations are injected into historical latent states and into the corresponding Gaussian memory, exposing the denoiser to prediction-corrupted memory. The reported one-step deviation approximation reaches approximately $\mathbf{z}$9 of the error magnitude obtained from explicit ODE simulation, while synthesized deviations outperform generic Gaussian noise, blur, and color-shift corruptions (Chen et al., 29 May 2026).

“L3G-World” uses rectified-flow training for the video DiT and on-policy flow-map distillation for real-time inference. The L3G-Mem updater is trained to spawn Gaussians in high-error or uncovered regions and to predict residual updates for existing visible Gaussians. The memory is therefore jointly optimized for spatial consistency and direct latent conditioning (Choi et al., 29 Oct 2026).

Self-supervised representation learning can provide an L3G encoder. “Gaussian-JEPA” predicts latent representations of held-out Gaussian token blocks rather than reconstructing raw Gaussian attributes. It uses an online encoder, an exponential-moving-average target encoder, multi-scale target blocks, complementary target projections, and feature-space grounding (Ren et al., 16 Aug 2026). On Gaussian resampling, the method reports relative drift of $\mathbf{x}$0 versus $\mathbf{x}$1 for Gaussian-MAE; under 55% missing groups, it reports R@1 of $\mathbf{x}$2 versus $\mathbf{x}$3. The method is not a memory system, but its resampling invariance and partial-observation robustness are desirable properties for one.

For structured scene codes, “TriFold” combines VAE reconstruction, occupancy prediction, local set decoding, Sinkhorn optimal-transport matching, rendering supervision, phantom-voxel penalties, and KL regularization (Wang et al., 16 Oct 2026). Its decoder predicts variable numbers of Gaussian candidates in occupied voxels, while the latent maintains a fixed triplane layout. “CodecSplat” instead uses a hierarchical learned feature codec with entropy modeling and a rate-distortion objective, storing quantized latent symbols and camera metadata rather than explicit Gaussian primitives (Yu et al., 25 May 2026).

6. Applications, limitations, and research directions

Applications

L3G architectures support several application classes:

• Long-horizon visual world generation: direct latent rendering supplies spatially aligned conditioning without repeated VAE conversion, as demonstrated by Robust Dreamer and L3G-World.
• Streaming novel-view reconstruction: StreamSplat maintains a causal latent cache whose quality improves as calibrated observations accumulate.
• Open-vocabulary semantic mapping: LatentAM associates Gaussian geometry with compact semantic queries and reconstructs embeddings from a shared VLM dictionary.
• Long-horizon robotic manipulation: GaussMemory couples task-conditioned retrieval with learned memory updates, insertion, and pruning.
• Compact scene storage and transmission: CodecSplat stores entropy-coded latent features from which feed-forward Gaussian scenes are regenerated.
• Object retrieval and completion: Gaussian-JEPA provides resampling-stable and partial-observation-robust features, while structured patch and triplane models provide local latent units for completion and generation.
• Dynamic scene reconstruction: deformable Gaussian representations use temporal deformation fields, learned masks, and motion-consistency constraints to reduce memory and temporal noise (Zhang et al., 2024).

Limitations

Despite their common terminology, current systems differ substantially in what they retain and how faithfully they recall it. Generative latent codes can produce plausible but historically incorrect completions. The latent diffusion model in (Henderson et al., 2024) explicitly performs stochastic completion, and Prometheus similarly hallucinates unseen regions rather than preserving a guaranteed historical scene (Yang et al., 2024). A memory system intended for factual recall therefore requires provenance, confidence, and an explicit distinction between observed and generated content.

Camera calibration and pose quality remain central. StreamSplat, LatentAM, and several feed-forward reconstruction systems assume calibrated cameras or an external tracker. Pose errors can misproject cached geometry, corrupt voxel assignment, and create persistent artifacts. Monocular depth introduces scale and shift ambiguity, while occlusions, reflective surfaces, textureless regions, thin structures, and topology changes remain difficult.

Memory growth is controlled but not eliminated. Voxel caches grow with explored spatial extent; active-local strategies bound GPU working sets but not total CPU storage; patch and triplane representations use bounded canonical domains; and Gaussian memories may still become large in very extensive environments. Fixed voxel sizes, fixed pivot counts, and fixed latent capacities can merge distinct surfaces or discard rare appearance modes.

Latent compression also creates decoder dependence. CodecSplat, TriFold, and other latent codecs do not preserve the exact original Gaussian set. Their stored codes are meaningful only together with shared decoder parameters. This improves compactness but complicates versioning, cross-model interoperability, exact editing, and long-term archival.

Dynamic environments present additional difficulties. Deformable Gaussian fields can model time-dependent offsets, and GaussMemory can update manipulated objects, but many systems remain static-scene or structured-motion models. Fluids, fire, topology-changing objects, illumination changes, object identity swaps, and severe non-rigid motion remain incompletely addressed.

Open directions

Several directions follow from the reported systems:

1. Hierarchical and heterogeneous memory: combine explicit coarse geometry, latent patch or voxel blocks, semantic dictionaries, and decoded active Gaussians.
2. Confidence and provenance: attach observation age, uncertainty, source views, and observed-versus-generated status to Gaussian or latent entries.
3. Adaptive spatial resolution: allocate finer voxels, more pivots, or more latent slots to high-error, task-relevant, or high-density regions.
4. Joint geometry–latent optimization: train memory writing, latent formation, Gaussian decoding, and downstream diffusion or policy objectives together.
5. Temporal and object identity management: integrate persistent IDs, motion models, object-level grouping, and scene versioning.
6. Selective decoding and hardware support: combine L3G with memory-centric streaming, hierarchical filtering, codebook caches, and partial Gaussian materialization.
7. Cross-model semantic interoperability: use dictionaries, shared latent spaces, or adapters to support multiple VLMs and changing downstream tasks.
8. Retrieval benchmarks: evaluate spatial recall, long-horizon consistency, exact geometry preservation, semantic retrieval, update stability, and memory cost separately from image metrics.
9. Memory-aware rate-distortion objectives: optimize rendering, semantic fidelity, update cost, and bit-level storage jointly rather than treating Gaussian count as the sole budget.
10. Uncertainty-aware generation: distinguish faithful reconstruction from plausible completion and allow downstream systems to act differently on observed and hypothesized geometry.

L3G is therefore best understood as an architectural family rather than a single representation. Its common abstraction is

$\mathbf{x}$4

The strongest instantiations combine world-space addressing, confidence-aware writes, learned refinement, direct latent rendering, and task- or view-conditioned readout. The principal unresolved problem is to retain the efficiency and generative flexibility of latent Gaussian representations while providing the fidelity, uncertainty, temporal identity, and update semantics required of a persistent three-dimensional memory.