Latent-to-4D: Dynamic 3D Representation Methods

• Latent-to-4D is a family of methods that decode compact latent vectors, sets, matrices, diffusion states, or point clouds into time-varying 3D outputs such as meshes, implicit fields, Gaussian scenes, radar distributions, and occupancy volumes.
• These systems improve efficiency by separating latent generation from final decoding, using temporal attention, joint diffusion, canonical-plus-deformation models, or continuous-time functions to preserve geometry, appearance, motion, and temporal consistency.
• Applications include human-motion synthesis, dynamic-scene reconstruction, novel-view rendering, radar and cardiac-data generation, beam diagnostics, tracking, and visual reasoning, although performance remains limited by ambiguity, canonical-geometry errors, computational cost, topology changes, and weak physical guarantees.

Latent-to-4D denotes a family of methods that generate, reconstruct, render, or reason about time-varying three-dimensional entities through compact latent representations. The defining computation maps latent variables—global vectors, local latent sets, latent matrices, diffusion latents, persistent perception tokens, or latent point clouds—into a temporally ordered 3D state such as meshes, implicit surfaces, Gaussian primitives, voxel fields, radar distributions, or dynamic visual representations. The term therefore encompasses substantially different output ontologies: dense human meshes, continuous implicit fields, deformable Gaussian scenes, explicit mesh trajectories, panoptic occupancy volumes, physical phase-space densities, and latent visual states for dynamic reasoning. Across these variants, the central objective is to avoid direct modeling of the full spatial-temporal object while preserving geometric, appearance, motion, and temporal-consistency properties.

1. Conceptual foundations and representation regimes

A latent-to-4D system separates the representation used for generation from the representation used for final output. The latent may be decoded into a sequence of dense meshes, as in the structured latent space for human motion (Marsot et al., 2021); into local signed-distance fields for clothed humans (Jiang et al., 2022); into a matrix of per-frame shape and color codes (Yang et al., 2024); or into Gaussian primitives with time-dependent attributes (Shi et al., 9 Jun 2025). In other systems, the latent is decoded into a dynamic occupancy volume, a radar point cloud, or a rendered video rather than a conventional geometric asset.

The term “4D” generally refers to a 3D state indexed by time. Depending on the domain, the temporal state may be represented explicitly or implicitly:

• Explicit mesh trajectories: a canonical mesh retains fixed connectivity while its vertices are assigned time-varying positions, as in DriveAnyMesh (Shi et al., 9 Jun 2025) and Motion 3-to-4 (Chen et al., 20 Jan 2026).
• Implicit dynamic surfaces: a signed-distance or occupancy function is queried at spatial and temporal coordinates, as in LoRD (Jiang et al., 2022) and Motion2VecSets (Cao et al., 2024).
• Dynamic Gaussian fields: canonical Gaussian primitives receive temporal changes in position, scale, rotation, opacity, or appearance, as in Gaussian Variation Field Diffusion (Zhang et al., 31 Jul 2025), Diff4Splat (Pan et al., 1 Nov 2025), and Orster-based 4D generation (Liu et al., 5 Mar 2026).
• Matrixized or tensorized latent states: a temporal sequence is compressed into a latent matrix or jointly modeled tensor, as in the matrixized volumetric generator (Yang et al., 2024) and CardioDiT (Seyfarth et al., 26 Mar 2026).
• Sparse latent point or Gaussian representations: learned point sets or feature-bearing Gaussians are decoded into radar or occupancy representations, as in 4D-RaDiff (Kwok et al., 16 Dec 2025) and LaGS (Luz et al., 26 Feb 2026).
• Latent visual states without explicit geometry: persistent hidden tokens encode dynamic visual semantics for reasoning rather than reconstructing a physical scene, as in 4DThinker (Chen et al., 7 May 2026).
• Latent conditioning for video generation: a large 4D perception model supplies persistent geometric features that condition novel-view video generation without explicit reconstruction, as in LaVR (Xie et al., 21 Jan 2026).

This diversity distinguishes latent-to-4D from a single architectural family. Its common structure is a learned low-dimensional or compressed state whose decoding operation produces a temporally organized representation.

A useful abstraction is:

$$\text{observation or condition} \longrightarrow \text{latent representation} \longrightarrow \text{temporal decoder} \longrightarrow \text{4D output}.$$

The observation may be text, an image, a monocular video, a point-cloud sequence, a camera trajectory, a 3D mesh, a radar scene, an accelerator condition vector, or an incomplete projection. The output may be deterministic or stochastic, explicit or implicit, and directly renderable or intended primarily for downstream reasoning.

2. Latent representations and spatial factorization

Global latent vectors

The earliest form in the supplied research is a low-dimensional global code representing an entire motion sequence. The structured human-motion model maps a pair $(z,\beta)$ to a sequence of dense SMPL meshes, where $z$ captures motion-related information and $\beta$ supplies morphology (Marsot et al., 2021). A single latent vector represents pose evolution, global translation, temporal evolution, and duration, while the decoder generates multiple dense meshes and timestamps.

Global codes provide compactness and enable interpolation, but they must compress all spatial regions and deformation modes into one bottleneck. Motion2VecSets reports that replacing a latent set by a single code substantially degrades reconstruction, particularly for unseen identities and motions (Cao et al., 2024). The reported full set model achieves higher unseen-motion IoU and lower Chamfer and correspondence errors than the $M=1$ variant.

Local latent sets

Local latent sets distribute representational capacity among learned elements. LoRD decomposes a clothed human into $K=2127$ overlapping spherical parts, with a default radius of $5$ cm and local motion, shape, and optional texture codes of dimension $128$ (Jiang et al., 2022). Each local field predicts deformation toward a canonical coordinate system and evaluates a canonical SDF. Overlapping fields are averaged to improve continuity at boundaries.

Motion2VecSets uses $M$ learned vectors for shape and deformation rather than one global code. The vectors are not observed points, mesh vertices, or explicitly semantic body parts. Their spatial interpretation emerges through positional encoding, cross-attention, and consistent indexing across frames (Cao et al., 2024). Temporal attention operates on corresponding latent indices, allowing a local deformation element to exchange information across time.

Local latent sets are particularly suited to high-frequency geometry, non-linear deformation, and incomplete observations. Their principal trade-off is that global sampling and scene-level interpolation become less straightforward than with a single latent vector.

Matrixized latent representations

The matrixized representation in the volumetric 4D generator concatenates per-frame shape and color latents into a temporal matrix. With shape and color dimensions both equal to $256$ and $(z,\beta)$0 temporal positions, the principal 4D representation is a $(z,\beta)$1 matrix (Yang et al., 2024). Diffusion operates on this matrix rather than directly on volumetric SDFs, meshes, or radiance fields. Geometry is decoded through an eight-layer SDF MLP followed by a six-layer volumetric 3D U-Net, while color is decoded by a separate eight-layer MLP and attached to reconstructed surface coordinates.

Matrixization reduces the dimensionality of diffusion while retaining temporal relationships between frame-wise shape and appearance codes. Temporal self-attention can model dependencies among frames, channels, and shape-color pairs. The final mesh is obtained only after diffusion through SDF decoding, marching cubes, and color evaluation.

Sparse latent point and Gaussian representations

SLS4D models dynamic radiance fields through dense temporal features and sparse spatial latent codes (Feng et al., 2023). A one-dimensional sequence of learnable time slots captures temporally dense deformation, while a dictionary of spatial codes is queried through attention. The spatial representation avoids allocating features to empty regions of a dense grid. For the reported D-NeRF configuration, the spatial dictionary contains $(z,\beta)$2 codes with feature dimension $(z,\beta)$3 and $(z,\beta)$4 time slots.

In Gaussian Variation Field Diffusion, a canonical Gaussian representation is separated from temporal variation. The canonical field $(z,\beta)$5 is generated once, while a compact latent sequence describes changes in Gaussian position, scale, rotation, color, and opacity (Zhang et al., 31 Jul 2025). The VAE compresses $(z,\beta)$6 mesh-point displacements into $(z,\beta)$7 tokens with feature dimension $(z,\beta)$8, and the decoder expands them into per-Gaussian variations.

Diff4Splat similarly separates canonical Gaussian structure from deformation, but uses a video latent and a Latent Dynamic Reconstruction Model to predict a canonical Gaussian field and time-dependent displacement, rotation, and scale adjustments (Pan et al., 1 Nov 2025). The resulting representation is directly compatible with Gaussian-splatting rasterization.

3. Temporal modeling and coherence mechanisms

Temporal structure is introduced through several non-equivalent mechanisms.

Timestamp or time-coordinate conditioning

The human-motion model uniformly samples sequences into a fixed number of anchor frames and explicitly reconstructs timestamps (Marsot et al., 2021). Duration is represented through the predicted timestamp sequence, allowing semantically related motions with different temporal unfoldings to occupy nearby latent regions. The model assumes constant motion between anchor meshes and does not report a learned continuous-time neural field or explicit temporal smoothness penalty.

LoRD uses normalized continuous time $(z,\beta)$9 as an input to its local motion network (Jiang et al., 2022). The local deformation function maps $z$0 directly to a canonical coordinate rather than composing frame-to-frame transformations. This avoids the same form of sequential tracking accumulation associated with chained deformations.

Temporal attention

Temporal attention is used when latent elements or frames have an explicit correspondence structure. Motion2VecSets applies interleaved spatial, conditional, and temporal attention. Spatial attention aggregates deformation elements within a frame, while temporal attention compares corresponding latent indices across frames (Mahto, 2023). The factorized attention reduces the cost from approximately $z$1 to $z$2.

DriveAnyMesh applies spatial self-attention within latent frames and temporal self-attention across corresponding latent tokens (Shi et al., 9 Jun 2025). Its transformer diffusion model generates a sequence of latent sets, which the VAE decodes into point trajectories. Motion 3-to-4 instead alternates global attention across frames with frame-wise attention within each frame, enabling variable-length processing and evaluation on sequences exceeding $z$3 frames (Chen et al., 20 Jan 2026).

CardioDiT represents depth, in-plane space, and time jointly in four-dimensional latent patches. Its tokens cover latent depth, height, width, and temporal coordinates, and global self-attention operates over the complete sequence (Seyfarth et al., 26 Mar 2026). This contrasts with slice-wise factorization and channel-merged 3D+$z$4 processing.

Joint sequence diffusion

Several methods diffuse a complete temporal latent object rather than independently generating frames. Motion2VecSets synchronously denoises the full deformation latent collection. The matrixized volumetric generator diffuses all frame-wise shape and color codes jointly (Yang et al., 2024). Gaussian Variation Field Diffusion applies temporal-aware diffusion to a $z$5 latent variation field (Zhang et al., 31 Jul 2025). Diff4Splat uses a video diffusion transformer whose latent contains appearance, multi-view structure, and temporal information before geometric lifting (Pan et al., 1 Nov 2025).

Joint diffusion encourages temporal coherence through shared latent processing, but the supplied papers differ in whether they also use explicit temporal objectives. SLS4D includes a one-dimensional total-variation penalty over time-slot features (Feng et al., 2023). Diff4Splat includes motion supervision and rendered geometric losses. Motion 3-to-4 reports direct trajectory supervision but no explicit temporal finite-difference or acceleration loss (Chen et al., 20 Jan 2026). The matrixized volumetric generator reports no separate temporal reconstruction, optical-flow, or vertex-correspondence loss (Yang et al., 2024).

Query and memory-based temporal identity

LaGS does not make each Gaussian a persistent temporal object. Instead, sparse Gaussian features are splatted into voxel features, and semantic, detection, and track queries are propagated through time (Luz et al., 26 Feb 2026). Persistent identity is therefore stored in query memory and temporal refinement rather than in the Gaussian parameters.

4DThinker also represents dynamic evolution through recurrently generated latent imagery blocks. The previous hidden state is used as input at latent positions, allowing the model to generate an ordered sequence of internal visual states (Chen et al., 7 May 2026). These states encode dynamic visual semantics but do not provide metric 3D coordinates or explicit trajectories.

4. Diffusion, decoding, and conditioning

Latent-to-4D systems commonly use diffusion to model ambiguous or high-dimensional distributions. The diffusion process may be applied to latent vectors, latent sets, matrices, Gaussian variation fields, video latents, or radar point-cloud latents.

Latent diffusion over geometry and motion

Motion2VecSets uses EDM-style denoising over shape and deformation latent sets. Shape and deformation autoencoders are trained first, after which diffusion models learn to denoise the latent representations conditioned on noisy or partial point-cloud observations (Cao et al., 2024). Multiple initial noise samples can generate different plausible completions, which is relevant when the observations do not determine hidden geometry.

4D-RaDiff uses a point-based VAE to compress radar point clouds containing spatial position, compensated Doppler velocity, and RCS into a latent point cloud with latent feature dimension $z$6 (Kwok et al., 16 Dec 2025). Separate diffusion models generate foreground radar points conditioned on boxes, classes, and velocities, and background radar points conditioned on LiDAR features. The decoded outputs are fused into a radar scene.

PhaseFlow4D applies conditional latent diffusion to a $z$7 latent representation of a $z$8 transverse beam phase-space density (Scheinker et al., 4 Apr 2026). The VAE decodes one complete four-dimensional density, after which six two-dimensional projections are analytically marginalized. The single decoded tensor provides projection consistency by construction.

Video-diffusion priors and geometric lifting

In systems that start from images or video, diffusion often supplies appearance and motion hypotheses while a second model lifts them into geometry. DriveAnyMesh uses a monocular video and an initial mesh or point cloud to generate latent sets that decode into explicit point trajectories (Shi et al., 9 Jun 2025). The original mesh topology and vertex ordering are retained, making the output directly usable in rasterization-based engines.

Gaussian Variation Field Diffusion generates a canonical Gaussian field and a compact variation latent from video (Zhang et al., 31 Jul 2025). DINOv2 features condition the variation-field diffusion transformer, while canonical Gaussian positions supply spatial correspondence. The use of additive variation supports efficient generation but is less suitable for topology-changing events.

Diff4Splat uses a pretrained CogVideoX-style video diffusion model and a 3D causal VAE to obtain a compact video latent. Its LDRM converts latent and camera tokens into a canonical Gaussian field and deformation parameters (Pan et al., 1 Nov 2025). The method uses flow matching with a 30-step inference procedure and predicts time-varying Gaussian position, rotation, and scale changes.

Trans4D uses diffusion models differently: Stable Diffusion 2.1, MVDream, and video diffusion models supply score-distillation priors, while the final representation is explicit dynamic 3D Gaussian Splatting (Zeng et al., 2024). An MLLM first plans object decomposition, trajectories, event timing, and transformations. A Transition Network modulates point visibility to represent appearance, disappearance, and large shape changes. This is a language-planned and diffusion-supervised explicit system rather than a general persistent latent scene decoder.

Geometry-aware latent conditioning

LaVR uses the persistent state of CUT3R as a latent geometric condition for novel-view video generation (Xie et al., 21 Jan 2026). CUT3R states encode scene structure, camera motion, and temporal information, but LaVR does not decode them into explicit depth or point maps. Instead, a CUT3R adapter compresses and groups the state tokens before inserting them into a video diffusion transformer together with source and target camera poses.

This design places latent geometry between explicit reconstruction and geometry-free video generation. The latent state acts as a soft spatial constraint: it encourages consistent parallax and identity while allowing the video prior to repair uncertain or missing content.

Appearance texturing in latent UV space

Tex4D operates on an already available animated mesh sequence and generates temporally and multi-view consistent textures (Bao et al., 2024). A depth-conditioned video diffusion model produces view-space latents, which are inverse-rendered into a shared UV space. UV-space latent aggregation synchronizes views, while a modified DDIM update compensates for variance reduction caused by averaging. A reference UV texture propagates appearance through self-occluded regions and across frames.

Tex4D does not generate geometry. Its latent-to-4D operation is an appearance lift:

$z$9

5. Decoding into renderable and analyzable 4D outputs

The decoder determines whether a latent-to-4D system yields an explicit asset, an implicit field, a volumetric tensor, or a visual reasoning state.

Mesh and surface decoding

The structured human-motion model decodes latent motion and morphology into SMPL pose and translation parameters and then applies the SMPL function to obtain dense meshes at multiple timestamps (Marsot et al., 2021). Motion 3-to-4 and DriveAnyMesh decode latent states into point positions, which are assigned to canonical mesh vertices while preserving face connectivity (Shi et al., 9 Jun 2025, Chen et al., 20 Jan 2026).

Motion2VecSets decodes a shape latent set into an occupancy field for the reference surface. Deformation latent sets are then queried to produce offsets for reference points, yielding a sequence of surfaces with preserved correspondence (Mahto, 2023). LoRD extracts zero-level surfaces from averaged local SDFs using Marching Cubes (Jiang et al., 2022).

Gaussian and radiance-field decoding

SLS4D first predicts a time-dependent displacement, queries a sparse spatial latent dictionary through attention, and decodes density and view-dependent color for volume rendering (Feng et al., 2023). Gaussian Variation Field Diffusion adds decoded variations to canonical Gaussian parameters and renders the resulting fields through Gaussian Splatting (Zhang et al., 31 Jul 2025).

Diff4Splat predicts canonical Gaussian attributes and deformation parameters. At time $\beta$0, Gaussian centers, rotations, and scales are updated relative to their canonical values, and the field is rendered through differentiable Gaussian rasterization (Pan et al., 1 Nov 2025). Orster-based 4D generation constructs an ST-HexPlane deformation representation whose queried features decode dynamic Gaussian parameters (Liu et al., 5 Mar 2026).

Trans4D uses explicit Gaussian models for each decomposed object or object phase. Its deformation network changes point position and orientation, while the Transition Network determines point presence over time (Zeng et al., 2024). The resulting collection of dynamic Gaussian objects represents scene-level interactions, transformations, and object appearance or disappearance.

Voxel, radar, and physical-distribution decoding

LaGS decodes sparse latent Gaussians into dense voxel features and then applies a mask-based panoptic decoder (Luz et al., 26 Feb 2026). Semantic, detection, and track queries generate voxel masks; query propagation supplies instance identity over time. The Gaussians are an intermediate feature representation rather than the final scene representation.

4D-RaDiff decodes latent point clouds into radar points with $\beta$1 attributes (Kwok et al., 16 Dec 2025). Foreground and background outputs are generated separately and fused into a complete synthetic radar scene.

PhaseFlow4D decodes a latent into a complete $\beta$2 beam phase-space density. Every two-dimensional observable is derived by summation over the other two coordinates (Scheinker et al., 4 Apr 2026). This guarantees that the generated projections are marginals of a common decoded distribution, although it does not guarantee exact physical correctness or unique reconstruction from one projection.

CardioDiT decodes a jointly denoised latent tensor into slice-wise cine CMR volumes (Seyfarth et al., 26 Mar 2026). Four-dimensional patchification and global attention couple depth, in-plane spatial structure, and cardiac time before VQ-GAN decoding.

Latent states for reasoning

4DThinker does not decode geometry. It replaces imagery placeholders in a language-model reasoning trace with continuous visual latent blocks derived from temporally ordered mask overlays (Chen et al., 7 May 2026). These latent blocks allow the VLM to internally simulate dynamic visual states, including camera motion, object motion, apparent scale, speed, and camera-object interaction.

The output is a textual answer, while the latent imagery functions as an internal dynamic representation. Its “4D” interpretation is semantic and temporal rather than metric and geometric.

6. Applications, evaluation, and limitations

Latent-to-4D systems support applications including motion synthesis, completion, tracking, novel-view rendering, mesh animation, dynamic scene generation, radar-data augmentation, cardiac-image synthesis, beam diagnostics, and dynamic spatial reasoning.

Reported evaluation regimes vary according to the output representation. Human-motion and shape-reconstruction methods use Chamfer distance, mean squared mesh-coordinate error, correspondence error, IoU, F-Score, surface-normal consistency, PSNR, SSIM, and LPIPS (Marsot et al., 2021, Jiang et al., 2022, Cao et al., 2024). Video and Gaussian-scene methods use PSNR, SSIM, LPIPS, CLIP similarity, FVD, MMD, coverage, 1-NNA, and user studies (Feng et al., 2023, Yang et al., 2024, Zeng et al., 2024, Zhang et al., 31 Jul 2025). CardioDiT evaluates FID, precision, recall, inter-slice $\beta$3-SSIM, end-diastolic alignment, ejection-fraction distributions, and Wasserstein distance (Seyfarth et al., 26 Mar 2026). LaGS uses semantic occupancy mIoU, association quality, spatiotemporal quality, and binary occupancy IoU (Luz et al., 26 Feb 2026). PhaseFlow4D evaluates six physical projections and inferred beam widths rather than visual similarity metrics (Scheinker et al., 4 Apr 2026).

The principal limitations are representation-dependent.

Ambiguous inverse problems: monocular video, sparse point clouds, incomplete projections, and single-image inputs do not uniquely determine hidden geometry or motion. Diffusion priors select plausible completions but do not establish uniqueness. PhaseFlow4D explicitly notes that one $\beta$4 projection can correspond to multiple four-dimensional distributions (Scheinker et al., 4 Apr 2026).

Dependence on canonical geometry: DriveAnyMesh, Motion 3-to-4, Gaussian Variation Field Diffusion, and Diff4Splat rely on an initial mesh or canonical Gaussian field. Errors in the first-frame reconstruction can propagate through the entire animation (Zhang et al., 31 Jul 2025, Pan et al., 1 Nov 2025, Chen et al., 20 Jan 2026).

Topology and correspondence constraints: mesh-based methods preserve topology and therefore cannot naturally represent tearing, splitting, or newly appearing parts. Local implicit and Gaussian methods can support larger deformation, but may lack explicit vertex correspondence or persistent identity. Motion 3-to-4 reports vertex sticking and topology-change limitations (Chen et al., 20 Jan 2026). LaGS stores temporal identity in queries rather than Gaussians, making tracking dependent on query propagation (Luz et al., 26 Feb 2026).

Computational cost: latent representations reduce the dimension of generation but may shift computation to decoding, iterative denoising, or test-time optimization. LoRD requires approximately 15 minutes for a 17-frame sequence on an RTX 2080Ti, while Tex4D requires approximately 30 minutes for 24 key frames on an RTX A6000 Ada (Jiang et al., 2022, Bao et al., 2024). Feed-forward systems such as DriveAnyMesh and Diff4Splat reduce inference time but rely more heavily on learned priors (Shi et al., 9 Jun 2025, Pan et al., 1 Nov 2025).

Dataset and domain dependence: many systems are trained on synthetic or restricted motion datasets. Motion3-to-4 and Gaussian Variation Field Diffusion use curated Objaverse-derived data, while CardioDiT addresses cine CMR and 4D-RaDiff addresses simulated or sensor-specific radar distributions (Chen et al., 20 Jan 2026, Zhang et al., 31 Jul 2025, Seyfarth et al., 26 Mar 2026, Kwok et al., 16 Dec 2025). Generalization to unusual motion, transparent objects, severe occlusion, rare materials, real-world sensor artifacts, or out-of-distribution categories remains limited.

Incomplete physical guarantees: architectural consistency does not necessarily imply physical validity. PhaseFlow4D guarantees that projections originate from a common decoded tensor, but not that the tensor satisfies all beam-dynamics equations (Scheinker et al., 4 Apr 2026). Trans4D uses physics-aware language planning but no explicit physics engine, collision loss, or conservation law (Zeng et al., 2024). Many visual systems use temporal attention or shared deformation functions without explicit acceleration, contact, optical-flow, or physical-dynamics constraints.

Fixed temporal discretization: many methods operate on fixed numbers of frames or anchor positions. Timestamp encoding, frame-wise transformers, key-frame texture synthesis, and cyclic temporal repetition provide practical standardization but can miss abrupt events or introduce artificial periodicity (Marsot et al., 2021, Bao et al., 2024, Seyfarth et al., 26 Mar 2026).

The research trajectory suggests a progression from global latent vectors toward structured latent fields: local latent sets, temporally indexed matrices, sparse Gaussian dictionaries, persistent scene tokens, and factorized spatial-temporal representations. The central design problem is the allocation of capacity between spatial detail, temporal evolution, conditioning information, and uncertainty. Global codes favor compact sampling; local sets preserve detail; canonical-plus-deformation models preserve correspondence; joint diffusion represents ambiguity; and explicit decoders provide renderable or physically analyzable outputs.

A latent-to-4D system is therefore best characterized not by a particular neural architecture, but by the role of its latent state: it must serve as a compressed carrier of temporally organized information whose decoding produces, constrains, or reasons about a coherent evolving 3D entity.