---
title: Mesh Latent Diffusion Models
url: https://www.emergentmind.com/topics/mesh-latent-diffusion-model-mesh-ldm
type: topic
---

# Mesh Latent Diffusion Models

A Mesh Latent Diffusion Model (Mesh LDM) is a generative model that synthesizes meshes, mesh-conditioned fields, or mesh-decoded dynamics by performing stochastic generation in a learned latent space rather than by diffusing directly over raw vertices, faces, or connectivity. In recent literature, the term covers several related but non-identical constructions: latent diffusion over hand-mesh codes from a SpiralNet++ autoencoder [2510.00527], latent diffusion over left-ventricular mesh embeddings learned by MeshVAE [2508.14122], conditional diffusion over Fourier-coefficient tokens derived from CoMA latents for 3D+t cardiac meshes [2606.03827], structured rectified-flow generation over part-wise mesh latent tokens inherited from TripoSG [2506.05573], diffusion over triplane latents that decode to meshes through DMTet [2303.05371], and intrinsic diffusion over tangent-vector texture field latents on a fixed mesh surface [2312.09250]. This suggests that Mesh LDM is best treated as a family of mesh-decoding latent generative models rather than as a single canonical architecture.

## 1. Definition and conceptual boundaries

The strict core of a Mesh LDM is the separation of representation learning from generative modeling: a mesh, or a mesh-derived object, is first encoded into a compact latent representation; a diffusion or diffusion-like prior is then trained in that latent space; finally, a decoder maps latent samples back to an explicit mesh, a mesh sequence, or a mesh-surface field. In this sense, the latent object may be a vector, a token set, a triplane tensor, Fourier coefficient tokens, or an intrinsic surface field, provided that decoding is mesh-relevant and the stochastic process acts on the learned latent rather than on raw mesh coordinates [2510.00527].

The literature also contains several important boundary cases. “MeshDiffusion” is diffusion for mesh generation, but not latent diffusion, because it operates directly in deformable tetrahedral-grid attribute space and has no learned latent bottleneck [2303.08133]. “LDM: Large Tensorial SDF Model for Textured Mesh Generation” uses diffusion only in an upstream multi-view image stage; its 3D generator is a feed-forward tensorial SDF predictor, so its “LDM” acronym does not mean latent diffusion model [2405.14580]. PolycubeNet uses conditional diffusion on polycube point clouds with dual latent-token bottlenecks for efficiency, but the diffusion variable remains the output point cloud rather than an autoencoded mesh latent [2605.20274]. “Controllable Mesh Generation Through Sparse Latent Point Diffusion Models” and “Drive Any Mesh” are likewise indirect or hybrid cases: both generate through latent point or latent-set proxies that are later converted into meshes or mesh deformations, rather than through a mesh-native latent prior in the narrow sense [2303.07938], [2506.07489].

These distinctions matter because “mesh diffusion” and “mesh latent diffusion” are not interchangeable. A model may generate explicit meshes without being latent; it may use latent computation without diffusing a mesh latent; or it may be mesh-adjacent while actually modeling an intermediate proxy such as a point cloud, triplane, polycube, or tangent field.

## 2. Latent representations and decoding pathways

Across the literature, Mesh LDMs differ most sharply in what they choose to encode and what they decode. Some operate on global shape codes, some on structured token sets, and some on surface-attached fields.

Representative latent objects are summarized below.

| Model | Latent object | Decoded output |
|---|---|---|
| PartCrafter [2506.05573] | Part token sets \(\boldsymbol{z}_i \in \mathbb{R}^{K\times C}\), concatenated as \(\boldsymbol{\mathcal{Z}}\in\mathbb{R}^{NK\times C}\) | Multiple mesh parts \(\mathbf{p}_i=\{\mathcal{V}_i,\mathcal{F}_i\}\) |
| Mesh LDM for hand reconstruction [2510.00527] | 168-dimensional mesh latent vector \(x\in\mathbb{R}^{168}\) | Hand mesh vertices \(\hat V_0\in\mathbb{R}^{778\times 3}\) |
| MeshLDM for LV anatomy [2508.14122] | MeshVAE latent \(z\in\mathbb{R}^{16}\) | LV surface mesh on a shared template |
| 4D F-MeshLDM [2606.03827] | Fourier coefficient tokens \(\Theta\in\mathbb{R}^{B\times D}\), with \(B=25\), \(D=64\) | 3D+t biventricular mesh sequence |
| 3DGen [2303.05371] | Rolled-out triplane latent \(z\in\mathbb{R}^{H\times 3W\times C}\) | Textured mesh via SDF/deformation and DMTet |
| FLDM [2312.09250] | Tangent-vector field latents \(Z\in TM^d\), with \(d=8\) | Texture on a fixed or approximately isometric mesh |

In the hand-reconstruction Mesh LDM, the latent is defined by a SpiralNet++ Mesh AutoEncoder: \(x=\mathcal E(V)\), \(\hat V=\mathcal D(x)\), with \(x\in\mathbb R^{168}\) and \(V\in\mathbb R^{778\times 3}\) [2510.00527]. In the cardiac MeshLDM for left-ventricular anatomy, the encoder-decoder pair is \(E:X\mapsto z\), \(D:z\mapsto \hat X\), with \(z\in\mathbb R^{16}\), and the model relies on meshes that share the same number of vertices and consistent ordering across samples [2508.14122]. In 4D F-MeshLDM, the latent is first framewise, \(z_{s,u}\in\mathbb R^{64}\), and is then reparameterized over time by a truncated Fourier series, yielding a coefficient matrix \(\Theta_s\in\mathbb R^{25\times 64}\) that becomes the actual diffusion object [2606.03827].

PartCrafter replaces a monolithic whole-object latent set with a structured collection of part-specific token sets. Given a single RGB image \(\mathbf c\) and a user-specified number of parts \(N\), it generates \(\mathcal O=\{\mathbf p_i\}_{i=1}^N\), where each \(\mathbf p_i=\{\mathcal V_i,\mathcal F_i\}\), all in the common canonical coordinate system \([-1,1]^3\). Each part receives \(K=512\) tokens, and the global latent is \(\boldsymbol{\mathcal Z}=\{\boldsymbol z_i\}_{i=1}^N\in\mathbb R^{NK\times C}\) [2506.05573]. In 3DGen, the latent is neither a mesh vector nor a graph token set, but a Gaussian triplane tensor that is rolled out into \(\mathbb R^{H\times 3W\times C}\) for diffusion and later decoded into SDF and deformation values on a tetrahedral grid before DMTet extraction [2303.05371]. In FLDM, by contrast, the geometry is fixed and the latent object is a vertexwise tangent-vector field \(Z\in TM^d\) encoding texture intrinsically on the mesh surface [2312.09250].

The decoder stage is correspondingly heterogeneous. Some decoders reconstruct vertex coordinates directly from latent vectors; some decode each part latent into a separate mesh; some decode to SDF fields and then extract meshes; some decode latent trajectories framewise; and some decode to texture values on the surface. The commonality is not the decoder type, but the fact that the learned latent is the domain of the generative prior.

## 3. Generative objectives and denoiser architectures

Mesh LDMs do not share a single denoising objective. The most common formulation is DDPM-style latent diffusion, but rectified flow and other variants are also used.

The hand Mesh LDM in the cascaded diffusion framework uses the standard latent DDPM forward process,
\[
x_t = \sqrt{\bar\alpha_t}\, x_0 + \sqrt{1-\bar\alpha_t}\,\epsilon,\qquad \epsilon\sim\mathcal N(0,I),
\]
and directly predicts the clean latent with
\[
\mathcal L_{DDPM}=\|x_0-\hat x_0\|^2.
\]
Its denoiser is DiT-based. The supplement specifies that \(x_t\in\mathbb R^{168}\) is repeated 21 times and concatenated with the 21-joint 3D hand skeleton to produce a tensor of shape \(\mathbb R^{171\times 21}\), projected to hidden dimension 512, with cross-attention to four levels of image features and Adaptive Layer Normalization [2510.00527].

The left-ventricular MeshLDM also adopts a DDPM-style latent formulation, but with a much smaller latent and a much lighter denoiser. Its diffusion variable is \(z\in\mathbb R^{16}\), the scheduler uses \(T=1000\) steps with a linear schedule from \(\beta_1=0.0001\) to \(\beta_T=0.02\), and the denoiser is a 6-layer fully connected network with Swish-1 activations and sinusoidal timestep embeddings [2508.14122]. This is an important counterexample to the assumption that Mesh LDMs require spatial UNets or large transformers; in template-aligned medical anatomy, a compact latent and an MLP prior were sufficient to generate clinically plausible meshes.

PartCrafter is architecturally in the Mesh LDM family but does not use standard DDPM noise prediction. It inherits TripoSG’s rectified-flow formulation, with perturbation
\[
\boldsymbol{\mathcal Z_t}=t\boldsymbol{\mathcal Z_0}+(1-t)\boldsymbol\epsilon,
\]
target velocity \(\boldsymbol\epsilon-\boldsymbol{\mathcal Z_0}\), and flow-matching loss
\[
\mathcal L_{\text{flow}}=\mathbb E_{\boldsymbol{\mathcal Z_0},\boldsymbol\epsilon,t}\left[\left\|\left(\boldsymbol\epsilon-\boldsymbol{\mathcal Z_0}\right)-\mathbf v_\theta(\boldsymbol{\mathcal Z_t},t,\mathbf c)\right\|^2\right].
\]
Its denoiser is a hierarchical “local-global denoising transformer” that alternates part-local attention and global attention across the 21 DiT blocks, with DINOv2 image features injected by cross-attention at both levels [2506.05573].

4D F-MeshLDM returns to a DDPM prior, but its diffusion object is the Fourier coefficient matrix \(\Theta\in\mathbb R^{B\times D}\), treated as a sequence of frequency tokens. The training loss is
\[
\mathcal L_{\mathrm{Diff}}=\mathbb E_{\Theta_0,\epsilon,n,c}\left[\|\epsilon-\epsilon_\psi(\Theta_n,n,c)\|_2^2\right],
\]
and the denoiser is a Transformer with width 256, depth 8, 8 heads, 500 diffusion steps, learnable Fourier-index embeddings, and AdaLN conditioning on age and gender [2606.03827].

Other variants show that the latent denoiser can also be a 3D-aware UNet over rolled-out triplanes, as in 3DGen, which trains a 1000-step latent diffusion model with a cosine schedule and \(v\)-prediction over triplane latents [2303.05371], or a fully intrinsic surface network built from field convolutions, as in FLDM, which performs \(\epsilon\)-prediction DDPM directly in tangent-bundle latent space on the mesh surface [2312.09250]. The architectural design space is therefore broad: transformer blocks, MLP priors, image-style UNets, and intrinsic field-convolution UNets have all been used successfully.

## 4. Structured, conditional, and dynamic extensions

A major development in the area is the move from monolithic single-shape generation to structured, conditional, and dynamic mesh latent priors.

PartCrafter is the clearest structured extension. It is described as the first structured 3D generative model that jointly synthesizes multiple semantically meaningful and geometrically distinct 3D meshes from a single RGB image. It does not require segmentation masks, segmented crops, or a separate segmentation network at inference. Instead, decomposition is built into the latent organization itself, with per-part identity embeddings and alternating local/global attention. For objects, the output parts correspond to object subcomponents; for scenes, the same machinery is reused with “parts” effectively becoming object instances [2506.05573].

The hand-reconstruction Mesh LDM is conditional rather than fully generative in the unconditional sense. It is the second stage of a coarse-to-fine cascade: a first diffusion model samples plausible 3D joints from 2D keypoints, and the Mesh LDM then reconstructs a dense 3D hand mesh conditioned on a sampled joint hypothesis and image features. Its central claim is “distribution-aware joint–mesh relationships”: the mesh model is not trained only on ground-truth joints, but on diverse sampled joints from the first-stage diffusion model, so it learns robustness to upstream ambiguity and self-occlusion [2510.00527].

4D F-MeshLDM extends the Mesh LDM notion into periodic 3D+t generation. Each frame of a biventricular mesh sequence is encoded by a CoMA-based VAE, the full latent trajectory is parameterized by a truncated Fourier series,
\[
z_{s,u}\approx a_{s,0}+\sum_{k=1}^{K-1}\left(a_{s,k}\cos(2\pi k\tau_u)+b_{s,k}\sin(2\pi k\tau_u)\right),
\]
and diffusion is learned over the coefficient tokens \(\Theta_s\). Because the normalized phase satisfies \(\tau_0=0\) and \(\tau_{T_s-1}=1\), inverse Fourier synthesis yields exactly periodic latent trajectories by construction [2606.03827].

DriveAnyMesh occupies a hybrid position. It denoises sequences of latent sets that jointly capture shape and motion information, then decodes them into point cloud trajectory sequences that drive an existing mesh over time. The output is directly compatible with rasterization-based rendering engines because topology is preserved by updating vertex positions while keeping connectivity fixed. The paper explicitly notes, however, that it is not fully mesh-native, since the latent does not explicitly encode mesh connectivity, geodesics, or physical constraints [2506.07489].

3DGen and FLDM show two other conditional directions. 3DGen performs conditional and unconditional generation of textured or untextured meshes by diffusing triplane latents and decoding them through a tetrahedral SDF/deformation field plus a color field [2303.05371]. FLDM is narrower in scope but more intrinsic: it is a single-textured-mesh paradigm in which diffusion operates on latent vector fields attached to the mesh surface, enabling unconditional texture variation, label-guided generation, inpainting, and approximately isometric generative transfer to new meshes [2312.09250].

## 5. Empirical landscape

The empirical record of Mesh LDMs is domain-specific, and results are best read relative to task definitions rather than as a single cross-paper leaderboard.

In structured mesh generation, PartCrafter reports that on Objaverse it achieves CD \(0.1726\), F-Score \(0.7472\), and IoU \(0.0359\), compared with HoloPart’s \(0.1916\), \(0.6916\), and \(0.0443\). On ABO it reports CD \(0.1047\), F-Score \(0.8617\), and IoU \(0.0243\), versus HoloPart’s \(0.1338\), \(0.8093\), and \(0.0449\). Runtime for 4-part objects on an H20 GPU is reported as 34s versus 18 minutes. The paper also notes that on 3D-Front scenes PartCrafter surpasses MIDI in CD and F-Score, especially on occluded subsets, while MIDI retains slightly better IoU because it is given ground-truth segmentation masks [2506.05573].

In probabilistic hand reconstruction, the Mesh LDM stage is supported by several ablations. Training on diverse sampled joint hypotheses performs better than training on off-the-shelf or averaged joints: on FreiHAND, the proposed setting reports P-MPJPE 5.0 and P-MPVPE 5.2, with a best-of-sampling result of 4.2 and 4.5. The latent-space design also outperforms MANO-space diffusion: on FreiHAND, latent diffusion yields P-MPJPE 5.00 and P-MPVPE 5.23, while MANO-space diffusion gives 5.70 and 5.82; on HO3Dv2, the comparison is 7.50 and 7.52 versus 7.81 and 7.81. The paper further reports Pearson correlations between joint error and mesh error of 0.93 on FreiHAND and 0.80 on HO3Dv2, indicating that mesh quality remains strongly dependent on the conditioning joint [2510.00527].

In medical anatomy synthesis, the static LV MeshLDM reports that generated meshes have an average difference in population mean of 2.4% across all clinical scores and both phases relative to the gold standard. At end-diastole, generated cavity volume is \(155.8\pm 29.2\) ml versus \(156.3\pm 43.0\) ml in the test set, and generated LV mass is \(123.1\pm 17.6\) g versus \(123.0\pm 30.6\) g. The same paper also reports limited diversity, particularly at end-systole, with 1-NNA and coverage indicating that generated and real distributions remain distinguishable [2508.14122]. In dynamic cardiac generation, 4D F-MeshLDM reports conditional-generation Specificity \(2.76\pm 0.52\) mm, Coverage@5 98.60%, and Seq. RMSE \(7.54\pm 2.75\) mm, while its cycle-consistency metrics are markedly smaller than those of recurrent and continuous-time baselines: volume consistency \(4.506\times 10^{-6}\ \mathrm{mL}^2\) and mesh consistency \((5.564\pm 0.309)\times 10^{-3}\ \mathrm{mm}\) [2606.03827].

For appearance-focused models, FLDM reports SIFID \(3.27\) and LPIPS \(1.15\) on single-textured-mesh generation, compared with Sin3DM’s SIFID \(6.58\) and LPIPS \(2.20\), and attributes this to intrinsic equivariance and tangent-field latents rather than to geometry generation [2312.09250]. For textured mesh generation from triplane latents, 3DGen reports unconditional textured-mesh FID of 36.27 on cars, 11.07 on chairs, and 12.34 on tables, outperforming GET3D variants, and reports image-conditioned geometry improvements in Chamfer-L1 and Shading FiD as well [2303.05371].

Taken together, these studies show that Mesh LDMs have been used for at least five distinct problem classes: monolithic mesh synthesis, structured multi-part generation, conditional dense reconstruction from articulated proxies, static medical anatomy synthesis, periodic 3D+t anatomy synthesis, and intrinsic texture generation on meshes.

## 6. Limitations, misconceptions, and open directions

Several limitations recur across the literature. A first is topology dependence. The medical MeshLDM and 4D F-MeshLDM both assume fixed topology and consistent vertex ordering across subjects; the 3D cardiac datasets are template-aligned, and the VAE backbones depend on that correspondence [2508.14122], [2606.03827]. 3DGen outputs explicit meshes, but topology is still mediated through a tetrahedral grid and DMTet extraction rather than generated as arbitrary connectivity [2303.05371]. FLDM does not generate geometry at all; it assumes the mesh is given and models only surface appearance [2312.09250].

A second limitation is that latent diffusion does not by itself solve structural ambiguity. In the hand Mesh LDM, most multimodality resides in the first-stage joint distribution, while the second-stage mesh model has limited variance for a fixed joint condition [2510.00527]. In PartCrafter, the number of parts is user-specified rather than inferred, the part semantics depend on mined asset decompositions, and the paper does not provide exact backbone latent dimensions, exact sampling-solver details, or guarantees of watertightness or manifoldness for generated parts [2506.05573]. In the static LV MeshLDM, diversity remains lower than in the real population, especially at end-systole, which the paper partly attributes to the small dataset size of 1034 meshes per phase [2508.14122].

A third issue is terminological. Not every mesh-producing diffusion system is a Mesh LDM in the strict sense. MeshDiffusion is a strong data-space diffusion baseline for explicit mesh extraction, but not a latent model [2303.08133]. The “LDM” acronym in “Large Tensorial SDF Model” is unrelated to latent diffusion [2405.14580]. PolycubeNet is best described as conditional point-cloud diffusion with latent-token computation, not as a standard latent diffusion model over compressed mesh codes [2605.20274]. This suggests that the term Mesh LDM should be used carefully, with explicit reference to what the latent variable actually represents.

Open directions are correspondingly clear in the source papers. PartCrafter raises questions about variable-cardinality prediction, larger structured datasets, and more formal part correspondence across samples [2506.05573]. The hand Mesh LDM points to stronger uncertainty modeling beyond joint-conditioned refinement and beyond dataset-specific autoencoders [2510.00527]. The cardiac models point toward relaxing fixed-topology assumptions, handling severe pathologies, and extending periodic priors beyond single-cycle synthesis [2508.14122], [2606.03827]. Texture-focused intrinsic models raise the possibility of broader surface-native latent diffusion beyond single-mesh regimes, but currently remain tied to fixed or approximately isometric geometry [2312.09250].

In that broader perspective, Mesh LDM denotes not a single recipe but a design principle: learn a mesh-relevant latent space, place the stochastic prior in that space, and use a decoder or extractor that preserves enough geometric or surface structure for the target application. The strongest current systems differ precisely in how they instantiate those three steps.

Source: https://www.emergentmind.com/topics/mesh-latent-diffusion-model-mesh-ldm