Skeletal Latent Diffusion Models

• Skeletal latent diffusion frameworks are generative models that operate in a latent space structured by explicit skeletal or anatomical priors.
• They utilize representational regimes such as joint trajectories, skeleton graphs, and skeleton-conditioned latent spaces to guide the diffusion process.
• Applications include motion generation, pose-conditioned image synthesis, and medical shape modeling, achieving improved structural fidelity over isotropic methods.

A skeletal latent diffusion framework is a class of generative models in which diffusion is carried out in a latent space organized by an explicit structural prior: articulated joints and kinematic trees for motion, skeleton graphs for rigged assets, keypoint geometry for image synthesis, or medial and branching skeletons for anatomical shape modeling. In these systems, the skeleton is not merely an output format. It determines latent tokenization, covariance structure, attention topology, conditioning interfaces, decoder design, or post-processing constraints. Representative realizations include universal skeletal motion generation for rigged meshes, cross-modal biomechanical generation with locally aligned latent manifolds, pose-conditioned image synthesis, canonical-skeleton motion in-betweening, skeleton-graph diffusion for rigged 3D asset creation, and skeleton-guided medical shape generation (Tao et al., 1 Jun 2026, Dey et al., 15 Mar 2025, Khameneh et al., 26 Apr 2026, Qin, 13 Apr 2025, Zhao et al., 10 Feb 2026, Zhang et al., 8 Mar 2026).

1. Conceptual definition and scope

The narrowest use of the term denotes models that diffuse directly over latent variables attached to joints, bones, or skeletal graphs. MotionDreamer is exemplary: given a rigged mesh $\mathcal{M}=\{\mathcal{V},\mathcal{F}\}$, a rest-pose skeleton $\mathcal{S}^0=(\mathcal{P}^0,\mathcal{G})$, and a monocular driving video $\mathcal{X}=\{I^l\}_{l=1}^L$, it formulates universal skeletal motion generation as conditional generative modeling over global joint trajectories $\mathcal{P}\in\mathbb{R}^{L\times J\times 3}$, linearly embeds them into a joint- and time-indexed latent $z_0$, performs DDPM-style diffusion in that latent space, and decodes back to global joint coordinates (Tao et al., 1 Jun 2026). Stroke3D uses the same principle for skeleton creation rather than motion: a Skeletal Graph VAE maps a 3D skeleton graph $G=(X,E)$ into a latent code, and a Skeletal Graph Diffusion Transformer generates that latent conditioned on 2D strokes and text before decoding it into a 3D skeleton (Zhao et al., 10 Feb 2026).

A broader use encompasses latent diffusion models in which the latent variable being diffused is not itself a skeleton, but the entire generative process is skeleton-conditioned. Pose-LDM synthesizes blanket-covered images in the latent space of a KL-regularized VAE while conditioning exclusively on pose keypoints $p\in\mathbb{R}^{K\times 2}$ or $\mathbb{R}^{K\times 3}$, so the skeleton functions as the primary geometric control signal rather than the diffusion state (Khameneh et al., 26 Apr 2026). In medical imaging, morpho-skeletal latent diffusion for coronary anatomy concatenates latent variables with a voxelized tree skeleton and morphology-aligned descriptors, and high-fidelity medical shape generation uses differentiable skeletonization to define the latent point set on which diffusion operates (Kadry et al., 2024, Zhang et al., 8 Mar 2026). This suggests that the literature supports both a strict definition—diffusion over skeletal latents—and a broader one in which diffusion is organized by skeletal structure.

The framework also spans multiple object types. In articulated graphics, “skeletal” usually means joints, hierarchies, and skinning. In anatomy modeling, the same word often refers to medial axes, vessel trees, or curve skeletons. The unifying property is structural abstraction: the latent space is constrained by a low-dimensional representation that captures topology and coarse geometry more directly than raw pixels, voxels, or vertex clouds (Zhang et al., 8 Mar 2026, Kadry et al., 2024).

2. Representational regimes and latent variables

Across the literature, three recurrent representational regimes appear. The first is the joint-trajectory regime, in which motion is represented as global Cartesian trajectories or jointwise latent codes. MotionDreamer adopts $\mathcal{P}\in\mathbb{R}^{L\times J\times 3}$ rather than relative rotations, arguing for better geometric alignment with 2D visual cues and easier trajectory matching, while SkeletonDiffusion encodes future motion into a latent tensor $\latentvar{}\in\mathbb{R}^{J\times L}$ with the first dimension tied one-to-one to joints (Tao et al., 1 Jun 2026, Curreli et al., 10 Jan 2025). The second is the graph-skeleton regime, where nodes are joints and edges are bones. Stroke3D represents a 3D skeleton as an undirected graph $\mathcal{S}^0=(\mathcal{P}^0,\mathcal{G})$0 with $\mathcal{S}^0=(\mathcal{P}^0,\mathcal{G})$1 and uses $\mathcal{S}^0=(\mathcal{P}^0,\mathcal{G})$2 explicitly in both encoder and decoder (Zhao et al., 10 Feb 2026). The third is the skeleton-conditioned latent-image or latent-volume regime, in which the diffused variable is an image or anatomical latent, but skeletons or centerlines remain indispensable for control and decoding. Pose-LDM and the coronary anatomy model fall into this category (Khameneh et al., 26 Apr 2026, Kadry et al., 2024).

Regime	Skeletal object	Latent or diffusion space
Joint-centric motion	Rest-pose skeleton $\mathcal{S}^0=(\mathcal{P}^0,\mathcal{G})$3 or per-joint latent codes	Latent tensor over joint trajectories or joint latents (Tao et al., 1 Jun 2026, Curreli et al., 10 Jan 2025)
Skeleton graph generation	Graph $\mathcal{S}^0=(\mathcal{P}^0,\mathcal{G})$4	VAE latent over skeleton geometry; diffusion on graph-conditioned codes (Zhao et al., 10 Feb 2026)
Skeleton-conditioned image/shape generation	2D keypoints, tree skeletons, medial skeletons	VAE image latents or skeletal latent point sets (Khameneh et al., 26 Apr 2026, Kadry et al., 2024, Zhang et al., 8 Mar 2026)

The canonical-skeleton formulation is a particularly important variant. In scalable motion in-betweening, all motions are first mapped into a fixed canonical human skeleton, and diffusion is performed only in that shared canonical space; stage two then adapts the generated motion to a target character through physics-based reinforcement learning (Qin, 13 Apr 2025). This suggests a useful distinction between intrinsic and extrinsic skeletal latents. Intrinsic latents are tied to the target rig itself, as in MotionDreamer. Extrinsic latents are tied to a universal canonical skeleton, as in motion in-betweening.

Medical-shape work introduces a further generalization: the latent need not be attached to anatomical joints. High-fidelity medical shape generation defines a latent point set

$\mathcal{S}^0=(\mathcal{P}^0,\mathcal{G})$5

where $\mathcal{S}^0=(\mathcal{P}^0,\mathcal{G})$6 contains skeleton coordinates and radii, and the remaining channels encode learned features aggregated from the surface (Zhang et al., 8 Mar 2026). Here the skeleton serves simultaneously as geometric scaffold, latent support, and decoding prior.

3. Diffusion formulations

Most skeletal latent diffusion frameworks adopt standard Gaussian diffusion in latent space. MotionDreamer uses the DDPM forward process

$\mathcal{S}^0=(\mathcal{P}^0,\mathcal{G})$7

with training objective

$\mathcal{S}^0=(\mathcal{P}^0,\mathcal{G})$8

where conditioning $\mathcal{S}^0=(\mathcal{P}^0,\mathcal{G})$9 includes video features, rest-pose structure, topology, and joint semantic features (Tao et al., 1 Jun 2026). Pose-LDM uses the same noise-prediction objective in the latent space of a VAE, with $\mathcal{X}=\{I^l\}_{l=1}^L$0 and conditioning supplied by pose embeddings rather than by source imagery (Khameneh et al., 26 Apr 2026). Stroke3D likewise trains its Skeletal Graph Diffusion Transformer with a DDPM-style latent loss over skeleton graph codes conditioned on $\mathcal{X}=\{I^l\}_{l=1}^L$1, $\mathcal{X}=\{I^l\}_{l=1}^L$2, and text (Zhao et al., 10 Feb 2026).

Not all frameworks use the same prediction target. In scalable motion in-betweening, the canonical-skeleton diffusion model predicts clean motion rather than noise and performs masked denoising in canonical skeletal space, with keyframe frames clamped by a binary mask $\mathcal{X}=\{I^l\}_{l=1}^L$3 throughout training and sampling (Qin, 13 Apr 2025). This is a diffusion-based inpainting formulation rather than an unconstrained motion prior.

A more substantial departure is SkeletonDiffusion, which replaces isotropic latent noise by a nonisotropic Gaussian aligned with the human kinematic graph. Starting from a skeleton adjacency matrix $\mathcal{X}=\{I^l\}_{l=1}^L$4, it constructs a correlation matrix

$\mathcal{X}=\{I^l\}_{l=1}^L$5

and then uses timestep-dependent covariance

$\mathcal{X}=\{I^l\}_{l=1}^L$6

Its diffusion loss becomes a Mahalanobis distance in principal-coordinate space rather than a simple isotropic Euclidean error (Curreli et al., 10 Jan 2025). This explicitly builds the skeleton into the stochastic process itself, not only into the network.

A related neighboring formulation is cross-modal biomechanical diffusion through local manifold alignment. There, diffusion is technically performed in data space for synchronized modalities such as joint angles and ground reaction forces, but encoder latents $\mathcal{X}=\{I^l\}_{l=1}^L$7 and $\mathcal{X}=\{I^l\}_{l=1}^L$8 are aligned at every diffusion step through first-order contrastive alignment and second-order covariance matching. The method therefore extends skeletal latent diffusion toward multimodal latent manifolds rather than a single latent trajectory space (Dey et al., 15 Mar 2025).

4. Structural priors, conditioning, and decoder constraints

Skeletal latent diffusion frameworks are distinguished less by diffusion mechanics than by how structural priors are injected. MotionDreamer combines four such priors. First, the rest pose $\mathcal{X}=\{I^l\}_{l=1}^L$9 is encoded into a structural latent $\mathcal{P}\in\mathbb{R}^{L\times J\times 3}$0 and concatenated to the noisy latent at every denoising step. Second, arbitrary topology $\mathcal{P}\in\mathbb{R}^{L\times J\times 3}$1 is encoded through Skeletal Attention, which uses a tree-distance matrix to bias self-attention toward structurally nearby joints. Third, mesh semantics are aggregated to joints by rendering the mesh from multiple views, extracting DINOv2 features, back-projecting them to vertices, and pooling them to joints with normalized skinning weights,

$\mathcal{P}\in\mathbb{R}^{L\times J\times 3}$2

Fourth, bidirectional video-skeleton fusion uses symmetrical cross-attention so that skeleton tokens query video features and video tokens query skeletal features during each denoising step (Tao et al., 1 Jun 2026).

Cross-modal biomechanical work arrives at a different but conceptually related prior. It assumes that synchronized modalities are observations of a shared locomotor dynamical system and enforces this assumption through Local Latent Manifold Alignment. Positive pairs are time-matched local windows $\mathcal{P}\in\mathbb{R}^{L\times J\times 3}$3, negatives are time-mismatched or cross-sequence windows, and the full LLMA loss combines a contrastive term with covariance matching. An additional energy regularizer encourages realistic temporal energy profiles (Dey et al., 15 Mar 2025). In this setting, the structural prior is not a rigid skeleton tree but a phase-consistent locomotor manifold.

Canonical-skeleton in-betweening externalizes structural control still further. Stage one learns a character-agnostic diffusion prior on a fixed canonical skeleton, whereas stage two uses a physics-based RL controller to track the canonical motion on a target character with different morphology, masses, and degrees of freedom (Qin, 13 Apr 2025). This decouples generative modeling from physical embodiment. A plausible implication is that skeletal latent diffusion can treat physics either as an in-model inductive bias, as in nonisotropic covariance, or as a downstream decoder, as in physics-based adaptation.

Medical-shape models use analogous mechanisms. The coronary anatomy framework concatenates latent variables with a skeletal heatmap $\mathcal{P}\in\mathbb{R}^{L\times J\times 3}$4 and morphology channels $\mathcal{P}\in\mathbb{R}^{L\times J\times 3}$5, and at sampling time introduces Adaptive Null Guidance, in which the null morphology is updated online by

$\mathcal{P}\in\mathbb{R}^{L\times J\times 3}$6

This allows continuous morphologic control without differentiating through the full measurement pipeline (Kadry et al., 2024). High-fidelity medical shape generation instead constrains the decoder by requiring predicted SDF values at skeletal points to match stored skeleton radii, thereby tying the implicit field back to the medial representation used to define the latent (Zhang et al., 8 Mar 2026).

5. Application domains and reported empirical behavior

The most visible application domain is category-agnostic animation. MotionDreamer targets video-driven animation of arbitrary rigged 3D assets and is trained on a dynamic dataset of approximately 20,000 diverse 3D models, more than 40,000 paired video-motion sequences, and more than 3.5M frames. Its stated scope spans humans, stylized humanoids, quadrupeds, multi-legged creatures, and fantasy shapes, and it uses 150 iterations of inverse kinematics at inference to enforce fixed bone lengths and rigid structure after latent denoising (Tao et al., 1 Jun 2026). Closely related work on canonical-skeleton in-betweening reports improved K-FID after physics-based adaptation across sparse-keyframe and long-horizon settings, including sequences of 500 and 1000 frames, while stage two reduces foot sliding, ground penetration, and self-intersections on stylized characters (Qin, 13 Apr 2025).

Human motion prediction provides a more controlled test bed for structural priors. SkeletonDiffusion reports better limb realism than isotropic baselines and argues that high diversity can be artificially inflated by inconsistent limb lengths. On AMASS, its reported mean stretching and jitter are lower than competing diffusion models, and it also improves zero-shot robustness on 3DPW and performance on FreeMan, a noisier RGB-derived motion dataset (Curreli et al., 10 Jan 2025). In cross-modal biomechanics, LLMA improves angles–GRF generation in the $\mathcal{P}\in\mathbb{R}^{L\times J\times 3}$7 direction from MSE $\mathcal{P}\in\mathbb{R}^{L\times J\times 3}$8, FID $\mathcal{P}\in\mathbb{R}^{L\times J\times 3}$9, and predictive error $z_0$0, while also improving probe accuracy over SimCLR, Barlow Twins, VICReg, and simple latent MSE alignment (Dey et al., 15 Mar 2025).

Skeletal conditioning also supports image generation tasks. Pose-LDM reframes blanket occlusion augmentation for in-bed pose estimation as geometry-conditioned latent diffusion from pose keypoints alone. In the covered-only evaluation, it reports PCK@0.1 of $z_0$1 and AP$z_0$2 of $z_0$3, compared with $z_0$4 and $z_0$5 for paired BBDM synthesis; on covered+uncovered data it reports PCK $z_0$6 and AP $z_0$7, approaching the fully supervised setting while avoiding paired uncovered-covered supervision (Khameneh et al., 26 Apr 2026). The critical architectural finding is that cross-attention conditioning outperforms spatial concatenation, and that raw 2D coordinates suffice without explicit heatmaps.

In rigged 3D asset creation, Stroke3D uses a two-stage pipeline in which skeleton generation precedes skeleton-conditioned mesh synthesis. Its skeleton-to-mesh stage combines TextuRig with SKA-DPO, increasing MeanInst SKA score from $z_0$8 for the TextuRig-enhanced model to $z_0$9 for the TextuRig + SKA-DPO + Stroke3D pipeline (Zhao et al., 10 Feb 2026). In anatomical generation, skeletal latent diffusion improves both quality and efficiency. High-fidelity medical shape generation reports MedSDF generation FID $G=(X,E)$0, KID $G=(X,E)$1, COV-CD $G=(X,E)$2, and generation time per sample of $G=(X,E)$3 s, while GeM3D requires $G=(X,E)$4 s per sample (Zhang et al., 8 Mar 2026). This suggests that skeleton-defined latent supports can be computationally advantageous even outside articulated motion.

A neighboring explicit-deformation formulation, DriveAnyMesh, does not require a rig or explicit skeleton but nevertheless shows how spatiotemporal latent-set diffusion can drive mesh animation from video. It reports PSNR $G=(X,E)$5, SSIM $G=(X,E)$6, LPIPS $G=(X,E)$7, and Chamfer distance $G=(X,E)$8, with inference around 10 seconds for 30 frames on a single Ascend 910B NPU (Shi et al., 9 Jun 2025). Although not a skeletal method in the strict sense, it functions as a transferable template for spatiotemporal latent diffusion over structured deformation tokens.

6. Limitations, evaluation issues, and future directions

A central misconception is that higher diversity necessarily means better motion modeling. SkeletonDiffusion explicitly shows that commonly used diversity metrics such as APD may reward limb-length inconsistency within a single sequence, so body realism metrics such as stretching and jitter are required to distinguish physically plausible diversity from artifact-driven variance (Curreli et al., 10 Jan 2025). This criticism generalizes beyond human motion: any skeletal latent diffusion framework that reports diversity without structural validity risks overstating generative quality.

Another recurring limitation concerns the assumptions built into the structural prior. LLMA assumes that modalities are synchronized views of one underlying dynamical system; if that assumption fails, alignment can become harmful (Dey et al., 15 Mar 2025). Pose-LDM depends on accurate conditioning poses and remains domain-specific to the SLP setting; it also prioritizes structural plausibility over photorealistic blankets (Khameneh et al., 26 Apr 2026). Stroke3D remains sensitive to rare concepts, side-view ambiguity, and incomplete or weak stroke constraints, while medical-shape generation reports topological discontinuities in the computed skeletons and identifies learning-based topology-aware skeletonization as future work (Zhao et al., 10 Feb 2026, Zhang et al., 8 Mar 2026).

The literature also indicates a tension between structural abstraction and exact embodiment. MotionDreamer requires post hoc inverse kinematics to enforce exact bone lengths, which means anatomical consistency is only partly internalized by the diffusion prior (Tao et al., 1 Jun 2026). Canonical-skeleton motion in-betweening preserves character agnosticism by moving physics into a second stage, but the trade-off is slight keyframe deviation during long autoregressive rollouts (Qin, 13 Apr 2025). Coronary anatomy generation resolves a related issue by shaping the latent space with topological interaction loss during VAE training rather than by imposing hard topological constraints during diffusion (Kadry et al., 2024).

Current directions implied by these works are relatively consistent. One direction is richer structure-aware noise and alignment models: nonisotropic joint covariance, local latent manifold alignment, or learned skeleton-aware guidance. A second is broader conditioning: text, video, multimodal biomechanics, or morphology-aligned measurements. A third is tighter coupling of diffusion with kinematics or physics, either through explicit post-processing, reinforcement-learning decoders, or decoder-side structural constraints. A fourth is data expansion: MotionDreamer’s large rigged-motion corpus, MedSDF’s 12,472 shapes across 14 anatomical categories, and TextuRig’s additional 6.8k textured rigged meshes all point to the importance of scale for cross-category generalization (Tao et al., 1 Jun 2026, Zhang et al., 8 Mar 2026, Zhao et al., 10 Feb 2026). Taken together, these trajectories indicate that a skeletal latent diffusion framework is best understood not as a single architecture, but as a design family whose common objective is to make diffusion respect topology, anatomy, or articulation at the level where generative uncertainty is actually modeled.