Stable Part Diffusion 4D (SP4D)

• Stable Part Diffusion 4D (SP4D) is a generative framework that creates consistent multi-view videos and kinematic segmentation maps from monocular inputs.
• It uses a dual-branch architecture with an RGB branch and a part branch, incorporating spatial color encoding and bidirectional fusion modules for enhanced coherence.
• The approach ensures kinematically meaningful decompositions, enabling automated 3D lifting, rigging, and animation with robust temporal and spatial consistency.

Stable Part Diffusion 4D (SP4D) is a diffusion-based generative modeling framework designed to synthesize temporally consistent, multi-view RGB videos in tandem with kinematic part segmentation maps from monocular inputs. Unlike conventional segmentation or video diffusion methods that focus on appearance-based or semantic cues, SP4D learns to generate kinematic parts—structural regions that are stable under object articulation, consistent across frames and views, and directly suited for downstream animation, rigging, and motion analysis. The method uses a dual-branch diffusion backbone, a spatial color encoding for parts, inter-branch fusion modules, and a contrastive regularization to ensure coherent, kinematically meaningful decompositions. The approach is validated on a large dataset of multi-part rigged 3D assets and demonstrates strong generalization to novel and real-world data (Zhang et al., 12 Sep 2025).

1. Architectural Overview of SP4D

SP4D is based on an adaptation and extension of a multi-view video diffusion backbone—specifically, an enhanced UNet structure as in SV4D 2.0 (Yao et al., 20 Mar 2025). The architecture comprises two concurrently operating branches:

• The RGB branch synthesizes photo‐realistic multi-view dynamic video frames.
• The part branch simultaneously produces time- and view-consistent kinematic part segmentation maps encoded as RGB-like images.

Both branches ingest a shared spatio-temporal latent tensor and utilize the same VAE for encoding and decoding, with positional encodings shared between them (encompassing camera parameters and temporal indices). Key architectural features include:

• Spatial Color Encoding: Kinematic part segmentation maps are encoded as images whose colors correspond to normalized 3D part centers, leading to continuous-valued outputs compatible with the VAE structure.
• Bidirectional Diffusion Fusion (BiDiFuse) Modules: Lightweight fusion layers facilitate information flow and cross-branch supervision throughout the model, inserted at each resolution in encoder, bottleneck, and decoder stages.
• Contrastive Part Consistency Regularization: A loss enforces that part representations remain temporally and spatially coherent across frames and views.

This integrated yet modular design allows RGB appearance and kinematic structure to be synthesized in a mutually consistent manner, which is critical for generating articulated objects suited for animation.

2. Kinematic Part Segmentation via Spatial Color Encoding

SP4D innovates on multi-part segmentation by encoding each kinematic part as a unique, deterministic RGB value that reflects its normalized 3D center within a unit cube. For a part $p$ with normalized position $(x_p, y_p, z_p)$, its color is mapped via a deterministic function to $(r_p, g_p, b_p)$ in $[0,1]^3$. This encoding provides:

• View and Temporal Consistency: The same part is always represented by the same color, regardless of viewpoint or articulation.
• Architectural Simplification: Both part and RGB branches can share latent VAE and generation modules.
• Postprocessing Flexibility: At inference, segmentation is recovered by clustering in continuous color space, avoiding reliance on discrete class labels or fixed part counts.

This mechanism is critical for flexibly supporting varying part granularities and ensuring downstream 3D lifting and rigging is easily performed.

3. Cross-branch Fusion and Training Objectives

The inter-branch BiDiFuse module enables bidirectional feature sharing, enhancing consistency between appearance and part structure. At any resolution, intermediate features $h^{(\mathrm{RGB})}$ and $h^{(\mathrm{Part})}$ are transformed as: $$\begin{aligned} h^{(\mathrm{RGB})}_{\text{fused}} &= h^{(\mathrm{RGB})} + \mathcal{F}([h^{(\mathrm{RGB})}, h^{(\mathrm{Part})}]) \ h^{(\mathrm{Part})}_{\text{fused}} &= h^{(\mathrm{Part})} + \mathcal{F}([h^{(\mathrm{RGB})}, h^{(\mathrm{Part})}]) \end{aligned}$$ where $\mathcal{F}$ is a $1\times1$ convolution with ReLU (feature concatenation in channel axis).

A contrastive part consistency loss based on InfoNCE pulls per-part region features (aggregated over pixels within each predicted part mask) closer if they correspond to the same physical part across frames/views, and pushes them apart if from different parts: $$\mathcal{L}_{\text{contrast}} = - \mathbb{E}_{i \in P, j \in P_i^+ } \left[ \log \frac{ \exp ( \operatorname{sim}(f_i, f_j)/\tau ) }{ \sum_{k \in P \setminus \{i\}} \exp ( \operatorname{sim}(f_i, f_k)/\tau ) } \right]$$ where $f_i$ is the feature vector for a part instance, $P$ is the set of all parts, and $P_i^+$ contains positives (same part, different frames/views).

This combination encourages alignment of part predictions in both space and time, overcoming issues suffered by traditional appearance-based segmentation, which is typically inconsistent under articulation or occlusion.

4. 3D Lifting Pipeline and Automatic Rigging

Outputs of SP4D (multi-view RGB frames and part segmentation maps) can be lifted into 3D assets prepared for animation via a structured pipeline:

1. 3D Mesh Reconstruction: Multi-view RGB predictions are processed by a reconstruction framework (such as Hunyuan 3D 2.0) to obtain an untextured mesh.
2. Part Assignment: Part segmentation masks are projected onto the reconstructed mesh vertices, and clustering (e.g., HDBSCAN) is used to assign part labels.
3. Harmonic Skinning: For each part $p$, a harmonic field $w_p$ is computed by solving the Laplacian on the mesh subject to boundary constraints (vertices assigned to part $p$ have $w_p=1$, others $w_p=0$):

$$\Delta w_p(x) = 0 \quad \text{on interior, with} \quad w_p(x) = b_p(x) \text{ on } \partial\Omega_p$$

This enables soft skinning weights suitable for skeletal animation.

This pipeline, enabled by SP4D’s consistent kinematic part maps, automates the challenging task of rigging arbitrary reconstructed objects.

5. KinematicParts20K Dataset and Supervision

SP4D is trained and evaluated on the KinematicParts20K dataset, comprising over 20,000 rigged 3D assets curated from Objaverse XL. Each object includes:

• Multi-view RGB video sequences (24 views, 24 frames/object)
• Aligned part segmentation videos, with parts derived from mesh skeletons and bone hierarchies, filtered and merged to limit granularity
• Rigging data (skeleton, bones, skinning weights)

This dataset provides dense view-temporal and kinematic supervision for learning articulation-aware segmentation and appearance generation in complex real-world and synthetic assets.

6. Empirical Performance and Generalization

SP4D outperforms canonical segmentation approaches—both 2D, such as SAM2 and DeepViT, and 3D, such as SAMPart3D and Segment Any Mesh—across mIoU, ARI, F1, and mean Accuracy when evaluated on multi-view and time-resolved metrics. Qualitatively, the generated kinematic part maps exhibit higher clarity and temporal consistency, facilitating effective downstream mesh rigging and animation. User studies confirm a strong preference for SP4D’s outputs—especially for articulation clarity and overall rigging readiness.

Generalization experiments show robust performance across diverse synthetic categories, real-world video sequences, and with rare or unusual object articulation. SP4D further enables new workflows for generating animation-ready digital assets from unconstrained video or minimal input.

7. Theoretical and Methodological Foundations

SP4D builds on three conceptual and methodological pillars:

• Geometric regularization via spatial color encoding endows the part branch with view-invariant correspondence, while latent sharing with the RGB branch leverages powerful pretrained appearance diffusion models.
• Bidirectional cross-branch fusion establishes mutual guidance, leading to expressive latent representations capturing both appearance and kinematic structure.
• Contrastive learning across spatio-temporal context enables the network to correctly track articulated parts through challenging motion, significantly improving stability and coherence.

This design is notable in that it emerges directly from coordinated advances in part-aware generative modeling, spatio-temporal diffusion, and geometric reasoning applied to high-dimensional video and 3D data.

---

In summary, Stable Part Diffusion 4D (SP4D) operationalizes part-aware, temporally stable multi-view generation using a diffusion framework that directly links video appearance synthesis, kinematic part segmentation, and downstream 3D asset creation with harmonically weighted rigging. It enables new directions for automated animation, motion analysis, and robust 4D content generation suitable for graphics, robotics, and vision applications (Zhang et al., 12 Sep 2025).