Articulated 3D Gaussians

• Articulated 3D Gaussians are explicit, fully differentiable volumetric models that approximate object geometry and appearance through mixtures of spatial 3D Gaussian functions dynamically transformed by pose-dependent kinematics.
• The methodology integrates differentiable rendering, skinning via both template-based and learned kinematic structures, and optimization using multi-view photometric, SSIM, and regularization losses.
• These models enable fast novel-pose synthesis and have proven effective in applications such as digital twins, robotic manipulation, and avatar reconstruction, outperforming earlier NeRF-based approaches.

Articulated 3D Gaussians are a class of explicit, fully differentiable, volumetric graphics models that approximate articulated object geometry and appearance using mixtures of spatial 3D Gaussian functions whose parameters are dynamically transformed under a pose-dependent kinematic model. This approach unifies high-fidelity appearance modeling with physically structured articulation, supporting applications in dynamic scene modeling, robotics, avatar reconstruction, and interactive rendering. Articulated 3D Gaussians enable fast, high-quality novel-pose synthesis and joint learning of geometry, motion, and visual attributes from images or videos, and have become foundational across recent advances in digital twin modeling, robotic manipulation, and neural performance capture.

1. Mathematical Formulation of Articulated 3D Gaussians

The core representational unit is a 3D anisotropic Gaussian, parameterized as

$$\mathcal{G}_i = (\mu^{(i)}, R^{(i)}, s^{(i)}, \eta^{(i)}, f^{(i)})$$

with $\mu^{(i)} \in \mathbb{R}^3$ denoting the center, $R^{(i)} \in SO(3)$ the orientation, $s^{(i)} \in \mathbb{R}_+^3$ the anisotropic scale, and $$\Sigma^{(i)} = R^{(i)} \operatorname{diag}(s^{(i)2}) R^{(i)T}$$ the covariance; $\eta^{(i)}$ is the scalar opacity or density; and $f^{(i)}$ are spherical harmonic coefficients for view-dependent color. In some frameworks, an explicit color vector $c^{(i)}$ replaces $f^{(i)}$ when isotropic or static color suffices. The spatial density field is modeled as a sum of $N$ such Gaussians:

$\mu^{(i)} \in \mathbb{R}^3$0

The joint geometric and appearance representation supports compositing in rendering pipelines via depth-sorted alpha blending.

Articulation is enacted through bone- or part-driven transformations, using either template-based or learned kinematic structures. Each Gaussian carries skinning weights $\mu^{(i)} \in \mathbb{R}^3$1 (for $\mu^{(i)} \in \mathbb{R}^3$2 parts or bones), and, under an articulated pose $\mu^{(i)} \in \mathbb{R}^3$3, transforms as:

$\mu^{(i)} \in \mathbb{R}^3$4

$\mu^{(i)} \in \mathbb{R}^3$5

with $\mu^{(i)} \in \mathbb{R}^3$6 the SE(3) transform for part/bone $\mu^{(i)} \in \mathbb{R}^3$7. This forward skinning supports both template priors (e.g., SMPL) and data-driven articulation (Lei et al., 2023, Liu et al., 26 Feb 2025, Hu et al., 2023).

Expanding the representation, latent bones or extra kinematic nodes are introduced to capture non-skeletal deformations such as clothing or fur, accompanied by learnable per-Gaussian skinning weights to these degrees of freedom (Lei et al., 2023).

2. Differentiable Rendering and Optimization

Rendering is performed by projecting posed Gaussians to the image plane as 2D ellipses, computing pixel-wise opacity and radiance via the projected densities:

$\mu^{(i)} \in \mathbb{R}^3$8

with $\mu^{(i)} \in \mathbb{R}^3$9 the camera extrinsic matrix, $R^{(i)} \in SO(3)$0 intrinsics, and $R^{(i)} \in SO(3)$1 the Jacobian of projection. Per-pixel color is composited in front-to-back order using:

$R^{(i)} \in SO(3)$2

where $R^{(i)} \in SO(3)$3 is the projected Gaussian's opacity at pixel $R^{(i)} \in SO(3)$4 (Lei et al., 2023, Pokhariya et al., 2023).

The whole pipeline is differentiable with respect to all Gaussian, skinning, and articulation parameters. Losses used for optimization include photometric (L1), structural similarity (SSIM), part-consistency, regularization on skinning weights, and (optionally) pose-refinement per frame. Some frameworks utilize SDF (Signed Distance Field) regularization to align Gaussian distributions with watertight surfaces, and geometric, ARAP, or physics-based priors to improve physical plausibility (Wu et al., 9 Mar 2025, Yu et al., 20 Jun 2025, Liu et al., 26 Feb 2025).

3. Kinematic Articulation and Skinning Structures

The kinematic model driving Gaussian articulation encompasses both category-driven and data-driven approaches. Template-based systems exploit prior skeletons such as SMPL for humans or SMAL for animals, providing a set of $R^{(i)} \in SO(3)$5 bone transforms and predefined skinning fields. More recent approaches learn both the skeleton topology and the bone transforms from the observed motion (Lei et al., 2023, Yu et al., 3 Jul 2025, Yu et al., 20 Jun 2025).

Advanced variants encode the assignment of Gaussians to articulated parts via softmaxed embeddings, supporting unsupervised discovery of part structure (Liu et al., 26 Feb 2025, Yu et al., 20 Jun 2025, Shen et al., 20 Aug 2025). Linear blend skinning (LBS) is the standard mechanism, but nonlinear extensions (e.g., via MLPs or additional latent bones) enhance flexibility for non-rigid or data-driven deformations (Lei et al., 2023, Wu et al., 4 Feb 2026).

Motion-aware canonical representation and physical constraints (contact, velocity consistency, vector-field alignment) are incorporated to ensure that per-part transformations remain consistent with real-world kinematics, enabling stable, collision-free articulation (Yu et al., 20 Jun 2025).

4. Initialization, Training Procedures, and Pipeline Structure

Articulated 3D Gaussian pipelines typically initialize Gaussians via template mesh sampling or by separate static reconstructions of different articulation states. Coarse-to-fine initialization, such as Hungarian matching of single-state Gaussians followed by creation of a canonical mid-pose field, supports robust alignment and part clustering (Liu et al., 26 Feb 2025, Shen et al., 20 Aug 2025). Unsupervised part discovery is performed by clustering Gaussian trajectories or latent part embeddings, augmented by Mahalanobis-distance-based assignments or smoothness-regularized MLPs (Liu et al., 26 Feb 2025, Chao et al., 28 Jun 2025).

Optimization proceeds in either staged or unified fashion, involving joint minimization over geometry (centers/covariances), appearance, skinning assignments, articulation parameters, and sometimes SDF fields. Losses typically include multi-view photometric consistency, part-segmentation terms, regularization for smoothness and physical plausibility, and, where applicable, trajectory or correspondence supervision (Shen et al., 20 Aug 2025, Yu et al., 3 Jul 2025, Liu et al., 26 Feb 2025).

Splitting, merging, and pruning operations, often guided by gradient magnitude and KL divergence between Gaussians, control model complexity and maintain high fidelity with minimal parameter count (Hu et al., 2023). Collision avoidance mechanisms, such as repel points near static bases, mitigate ghost interpenetrations in articulated configurations (Yu et al., 20 Jun 2025).

5. Applications and Benchmarking

Articulated 3D Gaussians underlie several state-of-the-art pipelines for human performance capture, robotics, part segmentation, manipulation, and editable avatar generation. Notable applications include:

• Monocular or multi-view capture of deformable humans, animals, and multi-part objects, supporting real-time novel-view or novel-pose synthesis at over 150 FPS (Lei et al., 2023, Hu et al., 2023, Wu et al., 4 Feb 2026, Liu et al., 26 Feb 2025).
• Interactive physical/visual modeling for articulated robotic manipulation, with accurate joint estimation and cross-embodiment digital twin transfer (Yu et al., 3 Jul 2025, Shen et al., 20 Aug 2025).
• Markerless hand–object interaction modeling for dense grasp contact maps, validated via large-scale multi-view datasets (Pokhariya et al., 2023).
• Complex multi-part digital twin creation with physically consistent joint structure and scalability up to 20 articulated bodies per object (Shen et al., 20 Aug 2025).
• Self-supervised, annotation-free reconstruction and decomposition from unconstrained video or RGB-D using trajectory and vision-language-clustered segmentation (Wang et al., 11 Jun 2025, Dai et al., 23 Mar 2026).

On benchmarks such as ZJU-MoCap, UBC-Fashion, and MPArt-90, articulated 3D Gaussian methods outperform NeRF-based and earlier parametric approaches in PSNR, SSIM, LPIPS, joint-axis and position error, and surface accuracy metrics, especially for multi-part and low-visibility cases (Lei et al., 2023, Shen et al., 20 Aug 2025, Liu et al., 26 Feb 2025).

6. Variants, Limitations, and Ongoing Directions

Current variants extend articulated 3D Gaussians with hybrid SDF alignment for sharper surfaces (Wu et al., 9 Mar 2025), data-driven skeleton extraction (Wu et al., 4 Feb 2026, Yao et al., 21 Mar 2025), or hierarchical skeleton/non-rigid refinement for editable 4D generation and motion editing (Wu et al., 4 Feb 2026). Parts are discovered entirely unsupervised via differentiable clustering and trajectory analysis, and motion is encoded through rigid kinematic trees, soft skinning, and, if needed, MLP-parameterized latent transformations.

Key limitations arise from the reliance on dense multi-view or RGB-D data for fully self-supervised part segmentation and fine-level contact/motion estimation, sensitivity to outlier trajectories, and the computational expense of large Gaussian sets (alleviated by split/merge/prune heuristics). Physical simulation and frictional contact modeling remain areas for further integration, as current frameworks focus primarily on kinematics and appearance.

Recent advances point toward unified, cross-embodiment systems supporting zero-shot model transfer, stable multi-object scenes, and interactive scene editing by embedding articulated 3D Gaussian fields within simulation and rendering frameworks (Yu et al., 3 Jul 2025, Shen et al., 20 Aug 2025, Wu et al., 4 Feb 2026).

---

For comprehensive mathematics, implementation strategies, and quantitative validation, see (Lei et al., 2023, Yu et al., 3 Jul 2025, Liu et al., 26 Feb 2025, Shen et al., 20 Aug 2025, Yu et al., 20 Jun 2025, Hu et al., 2023, Pokhariya et al., 2023, Wu et al., 4 Feb 2026, Wu et al., 9 Mar 2025, Chao et al., 28 Jun 2025, Wang et al., 11 Jun 2025, Guo et al., 11 Mar 2025, Ding et al., 2018).