Human Gaussian Splats (HUGS)

• Human Gaussian Splats (HUGS) are techniques that represent human avatars using anisotropic 3D Gaussians, encoding geometry, appearance, and semantic features.
• They leverage explicit Gaussian primitives and deformation methods such as LBS and DQS for real-time photorealistic rendering and dynamic pose editing.
• HUGS enable applications in telepresence, motion analysis, and avatar creation while achieving state-of-the-art performance in image synthesis and rendering speed.

Human Gaussian Splats (HUGS) represent a family of methods for human avatar modeling and rendering based on the 3D Gaussian Splatting (3DGS) paradigm. HUGS and its variants encode human geometry, appearance, and (in several systems) semantic, skeletal, or biomechanical features as explicit sets of anisotropic 3D Gaussians. The core innovation of HUGS approaches is the adaptation of fast, explicit, local Gaussian primitives—originally developed for high-fidelity, static scene reconstruction—to the highly dynamic, articulate requirements of human reconstruction, editing, and real-time photorealistic rendering. The framework supports novel-view and novel-pose synthesis and is generalizable to a wide variety of data regimes, from dense multi-view studios to monocular or even single-image generalizable pipelines.

1. Mathematical Foundations and Gaussian Representation

Central to all HUGS methods is the representation of the human body (and possibly the scene) as a set of $N$ anisotropic 3D Gaussians: $$G_i(\mathbf x) = \exp\Bigl(-\tfrac12(\mathbf x-\boldsymbol\mu_i)^{T}\Sigma_i^{-1} (\mathbf x-\boldsymbol\mu_i)\Bigr)$$ Here, each Gaussian is parameterized by:

• Center $\boldsymbol\mu_i \in \mathbb{R}^3$
• Covariance $\Sigma_i = R_i S_i^2 R_i^T$, with rotation $R_i$ (from a quaternion) and per-axis scale $S_i$
• Appearance: typically color $c_i$, encoded directly or as spherical harmonic coefficients, and opacity $\alpha_i$
• Optionally: a learned feature vector $f_i$ for semantic, biomechanical, or surface embeddings

Each Gaussian is projected into a camera image plane via a Jacobian $J$ and extrinsic/intrinsic matrices $$G_i(\mathbf x) = \exp\Bigl(-\tfrac12(\mathbf x-\boldsymbol\mu_i)^{T}\Sigma_i^{-1} (\mathbf x-\boldsymbol\mu_i)\Bigr)$$0, with: $$G_i(\mathbf x) = \exp\Bigl(-\tfrac12(\mathbf x-\boldsymbol\mu_i)^{T}\Sigma_i^{-1} (\mathbf x-\boldsymbol\mu_i)\Bigr)$$1 Splatting onto the image plane produces an oriented elliptical “footprint” per Gaussian. Image synthesis blends contributions front-to-back using alpha compositing: $$G_i(\mathbf x) = \exp\Bigl(-\tfrac12(\mathbf x-\boldsymbol\mu_i)^{T}\Sigma_i^{-1} (\mathbf x-\boldsymbol\mu_i)\Bigr)$$2 This approach underlies all real-time differentiable rasterization in HUGS and is extensible to per-pixel dense feature compositing for surface tracking or biomechanical semantics (Dey et al., 2024, Kocabas et al., 2023).

2. Articulation, Deformation, and Skinning

To support pose control and animation, HUGS methods deform the Gaussian cloud from a canonical configuration (typically SMPL rest pose) to a target body pose. The canonical-to-posed transformation is driven by Linear Blend Skinning (LBS), Dual Quaternion Skinning (DQS), or hybrid approaches:

• Each Gaussian $$G_i(\mathbf x) = \exp\Bigl(-\tfrac12(\mathbf x-\boldsymbol\mu_i)^{T}\Sigma_i^{-1} (\mathbf x-\boldsymbol\mu_i)\Bigr)$$3 inherits (and often learns to refine) a vector of skinning weights $$G_i(\mathbf x) = \exp\Bigl(-\tfrac12(\mathbf x-\boldsymbol\mu_i)^{T}\Sigma_i^{-1} (\mathbf x-\boldsymbol\mu_i)\Bigr)$$4 over $$G_i(\mathbf x) = \exp\Bigl(-\tfrac12(\mathbf x-\boldsymbol\mu_i)^{T}\Sigma_i^{-1} (\mathbf x-\boldsymbol\mu_i)\Bigr)$$5 body joints.
• The deformed center is given by:

$$G_i(\mathbf x) = \exp\Bigl(-\tfrac12(\mathbf x-\boldsymbol\mu_i)^{T}\Sigma_i^{-1} (\mathbf x-\boldsymbol\mu_i)\Bigr)$$6

where $$G_i(\mathbf x) = \exp\Bigl(-\tfrac12(\mathbf x-\boldsymbol\mu_i)^{T}\Sigma_i^{-1} (\mathbf x-\boldsymbol\mu_i)\Bigr)$$7 are rotation and translation from joint $$G_i(\mathbf x) = \exp\Bigl(-\tfrac12(\mathbf x-\boldsymbol\mu_i)^{T}\Sigma_i^{-1} (\mathbf x-\boldsymbol\mu_i)\Bigr)$$8 given pose $$G_i(\mathbf x) = \exp\Bigl(-\tfrac12(\mathbf x-\boldsymbol\mu_i)^{T}\Sigma_i^{-1} (\mathbf x-\boldsymbol\mu_i)\Bigr)$$9.

Refinements include:

• Learning skinning weights per Gaussian via small MLPs, enabling fine adaptation to off-mesh regions (wrinkles, hair, cloth) (Hu et al., 2023, Kocabas et al., 2023).
• Local, per-Gaussian non-rigid offsets or latent-driven refinements (beyond pure LBS) (Moreau et al., 2023).
• Handling canonical pose misalignments or SMPL inaccuracies via pose-correction modules (Tian et al., 29 Apr 2025).

Several systems further decouple body and garment Gaussians (Mubashshira et al., 17 Apr 2026), introducing physics-based priors for cloth.

3. Feature Splatting, Semantics, and Biomechanics

HUGS supports not only geometry and appearance rendering but also the splatting of general learned feature vectors for tasks such as dense pose regression, skeletal keypoint localization, or part-based semantic segmentation. Each Gaussian may carry a feature embedding $\boldsymbol\mu_i \in \mathbb{R}^3$0 (e.g., DensePose, UV coordinates), which is splatted identically to color: $\boldsymbol\mu_i \in \mathbb{R}^3$1 A small decoder MLP maps per-pixel splatted features to desired semantic representations (UV, keypoints, body part). This integration enables real-time, simultaneous view synthesis and biomechanical or part-aware analysis (Dey et al., 2024, Wang et al., 2024). Semantic graph-based constraints (e.g., Hierarchical Semantic-Kinematic Graphs) regularize local neighborhood consistency and enforce part topology (Wang et al., 2024).

4. Reconstruction Pipelines and Architectures

Human Gaussian Splats pipelines can be organized into training-based, generalizable, or generative (diffusion-driven) methodologies:

Optimization-based / Canonical-Driven:

• SMPL mesh or vertices serve as anchoring scaffold; one Gaussian per mesh vertex (Hu et al., 2023, Wang et al., 12 Mar 2025)
• Gaussian positions/colors initialized from mesh or image data, then optimized for photometric and silhouette fidelity by differentiable splatting (Kocabas et al., 2023, Moreau et al., 2023)
• Adaptive split/merge/prune for density control (often KL-divergence measured) (Hu et al., 2023)

Feedforward / Generalizable:

• 2D U-Nets or transformer-based networks regress UV-parameterized or pixel-aligned splat parameters from sparse or single inputs (Kwon et al., 2024, Pan et al., 2024)
• Multi-scaffold approaches use concentric shells (offset along vertex normals) for fine-detail modeling beyond the body mesh (Kwon et al., 2024)
• Image-aligned point lifting, multi-stage coarse-to-fine regression exploit both pixel-level and voxel-level features (Xiao et al., 18 Mar 2025)
• Most methods efficiently composit tens of thousands of Gaussians per subject at real-time rates (>25 FPS) (Kocabas et al., 2023, Mubashshira et al., 17 Apr 2026)

Generative / Diffusion-Based:

• 3DGS parameters are generated by conditional diffusion models, either from single images, or directly from text prompts (Tang et al., 25 Jan 2025, Ivashechkin et al., 4 Jun 2025, Liu et al., 2023)
• Multi-stage or attribute-wise generation stabilizes optimization (Tang et al., 25 Jan 2025)
• Structure-aware guidance, annealed negative-prompt SDS, or semantic priors regulate density and artifact suppression
• Capable of high text-conditioned realism and diversity, real-time rendering, and orders-of-magnitude faster sample generation than NeRF/mesh counterparts (Tang et al., 25 Jan 2025, Ivashechkin et al., 4 Jun 2025)

5. Quantitative Evaluation and Performance

Human Gaussian Splats methods have established state-of-the-art results on standard benchmarks including NeuMan, ZJU-MoCap, THuman2.0, and MonoCap. Representative metrics:

• Novel view image synthesis: PSNR 32–36, SSIM 0.96–0.99, LPIPS 0.02–0.1 (Dey et al., 2024, Moreau et al., 2023, Hu et al., 2023, Pang et al., 2023)
• Dense pose regression: MSE 0.0017, 2D keypoints [email protected] = 0.8707, 3D MPJPE = 0.0704 m (Dey et al., 2024)
• Real-time rendering: 25–189 FPS depending on architecture and hardware (Dey et al., 2024, Hu et al., 2023, Mubashshira et al., 17 Apr 2026)
• Generalizability: Single-view and sparse-view methods (e.g. HumanSplat, RoGSplat, HuGDiffusion) outperform NeRF and previous feed-forward methods in perceptual and geometric quality under domain shift (Pan et al., 2024, Xiao et al., 18 Mar 2025, Tang et al., 25 Jan 2025).

A summary of results for recent methods can be organized as follows:

Method	PSNR↑	SSIM↑	LPIPS↓	FPS↑	Task/Notes
HFGaussian	32.43	0.973	0.030	24.4	Real-time, integrated biomechanics
GauHuman	31.34	0.965	0.030	189	1–2 min train, canonical + LBS
HuGS	32.49	0.984	0.019	80	Coarse-to-fine deformation
EfficientHuman	31.62	0.97	—	—	2D Gaussian surfels, <1 min train
Cloth-HUGS	31.0	0.975	0.028	60+	Disentangled body/cloth, physics
HumanSplat	24.0	0.918	0.055	150+	Single-image, prior-augmented
RoGSplat	28.9	0.962	0.043	5+	Sparse-view, coarse-to-fine pipeline

6. Applications, Limitations, and Future Directions

Human Gaussian Splats have enabled a broad spectrum of applications:

• Real-time telepresence and avatar streaming in AR/VR (Kocabas et al., 2023, Dey et al., 2024)
• Biomechanically informed motion analysis, including explicit skeleton/keypoint/dense pose estimation (Dey et al., 2024)
• Interactive editing, pose transfer, and avatar creation
• Text-guided 3D avatar synthesis for rapid prototyping and digital content creation (Ivashechkin et al., 4 Jun 2025, Liu et al., 2023)

Key limitations across variants include:

• Reliance on SMPL or parametric priors constrains generalization to non-canonical bodies and loose clothing (Kocabas et al., 2023, Kwon et al., 2024)
• Current methods focus on single-person articulation; multi-person/crowd modeling is still open (Dey et al., 2024, Hu et al., 2023)
• Temporal coherence for dynamic sequences is not yet consistently addressed, with most systems treating each frame independently (Dey et al., 2024, Mubashshira et al., 17 Apr 2026)
• Recovery of watertight geometry for downstream physics or simulation remains future work (Hu et al., 2023, Tian et al., 29 Apr 2025)

Directions for future research include:

• Multiperson/multi-instance HUGS models for group scenes (Dey et al., 2024, Hu et al., 2023)
• Explicit muscle/soft-tissue physics (4D Gaussian splatting) (Dey et al., 2024, Mubashshira et al., 17 Apr 2026)
• Joint mesh extraction and Gaussian optimization for simulation and editing (Hu et al., 2023, Tian et al., 29 Apr 2025)
• End-to-end, semantics-aware pipelines fusing part labels and hierarchical feature splatting (Wang et al., 2024, Dey et al., 2024)

7. Notable Variants and Recent Progress

Recent research has extended the original HUGS paradigm in multiple orthogonal directions:

• Hierarchical graph control and semantic-kinematic priors (Hierarchical Graph Human Gaussian Control) (Wang et al., 2024)
• Disentangling body and cloth with physics-based learning for loose garment realism (Cloth-HUGS) (Mubashshira et al., 17 Apr 2026)
• 2D surfel-based (EfficientHuman) and position-texture subdivision (SkinningGS) for memory/fidelity tradeoffs (Tian et al., 29 Apr 2025, Li et al., 25 Jun 2025)
• Sparse-view and single-image generalizable synthesis (HumanSplat, RoGSplat, HuGDiffusion) with diffusion and transformer conditioning (Pan et al., 2024, Xiao et al., 18 Mar 2025, Tang et al., 25 Jan 2025)
• Controllable text-to-3D human generation via prompt-driven 3DGS diffusion (Ivashechkin et al., 4 Jun 2025, Liu et al., 2023)

Together, these advances define Human Gaussian Splats as a versatile, high-performance family of techniques for human-centric modeling, reconstruction, and rendering, unifying explicit geometry, rich semantics, and practical computational efficiency.