Gaussian Refinement for Human-Scene Rendering

• Human-Scene Gaussian Refinement Optimization is a technique that uses explicit 3D Gaussian splats to represent dynamic human motion and static scene geometry for physically plausible neural rendering.
• It employs loss functions such as contact, separation, and temporal smoothness regularization, combined with semantic gating, to ensure accurate human-scene contact and deformation.
• The method leverages joint differentiable rendering and hyperparameter-free densification, achieving state-of-the-art performance in metrics like PSNR, SSIM, and real-time rendering speeds.

Human-Scene Gaussian Refinement Optimization constitutes the set of optimization methodologies, loss formulations, and implementation patterns by which explicit 3D Gaussian primitives—jointly representing dynamic humans and static scenes—are adjusted to enforce physical plausibility, contact, and high-fidelity interaction in neural rendering systems based on Gaussian Splatting. These methods include per-Gaussian translation optimization for human-scene contact (as in post-hoc refinement), semantic gating for selective deformation, loss-driven regularization for geometric consistency, perceptual and event-based supervision, hyperparameter-free densification, and joint optimization strategies grounded in differentiable rendering pipelines.

1. Joint Representation of Human and Scene with Gaussian Primitives

All modern human-scene Gaussian refinement approaches employ a unified parametric field composed of 3D Gaussian splats, with the dual aim of: (a) representing animatable humans whose geometry evolves via SMPL-driven or learned deformation, and (b) representing static background scenes. Each Gaussian is defined by its center $\mu_i \in \mathbb{R}^3$, covariance $\Sigma_i$, color $c_i$, and opacity $\alpha_i$ (Kocabas et al., 2023, Mir et al., 13 Nov 2025, Li et al., 25 Jun 2025, Yin et al., 23 Sep 2025).

The core difference between human and scene Gaussians resides in their animation and deformation properties:

• Human Gaussians typically undergo pose-dependent deformation via Linear Blend Skinning (LBS) with SMPL weights, potentially augmented by learned offsets for cloth/hair (Kocabas et al., 2023, Li et al., 25 Jun 2025).
• Scene Gaussians remain static; they do not participate in skinning or deformation, and feature vectors for background appearance are handled independently.

Semantic logit assignments or segmentation masks—including learnable logits $s_i$ subject to binary thresholding—enable explicit gating: only designated human Gaussians receive deformation, with semantic separation enforced by masked losses, contact constraints, or dual color/appearance MLPs (Yin et al., 23 Sep 2025, Kocabas et al., 2023).

2. Refinement Optimization: Contact Enforcement, Separation, and Smoothness

To achieve realistic interaction at the human-scene boundary (e.g., foot-floor contacts, avoidance of penetrations), refinement optimization operates post motion synthesis by updating only a subset of human Gaussian centers via per-frame translation variables $T_{k,t} \in \mathbb{R}^3$ (Mir et al., 13 Nov 2025).

The refinement objective comprises:

• Contact Loss (for detected contacts): Minimizes squared soft nearest-neighbor scene-Gaussian distance,

$$L_{\text{contact}}(k,t) = d_\beta(x_{k,t}^P + T_{k,t})^2$$

• Separation Loss (for non-contacts): Penalizes insufficient clearance,

$$L_{\text{separation}}(k,t) = [h_r(d_\beta(x_{k,t}^P + T_{k,t}))]^2$$

where $h_r(d) = \max(0, r-d)$ and $d_\beta(x)$ is a soft minimum over scene shots.

• Temporal Smoothness Regularization: First-order difference penalty on subsequent translations,

$$L_{\text{smooth}} = \lambda_t \sum_{k} \sum_{t=2}^{T} \|T_{k,t} - T_{k,t-1}\|_2^2$$

The total refinement loss is summed over contact candidates and frames, with only the means updated; covariances, opacities, colors—and all scene Gaussians—remain fixed (Mir et al., 13 Nov 2025). Typical optimization employs Adam with a small set of variables (contacted Gaussians across $T$ frames), converging within 50–100 gradient steps.

3. Semantic Gating and Deformation Models

In pipelines such as Event-guided 3DGS (Yin et al., 23 Sep 2025), unified semantic labeling (logit-based) enables selective deformation:

• Gaussians with thresholded $s_i = 1$ are designated human—receiving both non-rigid and LBS-based deformation via networks $\Phi_{\psi_{nr}}$ (non-rigid) and $\Phi_{\psi_r}$ (LBS, Eq (2)). For example:

$$G_{o,i} = s_i \cdot \Phi_{\psi_r}(\Phi_{\psi_{nr}}(G_{c,i}, z_p); \{B_b\})$$

where $z_p$ denotes current pose code.

• Scene Gaussians ($s_i = 0$) remain undeformed, copying their canonical properties into observation space.

Densification and appearance modeling are likewise gated, with human and background MLPs receiving separate inputs. Scene-adaptive perceptual densification strategies further refine distribution by allocating granularity according to perceptual sensitivity (Zhou et al., 14 Jun 2025).

4. Regularization, Perceptual & Event-Guided Losses

Refinement is stabilized by regularization terms enforcing geometric coherence. For example, GPS-Gaussian+ (Zhou et al., 2024) introduces bidirectional Chamfer loss between two view sets:

$$L_{CD} = \frac{1}{|P^l|}\sum_{p_l \in P^l} \min_{p_r \in P^r} \|p_l-p_r\|_2 + \frac{1}{|P^r|}\sum_{p_r \in P^r} \min_{p_l \in P^l} \|p_r-p_l\|_2$$

with scalar weighting (e.g., $\alpha \approx 0.5$).

Event-guided Gaussian Splatting (Yin et al., 23 Sep 2025) leverages an event-guided loss comparing simulated log-brightness differences between frames against an event stream:

$$L_{\text{event}} = w_{ev} \cdot \left\| \frac{\Delta\mathcal{L}}{\|\Delta\mathcal{L}\|_F} - \frac{E_{gt}}{\|E_{gt}\|_F} \right\|_1$$

where $\Delta\mathcal{L}(u)$ is computed from rendered sRGB frames raised to 2.2.

Perceptual-GS (Zhou et al., 14 Jun 2025) employs a dual-branch loss, including a sensitivity alignment (binary cross-entropy) ensuring densification aligns with human visual sensitivity:

$$\mathcal{L}_v^S := \mathrm{BCE}\left(\mathcal{R}_v^S, \mathcal{I}_v^S\right)$$

5. Hyperparameter-Free Densification and Initialization

SkinningGS (Li et al., 25 Jun 2025) introduces a position texture (UV-mapped over SMPL surface) whereby exactly one Gaussian is seeded per texture texel, with skinning weights interpolated from triangle barycentric coordinates. This procedure obviates the need for ad hoc splitting thresholds, leading to adjustable density via texture resolution alone.

Human features (colors, geometry offsets, scales) are predicted by a fully-convolutional Power-of-Points (PoP) network over this texture, enabling direct conversion to Gaussian sets suitable for LBS skinning, rendering, and joint optimization. Background Gaussians are initialized independently (COLMAP/MVS), remaining strictly static.

6. Optimization Pipeline and Empirical Outcomes

All contemporary systems utilize fully differentiable, joint optimization of splat parameters (means, covariances, colors, opacities), deformation network weights, skinning weights, and, where applicable, feature network (MLP/CNN/PoP) parameters. The typical loss composites encompass:

• Image-space photometric losses ($L_1$, SSIM, LPIPS), sometimes VGG or patch-based
• Human-only photometric losses via segmentation masks
• Skinning-weight or feature regularizers
• Event or perceptual branch losses

Optimization is conducted via Adam, with learning rates and scheduling tuned for splat and network parameters (higher LR for means/covariances, lower for deformation nets). Densification (split, clone, prune) occurs periodically based on either geometric or perceptual metrics. For instance, HUGS (Kocabas et al., 2023) performs densification every 600 steps, ending with ~200K Gaussians per semantic region and training complete in ≈0.5 h.

Typical performance outcomes:

• GPS-Gaussian+ (Zhou et al., 2024): $33.7$ PSNR, $0.971$ SSIM, $0.041$ LPIPS at $25$ FPS for human-scene data
• SkinningGS (Li et al., 25 Jun 2025): $>$100 FPS rendering, %%%%29$$L_{\text{contact}}(k,t) = d_\beta(x_{k,t}^P + T_{k,t})^2$$30%%%% HUGS speed, fewer splats per human with competitive or superior reconstruction metrics
• HUGS (Kocabas et al., 2023): $60$ FPS, state-of-the-art metrics on NeuMan/ZJU-MoCap

Refinement (as in AHA! (Mir et al., 13 Nov 2025)) specifically reduces foot–floor penetrations and improves temporal contact stability; ablation studies show a 5–10 point drop in human preference scores when omitted. Event-guided refinement sharpens dynamic reconstruction in fast motion scenarios where RGB-only photometric losses fail.

7. Practical Implications and Extensions

Human-Scene Gaussian Refinement Optimization has enabled a leap in physical plausibility and photorealism for free-viewpoint and animated rendering from sparse views, monocular event cameras, or RGB videos. Extensions to animal scenes, as demonstrated by SkinningGS (Li et al., 25 Jun 2025), are feasible given an accurate poseable model.

The decoupling of motion synthesis from rendering, semantic gating of deformation, and post-hoc translation of contact Gaussians constitute the foundational steps in modern pipelines. Hyperparameter-free approaches and perception/event-based losses further automate quality and efficiency.

A plausible implication is that these joint, fully-differentiable optimization paradigms—integrating explicit parametric deformation with contact-aware refinement—will underpin future methods for real-time interactive rendering, multi-agent scenes, and neuro-symbolic integration for semantic scene understanding in dynamic scenarios.