SkelSplat: Robust Multi-view 3D Human Pose Estimation with Differentiable Gaussian Rendering (2511.08294v1)

Abstract: Accurate 3D human pose estimation is fundamental for applications such as augmented reality and human-robot interaction. State-of-the-art multi-view methods learn to fuse predictions across views by training on large annotated datasets, leading to poor generalization when the test scenario differs. To overcome these limitations, we propose SkelSplat, a novel framework for multi-view 3D human pose estimation based on differentiable Gaussian rendering. Human pose is modeled as a skeleton of 3D Gaussians, one per joint, optimized via differentiable rendering to enable seamless fusion of arbitrary camera views without 3D ground-truth supervision. Since Gaussian Splatting was originally designed for dense scene reconstruction, we propose a novel one-hot encoding scheme that enables independent optimization of human joints. SkelSplat outperforms approaches that do not rely on 3D ground truth in Human3.6M and CMU, while reducing the cross-dataset error up to 47.8% compared to learning-based methods. Experiments on Human3.6M-Occ and Occlusion-Person demonstrate robustness to occlusions, without scenario-specific fine-tuning. Our project page is available here: https://skelsplat.github.io.

• The paper introduces SkelSplat, a framework that leverages differentiable Gaussian rendering to optimize independent 3D Gaussians for human joint positions without relying on 3D ground truth.
• It employs one-hot joint encoding and accumulates gradients across multiple views, ensuring projective consistency and anatomical symmetry in pose estimation.
• Experimental results show that SkelSplat achieves superior generalization and robustness to occlusion on benchmark datasets while eliminating dataset-specific retraining.

SkelSplat: Robust Multi-View 3D Human Pose Estimation with Differentiable Gaussian Rendering

Introduction

The paper introduces SkelSplat, a multi-view 3D human pose estimation framework that leverages differentiable Gaussian rendering to achieve robust, annotation-free pose inference across varying scenarios and camera setups. Traditional learning-based fusion methods for multi-view pose estimation suffer from poor generalization, as they depend heavily on dataset-specific training and camera configurations. SkelSplat addresses these limitations by treating each human joint as an independently optimized 3D Gaussian, obviating the need for 3D ground truth during deployment and requiring only 2D keypoint detections as supervision.

Figure 1: SkelSplat 3D Gaussian joints are optimized (red) by aligning renderings with 2D detection heatmaps (green arrows).

Methodology

Gaussian Splatting for Skeletal Representation

SkelSplat builds on Gaussian Splatting, previously developed for dense scene modeling, and repurposes it for 3D skeleton pose estimation. Each human joint is modeled as an anisotropic 3D Gaussian ($gs_j = (\mu_j, \Sigma_j)$, representing mean position and covariance), initialized according to a coarse 3D pose estimate and independently optimized through differentiable rendering. The choice of Gaussians enables projective consistency across views and provides a natural means for representing spatial uncertainty for each joint.

Differentiable Rendering and One-Hot Joint Encoding

Unlike the original Gaussian Splatting, which operates in the RGB domain, SkelSplat employs a one-hot encoding scheme for joint identity. Each Gaussian's appearance vector $c_j$ is redefined to activate only its corresponding channel, and the rendering function is modified to output an $N$-channel image with each channel representing a distinct joint.

Figure 2: One-hot encoding, each joint rendered to its channel.

The rendered image is compared to pseudo-ground-truth heatmaps generated from 2D detector outputs, with 2D covariances derived by projecting 3D Gaussian covariances into the image plane using the camera intrinsics and extrinsics. This ensures the optimization is geometrically consistent and agnostic to the choice of input cameras.

Supervision and Loss Functions

Optimization proceeds via a joint loss comprising:

• Masked Multi-View $\ell_2$ Loss: The difference between the rendered $N$-channel heatmaps and the pseudo-ground-truth, computed only at nonzero pixels.
• 3D Structural Symmetry Loss: Encourages anatomical plausibility by penalizing differences in lengths between symmetric limbs (e.g., left/right arms and legs).
• Gradient Accumulation Across Views: Unlike per-view updates in classic Gaussian Splatting, SkelSplat accumulates gradients over all available views, which empirically improves robustness to occlusion and cross-view inconsistencies.

Overall Pipeline

Given multi-view images and their 2D keypoint detections, SkelSplat initializes a joint-wise Gaussian skeleton. It iteratively refines the joint locations and covariances, minimizing the total loss until convergence.

Figure 3: Overview of the SkelSplat framework, showing initialization, differentiable rendering into heatmaps, and loss-based refinement using multi-view cues.

Experimental Results

Datasets and Metrics

SkelSplat is benchmarked on Human3.6M, CMU Panoptic Studio, Occlusion-Person, and Human3.6M-Occ datasets, using standard MPJPE (Mean Per Joint Position Error) in millimeters as the primary metric, with an emphasis on absolute positioning.

Performance and Generalization

Strong results are reported across all settings:

• On Human3.6M, SkelSplat achieves 20.3 mm MPJPE using ResNet-152 2D detections, outperforming all methods that do not use 3D supervision, and rivaling fully supervised baselines.
• When evaluated on Human3.6M with models trained on CMU Panoptic, learning-based approaches suffer a substantial increase in error (up to 47.8% higher), while SkelSplat—trained on neither—achieves the lowest cross-dataset error, evidencing strong out-of-distribution generalization.
• On CMU Panoptic Studio, SkelSplat’s 20.9 mm error outperforms cross-scene generalization baselines, retaining accuracy in the face of novel scene and camera configurations.

Occlusion Robustness

Experiments on Human3.6M-Occ and Occlusion-Person demonstrate superior performance under occlusions, without any scenario-specific fine-tuning or re-training. For instance, on Human3.6M-Occ-3 (three views occluded), SkelSplat achieves 27.0 mm error (ResNet-152 2D), outperforming algebraic triangulation and recent robust fusion methods. This robustness is attributed to the multi-view optimization and independently supervised Gaussian joints.

Figure 4: SkelSplat results (blue) on Human3.6M-Occ-3 vs ground-truth (green); failures typically occur under severe occlusion (right).

Ablations

Key ablation studies highlight:

• The one-hot joint encoding yields a 7.9–10% reduction in error versus RGB/single-channel encoding.
• Accumulating gradients across multiple views (vs single-view) reduces error by up to 2.7 mm.
• Performance is robust to realistic initialization noise (errors below 15% increase for up to 40mm noise).
• Using more views improves accuracy (from 20.9 mm with 4 views to 15.6 mm with 8 views).
• Enlarged covariance for frequently occluded joints improves robustness under occlusion but must be balanced to avoid loss of spatial precision.

Practical Implications and Limitations

SkelSplat’s key advantage is its independence from dataset- and camera-specific training. In practice, this enables scalable application across environments with arbitrary camera arrangements and subjects, requiring only 2D detectors and camera calibration for operation.

Resource requirements are moderate: inference takes approximately 3 seconds for a 4-view setup, as each joint requires separate rendering due to the one-hot channel assignment. While this is not real-time, the approach is suitable for batch and offline applications. No retraining or extensive ground-truth collection is needed for each new scene.

A potential extension is real-time or multi-person pose estimation. Further compact channel encodings or joint hierarchies could reduce computation. The method also provides per-joint covariance, which could serve as an uncertainty measure, although correlation with prediction quality requires further paper.

Theoretical Contributions

The main theoretical insight is the reinterpretation of 3D human pose estimation as a differentiable geometric optimization problem, rather than a dataset-specific learning task. The Gaussian skeleton provides a continuous, uncertainty-aware joint representation that aligns naturally with multi-view constraints. The novel one-hot rendering adapts dense 3D rendering techniques to the discrete, structured domain of human skeletons.

Conclusion

SkelSplat presents a flexible, differentiable framework for multi-view 3D human pose estimation based on Gaussian Splatting with a novel one-hot encoding. It achieves state-of-the-art results for methods without 3D supervision, generalizes robustly across camera setups and occlusion levels, and dispenses with retraining or domain adaptation. Current limitations include computational cost and the single-person focus; future work may address efficient rendering, multi-person tracking, and leveraging the uncertainty quantification implicit in per-joint covariances for downstream reasoning or active data selection.