Human Gaussian Splatting: Real-time Rendering of Animatable Avatars (2311.17113v2)

Abstract: This work addresses the problem of real-time rendering of photorealistic human body avatars learned from multi-view videos. While the classical approaches to model and render virtual humans generally use a textured mesh, recent research has developed neural body representations that achieve impressive visual quality. However, these models are difficult to render in real-time and their quality degrades when the character is animated with body poses different than the training observations. We propose an animatable human model based on 3D Gaussian Splatting, that has recently emerged as a very efficient alternative to neural radiance fields. The body is represented by a set of gaussian primitives in a canonical space which is deformed with a coarse to fine approach that combines forward skinning and local non-rigid refinement. We describe how to learn our Human Gaussian Splatting (HuGS) model in an end-to-end fashion from multi-view observations, and evaluate it against the state-of-the-art approaches for novel pose synthesis of clothed body. Our method achieves 1.5 dB PSNR improvement over the state-of-the-art on THuman4 dataset while being able to render in real-time (80 fps for 512x512 resolution).

• The paper introduces a novel 3D Gaussian Splatting method that represents human bodies using Gaussian primitives in a canonical space for high-fidelity rendering.
• It employs a two-step deformation technique combining forward skinning with local non-rigid refinement to accurately capture both broad and fine body motions.
• The approach delivers real-time performance at 80 FPS and a 1.5 dB PSNR improvement on the THuman4 dataset compared to traditional methods.

The paper "Human Gaussian Splatting: Real-time Rendering of Animatable Avatars" addresses the challenge of real-time rendering of photorealistic avatars, particularly focusing on achieving high visual quality for animatable human models. Traditional methods largely relied on textured meshes for modeling virtual humans, but these often fall short in rendering quality and real-time performance, especially when the avatars are animated with new poses.

Key contributions of this work include the use of a novel technique called 3D Gaussian Splatting, which has gained traction as an efficient alternative to neural radiance fields. The researchers formulated an animatable human model that uses Gaussian primitives to represent the human body in a canonical space. This model employs a two-pronged approach combining forward skinning and local non-rigid refinement to handle body deformations efficiently.

The methodology includes:

1. Representation: The human body is represented by a cloud of Gaussian splats, which are defined in a canonical pose. This allows the model to handle various body shapes more flexibly.
2. Deformation Technique: The transformation from the canonical space to the animated pose is achieved through a coarse-to-fine deformation strategy. It starts with forward skinning for broad deformations and follows up with local refinements to capture finer details.
3. Training and Rendering: The model is trained end-to-end using multi-view video data. This approach helps the model learn accurate geometric and photometric details required for high-fidelity rendering.

The authors report substantial improvements over existing methods, with a 1.5 dB increase in Peak Signal-to-Noise Ratio (PSNR) on the THuman4 dataset, a benchmark for evaluating synthetic novel pose synthesis. Notably, their approach supports real-time rendering capabilities, achieving approximately 80 frames per second at a resolution of 512x512, which marks a significant advancement in efficiency compared to prior neural representations.

Overall, the paper showcases a breakthrough in creating high-quality, animatable human models that can be rendered in real-time, advancing the possibilities for applications in virtual reality, gaming, and digital human creation.