Relightable Gaussian Codec Avatars (2312.03704v2)

Abstract: The fidelity of relighting is bounded by both geometry and appearance representations. For geometry, both mesh and volumetric approaches have difficulty modeling intricate structures like 3D hair geometry. For appearance, existing relighting models are limited in fidelity and often too slow to render in real-time with high-resolution continuous environments. In this work, we present Relightable Gaussian Codec Avatars, a method to build high-fidelity relightable head avatars that can be animated to generate novel expressions. Our geometry model based on 3D Gaussians can capture 3D-consistent sub-millimeter details such as hair strands and pores on dynamic face sequences. To support diverse materials of human heads such as the eyes, skin, and hair in a unified manner, we present a novel relightable appearance model based on learnable radiance transfer. Together with global illumination-aware spherical harmonics for the diffuse components, we achieve real-time relighting with all-frequency reflections using spherical Gaussians. This appearance model can be efficiently relit under both point light and continuous illumination. We further improve the fidelity of eye reflections and enable explicit gaze control by introducing relightable explicit eye models. Our method outperforms existing approaches without compromising real-time performance. We also demonstrate real-time relighting of avatars on a tethered consumer VR headset, showcasing the efficiency and fidelity of our avatars.

• The paper presents a method that uses a 3D Gaussian geometric representation for capturing fine details like hair and skin, enabling real-time rendering with dynamic expressions.
• It introduces a learned radiance transfer appearance model that integrates diffuse spherical harmonics and specular Gaussian components for high-fidelity rendering under varied lighting conditions.
• The research also proposes a relightable explicit eye model to enhance photorealism, outperforming existing methods in both PSNR and SSIM metrics for complex avatar animations.

Relightable Gaussian Codec Avatars: A Technical Overview

The paper presents a novel approach titled "Relightable Gaussian Codec Avatars." This research introduces a method for constructing high-fidelity, animatable, and relightable head avatars using a combination of 3D Gaussian geometric representation and a sophisticated relightable appearance model. The primary goals of this work are to overcome the challenges of real-time relightable avatars and to enhance the rendering performance without sacrificing visual fidelity, particularly when modeling complex human head features such as hair, eyes, and skin.

Key Contributions

1. 3D Gaussian Geometric Representation: The authors utilize a 3D Gaussian representation for geometry, which allows the modeling of fine details such as hair strands and skin pores that are difficult to capture with traditional mesh or volumetric approaches. This representation facilitates real-time rendering using a technique known as splatting. The Gaussians are parameterized based on a shared UV texture map of a template mesh, enabling efficient rendering of dynamic expressions in real time.
2. Learned Radiance Transfer Appearance Model: The appearance model is based on learnable radiance transfer, combining diffuse spherical harmonics and specular spherical Gaussians. This design supports high-fidelity rendering with spatially all-frequency reflections, ensuring detail preservation under diverse lighting conditions. The model integrates efficiently with both point lights and continuous illumination, making it suitable for real-time applications.
3. Relightable Explicit Eye Model: To enhance gaze control and eye reflection fidelity, the paper introduces a relightable explicit eye model. This supports explicit control of the eyeballs separate from facial movements, crucial for photorealistic avatar representation under varying environmental lights.

Numerical Evaluation

The paper showcases the efficacy of its approach through quantitative and qualitative experiments. The proposed model consistently outperforms existing methods in measures such as PSNR and SSIM, both for different geometric and appearance model combinations. The combination of 3D Gaussians with the proposed relightable appearance model excels in rendering photorealistic avatars under novel illuminations, including point light sources and high-resolution environment maps, demonstrating superior performance in maintaining temporal coherence and capturing fine details like specular reflections on eyes and hair strands.

Practical and Theoretical Implications

The practical implications of this research are far-reaching, especially for applications in virtual reality (VR), gaming, and telecommunication, where real-time and photorealism are paramount. The ability to drive avatars with accurate reflections and lighting under various environmental conditions paves the way for more immersive and realistic user experiences.

On a theoretical level, this work advances the understanding of radiance transfer in dynamic avatars and geometric representations, challenging current limitations in avatar modeling concerning light transport and material representation. This could potentially stimulate further research into integrating more complex material models and expanding applicability to broader in-the-wild captures.

Future Developments and Considerations

While the paper presents a significant advancement, future developments could address the scalability of the method, particularly for larger scenes or multiple avatars rendered simultaneously. Furthermore, the dependency on precise initial tracking and specific capture setups may limit immediate applicability in more uncontrolled environments. Future research could focus on reducing pre-processing needs and adapting models for more generalized input data.

In conclusion, "Relightable Gaussian Codec Avatars" presents a technically robust framework that significantly enhances avatar realism and interactivity in real-time rendering contexts, laying the groundwork for more sophisticated avatar applications in the foreseeable future.