URAvatar: Universal Relightable Gaussian Codec Avatars (2410.24223v1)

Abstract: We present a new approach to creating photorealistic and relightable head avatars from a phone scan with unknown illumination. The reconstructed avatars can be animated and relit in real time with the global illumination of diverse environments. Unlike existing approaches that estimate parametric reflectance parameters via inverse rendering, our approach directly models learnable radiance transfer that incorporates global light transport in an efficient manner for real-time rendering. However, learning such a complex light transport that can generalize across identities is non-trivial. A phone scan in a single environment lacks sufficient information to infer how the head would appear in general environments. To address this, we build a universal relightable avatar model represented by 3D Gaussians. We train on hundreds of high-quality multi-view human scans with controllable point lights. High-resolution geometric guidance further enhances the reconstruction accuracy and generalization. Once trained, we finetune the pretrained model on a phone scan using inverse rendering to obtain a personalized relightable avatar. Our experiments establish the efficacy of our design, outperforming existing approaches while retaining real-time rendering capability.

• The paper introduces a novel 3D Gaussian representation combined with end-to-end learnable radiance transfer to render highly detailed, photorealistic avatars.
• The paper leverages single-phone scans and multi-view data to achieve consistent relighting and accurate geometric tracking under diverse lighting conditions.
• The paper demonstrates significant improvements in rendering fidelity, outperforming prior methods as evidenced by metrics such as Mean Absolute Error and LPIPS.

Overview of "URAvatar: Universal Relightable Gaussian Codec Avatars"

The paper "URAvatar: Universal Relightable Gaussian Codec Avatars" addresses the creation of photorealistic, relightable head avatars using a single-phone scan, aimed at rendering realistic avatars that are consistent across different lighting conditions, identities, and expressions. The proposed approach contributes to the domain of 3D graphics and neural rendering by advancing the methodologies involved in avatar creation with complex lighting dynamics and reduced reliance on extensive capture systems.

The primary innovation of this research lies in its utilization of 3D Gaussians for geometric representation and learnable radiance transfer for appearance modeling. Unlike traditional rendering methods that decompose lighting parameters for diffuse and specular components via parametric reflectance, the paper presents an efficient model trained on hundreds of multi-view human scans. This model minimizes the gap between conventional studio-quality avatars and those generated from minimal input data, such as a cellphone scan, achieving notable improvements in real-time rendering fidelity.

Key Contributions and Methodology

1. 3D Gaussian Representation:
   • The paper demonstrates the use of 3D Gaussians to handle the intricate geometry of human heads efficiently. This method facilitates a high degree of detail without resorting to computationally expensive operations typical of traditional mesh or voxel-based models.
2. Learnable Radiance Transfer:
   • A critical aspect of the proposed framework is the radiance transfer function, which is directly learned in an end-to-end manner to account for global light transport. This makes the model adept at handling multiple light bounces and complex materials, such as skin and hair, which exhibit significant scattering and reflectance properties.
3. Universal Relightable Prior:
   • The research introduces a universal avatar model that generalizes across identities by capturing shared characteristics through multi-identity training. This model enhances personalization by combining large-scale data with personalized finetuning for new identities using inverse rendering techniques.
4. High-Quality Tracking and Albedo Estimation:
   • High-resolution geometric tracking and sophisticated albedo estimation support the model in maintaining visual accuracy and detail when repurposed for personalized avatars. The inclusion of such detailed preprocessing steps plays a significant role in the final relighting and rendering quality.

Experimental Evaluations and Results

The experimental setup includes meticulously captured studio and phone scan datasets, enabling quantitative and qualitative evaluation of relighting accuracy and rendering performance. URAvatar significantly outperforms prior approaches, such as the FLARE method, in metrics like Mean Absolute Error and LPIPS, underscoring the efficacy of its innovative learning components.

Additionally, ablation studies highlight the improvements brought by specific architectural choices, such as the unified specular visibility decoder for authentic eye reflections and the use of identity-conditioned biases for detailed expression modeling.

Implications and Future Work

The practical implications of this research pave the way for more accessible virtual communication technologies, where users can effortlessly create realistic avatars from readily available hardware like smartphones. The theoretical advancements also suggest potential improvements in the understanding of light physics and appearance modeling in neural avatars, posing exciting possibilities for future extensions in real-time 3D graphics.

Potential future directions could explore reducing the computational burden of personalization, further generalizing the lighting model to include dynamic environmental changes, or extending avatar dynamics to cover full-body renditions with similar relightable characteristics.

In conclusion, the paper presents meaningful advancements in photorealistic avatar creation, offering a promising vision for integrating realistic digital selves into immersive communication platforms.