BecomingLit: Relightable Gaussian Avatars with Hybrid Neural Shading (2506.06271v1)

Abstract: We introduce BecomingLit, a novel method for reconstructing relightable, high-resolution head avatars that can be rendered from novel viewpoints at interactive rates. Therefore, we propose a new low-cost light stage capture setup, tailored specifically towards capturing faces. Using this setup, we collect a novel dataset consisting of diverse multi-view sequences of numerous subjects under varying illumination conditions and facial expressions. By leveraging our new dataset, we introduce a new relightable avatar representation based on 3D Gaussian primitives that we animate with a parametric head model and an expression-dependent dynamics module. We propose a new hybrid neural shading approach, combining a neural diffuse BRDF with an analytical specular term. Our method reconstructs disentangled materials from our dynamic light stage recordings and enables all-frequency relighting of our avatars with both point lights and environment maps. In addition, our avatars can easily be animated and controlled from monocular videos. We validate our approach in extensive experiments on our dataset, where we consistently outperform existing state-of-the-art methods in relighting and reenactment by a significant margin.

• The paper introduces a novel methodology that uses Gaussian primitives combined with hybrid neural shading to generate relightable, high-resolution avatars.
• It achieves superior performance in relighting and self-reenactment tasks, demonstrating significant improvements in PSNR, SSIM, and LPIPS metrics.
• The approach leverages an accessible low-cost light stage dataset, opening new avenues for research in photorealistic facial appearance reconstruction.

BecomingLit: Relightable Gaussian Avatars with Hybrid Neural Shading

The academic paper "BecomingLit: Relightable Gaussian Avatars with Hybrid Neural Shading" presents a significant advancement in the field of computer vision, particularly in the domain of photorealistic avatar creation. The authors introduce a novel methodology for reconstructing relightable, high-resolution head avatars that can be animated and rendered from novel viewpoints at interactive rates, addressing the need for realistic avatar creation critical for applications in virtual reality, cinematography, and other graphical interfaces.

The cornerstone of this methodology is the use of Gaussian primitives to model head geometries and a hybrid neural shading approach that combines neural rendering techniques with analytical models. This fusion enables highly detailed photorealistic avatars, whose materials can be disentangled and animated with striking realism. Furthermore, the avatars support all-frequency relighting, allowing them to be seamlessly integrated into different lighting environments, both with point lights and environment maps.

A major contribution of the paper is the introduction of a new dataset obtained using a low-cost light stage capture setup tailored specifically for capturing faces. This OLAT dataset comprises high-resolution, high-frame-rate, multi-view sequences of participants exhibiting various facial expressions under controlled lighting conditions. The dataset's significance lies in its potential to stimulate further research in avatar and facial appearance reconstruction, particularly given the scarcity of publicly accessible and free-to-use datasets for studying facial appearance reconstruction.

The paper thoroughly validates the proposed approach, demonstrating that BecomingLit outperforms existing state-of-the-art methods in both relighting and self-reenactment tasks by a significant margin. Notably, strong numerical results in PSNR, SSIM, and LPIPS metrics reflect the superior quality of the avatar generation compared to alternative methods.

From a technical standpoint, the authors engender a unique geometry and appearance model wherein 3D Gaussian primitives are animated with expression-dependent dynamics modules. Additionally, they harness a hybrid neural shading strategy to effectively capture complex facial reflection behaviors by learning a neural diffuse BRDF alongside an analytical specular term. This innovative shading model disentangles materials from dynamic light stage recordings, enhancing the realism of the rendered avatars.

The paper also addresses practical implications, suggesting that the economical setup required by BecomingLit makes the technology more accessible and potentially transformative for consumer-level virtual reality applications. It paves the way for more realistic avatars in everyday scenarios, like video calling and social media interactions, enhancing user experiences through authentic facial representations.

The research further hints at theoretical implications, proposing that the captured dataset could foster advanced studies in facial appearance modeling. Particularly, the data's granularity might help refine facial geometry and light reflection models, improving the synthesis of dynamic facial details and expressions in computational models. Future developments could explore the expansion of the dataset and its use in building comprehensive appearance priors for facial modeling.

In conclusion, while the paper presents impressive strides in avatar reconstruction technology, it acknowledges certain limitations such as reliance on comprehensive training expressions and potential sensitivity to tracking failures in FLAME geometry. Ethical considerations are duly noted, emphasizing the responsible distribution and use of the dataset to prevent misuse. Overall, the methodology opens promising avenues for research, particularly in enabling accessible, photorealistic avatars that could fundamentally reshape interactions in virtual environments.