Instant Facial Gaussians Translator for Relightable and Interactable Facial Rendering (2409.07441v2)

Abstract: We propose GauFace, a novel Gaussian Splatting representation, tailored for efficient animation and rendering of physically-based facial assets. Leveraging strong geometric priors and constrained optimization, GauFace ensures a neat and structured Gaussian representation, delivering high fidelity and real-time facial interaction of 30fps@1440p on a Snapdragon 8 Gen 2 mobile platform. Then, we introduce TransGS, a diffusion transformer that instantly translates physically-based facial assets into the corresponding GauFace representations. Specifically, we adopt a patch-based pipeline to handle the vast number of Gaussians effectively. We also introduce a novel pixel-aligned sampling scheme with UV positional encoding to ensure the throughput and rendering quality of GauFace assets generated by our TransGS. Once trained, TransGS can instantly translate facial assets with lighting conditions to GauFace representation, With the rich conditioning modalities, it also enables editing and animation capabilities reminiscent of traditional CG pipelines. We conduct extensive evaluations and user studies, compared to traditional offline and online renderers, as well as recent neural rendering methods, which demonstrate the superior performance of our approach for facial asset rendering. We also showcase diverse immersive applications of facial assets using our TransGS approach and GauFace representation, across various platforms like PCs, phones and even VR headsets.

• The paper introduces GauFace, converting PBR facial assets into Gaussian representations to enable real-time, high-fidelity facial rendering.
• It employs a diffusion transformer with Pixel Aligned Sampling to manage detailed avatar generation over UV space efficiently.
• Extensive evaluations demonstrate that TransGS achieves rendering quality on par with offline methods while supporting interactive applications.

Insightful Overview of Instant Facial Gaussians Translator for Relightable and Interactable Facial Rendering

The paper introduces a novel approach called "Instant Facial Gaussians Translator" (TransGS), designed for the efficient generation and rendering of facial avatars leveraging Gaussian splatting techniques. This method bridges the gap between traditional physically-based rendering (PBR) facial assets and real-time Gaussian splatting, enabling high-fidelity, real-time interactions across multiple platforms.

Summary of Methodologies

The core contribution of the paper is the introduction of a new representation termed "GauFace." This representation efficiently converts PBR facial assets into Gaussian assets. GauFace leverages established geometric priors and constrained optimization to maintain a structured Gaussian representation, potentially overcoming the limitations posed by implicit methods in terms of rendering fidelity and interaction. By anchoring Gaussian splatting with physically-based attributes, the paper demonstrates high-quality rendering comparable to offline methods, but with the potential application in real-time interactive environments.

TransGS is a generative model based on a diffusion transformer architecture that efficiently maps PBR features to Gaussian representations. The model employs a patch-based learning strategy, facilitating the handling of vast numbers of Gaussian points necessary for detailed avatar generation. A key innovation here is the use of Pixel Aligned Sampling (PAS), which maintains a consistent distribution of Gaussian points over the UV space, making the generated assets suitable for generative tasks without sacrificing performance.

Evaluation and Results

The paper substantiates its claims with rigorous evaluations. Extensive user studies were conducted, comparing the rendering quality of TransGS-generated assets against established renderers such as Blender Cycles, Blender EEVEE, and Unity3D’s built-in pipeline. The findings illustrate that the TransGS-generated assets achieve aesthetic qualities closely akin to Blender Cycles, with significant improvements over interactive renderers, thus validating the approach's visual fidelity.

Moreover, when benchmarked against recent neural rendering methods, such as Gaussian Avatars (GA) and INSTA, the TransGS assets demonstrate superior performance in terms of PSNR, SSIM, and LPIPS metrics. Notably, the approach provides highly efficient generation times (a few seconds) while delivering competitive rendering quality, a significant contrast to other methods that require substantial data preparation and optimization times.

Implications and Future Directions

The implications of this research are substantial both practically and theoretically. Practically, TransGS can drastically enhance the realism and interactive capabilities of digital avatars in applications ranging from virtual reality to gaming and beyond. It provides a scalable solution for platforms with varying computational capabilities, as demonstrated by the successfully implemented cross-platform renderer that performs real-time computations on devices, including mobile platforms.

Theoretically, the paper opens several avenues for future research. The explicit Gaussian representation facilitates straightforward integration with existing CG pipelines, a feature not yet fully exploited. Moreover, future work could explore the augmentation of TransGS to include hair and more complex facial features, potentially improving relighting capabilities in real-time and incorporating globally correlated shading across different facial components.

In conclusion, the Instant Facial Gaussians Translator represents a step forward in avatar rendering technology, offering a balanced convergence of performance, quality, and interaction. It mitigates the traditional trade-offs between online speed and rendering fidelity, suggesting a promising direction for future advancements in real-time computer-generated imagery.