EVA-Gaussian: 3D Gaussian-based Real-time Human Novel View Synthesis under Diverse Camera Settings (2410.01425v1)

Abstract: The feed-forward based 3D Gaussian Splatting method has demonstrated exceptional capability in real-time human novel view synthesis. However, existing approaches are restricted to dense viewpoint settings, which limits their flexibility in free-viewpoint rendering across a wide range of camera view angle discrepancies. To address this limitation, we propose a real-time pipeline named EVA-Gaussian for 3D human novel view synthesis across diverse camera settings. Specifically, we first introduce an Efficient cross-View Attention (EVA) module to accurately estimate the position of each 3D Gaussian from the source images. Then, we integrate the source images with the estimated Gaussian position map to predict the attributes and feature embeddings of the 3D Gaussians. Moreover, we employ a recurrent feature refiner to correct artifacts caused by geometric errors in position estimation and enhance visual fidelity.To further improve synthesis quality, we incorporate a powerful anchor loss function for both 3D Gaussian attributes and human face landmarks. Experimental results on the THuman2.0 and THumansit datasets showcase the superiority of our EVA-Gaussian approach in rendering quality across diverse camera settings. Project page: https://zhenliuzju.github.io/huyingdong/EVA-Gaussian.

• The paper introduces EVA-Gaussian, a novel pipeline combining efficient cross-view attention and Gaussian attribute estimation for real-time 3D human reconstruction.
• It employs a recurrent U-Net with anchor loss regularization to correct artifacts, achieving superior PSNR, SSIM, and LPIPS scores on benchmark datasets.
• The approach proves effective under sparse and diverse camera setups, paving the way for advancements in AR/VR and real-time communication applications.

EVA-Gaussian: 3D Gaussian-based Real-time Human Novel View Synthesis under Diverse Camera Settings

The paper "EVA-Gaussian: 3D Gaussian-based Real-time Human Novel View Synthesis under Diverse Camera Settings" introduces a robust methodology to address the challenges in 3D human reconstruction and novel view synthesis. This research proposes a new pipeline named EVA-Gaussian, which demonstrates significant improvements in rendering quality and efficiency, particularly under sparse camera settings diversely positioned.

Summary

The field of 3D reconstruction and novel view synthesis has seen substantial advancements, yet existing methodologies often rely on dense viewpoint settings or precise templates. These constraints limit the flexibility needed for real-time applications such as AR/VR and holographic communication. The proposed EVA-Gaussian pipeline addresses these limitations by incorporating three key innovations:

1. Efficient cross-View Attention (EVA) Module: This module enhances the estimation of 3D Gaussian positions from source images. The network utilizes U-Net with a dedicated attention mechanism for accurate multi-view correspondence retrieval, even in sparse camera setups.
2. Gaussian Attribute Estimation: Once the positions are estimated, the remaining attributes—such as opacity, scales, quaternions, and additional features—are predicted using a shallow U-Net. This combination ensures comprehensive attribute estimation for each 3D Gaussian.
3. Recurrent Feature Refinement: To correct artifacts caused by initial position estimation errors, a recurrent U-Net refines the rendered images in a recurrent manner. This process, along with an anchor loss function, further improves the visual fidelity of the synthesized views.

Results and Performance

The paper presents experimental results on the THuman2.0 and THumanSit datasets, showcasing superior performance of the EVA-Gaussian method over existing approaches such as GPS-Gaussian and ENeRF. For instance, EVA-Gaussian achieved the highest PSNR and SSIM scores, and the lowest LPIPS scores, while maintaining competitive inference times. Some key numerical outcomes were:

• On THuman2.0 with an angle change of 45°, EVA-Gaussian achieved a PSNR of 31.11, outperforming GPS-Gaussian (30.30) and ENeRF (29.62).
• On THumansit with a similar setting, EVA-Gaussian reached a PSNR of 29.16 compared to GPS-Gaussian's 28.02 and ENeRF's 27.06.

These results illustrate EVA-Gaussian’s prowess in rendering high-fidelity images even with substantial viewpoint changes (e.g., 90°).

Methodological Contributions

The paper's primary contributions can be outlined as follows:

• Pipeline Innovation: EVA-Gaussian integrates multi-view 3D Gaussian position estimation with efficient attention mechanisms, followed by attribute and recurrent feature refinement, ensuring robust and efficient 3D human reconstruction.
• Cross-view Attention Mechanism: By embedding an efficient cross-view attention mechanism at various scales within the U-Net architecture, the proposed method effectively handles diverse viewing angles, significantly enhancing depth and Gaussian position estimation accuracy.
• Anchor Loss Regularization: Introducing an anchor loss that penalizes inconsistencies in Gaussian attributes and face landmarks ensures higher accuracy in critical regions like human faces, thereby enhancing overall image fidelity.

Practical and Theoretical Implications

Practically, EVA-Gaussian’s ability to reconstruct high-fidelity 3D human models in real-time can revolutionize applications in AR/VR, holographic communications, and real-time teaching. These applications demand swift and accurate reconstructions under variable camera settings, aligning perfectly with EVA-Gaussian’s capabilities.

Theoretically, this work advances the understanding of how real-time 3D reconstruction can be achieved using Gaussian representations and deep learning. The cross-view attention approach combined with recurrent refinement introduces a novel way to enhance multi-view consistency and depth accuracy, setting a new benchmark for future research.

Future Directions

Future research may focus on optimizing the computational efficiency of the attention module to handle even higher resolution images and more extensive input views. Additionally, integrating depth information or overlap area detection techniques could further reduce redundancy and improve overall system performance. Exploring these directions can help in refining and expanding the applicability of the EVA-Gaussian approach across various real-world scenarios.

In summary, EVA-Gaussian represents a significant step forward in the field of 3D Gaussian-based human reconstruction and novel view synthesis. Its methodological innovations and superior performance metrics underline its potential for practical applications and pave the way for further advancements in this research area.