Neural Human Performer: Learning Generalizable Radiance Fields for Human Performance Rendering (2109.07448v1)

Abstract: In this paper, we aim at synthesizing a free-viewpoint video of an arbitrary human performance using sparse multi-view cameras. Recently, several works have addressed this problem by learning person-specific neural radiance fields (NeRF) to capture the appearance of a particular human. In parallel, some work proposed to use pixel-aligned features to generalize radiance fields to arbitrary new scenes and objects. Adopting such generalization approaches to humans, however, is highly challenging due to the heavy occlusions and dynamic articulations of body parts. To tackle this, we propose Neural Human Performer, a novel approach that learns generalizable neural radiance fields based on a parametric human body model for robust performance capture. Specifically, we first introduce a temporal transformer that aggregates tracked visual features based on the skeletal body motion over time. Moreover, a multi-view transformer is proposed to perform cross-attention between the temporally-fused features and the pixel-aligned features at each time step to integrate observations on the fly from multiple views. Experiments on the ZJU-MoCap and AIST datasets show that our method significantly outperforms recent generalizable NeRF methods on unseen identities and poses. The video results and code are available at https://youngjoongunc.github.io/nhp.

• The paper introduces generalizable neural radiance fields that integrate parametric human models to render human performance from limited multi-view inputs.
• It employs advanced temporal and multi-view transformers to fuse skeletal motion and pixel-aligned features for enhanced rendering quality.
• Experimental results on ZJU-MoCap and AIST datasets show significant improvements in PSNR and SSIM over existing methods.

Overview of "Neural Human Performer: Learning Generalizable Radiance Fields for Human Performance Rendering"

The paper "Neural Human Performer" addresses the complex task of rendering free-viewpoint video of arbitrary human performances with sparse multi-view camera inputs. This task holds significant relevance in fields such as telepresence, mixed reality, and gaming. Traditional methods for this task often rely on expensive and dense camera setups or accurate depth sensors, limiting their scalability and applicability. The paper proposes a novel approach that aims to overcome these limitations by efficiently synthesizing high-fidelity video from limited data, utilizing a novel combination of neural radiance fields (NeRF) and parametric human body models (e.g., SMPL).

Key Contributions

The paper presents several key contributions to the field of human performance rendering:

1. Generalizable Neural Radiance Fields: The authors introduce the concept of learning generalizable neural radiance fields that encode human performance. Unlike person-specific NeRF solutions, this approach aims to generalize across different human identities and poses using a robust parametric human model.
2. Temporal and Multi-View Transformations: Central to the proposed solution are advanced transformer architectures: a temporal transformer and a multi-view transformer. The temporal transformer is designed to integrate visual features derived from skeletal motion over time, while the multi-view transformer performs cross-attention between temporally-fused and pixel-aligned features. This enables adaptive aggregation of observations for improved rendering quality.
3. Experimental Validation: The method undergoes rigorous evaluation using the ZJU-MoCap and AIST datasets, demonstrating superior performance over recent generalizable NeRF methods such as Pixel-NeRF and PVA. Notably, the proposed method even outperforms person-specific approaches when tested on novel poses, highlighting its robust generalization capabilities.

Numerical Results and Implications

• The proposed method achieves a significant improvement in performance, reaching PSNR and SSIM scores that surpass those of competing methods. For example, it shows improvements of over +3 PSNR against Neural Body in certain settings, underscoring the viability of the architecture in handling unseen identities and poses.
• It also excels in 3D reconstruction tasks, providing high-quality, view-consistent outputs.

Theoretical and Practical Implications

Theoretically, the paper advances the understanding of how to integrate neural representations with sophisticated human body models to address the inherent challenges of occlusion and dynamic articulation in human performance. Practically, this work paves the way for scalable and cost-effective applications in interactive 3D environments, which require high-fidelity human renderings under varied view conditions.

Future Prospects

Future research could explore several directions inspired by this paper:

• Refinement and Optimization: Further optimization of the transformers and incorporation of more advanced body models could enhance precision and run-time efficiency.
• Real-World Application and Testing: Implementing the method in real-world scenarios with dynamic and uncontrolled environments would test its robustness and potential adaptations required for commercial applications.
• Cross-Domain Applications: The integration of generalizable radiance fields could extend beyond human performance to other articulated figures in diverse domains from robotics to biomechanics.

In conclusion, the "Neural Human Performer" presents a sophisticated approach towards rendering human performances under sparse-input constraints, offering meaningful contributions both in advancing theoretical frameworks and addressing practical challenges in modern graphics and computer vision applications.