HumanNeRF: Free-viewpoint Rendering of Moving People from Monocular Video (2201.04127v2)

Abstract: We introduce a free-viewpoint rendering method -- HumanNeRF -- that works on a given monocular video of a human performing complex body motions, e.g. a video from YouTube. Our method enables pausing the video at any frame and rendering the subject from arbitrary new camera viewpoints or even a full 360-degree camera path for that particular frame and body pose. This task is particularly challenging, as it requires synthesizing photorealistic details of the body, as seen from various camera angles that may not exist in the input video, as well as synthesizing fine details such as cloth folds and facial appearance. Our method optimizes for a volumetric representation of the person in a canonical T-pose, in concert with a motion field that maps the estimated canonical representation to every frame of the video via backward warps. The motion field is decomposed into skeletal rigid and non-rigid motions, produced by deep networks. We show significant performance improvements over prior work, and compelling examples of free-viewpoint renderings from monocular video of moving humans in challenging uncontrolled capture scenarios.

• The paper introduces HumanNeRF, a novel method that decomposes human motion into skeletal and non-rigid components for accurate 3D rendering.
• It employs a canonical T-pose representation and motion field mapping to convert monocular video into flexible, photorealistic views.
• The approach delivers significant improvements in PSNR, SSIM, and LPIPS metrics, outperforming previous state-of-the-art techniques.

HumanNeRF: Free-Viewpoint Rendering from Monocular Video

The paper "HumanNeRF: Free-viewpoint Rendering of Moving People from Monocular Video" investigates a novel method for creating free-viewpoint renderings of humans from single-camera video captures. This work addresses the longstanding challenge in computer vision and graphics of synthesizing photorealistic images from arbitrary viewpoints of a subject performing complex body movements. The proposed method, HumanNeRF, exhibits substantial improvements over previous efforts in this domain, leveraging neural radiance fields (NeRF) to achieve high-quality and realistic renderings of dynamic human figures, even when capturing videos in uncontrolled environments.

Technical Contributions

HumanNeRF introduces a sophisticated approach to map monocular video inputs into a flexible 3D representation, enabling the synthesis of views from any angle. The core of the method is the decomposition of human motion into two components: skeletal rigid motion and non-rigid motion. This decomposition allows for accurate modeling of both the coarse body posture and the subtle movements caused by clothing dynamics and facial expressions:

1. Canonical Volume Representation: The method formulates human appearance in a canonical T-pose using an MLP-based continuous volumetric representation. This acts as a reference morphology that simplifies subsequent motion parameterization.
2. Motion Field Mapping: A crucial innovation in this work is the mapping of video frames to the canonical pose via a motion field. This field decomposes motion into skeletal motion, captured through linear blend skinning, and non-rigid deformation, modeled using another MLP. This design choice effectively manages the dynamic aspects of human movements while maintaining a coherent geometric structure.
3. Pose Correction Mechanism: Recognizing the inaccuracies in common pose estimation algorithms, HumanNeRF incorporates an optimization-based refinement of poses. This refinement increases the fidelity of rendered sequences when compared to raw pose estimations drawn from off-the-shelf models.

HumanNeRF further employs a novel training strategy that delays the optimization of the non-rigid motion component. This strategy prevents overfitting and ensures that skeletal motion is properly learned before introducing more complex deformations, enhancing the generalization capabilities of the model.

Evaluation and Results

The paper reports significant quantitative improvements over prior state-of-the-art methods, particularly noted on the ZJU-MoCap dataset. HumanNeRF consistently yields better PSNR, SSIM, and perceptual loss scores (LPIPS) compared to Neural Body, a closely related method. These metrics underline HumanNeRF's capability to maintain high-fidelity reconstructions even when rendered from novel viewpoints not originally observed in input videos.

Notably, HumanNeRF demonstrates robust performance on challenging uncontrolled scenarios, such as YouTube videos, effectively overcoming issues with limited frame rates and suboptimal lighting conditions. This flexibility is a considerable advantage over previous methods, which often required multi-view systems and carefully controlled environments.

Implications and Future Work

The implications of HumanNeRF span both practical applications and theoretical research advancements. Practically, this technology could enable new functionalities in media production, virtual reality, and telepresence, where capturing dynamic, lifelike representations of people is essential. Theoretically, HumanNeRF enriches the understanding of combining neural representations with motion modeling and opens avenues for future exploration in more versatile and computationally efficient renderings.

Future work might focus on reducing the computational complexity of the proposed system, enhancing its scalability for real-time applications, and extending the underlying models to account for even more complex dynamic interactions, such as those influenced by environmental factors or interactions with other dynamic subjects.

In conclusion, HumanNeRF stands as a significant contribution to the advancement of free-viewpoint rendering, bridging the gap between monocular video inputs and high-quality 3D representations with remarkable precision and flexibility.