NeRSemble: Multi-view Radiance Field Reconstruction of Human Heads (2305.03027v1)

Abstract: We focus on reconstructing high-fidelity radiance fields of human heads, capturing their animations over time, and synthesizing re-renderings from novel viewpoints at arbitrary time steps. To this end, we propose a new multi-view capture setup composed of 16 calibrated machine vision cameras that record time-synchronized images at 7.1 MP resolution and 73 frames per second. With our setup, we collect a new dataset of over 4700 high-resolution, high-framerate sequences of more than 220 human heads, from which we introduce a new human head reconstruction benchmark. The recorded sequences cover a wide range of facial dynamics, including head motions, natural expressions, emotions, and spoken language. In order to reconstruct high-fidelity human heads, we propose Dynamic Neural Radiance Fields using Hash Ensembles (NeRSemble). We represent scene dynamics by combining a deformation field and an ensemble of 3D multi-resolution hash encodings. The deformation field allows for precise modeling of simple scene movements, while the ensemble of hash encodings helps to represent complex dynamics. As a result, we obtain radiance field representations of human heads that capture motion over time and facilitate re-rendering of arbitrary novel viewpoints. In a series of experiments, we explore the design choices of our method and demonstrate that our approach outperforms state-of-the-art dynamic radiance field approaches by a significant margin.

• The paper introduces a novel multi-view neural radiance field method that integrates deformation fields and hash grid ensembles to capture complex facial dynamics.
• The methodology employs a warm-up phase and depth supervision to enhance spatial alignment and improve reconstruction fidelity using high-resolution data.
• Experimental results demonstrate superior performance in PSNR, SSIM, and LPIPS metrics compared to state-of-the-art methods, setting new benchmarks in human head rendering.

An Examination of NeRSemble: Multi-view Radiance Field Reconstruction of Human Heads

Introduction

The paper "NeRSemble: Multi-view Radiance Field Reconstruction of Human Heads" presents an innovative approach to photo-realistic rendering of dynamic human heads using multi-view video data. NeRSemble introduces a method combining deformation fields and hash grid ensembles to effectively capture complex facial dynamics and enable novel view synthesis (NVS). This document provides a detailed analysis of the methodology, dataset, and results presented in the paper.

Methodology

NeRSemble leverages multi-view video data captured through a sophisticated setup involving 16 synchronized cameras with high resolution and frame rates. This setup enables the recording of intricate facial motions, expressions, and speech dynamics, contributing significantly to the robustness of the dataset.

The core of NeRSemble's approach is its Dynamic Neural Radiance Fields using Hash Ensembles. This technique models scene dynamics by combining a deformation field with an ensemble of 3D hash encodings. The deformation field accounts for simplistic movement, providing spatial alignment across frames, whereas the hash grid ensemble enables the representation of highly detailed dynamics and non-rigid deformations.

A crucial part of the method is the incorporation of a warm-up phase in the training process. This phase focuses on optimizing the deformation field isolated from other components, ensuring meaningful learning of spatial correspondences. Additionally, depth supervision through traditional methods such as COLMAP is utilized to provide geometry constraints, enhancing the fidelity of the reconstructions.

Dataset

A key contribution of the paper is the release of a novel multi-view video dataset encompassing 4734 sequences from 222 subjects. The captured data spans various facial expressions, emotions, and challenging head movements, recorded at a resolution of 3208 x 2200 and 73 fps. This dataset exceeds the capabilities of existing databases in terms of resolution and temporal granularity, setting a new standard for multi-view video data related to NVS tasks.

Results and Comparisons

NeRSemble demonstrates superior performance compared to other state-of-the-art dynamic radiance field methods, such as Nerfies, HyperNeRF, and DyNeRF, especially in terms of high-frequency detail accuracy and temporal consistency. The novel hash ensemble approach employed by NeRSemble provides significant improvements across challenging expressions and motion scenarios.

Quantitatively, NeRSemble excels in terms of PSNR, SSIM, and LPIPS metrics, indicating superior reconstruction quality and temporal coherence. Additionally, experiments involving face-specific methods—Neural Head Avatars (NHA) and NeRFace—further underline NeRSemble’s capabilities in achieving detailed and realistic renderings without relying on predefined geometric models.

Implications and Future Work

The proposed NeRSemble framework and dataset offer substantial contributions to the fields of graphics and AI, particularly in digital avatar construction and VR applications. The insights gained from NeRSemble's dynamic scene modeling could inform future research directions in improving the efficiency and generalization capabilities of neural radiance fields in dynamic settings.

Future work may explore integrating learned generative priors for enhanced monocular view synthesis and exploring applications beyond human heads, such as complex scene reconstructions. With the public availability of the dataset and accompanying benchmark, NeRSemble sets a foundation for advancing research in photo-realistic rendering and multi-view video synthesis, fostering developments across AI-powered digital human technologies.