NeRSemble Dataset: Dynamic NeRF Benchmark

• NeRSemble Dataset is a comprehensive multi-view benchmark designed for high-fidelity radiance field reconstruction of human heads with dynamic facial expressions and fine geometric details.
• It employs a custom rig of 16 synchronized industrial cameras capturing over 31 million high-resolution frames and utilizes rigorous calibration methods for sub-millimeter accuracy.
• The dataset supports advanced research in dynamic neural radiance field modeling with extensive subject diversity and evaluation metrics including PSNR, SSIM, LPIPS, and JOD.

The NeRSemble Dataset is a large-scale benchmark designed for high-fidelity multi-view radiance field reconstruction of human heads, with an emphasis on capturing dynamic facial movements, subtle expressions, and fine-grained geometry under controlled conditions. Developed to drive progress in dynamic neural radiance field (NeRF) modeling, NeRSemble combines extensive subject diversity, tightly calibrated hardware, and comprehensive auxiliary data to establish a new standard for novel-view synthesis and head animation research (Kirschstein et al., 2023).

1. Dataset Composition and Demographics

NeRSemble comprises 4,734 multi-view video sequences of 222 unique subjects, totaling 31.7 million frames over approximately 7.5 hours of recorded footage. Each subject is recorded in 25 short sessions, aggregating to roughly three minutes per individual. Motion protocols are systematically designed to span a spectrum of facial muscle activations and articulations, including:

• Nine targeted expression sequences (e.g., eyebrow raises, cheek puffs) to engage distinct facial muscle groups
• A rapid-motion hair sequence
• Four controlled emotion sequences (e.g., anger, joy)
• Ten phonetically rich spoken sentences with synchronized high-quality audio
• One free-expression session allowing unrestricted facial and head movement

Subject demographics exhibit balance: the dataset consists of 157 males and 65 females, with age distribution ranging from late teens to senior adults. Ethnicity, self-reported by participants, encompasses Caucasian, East Asian, South Asian, African, and Hispanic backgrounds. This extensive coverage supports research into robust radiance field approaches that generalize across diverse physiognomies, skin tones, and dynamic conditions (Kirschstein et al., 2023).

2. Multi-View Capture Hardware and Calibration

Data acquisition leverages a bespoke rig consisting of 16 industrial machine-vision cameras, each providing synchronized global shutter images at 3,208×2,200 px (7.1 MP) and 73 frames per second with 3 ms exposure. The camera arrangement forms a semicircular arc spanning 93° horizontally and 32° vertically, offering dense multi-view coverage. Calibration achieves sub-millimeter pose accuracy through:

• Intrinsic and extrinsic parameter estimation using a fine checkerboard pattern
• Global bundle adjustment across views

Synchrony is guaranteed with all cameras connected to a Precision Time Protocol (PTP) clock, achieving sub-microsecond drift. Illumination is controlled by eight high-power, diffused LED panels. Each capture session employs f/8 aperture to maximize depth-of-field and mitigate motion blur; opal diffusion suppresses specular skin highlights.

Color calibration utilizes a reference checker for every sequence, solving per-camera white-balance and gamma correction. This ensures colorimetrically consistent imaging across all viewpoints—a critical requirement for color-accurate radiance field training (Kirschstein et al., 2023).

3. Data Modalities, Preprocessing, and Auxiliary Files

The dataset delivers lossless raw RGB video at native sensor resolution in a linear RGB space, supplemented by a suite of auxiliary data and metadata necessary for geometric and photometric consistency:

• Camera parameters: Intrinsics ($K$) and extrinsics ($[R|t]$) for all 16 views, provided in plain-text per session, as determined by global bundle adjustment.
• Foreground/background segmentation: High-precision alpha mattes for every frame, computed by BackgroundMatting v2 and stored as 32-bit float TIFFs. To enable robust matting, a static white-wall background is recorded before each session.
• Depth maps: Dense, per-view depth inferred via the COLMAP multi-view stereo pipeline, with reliability filtering (invalidation for pixels visible from fewer than three views).
• Color transforms: Each camera’s gamma ($\gamma$) and white-balance scaling from sRGB to linear RGB, computed from the color-checker, are released for downstream correction.

All files are organized following a consistent naming convention and directory structure, enabling systematic indexing of subject, sequence, and camera triplets.

4. Benchmark Splits, Metrics, and Baselines

NeRSemble defines deterministic splits for training, validation, and testing to support rigorous benchmarking:

• Of the 16 camera views, 12 are allocated to training and four spatially distributed views—including extreme angles—are reserved for evaluation.
• The validation set consists of 10 sequences (300–500 frames each), with metrics averaged across 15 uniformly sampled frames per sequence.
• Evaluation metrics comprise:
  • Peak Signal-to-Noise Ratio (PSNR)
  • Structural Similarity Index (SSIM)
  • Learned Perceptual Image Patch Similarity (LPIPS)
  • Just-Objectionable-Difference (JOD), a video quality metric quantifying temporal coherence

Comparative baselines include static and dynamic NeRFs (Poisson Surface Reconstruction on COLMAP points; Instant NGP NeRF per frame; Nerfies; HyperNeRF; DyNeRF), as well as methods leveraging face-specific 3DMM priors (Neural Head Avatars, NeRFace). Reported experiments demonstrate that the NeRSemble Dynamic Neural Radiance Field approach surpasses these baselines in both spatial and temporal dimensions (Kirschstein et al., 2023).

Benchmark Splits and Metric Summary

Split	Cameras Used	Frames per Seq.	Metrics (computed on)
Training	12 (semicircular)	n/a	n/a
Validation	4 (held-out)	15 × 10 seqs	PSNR, SSIM, LPIPS, JOD
Testing	4 (held-out)	specified in release	PSNR, SSIM, LPIPS, JOD

5. Camera Model and Calibration Equations

The cameras are modeled as pinhole devices with all radial-tangential distortion removed. The mathematical projection model is as follows:

• A 3D point $X$ in homogeneous coordinates projects onto image plane coordinate $x$ via $x = K [R|t] X$, with $K \in \mathbb{R}^{3 \times 3}$ the intrinsic matrix and $[R|t] \in SE(3)$ the extrinsic transformation from world to camera coordinates.
• For photometric consistency, each image is corrected with per-camera white-balance and gamma ($\gamma$) to linearize rgb values:

$$I_\mathrm{lin}(x) = s \circ (I_\mathrm{raw}(x))^\gamma$$

where $[R|t]$0 denotes the scale factor derived from the color checker.

Published calibration files, color transforms, and lookup tables are included to permit faithful reproduction of capture conditions—a necessity for high-precision radiance field training and geometric inference (Kirschstein et al., 2023).

6. Access, Licensing, and Ethical Compliance

All 4,734 video sequences, with their auxiliary data (calibration, depth maps, masks), are publicly released for academic research via the project website (https://tobias-kirschstein.github.io/nersemble). Distribution is subject to a Creative Commons Attribution-NonCommercial-ShareAlike license, permitting unrestricted non-commercial use and requiring derivative datasets to be released under equivalent terms. Users must accept a data-use agreement incorporating the following features:

• Full GDPR compliance
• Participants’ irrevocable right to withdraw consent and have their data deleted
• Mandatory share-alike for derivative works

Baseline code for dataset loading, training, and evaluation is provided under the MIT license to facilitate reproducibility. These arrangements position NeRSemble as a resource for both benchmarking and methodological innovation in dynamic multi-view reconstruction and neural radiance field research (Kirschstein et al., 2023).