ACC-NVS1: Human-Object Interaction Dataset

• ACC-NVS1 is a multi-view RGB-D dataset that captures dynamic human–object interactions under controlled lab conditions using 30 synchronized cameras.
• It provides high-resolution 4K sequences with detailed intrinsic and extrinsic calibrations, enabling accurate multi-modal fusion and occlusion-aware reconstruction.
• Benchmark results reveal significant overfitting challenges for current NVS methods, highlighting the dataset's role in advancing robust, occlusion-aware algorithms.

ACC-NVS1 is an acronym referring to two distinct large-scale multi-view datasets for @@@@1@@@@ (NVS), each targeting markedly different domains. The first, described in "A New People-Object Interaction Dataset and NVS Benchmarks" (Guo et al., 2024), focuses on dynamic human–object interactions captured under controlled laboratory conditions with RGB-D sensors. The second, presented in "Accenture-NVS1: A Novel View Synthesis Dataset" (Sugg et al., 24 Mar 2025), emphasizes urban-scale airborne and ground imagery for radiance field modeling. This article focuses on the former, which is the definitive collection for studying NVS in multi-person, multi-object human interaction scenes.

1. Scope and Motivation

ACC-NVS1 was developed to address key deficiencies in extant people–object interaction datasets, specifically the lack of synchronized multi-view RGB-D data, limited camera viewpoints, poor temporal calibration, and insufficient resolution. It provides an experimental foundation for new algorithms in dynamic scene synthesis, people–object interaction modeling, and occlusion-aware reconstruction. Its coverage includes both single- and multi-person activities involving a broad spectrum of everyday objects, capturing interaction complexity from simple gestures to severe occlusions (Guo et al., 2024).

2. Capture Setup and Data Composition

Data was recorded on a 2.5 m diameter circular stage using 30 Azure Kinect RGB-D sensors uniformly distributed in azimuth, guaranteeing an angular spacing of 12° between adjacent views and substantial overlapping fields of view for fusion. These Kinects are organized into five physical "modules," each a 3×2 grid, yielding systematic spatial coverage.

Each of the 38 dynamic sequences was acquired at 4K (3840×2160) resolution and 25 FPS, with accurate hardware synchronization using master controller–triggered TTL lines. The dataset includes:

• 1 empty-stage sequence for background modeling
• 1 calibration sequence with calibration chessboard patterns
• 23 single-person + object sequences with interactions such as "flipping through a book," "opening an umbrella," and "typing on a laptop"
• 11 two-person collaborative tasks such as "moving a table together" and "sweeping the floor"
• 2 three-person group interaction scenarios

Sequence durations range from 1 s to 19 s, facilitating research on both brief and sustained interaction episodes.

3. Modalities, Calibration, and Annotations

ACC-NVS1 provides a suite of spatial and structural modalities for each sequence:

• RGB and Depth: Both modalities at 4K, 25 FPS, with depth frames pre-aligned to RGB using Azure SDK routines.
• Intrinsic Calibration: Per-camera intrinsic matrices $K^i$ encoded in standard OpenCV form, including principal point $(c_x, c_y)$ and focal lengths $(f_x, f_y)$, with distortion models ($k_1, k_2, p_1, p_2$).
• Extrinsic Calibration: Rigid transformations $[R^i|T^i]$ in a common world frame established via pairwise module alignment and the Kabsch algorithm.
• Foreground Masks: Dense, per-frame high-resolution α-mattes computed using background reference and matting [Lin et al. 2021], stored as 16-bit alpha PNGs.
• Point Clouds: Per-view sparse clouds (from depth/rgb/calibration fusion), globally merged using ICP and step-discontinuity-constrained filtering [Zhou et al. 2021].
• Mesh Reconstructions: Surface meshes from temporally fused TSDF volumes, with Marching Cubes meshing [Zhou et al. 2022].
• SMPL Body Models: Gender-neutral SMPL parameters ($\beta\in\mathbb{R}^{10}$, $\theta\in\mathbb{R}^{72}$) with mesh vertices and semantic joints, sampled at ≈1 Hz and estimated via MMHuman3D.

Annotations are distributed in JSON (calibration) and NumPy NPZ (SMPL/mesh/pc). All video data is HEVC-encoded. Point clouds and meshes are provided in PLY and OBJ formats, respectively.

4. Organization, Access, and Workflow

The recommended directory structure is as follows:

Directory	Contents	Format
sequences/	Per-sequence (per-activity) RGB, depth, mask frames	MP4 (rgb), PNG (depth/mask)
calibration/	Intrinsic/extrinsic parameters for all cameras	intrinsics.json, extrinsics.json
priors/ (pc, mesh, smpl)	Sparse clouds, meshes, SMPL parameters (sampled)	PLY, OBJ, NPZ

The dataset requires significant compute resources (≥32 GB RAM, ≥12 GB VRAM GPU for NeRF/3DGS), and is accessed via standard Python (OpenCV, pyk4a, NumPy), Open3D, trimesh, and MMHuman3D software. Usage is governed by the GPL-3.0 license and the data are hosted at https://github.com/sjtu-medialab/People-Ojbect-Interaction-Dataset.

5. Train/Test Splits, Benchmarking, and Metrics

While no fixed splits are mandated, the canonical protocol benchmarks three sequences, choosing frames 5/10/10. For each, 25 cameras (except indices ≡ 0 mod 5) form the training set, with the remaining 5 as test views. Evaluation follows established NVS conventions:

• PSNR (Peak Signal-to-Noise Ratio, higher is better)
• SSIM (Structural Similarity, [0,1], higher is better)
• LPIPS (Learned Perceptual Image Patch Similarity, lower is better)

On the three scene benchmarks, results for representative SOTA models were reported as follows:

Method	Train PSNR	Train SSIM	Train LPIPS	Test PSNR	Test SSIM	Test LPIPS
TensoRF	33.47	0.97	0.08	9.85	0.82	0.40
K-Planes	28.50	0.95	0.11	11.54	0.79	0.43
3DGS	22.80	0.93	0.08	11.22	0.51	0.28

These figures evidence severe overfitting by current approaches, indicating the unique challenge inherent to dynamic, occluded, human–object scenes in ACC-NVS1 (Guo et al., 2024).

6. Best Practices, Limitations, and Common Pitfalls

• Synchronization is hardware-enforced; temporal offsets are negligible.
• Lighting is uniform (indoor fluorescent), but shadows and specularities are nontrivial; matting is robust but not infallible.
• Multi-person occlusion is pervasive (up to 3 people), exacerbating ambiguity for RGB-only models.
• Primary limitations: Maximum sequence duration is 19 s; depth artifacts at scene/object edges, and SMPL error rates increase under severe occlusion.
• Recommended pipeline:
  • Utilize depth and mask channels for segmentation and boundary localization.
  • Incorporate SMPL and mesh priors for geometric regularization.
  • Employ frame sub-sampling for static viewpoint NVS tasks.
  • Attack the reconstruction problem via multi-modal losses, e.g.,

  $$L_{total} = \lambda_{RGB}\|I_{pred} - I_{gt}\|^2 + \lambda_{depth}\|D_{pred} - D_{gt}\|^1 + \lambda_{mask} \operatorname{BCE}(\alpha_{pred}, \alpha_{gt}) + \lambda_{smpl} \|\theta_{pred} - \theta_{gt}\|^2$$

  as suggested in the dataset paper.

7. Research Directions and Impact

ACC-NVS1 represents the highest-resolution, most richly annotated dataset for dynamic, occluded human–object interaction NVS research. It facilitates rigorous evaluation of NVS, scene understanding, and interaction-aware geometry inference. A plausible implication is that the dataset will drive progress in overfitting-resistant, occlusion-aware approaches, inspire cross-modal loss functions, and offer new testbeds for multi-person SMPL-based pipelines. The unique multi-view, multi-modal design sets a new experimental standard for the field (Guo et al., 2024).