---
title: 'ACC-NVS1: Human-Object Interaction Dataset'
url: https://www.emergentmind.com/topics/acc-nvs1-dataset
type: topic
---

# ACC-NVS1: Human-Object Interaction Dataset

ACC-NVS1 is an acronym referring to two distinct large-scale multi-view datasets for novel view synthesis (NVS), each targeting markedly different domains. The first, described in "A New People-Object Interaction Dataset and NVS Benchmarks" [2409.12980], focuses on dynamic human–object interactions captured under controlled laboratory conditions with RGB-D sensors. The second, presented in "Accenture-NVS1: A Novel View Synthesis Dataset" [2503.18711], emphasizes urban-scale airborne and ground imagery for radiance field modeling. This article focuses on the former, which is the definitive collection for studying NVS in multi-person, multi-object human interaction scenes.

## 1. Scope and Motivation

ACC-NVS1 was developed to address key deficiencies in extant people–object interaction datasets, specifically the lack of synchronized multi-view RGB-D data, limited camera viewpoints, poor temporal calibration, and insufficient resolution. It provides an experimental foundation for new algorithms in dynamic scene synthesis, people–object interaction modeling, and occlusion-aware reconstruction. Its coverage includes both single- and multi-person activities involving a broad spectrum of everyday objects, capturing interaction complexity from simple gestures to severe occlusions [2409.12980].

## 2. Capture Setup and Data Composition

Data was recorded on a 2.5 m diameter circular stage using 30 Azure Kinect RGB-D sensors uniformly distributed in azimuth, guaranteeing an angular spacing of 12° between adjacent views and substantial overlapping fields of view for fusion. These Kinects are organized into five physical "modules," each a 3×2 grid, yielding systematic spatial coverage.

Each of the 38 dynamic sequences was acquired at 4K (3840×2160) resolution and 25 FPS, with accurate hardware synchronization using master controller–triggered TTL lines. The dataset includes:

- 1 empty-stage sequence for background modeling
- 1 calibration sequence with calibration chessboard patterns
- 23 single-person + object sequences with interactions such as "flipping through a book," "opening an umbrella," and "typing on a laptop"
- 11 two-person collaborative tasks such as "moving a table together" and "sweeping the floor"
- 2 three-person group interaction scenarios

Sequence durations range from 1 s to 19 s, facilitating research on both brief and sustained interaction episodes.

## 3. Modalities, Calibration, and Annotations

ACC-NVS1 provides a suite of spatial and structural modalities for each sequence:

- **RGB and Depth:** Both modalities at 4K, 25 FPS, with depth frames pre-aligned to RGB using Azure SDK routines.
- **Intrinsic Calibration:** Per-camera intrinsic matrices $K^i$ encoded in standard OpenCV form, including principal point $(c_x, c_y)$ and focal lengths $(f_x, f_y)$, with distortion models ($k_1, k_2, p_1, p_2$).
- **Extrinsic Calibration:** Rigid transformations $[R^i|T^i]$ in a common world frame established via pairwise module alignment and the Kabsch algorithm.
- **Foreground Masks:** Dense, per-frame high-resolution α-mattes computed using background reference and matting [Lin et al. 2021], stored as 16-bit alpha PNGs.
- **Point Clouds:** Per-view sparse clouds (from depth/rgb/calibration fusion), globally merged using ICP and step-discontinuity-constrained filtering [Zhou et al. 2021].
- **Mesh Reconstructions:** Surface meshes from temporally fused TSDF volumes, with Marching Cubes meshing [Zhou et al. 2022].
- **SMPL Body Models:** Gender-neutral SMPL parameters ($\beta\in\mathbb{R}^{10}$, $\theta\in\mathbb{R}^{72}$) with mesh vertices and semantic joints, sampled at ≈1 Hz and estimated via MMHuman3D.

Annotations are distributed in JSON (calibration) and NumPy NPZ (SMPL/mesh/pc). All video data is HEVC-encoded. Point clouds and meshes are provided in PLY and OBJ formats, respectively.

## 4. Organization, Access, and Workflow

The recommended directory structure is as follows:

| Directory                | Contents                                              | Format                         |
|--------------------------|------------------------------------------------------|--------------------------------|
| sequences/               | Per-sequence (per-activity) RGB, depth, mask frames | MP4 (rgb), PNG (depth/mask)    |
| calibration/             | Intrinsic/extrinsic parameters for all cameras       | intrinsics.json, extrinsics.json|
| priors/ (pc, mesh, smpl) | Sparse clouds, meshes, SMPL parameters (sampled)     | PLY, OBJ, NPZ                  |

The dataset requires significant compute resources (≥32 GB RAM, ≥12 GB VRAM GPU for NeRF/3DGS), and is accessed via standard Python (OpenCV, pyk4a, NumPy), Open3D, trimesh, and MMHuman3D software. Usage is governed by the GPL-3.0 license and the data are hosted at https://github.com/sjtu-medialab/People-Ojbect-Interaction-Dataset.

## 5. Train/Test Splits, Benchmarking, and Metrics

While no fixed splits are mandated, the canonical protocol benchmarks three sequences, choosing frames 5/10/10. For each, 25 cameras (except indices ≡ 0 mod 5) form the training set, with the remaining 5 as test views. Evaluation follows established NVS conventions:

- **PSNR** (Peak Signal-to-Noise Ratio, higher is better)
- **SSIM** (Structural Similarity, [0,1], higher is better)
- **LPIPS** (Learned Perceptual Image Patch Similarity, lower is better)

On the three scene benchmarks, results for representative SOTA models were reported as follows:

| Method    | Train PSNR | Train SSIM | Train LPIPS | Test PSNR | Test SSIM | Test LPIPS |
|-----------|------------|------------|-------------|-----------|-----------|------------|
| TensoRF   | 33.47      | 0.97       | 0.08        | 9.85      | 0.82      | 0.40       |
| K-Planes  | 28.50      | 0.95       | 0.11        | 11.54     | 0.79      | 0.43       |
| 3DGS      | 22.80      | 0.93       | 0.08        | 11.22     | 0.51      | 0.28       |

These figures evidence severe overfitting by current approaches, indicating the unique challenge inherent to dynamic, occluded, human–object scenes in ACC-NVS1 [2409.12980].

## 6. Best Practices, Limitations, and Common Pitfalls

- **Synchronization** is hardware-enforced; temporal offsets are negligible.
- **Lighting** is uniform (indoor fluorescent), but shadows and specularities are nontrivial; matting is robust but not infallible.
- **Multi-person occlusion** is pervasive (up to 3 people), exacerbating ambiguity for RGB-only models.
- **Primary limitations**: Maximum sequence duration is 19 s; depth artifacts at scene/object edges, and SMPL error rates increase under severe occlusion.
- **Recommended pipeline**:
  - Utilize depth and mask channels for segmentation and boundary localization.
  - Incorporate SMPL and mesh priors for geometric regularization.
  - Employ frame sub-sampling for static viewpoint NVS tasks.
  - Attack the reconstruction problem via multi-modal losses, e.g.,
    $$
    L_{total} = \lambda_{RGB}\|I_{pred} - I_{gt}\|^2 + \lambda_{depth}\|D_{pred} - D_{gt}\|^1 + \lambda_{mask} \operatorname{BCE}(\alpha_{pred}, \alpha_{gt}) + \lambda_{smpl} \|\theta_{pred} - \theta_{gt}\|^2
    $$
    as suggested in the dataset paper.

## 7. Research Directions and Impact

ACC-NVS1 represents the highest-resolution, most richly annotated dataset for dynamic, occluded human–object interaction NVS research. It facilitates rigorous evaluation of NVS, scene understanding, and interaction-aware geometry inference. *A plausible implication is* that the dataset will drive progress in overfitting-resistant, occlusion-aware approaches, inspire cross-modal loss functions, and offer new testbeds for multi-person SMPL-based pipelines. The unique multi-view, multi-modal design sets a new experimental standard for the field [2409.12980].

Source: https://www.emergentmind.com/topics/acc-nvs1-dataset