ArtHOI-RGBD: Controlled RGB-D Benchmark

• The paper presents ArtHOI-RGBD as a controlled RGB-D benchmark offering metric depth, detailed 3D motion annotations, and hand-object contact labels for 4D reconstruction evaluation.
• Methodologically, it leverages synchronized RGB-D captures and full object scans to provide precise geometric guidance and part-wise motion data of articulated objects.
• Experimental results demonstrate improved reconstruction metrics and alignment scores, validating the benchmark’s role in advancing evaluation of hand-object interactions.

ArtHOI-RGBD is one of the two datasets introduced with “ArtHOI: Taming Foundation Models for Monocular 4D Reconstruction of Hand-Articulated-Object Interactions” (Wang et al., 26 Mar 2026). It is a small, carefully captured benchmark of hand–articulated-object interaction sequences recorded with an RGB-D sensor, described as “ArtHOI‑RGBD, comprising RGBD videos captured with a RealSense camera…”. Its role in the work is threefold: to provide metric depth supervision and geometric guidance for evaluating 4D articulated object reconstruction and hand–object interaction alignment, to supply ground-truth annotations of part-wise articulated motion, and to offer ground-truth contact labels. In the context of monocular 4D reconstruction of hand–articulated-object interactions, it fills a gap because there was previously no small, controlled RGB-D benchmark that simultaneously provides accurate depth, annotated articulated motions, and HOI contact labels on real interaction videos (Wang et al., 26 Mar 2026).

1. Benchmark definition and scope

ArtHOI-RGBD is a controlled evaluation benchmark rather than a large-scale training corpus. The dataset consists of five demonstration sequences of common articulated objects, and each sequence corresponds to a distinct real physical object. The paper does not define an explicit train/val/test split; instead, all five sequences are treated as evaluation cases for 4D articulated object reconstruction and HOI alignment.

Its design emphasis is metric reliability rather than environmental diversity. The scenes are indoor manipulation sequences in which a human interacts with common tabletop articulated objects. The paper does not describe major variation in backgrounds or lighting. This emphasis is deliberate: accurate geometry, articulated motion annotation, and contact labeling are treated as the central requirements for quantitative evaluation.

Within the broader ArtHOI study, ArtHOI-RGBD is paired with RSRD and ArtHOI-Wild. ArtHOI-RGBD provides the controlled metric setting, whereas ArtHOI-Wild provides RGB-only in-the-wild clips used to assess robustness under uncontrolled capture (Wang et al., 26 Mar 2026).

2. Capture protocol, sensing, and data modalities

ArtHOI-RGBD is captured using an Intel RealSense stereo RGB-D camera at $1280 \times 720$ resolution and 30 FPS (Wang et al., 26 Mar 2026). The recordings are monocular videos from a single viewpoint, but each frame contains both an RGB image and a synchronized depth map. Unlike ArtHOI-Wild, which relies on monocular depth estimation, ArtHOI-RGBD provides true metric depth from the sensor.

Because RealSense RGB-D provides calibrated intrinsics and extrinsics, the authors treat depth as metric and do not need to estimate metric scale and camera parameters for this dataset. Camera intrinsics $\mathbf{K}$ are not enumerated as a standalone released modality in the description, but they are required and used in all depth-to-3D operations, and the paper treats them as known for ArtHOI-RGBD.

Beyond RGB and depth, the dataset includes additional geometric resources. For each object, the authors additionally capture a surrounding scan to reconstruct a full ground-truth mesh. These full meshes are used for evaluation metrics such as Chamfer distance and MSSD. ArtHOI-RGBD also includes part-wise 3D motion annotations and binary hand–object contact annotations.

The dataset is therefore multimodal in a precise reconstruction sense: synchronized RGB, metric depth, known camera geometry, full object meshes from surrounding scans, part-wise articulated motion annotations, and contact labels. This makes it suitable for evaluating both object-centric 4D reconstruction and interaction-centric alignment quality (Wang et al., 26 Mar 2026).

3. Object coverage, interaction content, and motion representation

ArtHOI-RGBD contains five object instances: Headphone, Scissor, Candy Box, CD Drive, and Stapler. These span several articulation types.

Object	Articulation type
Headphone	Multi-part wearable/device
Scissor	Two-link tool
Candy Box	Hinged container
CD Drive	Hinged container
Stapler	Hinged container

The recorded interactions are natural manipulation videos of a human operating these objects. The paper explicitly notes “challenging hand-part occlusions (e.g., Stapler) and part-part occlusions (e.g., CD Drive)”, indicating that the benchmark is not limited to clean, fully visible articulation. A plausible implication is that the dataset stresses precisely those failure modes—self-occlusion, mutual occlusion, and partial observability—that make monocular articulated HOI reconstruction ill-posed.

Within the ArtHOI method, articulated objects are represented as part-wise rigid bodies with per-part transformations in $SE(3)$. If the object mesh is partitioned into parts $p_k$, then motion over time is encoded as

$$\{\mathbf{T}_i^{p_k}\}_{i=1}^N, \qquad \mathbf{T}_i^{p_k} \in SE(3),$$

where $i$ indexes frames and $k$ indexes object parts. No explicit kinematic tree or joint-angle parameterization is described for ArtHOI-RGBD; articulation is represented implicitly through these per-part trajectories. The canonical object mesh used by the reconstruction method is obtained from an image-to-3D foundation model and then metricized and aligned, whereas the surrounding-scan mesh serves as ground truth for evaluation (Wang et al., 26 Mar 2026).

4. Annotation pipeline and ground-truth resources

The central annotation in ArtHOI-RGBD is part-wise 3D motion. Because ground-truth depth from the RealSense provides only partial surface observations, the authors develop a 3D annotation tool built on Viser to label part-wise object motions across frames for all five videos, using depth maps as geometric guidance (Wang et al., 26 Mar 2026). For each part $p_k$ and each frame $i$, annotators specify a rigid transform $\mathbf{T}_i^{p_k} \in SE(3)$ in world coordinates.

Depth enters both annotation and auxiliary processing through metric back-projection. The paper sketches the standard operation

$\mathbf{K}$0

where $\mathbf{K}$1 is the homogeneous pixel coordinate and $\mathbf{K}$2 is depth. The resulting point clouds are used to guide annotation and, more generally, to support coarse scale estimation by comparing the bounding-box extents of the back-projected object points with those of the normalized canonical mesh.

The surrounding scans provide complete object geometry, compensating for the fact that RealSense depth only observes visible surfaces. These full meshes are used to compute Chamfer distance, Maximum Symmetry-Aware Surface Distance, and F-score at 5 mm and 10 mm. The dataset also includes hand–object contact states for all used videos. These labels are at least per-frame binary contact annotations and are used both to evaluate interaction quality and to assess the MLLM-based contact reasoning module.

ArtHOI-RGBD itself does not store hand parameters such as MANO coefficients. In the reconstruction pipeline, however, the paper uses WiLoR to reconstruct MANO-based 4D hands parameterized by articulated hand joint poses $\mathbf{K}$3, hand shape $\mathbf{K}$4, and global transformations $\mathbf{K}$5. These are method-side predictions rather than dataset annotations (Wang et al., 26 Mar 2026).

5. Experimental role in the ArtHOI framework

ArtHOI-RGBD is central to the quantitative evaluation of the ArtHOI framework. For articulated object reconstruction, the paper compares EasyHOI, RSRD, and ArtHOI using Chamfer Distance (CD), Maximum Symmetry-Aware Surface Distance (MSSD), and F-score at 10 mm and 5 mm. The reported result is that ArtHOI achieves the lowest CD and MSSD and the highest F-scores on all five sequences. For the Scissor sequence, the paper reports approximately $\mathbf{K}$6 and F10 $\mathbf{K}$7 for EasyHOI, approximately $\mathbf{K}$8 and F10 $\mathbf{K}$9 for RSRD, and approximately $SE(3)$0 and F10 $SE(3)$1 for ArtHOI (Wang et al., 26 Mar 2026).

The dataset is also used to evaluate hand–object alignment through the Collision–Contact score $SE(3)$2, following Open3DHOI. On ArtHOI-RGBD, the reported scores are $SE(3)$3 for no alignment, $SE(3)$4 for RSRD + WiLoR, $SE(3)$5 for ArtHOI without MLLM, and $SE(3)$6 for ArtHOI with MLLM-based contact reasoning. These values show that the dataset supports sensitive evaluation of physically plausible HOI alignment.

A further use is the evaluation of canonical pose and scale optimization through Adaptive Sampling Refinement (ASR). On ArtHOI-RGBD, the paper reports mask IoU and optimization success rate for FoundationPose, Any6D, and ASR. The numbers are IoU $SE(3)$7, SR $SE(3)$8 for FoundationPose; IoU $SE(3)$9, SR $p_k$0 for Any6D; and IoU $p_k$1, SR $p_k$2 for ASR. Because ArtHOI-RGBD provides metric depth and masks, it offers a particularly stable setting for validating metric grounding and pose-scale refinement.

For MLLM-based contact reasoning, the paper notes that contact accuracy on ArtHOI-RGBD is “near 100%”. The result is described as sufficiently saturated that it is often omitted from ablation tables, and the dataset thus functions as a sanity check for controlled-condition contact reasoning rather than as the hardest contact benchmark (Wang et al., 26 Mar 2026).

6. Comparative position and limitations

ArtHOI-RGBD occupies a specific niche among RGB-D interaction datasets. ArtHOI-Wild, introduced alongside it, contains eight in-the-wild RGB-only clips from internet sources and smartphone recordings and is used to demonstrate robustness and generalization under uncontrolled capture, whereas ArtHOI-RGBD is the controlled RGB-D benchmark for quantitative evaluation (Wang et al., 26 Mar 2026). Relative to broader RGB-D resources, it is much smaller than HOI4D, which provides 2.4M RGB-D egocentric video frames over 4000 sequences with category-level object pose, 3D hand pose, action labels, and reconstructed meshes across 800 object instances from 16 categories (Liu et al., 2022). It is also different in purpose from the Human-Things Interactions dataset, which was introduced for dynamic view synthesis and consists of 43 multi-view RGBD video sequences of everyday activities (Wang et al., 2022).

Its contribution is therefore not breadth, subject diversity, or large-scale pretraining value. It is a tightly controlled, richly annotated metric benchmark tailored to articulated HOI reconstruction. This role is consistent with broader RGB-D dataset analyses that identified dense reconstructions of dynamic scenes and richer combinations of geometric and semantic annotation as underexplored directions (Firman, 2016).

The dataset’s limitations are clear from its construction. It contains only five sequences and five object instances, likely within a single indoor environment and with limited subject variety. Category coverage is narrow: Headphone, Scissor, Candy Box, CD Drive, and Stapler. Even with accurate metric depth, visible-surface sensing remains incomplete, which is why the authors additionally capture surrounding scans to reconstruct full ground-truth meshes. A plausible implication is that future extensions would benefit from more objects, more subjects, and finer-grained contact annotation, while preserving the dataset’s central property: metrically grounded, real-video evaluation of articulated hand–object interaction (Wang et al., 26 Mar 2026).