Arti4D: In-the-Wild Articulated Benchmark

• Arti4D is a comprehensive dataset offering annotated RGB-D sequences with detailed 3D kinematics and articulation axis labels for real-world object interactions.
• It supports rigorous evaluation through metrics like angular error and Euclidean position error, enabling robust assessment in dynamic, partially observable environments.
• The benchmark facilitates the development and validation of methods, such as ArtiPoint, that improve articulated object perception in unconstrained settings with occlusion and clutter.

Arti4D is an ego-centric, in-the-wild dataset and benchmark designed for scene-level articulated object interaction, focusing on the estimation of 3D kinematics and articulation modeling from challenging human demonstrations. Developed to support advances in articulated object perception and modeling, Arti4D provides annotated RGB-D video sequences of real-world articulated object manipulations, with ground-truth camera trajectories and explicit articulation axis labels. This resource enables robust algorithmic evaluation in unconstrained environments, where dynamic camera motion, partial observability, clutter, and occlusion are prevalent. Arti4D supports the development and assessment of methods such as ArtiPoint, which aim to bridge the gap between controlled benchmarks and practical, real-world robotic perception contexts (Werby et al., 1 Sep 2025).

1. Problem Setting and Rationale

Arti4D addresses the need for articulated object datasets and evaluation protocols that reflect real-world operation. Prior datasets for articulated object estimation are generally restricted to controlled setups featuring isolated objects, fixed viewpoints, and complete object state observation. These constraints are not representative of practical human/robot interactions, where:

• The camera viewpoint is dynamic and often ego-centric (e.g., hand-held or head-mounted).
• Objects are partially observable due to occlusion, interaction, and environmental clutter.
• Ground-truth kinematic models and part-level articulation axes are required for benchmarking articulation inference.

Arti4D is designed to capture these complexities, facilitating the development and validation of algorithms capable of estimating articulated object kinematics "in the wild." Partial observability, frequent occlusions, unconstrained manipulation, and handheld camera trajectories replicate the real embodied AI, service robot, or novel-object manipulation settings encountered in practice.

2. Dataset Composition and Annotations

Arti4D consists of 45 RGB-D video sequences of scene-level human-object interaction, distributed across four real-world environments (RR080, DR080, RH078, RH201). The dataset includes a total of 414 manually annotated human-object interaction segments, capturing a wide variety of articulated objects such as doors, drawers, cabinets, and appliances.

Key features of Arti4D:

• Modalities: Synchronized RGB video, aligned depth, and ground-truth camera poses with centimeter accuracy.
• Annotations: Each interaction segment is labeled with precise articulation axes, supporting both prismatic and revolute joint categories. Articulation parameters are encoded as twists in se(3), providing a mathematically rigorous reference.
• Difficulty Labels: Interactions are annotated as EASY or HARD, with HARD cases characterized by poor hand visibility, insufficient depth measurements, occlusion, or extensive missing object observations.
• Scene Context: Interactions are recorded within naturally cluttered real environments, frequently exhibiting multiple human actions per scene and repeated interactions with the same objects.

This structure provides high-fidelity benchmarking data for factors such as occlusion robustness, camera motion compensation, and articulated model inference under realistic conditions.

3. Evaluation Protocol and Metrics

Arti4D defines a comprehensive evaluation protocol for articulated object estimation methods. Each predicted interaction segment is matched to ground truth based on temporal intersection over union (IoU>0.5). The primary evaluation metrics include:

• Angular Error (revolute and prismatic axis direction): The error between the predicted axis direction $\hat{\mathbf{a}}_i$ and the corresponding ground-truth $\mathbf{a}_{\mathrm{gt}}$, calculated as

$$\Theta_{\mathrm{err},i} = \cos^{-1}\left(\frac{\mathbf{a}_{\mathrm{gt}}^\top \hat{\mathbf{a}}_i}{\|\mathbf{a}_{\mathrm{gt}}\|\,\|\hat{\mathbf{a}}_i\|}\right)$$

• Euclidean Position Error (revolute axis support point): The minimal perpendicular distance between the predicted axis and the ground-truth axis, formally,

$$d_i = \left|(\hat{\mathbf{p}}_i - \mathbf{p}_{\mathrm{gt}})^\top\,(\hat{\mathbf{a}}_i \times \mathbf{a}_{\mathrm{gt}})\right| / \|\hat{\mathbf{a}}_i \times \mathbf{a}_{\mathrm{gt}}\|$$

when the axes are not nearly parallel.

• Joint Type Accuracy: Categorical agreement between predicted and ground truth joint type.

Difficult cases are explicitly included, allowing methods to be assessed both in favorable (EASY) and adverse (HARD) scenarios.

4. Supported Methodologies and Benchmarking

Arti4D is designed as a reference testbed for articulated object estimation pipelines that operate on unconstrained ego-centric RGB-D data. The canonical method benchmarked on Arti4D is ArtiPoint (Werby et al., 1 Sep 2025), which combines:

• Interaction Detection: Hand segmentation triggers interaction intervals.
• Deep Point Tracking: GFTT (Shi-Tomasi) keypoints near detected hands are tracked across RGB-D frames using a deep any-point tracker (CoTracker3).
• 3D Trajectory Recovery: 2D tracks are lifted to 3D using depth, and compensated for camera motion with ground-truth odometry; trajectories are filtered and smoothed.
• Factor Graph Optimization: Aggregated, smoothed 3D point tracks are input into a factor graph optimizer (GTSAM-based), which fits a global articulated kinematic model (prismatic, revolute, or general screw axes).

Arti4D supports benchmarking of both classical [Sturm et al.] and recent learning-based articulated object estimators. Results reported on Arti4D enable direct comparison across levels of clutter, occlusion, and camera motion.

5. Empirical Findings and Use Cases

Benchmarking with ArtiPoint on Arti4D demonstrates substantially improved accuracy and occlusion robustness relative to prior art:

Method	Prismatic Angle (°)	Revolute Angle (°)	Revolute Pos. (m)	Prismatic Acc.	Revolute Acc.
ArtiPoint	14.54	17.14	0.07	0.68	0.98
ArtGS	52.63	59.48	0.28	0.30	0.76

Arti4D further reveals that factor-graph-based optimizers benefit significantly when provided with high-quality 3D point trajectories from robust tracking frontends. Ablation studies confirm the importance of trajectory filtering, smooth temporal denoising, and accurate camera pose estimation.

Arti4D’s difficulty labeling and diverse scene context expose frequent method failures due to missing depth, reflective objects, textureless parts, and challenging hand-object interaction dynamics. The dataset also supports downstream applications in scene understanding, mobile manipulation, and learning-from-demonstration by enabling kinematic model extraction in naturalistic tasks.

6. Limitations and Comparative Scope

Arti4D does not cover all possible articulated configurations. The annotation schema assumes relatively simple two-body kinematics and binary articulation types. Objects exhibiting multiple kinematic chains, closed loops, or complex coupled motions are not explicitly represented. The dataset’s RGB-D format requires reliable depth, challenging for metallic, glass, or low-texture materials. All sequences provide ground-truth odometry, but experiments in the companion paper suggest the benchmark remains informative when SLAM-derived poses are used, albeit with modest degradation.

Compared to earlier datasets and previous works (e.g., PartNet-Mobility), Arti4D uniquely targets ego-centric video, in-the-wild articulated scene interaction, with fine-grained axis annotation and explicit support for method evaluation under partial observability, real-world motion, and complex occlusion patterns.

7. Relevance for Broader Articulated Object Modeling and Future Directions

Arti4D provides a foundation for next-generation articulated object perception, enabling research that moves beyond instance-centric and category-constrained datasets. Its in-the-wild paradigm, combined with rigorous axis and pose annotation, facilitates:

• Development of algorithms robust to real-world observational constraints.
• Integration of articulated model estimation into larger embodied AI, robotics, and AR/VR pipelines.
• Extension toward more complex kinematic graphs, enhanced annotation schemas incorporating multiple interacting parts, and deeper semantic labeling.

Future directions suggested by the dataset’s design include adaptation to monocular-only settings, expansion to richer (e.g., 4D or functionally annotated) benchmarks, and synergistic use with datasets focused on interactive or functional object modeling (Peng et al., 13 Apr 2026). Arti4D sets a new empirical standard for realism and difficulty in articulated object estimation research.