ArtiPoint: Articulated Object Estimation

• ArtiPoint is an articulated object estimation framework using unconstrained, ego-centric RGB-D videos, integrating deep point tracking with factor graph optimization.
• It addresses 3D motion understanding by estimating part trajectories and articulation axes under dynamic camera motion, occlusions, and partial observability.
• The framework leverages the Arti4D dataset for robust benchmarking against state-of-the-art methods in articulated object perception.

Searching arXiv for ArtiPoint and closely related articulated object estimation/perception work. arXiv search: "Articulated Object Estimation in the Wild" ArtiPoint is an articulated object estimation framework for realistic, unconstrained ego-centric RGB-D video, designed to infer articulated object models under dynamic camera motion and partial observability. It is motivated by the observation that prior methods for articulation estimation have primarily focused on controlled settings, assuming either fixed camera viewpoints or direct observations of various object states, whereas humans can infer articulation by watching others manipulate objects. By combining deep point tracking with a factor graph optimization framework, ArtiPoint estimates articulated part trajectories and articulation axes directly from raw RGB-D videos, and it is introduced together with Arti4D, the first ego-centric in-the-wild dataset that captures articulated object interactions at a scene level, with articulation labels and ground-truth camera poses (Werby et al., 1 Sep 2025).

1. Problem setting and scope

ArtiPoint addresses 3D motion understanding for articulated objects in settings relevant to robotic scene understanding, mobile manipulation, and motion planning. The target regime is not a static laboratory setup, but in-the-wild human-object interaction recorded by a moving ego-centric RGB-D camera, with clutter, occlusions, dynamic viewpoint changes, and partial observability (Werby et al., 1 Sep 2025).

The framework is explicitly positioned against approaches that require either fixed viewpoints or direct observations of multiple object states. In this formulation, articulation is estimated from observation of manipulation rather than from isolated object scans. The resulting estimation problem is therefore simultaneously geometric and temporal: the method must identify manipulated object regions, track informative points across time, recover their 3D trajectories under camera motion, and fit an articulation model despite missing observations and depth noise.

A common misconception is that articulated object estimation in such settings reduces to static part segmentation. ArtiPoint instead targets articulated part trajectories and articulation axes from raw RGB-D video. This distinction matters because the operative signal is motion induced by interaction, not only geometric partitioning.

2. Estimation pipeline

The ArtiPoint pipeline begins with interaction segment extraction. Frames of human-object interaction are detected by segmenting hand presence with a segmentation model, smoothing detections over a window, and creating segments when a hand is present for $T_{min}$ to $T_{max}$ frames (Werby et al., 1 Sep 2025).

The next stage is deep point tracking, described as any-point tracking. The method samples 2D points near detected hand regions in each interaction frame, uses an instance segmentation model, MobileSAM, to mask candidate objects, extracts stable 2D keypoints via Shi-Tomasi/GFTT within these masks, and tracks them throughout the segment using CoTracker3. This yields 2D point tracks together with visibility over time.

ArtiPoint then performs 3D track estimation and filtering. Tracked 2D points are lifted into 3D using depth images and camera intrinsics, after which odometry compensation converts them into a global frame using precise camera pose information. Tracks are filtered if they are static, mostly occluded, or unreliable with high occlusion fractions. The remaining trajectories are smoothed to mitigate jitter from depth and tracking noise.

The final stage is factor graph optimization for articulation model estimation. The smoothed 3D point trajectories are converted into local pose sequences, and a factor graph jointly optimizes for an underlying articulation parameter and part trajectory. This formulation is used to fit prismatic and revolute motion models directly from noisy, partially observed tracks (Werby et al., 1 Sep 2025).

The pipeline is modular rather than end-to-end object-specific. The paper characterizes this as avoiding costly end-to-end object-specific learning and as relying on a combination of deep tracking and optimization-based model fitting.

3. Mathematical formulation

Trajectory smoothing is posed as an optimization problem over the lifted 3D track $\mathbf{p}$:

$$E(\mathbf{p}) = \sum_{t=1}^{T} v_t \|\mathbf{p}_t - \hat{\mathbf{p}}_t\|^2 + \lambda_{vel}\|\mathbf{p}_t - \mathbf{p}_{t-1}\|^2 + \lambda_{jerk}\|\mathbf{p}_t - 3\mathbf{p}_{t-1} + 3\mathbf{p}_{t-2} - \mathbf{p}_{t-3}\|^2,$$

where $v_t$ is visibility and $\lambda_{vel}, \lambda_{jerk}$ are regularization weights (Werby et al., 1 Sep 2025).

The articulation parameter is defined as

$$\xi = \{\omega, v\} \in \mathfrak{se}(3) \subseteq \mathbb{R}^6,$$

so that screw axes can represent prismatic, revolute, and screw joints. Pose evolution is expressed through the exponential map,

$g(\theta) = \exp(\xi \theta) \in SE(3),$

where $\theta$ is the configuration variable. In the articulation-fitting stage, ArtiPoint minimizes articulation-regularized reprojection errors across points and timesteps:

$$\hat{\xi}^*, \theta_1^*, \ldots, \theta_M^* = \operatorname*{argmin}_{\hat{\xi}, \theta_1, ..., \theta_M} \sum_{m=1}^{M} \sum_{j=1}^{|P_m|} \| p^{t+\epsilon}_j - \exp(\hat{\xi} \theta_m) \cdot p^t_j \|^2.$$

The implementation uses GTSAM within a factor graph framework. In practical terms, this means that articulation axes and motion parameters are estimated jointly rather than sequentially. This suggests that the method treats articulation as a constrained latent structure imposed on tracked motion, rather than as a post hoc regression from independent framewise pose estimates.

4. Arti4D dataset and evaluation protocol

Arti4D accompanies ArtiPoint and is described as the first ego-centric in-the-wild dataset targeting articulated object manipulation. It contains 45 RGB-D sequences across 4 diverse real-world scenes and 414 labeled human-object interactions. The data are captured by a handheld RGB-D camera following a human operator interacting with the environment (Werby et al., 1 Sep 2025).

Property	Specification
Sequences and scenes	45 RGB-D sequences across 4 diverse real-world scenes
Interactions	414 labeled human-object interactions
Annotations	Articulation axes, temporal segments, EASY/HARD difficulty, camera odometry

The annotations include precise 3D articulation axes stored as screw parameters in $T_{max}$0, manual temporal labels for interaction intervals, and a binary difficulty label, EASY or HARD, based on hand visibility, object visibility, depth quality, and extent of occlusion. Camera odometry is cm-accurate 6DoF, tracked externally with HTC Vive trackers. After recording, sequences are reconstructed via TSDF fusion for scene context; axes are labeled in annotated frames using Blender and then verified by a second annotator.

Evaluation matches predicted interactions to ground truth using segment IoU with threshold $T_{max}$1. The reported metrics are angular error, positional error for revolute joints, and joint type accuracy. Angular error is defined as

$T_{max}$2

Evaluation is reported across EASY and HARD interactions, and also under estimated versus ground-truth camera poses (Werby et al., 1 Sep 2025).

5. Benchmark results, ablations, and limitations

ArtiPoint is benchmarked on Arti4D against classical and learning-based baselines, including Sturm et al. [2011], Ditto, ArtGS, and Buchanan. The reported results are as follows (Werby et al., 1 Sep 2025).

Method	Reported metrics
ArtGS	52.29 prismatic [deg], 56.82 revolute [deg], 0.25 revolute $T_{max}$3 [m], 1.00 prismatic type acc., 0.00 revolute type acc.
Ditto	55.03 prismatic [deg], 60.89 revolute [deg], 0.29 revolute $T_{max}$4 [m], 0.00 prismatic type acc., 1.00 revolute type acc.
Sturm (w/ bbox baseline)	63.15 prismatic [deg], 57.98 revolute [deg], 1.34 revolute $T_{max}$5 [m], 0.00 prismatic type acc., 1.00 revolute type acc.
Buchanan (w/ bbox)	49.67 prismatic [deg], 58.82 revolute [deg], 0.22 revolute $T_{max}$6 [m], 0.00 prismatic type acc., 1.00 revolute type acc.
ArtiPoint (w/ Sturm backend)	26.85 prismatic [deg], 18.32 revolute [deg], 1.40 revolute $T_{max}$7 [m], 0.70 prismatic type acc., 0.96 revolute type acc.
ArtiPoint (indep. transforms)	15.60 prismatic [deg], 18.61 revolute [deg], 0.15 revolute $T_{max}$8 [m], 0.65 prismatic type acc., 0.96 revolute type acc.
ArtiPoint (ours, articulation-reg.)	14.54 prismatic [deg], 17.14 revolute [deg], 0.07 revolute $T_{max}$9 [m], 0.68 prismatic type acc., 0.98 revolute type acc.

The paper states that ArtiPoint outperforms all baselines by substantial margins, especially in joint parameter estimation and joint type classification, for both revolute and prismatic joints. It is also reported to remain robust across both EASY and HARD interactions, and to incur only minor degradation when estimated camera poses are used, for example from DROID-SLAM.

Ablation studies indicate that using learned keypoints (ORB) or coarser keyframe strides, or turning off smoothing, degrades performance. Accurate hand detection and reliable point tracking are described as critical for the pipeline. These findings localize the method’s empirical sensitivity: the strongest dependencies are in temporal segmentation and track quality rather than in a learned object prior.

The stated limitations are threefold. First, performance drops on keypoint-poor surfaces and when depth sensing fails, including transparent or reflective surfaces. Second, the formulation is limited to simple 2-body articulated models and does not handle more complex articulation topologies or higher DoF. Third, temporal segmentation depends on robust hand detection and may suffer when hands are not visible or are poorly segmented (Werby et al., 1 Sep 2025).

6. Relation to broader articulated object research and future directions

ArtiPoint occupies a distinct position within articulated object research. “Generalizable Articulated Object Perception with Superpoints” focuses on 3D point-cloud part segmentation, introducing a learnable, part-aware superpoint generation method and reporting AP50 scores of 77.9% for seen categories and $\mathbf{p}$0 for unseen categories on GAPartNet, with superior results in 5 out of 9 part categories for seen objects and all part categories for unseen objects (Yu et al., 2024). “RPMArt: Towards Robust Perception and Manipulation for Articulated Objects” instead addresses estimation of articulation parameters and affordance points from a single noisy point cloud, with zero-shot sim-to-real transfer after training only on synthetic data (Wang et al., 2024). “PAct: Part-Decomposed Single-View Articulated Object Generation” moves further downstream, generating articulated 3D assets from a single image with explicit part structure and motion, and reporting feed-forward inference at $\mathbf{p}$1 s/object (Liu et al., 16 Feb 2026).

Relative to these lines of work, ArtiPoint is centered on video-based articulation estimation from ego-centric interaction. It does not primarily solve part segmentation, single-view affordance prediction, or articulated asset generation. Instead, it estimates articulation from observed manipulation under dynamic camera motion. A plausible implication is that ArtiPoint can function as an intermediate layer between in-the-wild human demonstration and downstream robotic reasoning, whereas GAPS, RPMArt, and PAct address adjacent but different subproblems.

The paper identifies several future directions. These include generalizing from simple 2-body models to kinematic chains with multiple DoF or non-two-body links; improving tracking in featureless or depth-sparse scenarios, for example through monocular depth inpainting or learning-based 3D upsampling; moving beyond hand detection toward more advanced interaction localization; and pursuing real-time and on-robot deployment (Werby et al., 1 Sep 2025). The accompanying Arti4D dataset also suggests broader use as a benchmark for SLAM or odometry in dynamically changing scenes, because it provides accurate object motion and camera pose labels.