ComPose: When to Trust Hands for Object Pose Tracking

Published 22 May 2026 in cs.CV | (2605.23523v1)

Abstract: Reconstructing the motion of objects from videos is a key component for embodied AI and robot manipulation. While diverse approaches to object pose tracking have been studied, they rely heavily on strong external priors, such as depth data or 3D templates, and remain highly vulnerable to severe occlusions by hand grasps despite the use of explicit masks. In this work, we present ComPose, a 6DoF object tracking framework designed for hand-aware object pose estimation from RGB video. Rather than treating the hand purely as an occluder, our method harmonizes hand motions as a \textit{complementary cue} for object tracking. In detail, we recover a variety of object motions over time by combining object and hand cues from foundation models within a unified tracking pipeline. Here, ComPose adaptively selects informative hand joints, combines object- and hand-derived cues for motion estimation, and refines the resulting object motion using visible geometric evidence and a learned correction. We further enforce the temporal consistency over both rotation and translation, yielding stable 3D object trajectories over time without any external smoothing. Extensive experiments show that our method is accurate, efficient, and robust under severe hand occlusion and geometric ambiguity. In addition, the resulting trajectories can also effectively transfer to downstream robot manipulation by enabling robots to reconstruct human actions from online videos.

Abstract PDF Upgrade to Chat

Authors (8)

Summary

The paper proposes an adaptive fusion framework that combines dense object geometry and hand pose cues to accurately estimate 6DoF object pose under occlusions.
It introduces a learned gating mechanism that dynamically blends object-centric and hand-centric estimates to maintain robust performance even in ambiguous visual conditions.
Experimental results show superior trajectory smoothness and accuracy on several benchmarks, enabling direct transfer to robotic learning from unconstrained video input.

ComPose: Adaptive Fusion of Hand and Object Cues for 6DoF Object Pose Tracking

Introduction and Motivation

6DoF object pose estimation from monocular RGB video is a fundamental problem in visual perception and robotic manipulation, particularly for hand-held objects frequently subject to occlusion and geometric ambiguity. Traditional monocular and model-free pose estimators are limited by their reliance on visible object features, making them especially vulnerable to heavy hand occlusions or symmetric/texture-less object geometries. Prior methodologies employ depth data, CAD models, or require explicit object templates, constraining their applicability and generalization for real-world, template-free deployment.

ComPose proposes an adaptive, hand-aware tracking framework that uses both dense object geometry and articulated hand pose cues, dynamically weighting their contributions based on their momentary reliability. The system exploits recent advances in 3D vision foundation models and robust hand pose estimation, enabling temporally consistent and accurate object trajectories even when the object is largely occluded.

Figure 1: Robot manipulation guided by the internet video. (a) Given an instructional video, (b) ComPose predicts the expert’s brush trajectory and (c) directly transfers the estimated trajectory to a real robot.

Methodology

Foundation Model-Based and Hand-Aware Object Tracking

ComPose integrates two primary pose cues:

Object-centric geometric cue: Utilizing dense 3D geometry reconstructed from a sequence of images by a 3D foundation model (e.g., Flow3R), object point clouds are extracted from frame-wise masks (derived by SAM3), and relative rotations between consecutive frames are estimated via ICP-based alignment of visible object point clouds.
Hand-centric pose cue: The 3D hand pose, estimated per frame (e.g., via WiLoR), provides a reliable proxy for object motion during occlusion. By leveraging known correspondences across hand joints (via the MANO topology), the system computes relative object rotations from predicted hand joint motions using a weighted Procrustes alignment. Weights for each joint are adapted to the grasp configuration, highlighting those joints most likely transmitting rigid object motion.

Adaptive Cue Fusion via ComPose

Figure 2: An illustration of the proposed object tracking pipeline and its application to robot simulation.

The key architectural advance is an adaptive gating mechanism, which fuses object-based and hand-based relative pose estimates for each frame pair. A learned scalar $\alpha \in [0,1]$ dynamically blends the two rotations using SLERP in quaternion space, with $\alpha$ learned to privilege hand cues under occlusion and object cues under geometric reliability. Joint informativeness weights are simultaneously predicted for the hand cue.

For translation, the centroid of the object point cloud is used as a base estimate, with a learned residual offset that compensates for centroid bias due to partial visibility; this offset is bounded to prevent drift. Temporal consistency is enforced via explicit losses on the predicted pose sequence: accumulating errors, velocity smoothness, and residual bounding for translation.

Figure 3: Analysis on the predicted alpha and joint weights along with the input frames. Large $\alpha$ indicates heavier reliance on hand-based cue; higher joint weights concentrate on reliable contact points.

Training and Losses

ComPose operates on sliding windows of three consecutive frames, optimizing for both pairwise and composite pose consistency using a mix of rotation (log quaternion error), translation (L1 error in image space), consistency (triple-frame composition), and smoothness losses. Notably, all modules are trainable end-to-end via standard gradient-based optimization.

Experimental Results

Evaluation Protocols and Benchmarks

ComPose is evaluated on several standard hand-object interaction datasets, targeting both in-domain (DexYCB, HOI4D) and cross-dataset (HO3D-v2, OakInk-v1) generalization. The major metrics include relative/absolute translation and rotation errors (RTE, ARE, RRE, ATE), plus the temporal correlation coefficient (TCC) to assess trajectory smoothness.

Figure 4: Qualitative results from state-of-the-art models on DexYCB, HO3D-v2, and OakInk-v1. Images are zoomed for object focus.

ComPose outperforms prior RGB-only and template-based trackers on RRE and TCC metrics, demonstrating highly stable and accurate 3D trajectories under a wide range of occlusion and geometric ambiguity scenarios. The method is notably robust where object-only approaches fail, maintaining strong temporal consistency and reduced drift on longer sequences (e.g., HO3D-v2).

Ablations and Analysis

ComPose is rigorously ablated to measure the relative contributions of object-only, hand-only, and full fused tracking. Object-only tracking fails under occlusion; hand-only tracking is unreliable during dynamic grasps or finger articulation. The proposed joint adaptive fusion consistently yields the best performance and smoothest trajectories.

The gating coefficient $\alpha$ is empirically shown to adapt to the visibility of object geometry, being low when the object is unoccluded, and high in occluded or ambiguous regions. Joint weights focus on thumb and index contact points during stable grasps.

Practical Application to Robot Manipulation

ComPose's temporally smooth and accurate 6D trajectories enable direct transfer to robot controllers without post-processing or robot-specific optimization, enabling effective LfD from unconstrained human demonstrations in Internet videos.

Figure 5: Robot manipulation from internet videos. Predicted hand joints and mesh are overlaid, and trajectory is used by the robot without further optimization.

Figure 6: Robot manipulation from Internet video, mapping human-executed trajectories to a physical robot via ComPose.

The pipeline demarcates a simple mapping from predicted object pose in the camera frame, via hand-eye calibration and a (possibly fixed) grasp offset, to a robot end-effector sequence, which is tracked via an OSC on a real robot. The robustness and quality of the predicted trajectories allow not only the replay of object motion tasks, but also more sensitive manipulations such as liquid pouring (see supplementary bowl experiment).

Implications and Future Directions

The implications of ComPose are significant for embodied AI, manipulation learning, and general hand-object interaction understanding:

Removal of external priors: By using only RGB data and general foundation models, ComPose circumvents the need for CAD templates or depth data, facilitating deployment on arbitrary novel objects.
Bridging HCI and robotics: Direct LfD pipelines become practical, enabling robots to learn complex dynamic manipulations from in-the-wild videos.
Adaptive multimodal fusion: The method demonstrates the utility of adaptive hand-object reasoning and sets a precedent for further multimodal integration in pose tracking.

Open challenges include extremely severe occlusion, generalized grasp unknowns, or scenarios with poor image quality or minimal hand-object contact, where neither cue is reliable. Extending the approach to mode-agnostic hand-object contact modeling and richer context integration, as well as addressing long-horizon drift with stronger global re-anchoring, remain avenues for future work.

Conclusion

ComPose introduces an efficient, adaptive, and robust solution to the problem of 6DoF object tracking from monocular video under hand-object interaction. By harmonizing the strengths of object and hand cues via learned fusion, the method achieves high accuracy and temporal consistency where traditional approaches falter. Its direct applicability to robotic LfD pipelines without external post-processing marks a step forward in the interface between visual perception, video understanding, and embodied robotic action.