Point2Pose: Occlusion-Recovering 6D Pose Tracking and 3D Reconstruction for Multiple Unknown Objects Via 2D Point Trackers

Abstract: We present Point2Pose, a model-free method for causal 6D pose tracking of multiple rigid objects from monocular RGB-D video. Initialized only from sparse image points on the objects to be tracked, our approach tracks multiple unseen objects without requiring object CAD models or category priors. Point2Pose leverages a 2D point tracker to obtain long-range correspondences, enabling instant recovery after complete occlusion. Simultaneously, the system incrementally reconstructs an online Truncated Signed Distance Function (TSDF) representation of the tracked targets. Alongside the method, we introduce a new multi-object tracking dataset comprising both simulation and real-world sequences, with motion-capture ground truth for evaluation. Experiments show that Point2Pose achieves performance comparable to the state-of-the-art methods on a severe-occlusion benchmark, while additionally supporting multi-object tracking and recovery from complete occlusion, capabilities that are not supported by previous model-free tracking approaches.

• The paper introduces a causal, model-free 6D pose tracking method that leverages persistent 2D point tracking for long-range temporal association in multi-object environments.
• The method employs robust frame-to-map registration, multi-hypothesis RANSAC, and TSDF fusion to recover poses instantly after severe and full occlusion.
• The paper validates its approach on benchmarks including the new YCBMultiTrack dataset, achieving high ADD-S AUC and competitive Chamfer distances in dynamic scenes.

Point2Pose: Occlusion-Robust Model-Free 6D Multi-Object Pose Tracking and 3D Reconstruction via 2D Point Tracking

Problem Statement and Motivation

Point2Pose addresses the challenging task of causal 6-DoF pose tracking and object-centric 3D reconstruction of multiple previously unseen rigid objects solely from monocular RGB-D videos. This is a critical component for open-world robotic manipulation, where objects cannot be assumed to have CAD models or pre-defined category priors. Existing model-free methods predominantly support single-object tracking and often suffer catastrophic failures under severe occlusion or full object disappearance, lacking robust mechanisms for re-localization. Furthermore, most prior works either require multi-view scene reconstruction before tracking, depend on strong priors, or do not scale efficiently to multi-object or dynamic scenarios.

Method Overview

Point2Pose introduces a thoroughly model-free multi-object pose tracking and reconstruction pipeline, leveraging persistent 2D point tracking for long-horizon temporal association and robust frame-to-map registration for pose recovery and scene updating. The primary system components can be summarized as follows:

• Long-Range 2D Point Tracking: The framework uses a state-of-the-art 2D point tracker (e.g., BootsTAPIR, CoTracker) to follow user-initialized object-specific points through the image sequence, relying only on RGB observation. This enables persistent correspondences, bypassing brittle feature-matching across frames.
• Object-Centric Keypoint Maps and TSDF Fusion: Tracked points are lifted into 3D using the depth channel and incrementally fused into object-centric volumetric keypoint maps and dense TSDFs. This supports robust, model-free online object reconstruction.
• Multi-Hypothesis Map Registration and Pose Refinement: Frame-to-map pose estimation is cast as a robust point cloud registration problem, utilizing multi-hypothesis RANSAC to generate candidate solutions to handle perceptual aliasing and outlier contamination. Pose hypotheses are validated using TSDF consistency and further refined with iterative non-linear optimization.
• Graph-Based Global Map Optimization: A factor-graph is used to maintain global temporal consistency over keyframes and keypoints, mitigating drift and enforcing geometric constraints.
• Instantaneous Occlusion Recovery: Crucially, the persistent nature of 2D point tracks allows for immediate re-assignment and pose recovery when an object reappears, even after full occlusion or exiting the camera view, overcoming key limitations of prior methods.

  Figure 1: Overview of Point2Pose architecture, illustrating the parallel tracking, segmentation, TSDF fusion, and graph-based global optimization pipeline.

Dataset and Benchmark Contributions

Point2Pose is empirically validated on standard benchmarks (HO3D, YCBInEOAT), but uniquely introduces the YCBMultiTrack dataset—a novel benchmark for RGB-D-based, dynamic multi-object manipulation featuring both real and synthetic scenes, ground-truth 6D motion capture, and sequences with repeated, severe occlusion. This dataset is tailored for evaluating causal multi-object tracking with occlusion robustness.

Figure 2: Real-world sequence from YCBMultiTrack, demonstrating recovery from complete multi-object occlusion.

Figure 3: Synthetic YCBMultiTrack sequence showing mutual occlusion and independent motions.

Key Methodological Innovations

The distinguishing features of Point2Pose are:

• Model-Free Multi-Object Causal Tracking: No CAD models, object categories, or priors are required. Initialization uses only a sparse set of 2D image points per object, which is particularly relevant for open-world deployment.
• Occlusion Recovery Mechanism: The 2D tracker’s capacity for maintaining correspondences over long temporal gaps obviates brittle re-localization stages.
• Hybrid Map Association: Integrating point-level feature tracks, geometric keypoint curation (balancing spatial coverage and trackability), multi-hypothesis pose reasoning, and volumetric TSDF selection/refinement creates a robust system for complex, cluttered scenes.
• Online Integration and Consistency: Factor-graph optimization assures globally consistent keyframe and keypoint estimates, while incremental TSDF fusion yields steadily improving 3D models. This entire loop operates causally and online.

Experimental Analysis

Quantitative Results

On standard single-object benchmarks (HO3D, YCBInEOAT), Point2Pose achieves competitive ADD-S/ADD AUC and Chamfer metrics relative to BundleSDF, the leading prior work, across a range of manipulated objects and occlusion levels. Specifically:

• Occlusion Regimes: Point2Pose maintains state-of-the-art accuracy in severe and full occlusion regimes, whereas baseline methods often fail to track or require explicit re-initialization.
• Multi-Object Case: On the new YCBMultiTrack benchmark, Point2Pose provides robust, real-time tracking and reconstruction of multiple objects with dynamic motion and mutual occlusion. In real-world tests, baseline methods deteriorate rapidly with occlusion, while Point2Pose recovers object pose immediately as they re-enter view.
• Failure Modes: Textureless surfaces (Figure 4) and poor segmentation are identified as limiting factors; however, Point2Pose’s explicit handling of uncertainty and multi-hypothesis reasoning mitigates most catastrophic errors.

  Figure 4: Failure modes—unstable pose estimates and track uncertainty in low-texture environments compared to stable tracking in textured regions.

  

  Figure 5: Scene with a fully occluded object; Point2Pose recovers the correct pose after reappearance, unlike BundleSDF which fails to relocalize.

Ablation and Analysis

Ablation studies demonstrate the complementary value of multi-hypothesis registration, SDF-based pose refinement, and factor-graph optimization. The absence of any module leads to a consistent reduction in tracking accuracy, confirming the utility of each pipeline component.

The keypoint sampling method also impacts performance; hybrid strategies balancing feature detector confidence and geometrical coverage outperform both dense uniform and naive detector-only schemes.

Performance Metrics

• Occlusion Recovery: Point2Pose achieves near-instant pose recovery after full object disappearance, a property not shown by any prior model-free method.
• Tracking Accuracy: In multi-object, real-world YCBMultiTrack, ADD-S AUC remains high ($>89\%$ mean) for all tracked objects, substantially outperforming BundleSDF, which fails in the majority of severe occlusion cases.
• Reconstruction: The system delivers object-centric mesh reconstructions with Chamfer distances competitive with volumetric neural approaches.

Limitations

While robust, the approach relies on discriminative image textures for long-range tracking; large textureless regions and significant segmentation errors remain challenging and can degrade pose accuracy or keypoint map update. Memory and compute complexity scale with object count and number of tracks, though practical optimizations (point pruning, track subsampling) can help. The use of TSDF, while efficient, may provide lower-fidelity reconstructions compared to cutting-edge neural fields, though enabling neural fusion while maintaining causal, online operation remains an open challenge.

Implications and Future Directions

Point2Pose demonstrates that modern, discriminative, persistent 2D point tracking—when tightly integrated with uncertainty-aware map construction and optimization—resolves key bottlenecks in causal, model-free 6D multi-object tracking under real-world occlusion. Practically, this enables robot systems to interact with truly novel objects in clutter, with immediate recovery from visual loss, without CAD models or prior supervision—a key requirement for generalist in-the-wild manipulation.

Future research directions include replacing TSDF with more expressive neural field representations while preserving causal online operation and tight integration with tracking, improved strategies for textureless and ambiguous scenes (self-supervised enhancement, active querying), and efficient scaling to very large numbers of objects. Advances in robust instance segmentation, as well as exploiting additional modalities (e.g., tactile or language cues), could further increase resilience to perceptual failures.

Conclusion

Point2Pose sets a new standard for causal, model-free 6D pose tracking and 3D reconstruction in dynamic, multi-object, occlusion-prone environments without object models or priors. Its persistent 2D point tracking, uncertainty-aware mapping, robust multi-hypothesis registration, and online global optimization work synergistically to enable tracking and immediate occlusion recovery in settings previously inaccessible to model-free approaches. The introduction of the YCBMultiTrack benchmark fills a crucial evaluation gap, and the open-source code base will serve as a significant resource for the community.

(2604.10415)