Robust bimanual manipulation under heavy occlusion and perceptual noise

Develop a real-world bimanual manipulation system with two multi-fingered robot hands that remains robust to heavy in-hand visual occlusions caused by fingers surrounding the object and to perceptual noise from sensing, enabling reliable real-time manipulation.

Background

Within the task formulation, the authors highlight that in-hand manipulation induces severe occlusion as fingers envelop the object, making perception difficult. This complicates real-time control and state estimation for bimanual manipulation.

The paper proposes a sparse keypoint-based perception pipeline as a practical solution, but explicitly acknowledges that designing a robust real-world system under such occlusions and noise remains an open problem.

References

Heavy occlusion can occur during manipulation due to fingers surrounding the object in-hand. How to build a real robot system that can perform robust bimanual manipulation against such occlusion and perceptual noises remains an open problem.

— Twisting Lids Off with Two Hands  (2403.02338 - Lin et al., 2024) in Section 3 Task Formulation, Task Complexity, 1) In-Hand Perception Difficulty

Dexterous grasping in unstructured environments remains an open challenge in robotics.

— Grasping by interconnection: robust closing motions from coarse object templates  (2609.19228 - Vanderheyden et al., 16 Sep 2026) in Section 1, Introduction