Robust bimanual manipulation under heavy occlusion and perceptual noise

Develop a real-world bimanual manipulation system with two multi-fingered robot hands that remains robust to heavy in-hand visual occlusions caused by fingers surrounding the object and to perceptual noise from sensing, enabling reliable real-time manipulation.

Background

Within the task formulation, the authors highlight that in-hand manipulation induces severe occlusion as fingers envelop the object, making perception difficult. This complicates real-time control and state estimation for bimanual manipulation.

The paper proposes a sparse keypoint-based perception pipeline as a practical solution, but explicitly acknowledges that designing a robust real-world system under such occlusions and noise remains an open problem.