ACR: Attention Collaboration-based Regressor for Arbitrary Two-Hand Reconstruction (2303.05938v1)

Abstract: Reconstructing two hands from monocular RGB images is challenging due to frequent occlusion and mutual confusion. Existing methods mainly learn an entangled representation to encode two interacting hands, which are incredibly fragile to impaired interaction, such as truncated hands, separate hands, or external occlusion. This paper presents ACR (Attention Collaboration-based Regressor), which makes the first attempt to reconstruct hands in arbitrary scenarios. To achieve this, ACR explicitly mitigates interdependencies between hands and between parts by leveraging center and part-based attention for feature extraction. However, reducing interdependence helps release the input constraint while weakening the mutual reasoning about reconstructing the interacting hands. Thus, based on center attention, ACR also learns cross-hand prior that handle the interacting hands better. We evaluate our method on various types of hand reconstruction datasets. Our method significantly outperforms the best interacting-hand approaches on the InterHand2.6M dataset while yielding comparable performance with the state-of-the-art single-hand methods on the FreiHand dataset. More qualitative results on in-the-wild and hand-object interaction datasets and web images/videos further demonstrate the effectiveness of our approach for arbitrary hand reconstruction. Our code is available at https://github.com/ZhengdiYu/Arbitrary-Hands-3D-Reconstruction.

• The paper introduces ACR, an innovative model using an attention mechanism to disentangle hand representations and improve reconstruction accuracy under occlusions.
• It integrates an Attention Encoder and an Attention Collaboration-based Feature Aggregator to refine pose estimation for both isolated and interacting hand scenarios.
• Experimental results on the InterHand2.6M and FreiHand datasets demonstrate that ACR outperforms existing methods, enabling real-world AR/VR and HCI applications.

Insights into "ACR: Attention Collaboration-based Regressor for Arbitrary Two-Hand Reconstruction"

This paper introduces an innovative method for 3D hand pose and shape reconstruction from monocular RGB images, addressing the challenges posed by occlusions and complex interactions between two hands. The authors present the ACR (Attention Collaboration-based Regressor), which sets out to overcome the limitations of existing methods that typically rely on entangled representations vulnerable to breakdowns under imperfect conditions, such as truncated hands, separate hands, or external occlusions.

Methodological Innovation

The ACR model features an attention mechanism that significantly reduces interdependencies between hands and hand parts during feature extraction. This enables the reconstruction of hands in arbitrary scenarios without the excessive constraints of relying on mutual hand reasoning, which can weaken reconstruction accuracy when hands are interacting or occluded. The core components of ACR are the Attention Encoder (AE) and Attention Collaboration-based Feature Aggregator (ACFA). AE utilizes hand-center and part-based attention maps combined with a cross-hand prior map to provide visibility and improve initial hand-hand interactions predictions before regressing the hands separately. The ACFA then leverages these representations, integrating global and local features to collaboratively refine and regress each hand independently.

Experimental Validation

The efficacy of ACR is thoroughly validated across various datasets, most notably the InterHand2.6M and FreiHand datasets. ACR achieves state-of-the-art performance, markedly surpassing previous methods particularly noted for interacting hands. On the InterHand2.6M dataset, it significantly outperforms existing interacting-hand approaches and yields competitive results against leading single-hand methods on the FreiHand dataset. This demonstrates ACR’s capability of reconstructing hands with high accuracy regardless of complex interaction dynamics or spatial constraints.

Key Contributions and Implications

• Robust Representation Disentanglement: By disentangling the representations of different hand parts and leveraging center-based attention, ACR effectively reduces interdependencies and explicit constraints in hand interaction scenarios.
• Cross-hand Prior Reasoning: ACR's novel approach to cross-hand prior learning significantly enhances the model’s ability to maintain accuracies for interacting hands, addressing known issues in previous methods.
• Implications for Real-world Applications: The demonstrated effectiveness of ACR across in-the-wild scenarios opens up compelling opportunities for applications in AR/VR, human-computer interaction, and entertainment industries, where accurate hand reconstruction from single cameras is essential.

Future Work

The paper acknowledges certain limitations of the current approach, such as the occasional issue of mesh penetration. Future developments may explore leveraging more sophisticated depth reasoning techniques or incorporating alternative camera models to simulate hand translations more accurately, further enhancing the robustness and applicability of 3D hand reconstruction frameworks.

Overall, the paper presents a comprehensive and effective approach to arbitrary two-hand reconstruction, advancing both the theoretical foundations and the practical capabilities of hand pose estimation models.