Papers
Topics
Authors
Recent
Search
2000 character limit reached

HandMCM: Correspondence Mamba for Hand Pose

Updated 9 February 2026
  • The paper introduces HandMCM, a novel state-space model that fuses multi-modal inputs for accurate 3D hand keypoint localization.
  • It employs techniques like bidirectional gated state-space modeling and local token injection to enhance precision under occlusion.
  • Empirical evaluations on NYU, DexYCB, and HO3D datasets demonstrate significant improvements over existing methods in keypoint error reduction.

HandMCM, in the context of 3D hand pose estimation, is a multi-modal point cloud–based correspondence state space model designed for robust and accurate 3D hand keypoint localization, particularly under occlusion and in challenging scene conditions. HandMCM innovatively adapts the Mamba state-space model (SSM) architecture, introduces dynamic correspondence modeling, and fuses multi-modal features (RGB, depth, point cloud) to address the intrinsic complexity of hand articulation and keypoint visibility inherent to human-computer interaction tasks (Cheng et al., 2 Feb 2026).

1. Architectural Overview and Model Formulation

HandMCM processes three input modalities: a single-hand RGB image R∈RH×W×3R \in \mathbb{R}^{H \times W \times 3}, a depth image D∈RH×WD \in \mathbb{R}^{H \times W}, and a point cloud P∈RN×3P \in \mathbb{R}^{N \times 3} (sampled from the depth map). It constructs a fused feature representation using a multi-modal super point encoder:

  • 3D downsampling: PointNet++-style set convolution reduces PP to N′=N/2N' = N/2 super points P′∈RN′×3P' \in \mathbb{R}^{N' \times 3}, extracting geometric features Fp∈RN′×CpF_p \in \mathbb{R}^{N' \times C_p}.
  • 2D encoding: ResNet-based autoencoders generate Fd∈RH/2×W/2×CdF_d \in \mathbb{R}^{H/2 \times W/2 \times C_d} from DD and Frgb∈RH/2×W/2×CrgbF_\text{rgb} \in \mathbb{R}^{H/2 \times W/2 \times C_\text{rgb}} from D∈RH×WD \in \mathbb{R}^{H \times W}0. Each is projected onto D∈RH×WD \in \mathbb{R}^{H \times W}1 (via nearest-neighbor/barycentric interpolation), yielding D∈RH×WD \in \mathbb{R}^{H \times W}2 and D∈RH×WD \in \mathbb{R}^{H \times W}3.
  • Feature fusion: All modalities are concatenated D∈RH×WD \in \mathbb{R}^{H \times W}4.

Keypoint token extraction is achieved by aggregating D∈RH×WD \in \mathbb{R}^{H \times W}5 into a global context vector D∈RH×WD \in \mathbb{R}^{H \times W}6, followed by a bias-induced layer (BIL) that instantiates D∈RH×WD \in \mathbb{R}^{H \times W}7 learnable keypoint tokens D∈RH×WD \in \mathbb{R}^{H \times W}8 (each with a per-keypoint bias). An initial 3D keypoint estimate D∈RH×WD \in \mathbb{R}^{H \times W}9 is obtained through a linear regression head.

The core processing occurs in P∈RN×3P \in \mathbb{R}^{N \times 3}0 stacked “correspondence Mamba” blocks, each refining the tokens via SSM and correspondence modeling. Each block applies a bidirectional gated SSM (BiGS), outputting a dynamically learned correspondence map that models spatial relationships and interactions across keypoints.

2. Dynamic Correspondence Modeling via Bidirectional State-Space Models

HandMCM reframes the traditional keypoint association problem by treating the ordered keypoint tokens as a 1D sequence (as opposed to a fixed graphical topology). Within each block P∈RN×3P \in \mathbb{R}^{N \times 3}1:

  • Forward/Backward streams: A normalized input P∈RN×3P \in \mathbb{R}^{N \times 3}2 is projected via learned matrices. GELU activations and two separate projections yield P∈RN×3P \in \mathbb{R}^{N \times 3}3 (forward) and P∈RN×3P \in \mathbb{R}^{N \times 3}4 (backward), passed through dedicated SSM layers to produce P∈RN×3P \in \mathbb{R}^{N \times 3}5 and P∈RN×3P \in \mathbb{R}^{N \times 3}6.
  • Correspondence map: The outer product P∈RN×3P \in \mathbb{R}^{N \times 3}7 is linearly projected to form P∈RN×3P \in \mathbb{R}^{N \times 3}8, controlling spatial mixing among tokens.
  • Token update: Value projections P∈RN×3P \in \mathbb{R}^{N \times 3}9 are mixed via the correspondence map to yield PP0 (updated keypoint tokens) and projected to 3D coordinates PP1.

Local geometrical information is incorporated through a “local injection and filtering” mechanism. For each predicted keypoint, a local context is constructed from its PP2 nearest super points, concatenated with the keypoint’s global token and local geometric feature, and processed by a lightweight set-conv network. Local tokens are injected multiplicatively and combined through a learned gating mechanism, ensuring that each keypoint estimate is anchored to local evidence.

3. Multi-Modal Feature Integration and Projection

Comprehensive fusion of 2D and 3D cues is pivotal for occlusion robustness:

  • Point cloud super point features (PP3) are obtained by downsampling PP4.
  • Depth (PP5) and RGB (PP6) features are derived from independent ResNet autoencoders.
  • 2D features are aligned with 3D space by projecting onto the super points corresponding to PP7.
  • The concatenated final feature per super point combines spatial, depth, and appearance cues, yielding a highly expressive representation for the SSM-driven keypoint regression.

4. Optimization, Loss Functions, and Training Protocol

HandMCM employs a block-wise loss defined as

PP8

where PP9 is defined piecewise:

N′=N/2N' = N/20

The model is optimized using AdamW (N′=N/2N' = N/21) with a learning rate of N′=N/2N' = N/22 and batch size 32. Extensive data augmentation is applied, including random 3D rotation (±180°), scaling ([0.9, 1.1]), and translation (±10 mm). Epoch schedules are tailored for each benchmark (NYU: 40 epochs, HO3D: 24, DexYCB: 20, with staged decay).

5. Empirical Evaluation and Ablation Analysis

HandMCM was evaluated on NYU (depth-only), DexYCB, and HO3D (both RGBD) datasets with the following setup:

Dataset Modalities J (Keypoints) Metric HandMCM Previous SOTA
NYU Depth 14 Mean keypoint error (mm) 7.06 7.12 (HandDAGT)
DexYCB RGBD 21 Avg. MKE over S0–S3 (mm) 6.67 7.54 (K-Fusion)
HO3D RGBD 21 Mean keypoint error (cm) 1.71 1.79 (K-Fusion)

HandMCM demonstrates state-of-the-art performance, outperforming previous graph-based and standard SSM methods, especially under severe occlusion. Qualitative and quantitative ablations indicate that the combination of correspondence SSM and local token injection/filtering yields the most significant gains (e.g., reduction from 8.47 mm to 7.06 mm on NYU). Increasing Mamba block depth up to three layers yields optimal results.

6. Algorithmic Workflow

A high-level pseudocode sketch illustrates the sequential processing pipeline:

N′=N/2N' = N/23

7. Scientific Contributions, Limitations, and Prospects

HandMCM introduces the first correspondence-based adaptation of the Mamba SSM for 3D hand pose estimation, enabling robust dynamic modeling of keypoint relationships that flexibly adapts to severe occlusion. It advances the field through:

  • Dynamic modeling of hand keypoints via a bidirectional, scan-path SSM rather than a rigid graph.
  • Local information injection and filtering mechanisms, anchoring inference in per-keypoint geometry.
  • Multi-modal super point encoding, synthesizing depth, RGB, and 3D geometric cues for enhanced occlusion handling.

Noted limitations include the model's applicability to only single-hand or hand–object interaction scenarios; bimanual or closely interacting hands remain challenging. Prospective future work targets cross-hand correspondence modeling and efficient SSM variants for real-time AR/VR deployment (Cheng et al., 2 Feb 2026).

Definition Search Book Streamline Icon: https://streamlinehq.com
References (1)

Topic to Video (Beta)

No one has generated a video about this topic yet.

Whiteboard

No one has generated a whiteboard explanation for this topic yet.

Follow Topic

Get notified by email when new papers are published related to Correspondence Mamba.