Enhancing Hands in 3D Whole-Body Pose Estimation with Conditional Hands Modulator

Abstract: Accurately recovering hand poses within the body context remains a major challenge in 3D whole-body pose estimation. This difficulty arises from a fundamental supervision gap: whole-body pose estimators are trained on full-body datasets with limited hand diversity, while hand-only estimators, trained on hand-centric datasets, excel at detailed finger articulation but lack global body awareness. To address this, we propose Hand4Whole++, a modular framework that leverages the strengths of both pre-trained whole-body and hand pose estimators. We introduce CHAM (Conditional Hands Modulator), a lightweight module that modulates the whole-body feature stream using hand-specific features extracted from a pre-trained hand pose estimator. This modulation enables the whole-body model to predict wrist orientations that are both accurate and coherent with the upper-body kinematic structure, without retraining the full-body model. In parallel, we directly incorporate finger articulations and hand shapes predicted by the hand pose estimator, aligning them to the full-body mesh via differentiable rigid alignment. This design allows Hand4Whole++ to combine globally consistent body reasoning with fine-grained hand detail. Extensive experiments demonstrate that Hand4Whole++ substantially improves hand accuracy and enhances overall full-body pose quality.

• The paper introduces Hand4Whole++, which combines frozen SMPLer-X and WiLoR models with a Conditional Hands Modulator and differentiable finger and shape transfer to produce anatomically coherent hands.
• The method reduces hand MPVPE from 38.64 to 9.40 mm on InterHand2.6M and from 58.86 to 7.98 mm on ReInterHand while improving full-body accuracy on datasets such as AGORA and EHF.
• The results show that targeted modular adaptation generalizes better than fine-tuning or naïvely attaching hand predictions, although hand-only training leaves non-hand joints weakly constrained and increases runtime.

The supervision gap in whole-body hand estimation

Whole-body pose estimation methods that regress SMPL-X parameters—such as SMPLer-X (Turkar et al., 2023)—achieve strong global body accuracy, but their hand predictions lag behind dedicated hand estimators. The paper attributes this to a structural supervision mismatch: whole-body datasets (AGORA (Vedadi et al., 2021), ARCTIC (Georges et al., 2022), EHF) contain limited hand pose and shape diversity, while hand-centric datasets (InterHand2.6M (Dremin, 2020), ReInterHand (Krug, 2023), HIC) lack full-body annotations. Consequently, hand estimators like HaMeR (Li et al., 2023) and WiLoR (Li et al., 2024) excel at finger articulation in isolation but produce implausible results when hands interact or must be placed relative to the body, and naïvely attaching hand-estimator outputs to a body—as FrankMocap (Rong et al., 2020) does—yields anatomically invalid wrist configurations, particularly under occlusion.

The Hand4Whole++ framework

The proposed system composes two frozen pre-trained models—SMPLer-X-L32 for the whole body and WiLoR for the hands—plus two lightweight components: the Conditional Hands Modulator (CHAM) and a finger articulation and shape transfer module. The framework runs fully feed-forward at approximately 10 fps on an RTX A6000, with CHAM contributing only 10 ms (~10% of runtime).

CHAM. Hand regions are cropped at 256×256 and passed through WiLoR's ViT backbone; final-layer ViT features serve as conditional signals. These features receive 2D positional encodings derived from the original full-body image space (via inverse crop-and-resize), followed by a three-layer cross-attention Transformer encoder applied only when both hands are detected, enabling modeling of inter-hand relationships. The features are then split into left- and right-hand branches, each containing 24 independent 1×1 convolutions aligned one-to-one with SMPLer-X's 24 transformer blocks. Following ControlNet (Zhang et al., 2023), all convolutions are zero-initialized, so modulation begins from a neutral state. After inverse affine alignment to the full-body feature map, the two hand streams are merged via element-wise maximum and additively fused into the corresponding SMPLer-X blocks. Only CHAM is optimized during training; both estimators remain frozen. The design is architecture-agnostic for ViT-based whole-body backbones.

Finger articulation and hand shape transfer. Rather than trusting the whole-body model's finger predictions, the framework regenerates a canonical MANO (Alzamel et al., 2017) hand mesh from the hand estimator's finger pose and shape parameters, discards the hand estimator's wrist orientation, and rigidly aligns the mesh to the whole-body mesh using the wrist and the four MCP joints. The transformation is differentiable, allowing gradients to flow through CHAM's wrist prediction, and Laplacian smoothing mitigates seam artifacts at the hand boundary. The shape transfer is motivated by an ablation in the supplementary material showing MANO's hand shape space fits 3D scans with a mean point-to-point error of 1.34 mm versus 1.98 mm for SMPL-X hands, reflecting SMPL-X's shared body–hand–face latent space.

Training losses. CHAM is trained on InterHand2.6M, ReInterHand, ARCTIC, and AGORA for 4 epochs (~20 hours, single A6000) with equally weighted $\ell_1$ pose, shape, 2D, and 3D keypoint losses. Hand-only datasets are handled via forward-kinematic conversion of local wrist pose to global orientation, wrist-relative keypoint frames, $\ell_2$ shape regularization, and a body root pose regularizer enforcing an upright torso—a necessary prior since hand-only data lacks full-body supervision.

Empirical results

The central quantitative claims are strong. On hand-only benchmarks, Hand4Whole++ reduces hand MPVPE from 38.64 mm (original SMPLer-X) to 9.40 mm on InterHand2.6M, from 58.86 to 7.98 mm on ReInterHand, and from 32.43 to 17.72 mm on HIC, while simultaneously cutting MRRPE from 119.56 to 32.30 mm and 101.82 to 16.37 mm respectively. On full-body datasets it improves both full-body and hand MPVPE over the original SMPLer-X (e.g., AGORA full-body 76.84 vs. 85.61 mm; EHF hands 33.43 vs. 46.21 mm). Against hand-only specialists, it achieves MPVPE competitive with IntagHand (Yang et al., 2022) and InterWild (Wang et al., 2023)—methods purpose-built for interacting hands—while reporting dramatically lower MRRPE (e.g., 32.30 vs. 48.04 mm on InterHand2.6M), and it uniquely produces full-body outputs.

Two ablations substantiate the design choices. First, fine-tuning the whole-body model on the same limited data overfits: it improves hand-only benchmarks (e.g., 20.00 mm on InterHand2.6M) but degrades catastrophically on unseen full-body data (EHF full-body 126.34 vs. 61.24 mm), whereas CHAM preserves generalization. Second, comparison of combination strategies shows that copying wrist orientations from the hand estimator (FrankMocap-style) increases hand error to 100.59 mm on AGORA, while CHAM reduces it to 50.56 mm—and, notably, reduces full-body error from 84.76 to 76.88 mm. The paper argues CHAM implicitly corrects upstream shoulder and elbow joints by enforcing anatomically plausible hand placement within the kinematic chain. A third ablation shows finger transfer alone reduces hand MPVPE (e.g., InterHand2.6M 14.69 → 12.26 mm) with shape transfer yielding further gains to 9.40 mm; cross-attention in CHAM contributes consistent but smaller improvements.

Limitations and open questions

The paper concedes two limitations. Because hand-only datasets lack full-body annotations, non-hand joints are weakly supervised and may misalign with the image even when hand predictions are correct. Second, the reliance on two pre-trained models increases runtime relative to single-model approaches. Evaluation caveats also apply: HMR-Adapter could not be compared due to unavailable code, Multi-HMR required ground-truth bounding boxes in half of the qualitative cases, and the authors use ground-truth shape and gender for evaluation targets, yielding numbers that differ from those reported in prior papers using customized splits. Additionally, the paper notes preliminary, unvalidated observations of utility on hand-centric images such as egocentric views, leaving formal evaluation of such domains as an explicit open question. Whether CHAM's zero-initialized additive modulation scales to more than two conditional part-experts is likewise unexplored.

Conclusion

Hand4Whole++ addresses the hand-accuracy deficit in whole-body pose estimation through a frozen-backbone, modular design: CHAM injects hand-specific ViT features into a frozen whole-body estimator to produce anatomically coherent wrist orientations and improved upper-body kinematics, while differentiable rigid alignment transfers fine-grained finger articulation and expressive hand shapes from a frozen hand estimator. The results demonstrate that targeted part-aware adaptation under limited supervision can outperform both full fine-tuning and naïve model composition, and that hand enhancement propagates benefits to full-body pose quality—while the dependence on hand-only data for training leaves non-hand joints under-constrained.