AnyHand: Unified Hand Analysis & Applications

• AnyHand is a family of systems for 3D hand analysis, integrating pose estimation, dexterous grasping, and multi-hand reconstruction from RGB and RGB-D inputs.
• It features a large-scale synthetic dataset with aligned depth, arm context, dynamic lighting, and occlusion diversity to overcome limitations of real-world data.
• AnyHand also enables hand-agnostic robotic grasping and joint-driven 3D avatar rendering, leveraging universal geometry representations for real-time performance.

AnyHand is a research term used in several technically distinct but related senses in recent literature on hand analysis, manipulation, and rendering. It most directly names a large-scale synthetic RGB-D dataset for 3D hand pose estimation from RGB-only and RGB-D inputs, but it also denotes a hand-agnostic dexterous grasping formulation for different robotic hands, a single-stage strategy for reconstructing any number of hands from one RGB image, and a joint-driven 3D Gaussian Splatting representation for animatable hand avatars controlled by 21 3D keypoints (Si et al., 26 Mar 2026, Fang et al., 23 Feb 2025, Ren et al., 2022, Sun et al., 31 Jan 2025). Across these usages, the term consistently marks a generalization objective: broader data coverage, broader embodiment coverage, broader scene cardinality, or broader subject-specific deformation.

1. Terminological scope

A recurrent source of confusion is that AnyHand does not denote one unified task formulation. In the dataset paper, it is a proper name for a synthetic RGB-D corpus. In AnyDexGrasp, it refers to equipping different robotic hands with dexterous grasping policies using a single hand-agnostic perception model and a lightweight per-hand adaptation step. In SMHR, it denotes locating and reconstructing any number of hands from a single RGB image in one forward pass. In JGHand, it refers to an input-agnostic, 3D-keypoint-driven hand avatar that can be animated across poses and characters (Si et al., 26 Mar 2026, Fang et al., 23 Feb 2025, Ren et al., 2022, Sun et al., 31 Jan 2025).

Usage of “AnyHand”	Technical meaning	Reference
AnyHand dataset	Large-scale synthetic RGB-D data for 3D hand pose estimation	(Si et al., 26 Mar 2026)
AnyHand in AnyDexGrasp	General dexterous grasping for different robotic hands	(Fang et al., 23 Feb 2025)
AnyHand in SMHR	Single-stage reconstruction of multiple hands from one RGB image	(Ren et al., 2022)
AnyHand in JGHand	Joint-driven, real-time 3DGS hand avatar controllable by 3D keypoints	(Sun et al., 31 Jan 2025)

This suggests that “AnyHand” functions less as a single technical noun than as a family resemblance term for systems intended to remain valid under a source of variation that earlier pipelines treated as a special case.

2. Synthetic RGB-D dataset for 3D hand pose estimation

"AnyHand: A Large-Scale Synthetic Dataset for RGB(-D) Hand Pose Estimation" defines AnyHand as a synthetic dataset designed for “3D hand pose estimation … from either RGB-only or RGB-D inputs across diverse real-world scenarios” (Si et al., 26 Mar 2026). The motivating claim is that real captured datasets are “limited in coverage,” with annotation noise “especially under heavy occlusion,” while prior synthetic datasets rarely provide occlusions, arm details, and aligned depth together at scale. Early RHD has aligned depth but lacks arm context and objects and is small; ObMan has objects but limited realism and scale; newer large RGB sets such as Re:InterHand and RenderIH are “primarily released as RGB data without aligned depth, explicit modeling of arm and object occlusion.”

AnyHand addresses this bottleneck through two branches. The main text and Table 1 report “AnyHand-Single” at “2.5M” images and “AnyHand-Interact” at “4.1M” images. The appendix summary reports different counts: AnyHand-Single with “1.05M scenes, 2.1M images” and AnyHand-Interact with “2.1M scenes, 4.2M images.” The paper reports both sets of numbers as-is.

Branch	Main text and Table 1	Appendix summary
AnyHand-Single	2.5M images	1.05M scenes, 2.1M images
AnyHand-Interact	4.1M images	2.1M scenes, 4.2M images

The dataset stores “RGB, depth, foreground mask, 2D bounding boxes, together with camera intrinsics and extrinsics,” plus “precise 3D hand pose and shape parameters directly from the simulation.” The paper does not report object mesh or pose annotations, visibility or occlusion flags, or contact and penetration labels. Its diversity budget is unusually explicit: “47,438 MANO shape parameters $\beta$,” “10,240 unique hand appearances,” “254 high-quality human-body textures,” “MIT Indoor Scenes (536 images) and … 734 HDRI environment maps” in the main text, an appendix summary of “1,270 indoor images and HDRI maps,” and “more than 500k realistically textured objects” through Objaverse. Hand poses are sampled “on the fly” from the DPoser-Hand diffusion prior trained on FreiHAND, HO-3D, DexYCB, H2O, and Re:InterHand, and the interaction branch leverages GraspXL’s “over 10M physics-simulation-based hand-object interaction sequences.”

A key design choice is attached-arm context. The single-hand branch renders a hand-attached forearm aligned from SMPL or SMPL+H to the MANO wrist and textured with SMPLitex, including “both bare skin and clothing such as short- and long-sleeves.” The interaction branch explicitly models “realistic mutual occlusions between the hand and the manipulated object.” The paper’s central claim is therefore not only scale, but co-occurrence: aligned depth, arm context, object interaction, dynamic lighting, and occlusion diversity are provided together.

3. Generation pipeline, co-training, and empirical behavior

The generation pipeline uses SAPIEN’s ray-tracing renderer “to better capture realistic shading, cast shadows, and specular effects” (Si et al., 26 Mar 2026). Foregrounds are built from a MANO hand mesh with sampled shape $\beta$ and pose, a Handy-based texture with controlled hue and saturation perturbations, and a textured attached forearm. Backgrounds are either HDR environment maps or randomly cropped high-resolution patches from MIT Indoor, with foreground-background consistency enforced by correlating scene-light color statistics with background color statistics. Two views per scene are rendered from independently sampled camera poses; appendix settings specify FOV sampled uniformly from $30^\circ$–$40^\circ$, Gaussian-mixture hand-camera distances with means $0.6/0.7/1.0$ m and standard deviation $0.1$ m, and “Views per scene: 2.”

Depth is generated by combining “accurate metric depths” for hand and forearm rendered in SAPIEN with background depth estimated using MoGe-2. The two are “directly fuse[d] … in camera space to obtain a dense depth image.” The paper explicitly cautions that this is “not a ‘perfect’ ground truth depth map due to differences in camera intrinsics between the rendered foreground and background, as well as noise in the estimated background depth”; accordingly, a foreground mask is stored so that losses can be restricted to valid regions.

The dataset is evaluated through co-training rather than architectural redesign. For both HaMeR and WiLoR, the protocol is to augment the original training corpus with “6.6M synthetic samples generated by our AnyHand pipeline … while keeping the model architecture and training hyper-parameters identical to the official setups.” On FreiHAND, HaMeR improves from PA-MPJPE 6.0 mm to 5.54 mm and from PA-MPVPE 5.7 mm to 5.24 mm, with F@5 rising from 0.785 to 0.811 and F@15 from 0.990 to 0.993. WiLoR with AnyHand reaches PA-MPJPE 5.3 mm, PA-MPVPE 5.0 mm, F@5 0.827, and F@15 0.994. On HO-3D v2, HaMeR improves from PA-MPJPE 7.7 mm to 7.47 mm and from PA-MPVPE 7.9 mm to “$7.6_{76}$”; WiLoR with AnyHand reaches PA-MPJPE “$7.3_{55}$,” PA-MPVPE “$7.6_{24}$,” AUC_j 0.853, AUC_v 0.848, F@5 0.649, and F@15 0.984. On the out-of-domain HO-Cap benchmark, HaMeR improves from PA-MPJPE 4.94 mm to 4.66 mm and WiLoR from 5.02 mm to 4.69 mm, while the paper remarks that “the performance gains from adding AnyHand are substantially larger than the differences between architectures.”

The same paper extends AnyHand to RGB-D via a lightweight depth fusion module with “dual embedding branches to tokenize RGB and depth, followed by a lightweight bidirectional cross-attention module that exchanges information between the two modalities at corresponding image patches.” The fused tokens are concatenated with task tokens and passed through the remaining transformer blocks in a WiLoR backbone. On HO-3D v2, compared with Keypoint-Fusion at STA-MPJPE 1.87 cm and PA-MPJPE 0.94 cm, the proposed RGB-D model reaches Real-only STA-MPJPE “$1.20_1$” and PA-MPJPE “$\beta$0,” Real + AnyHand STA-MPJPE “$\beta$1” and PA-MPJPE “$\beta$2,” and, with MoGe-2 estimated depth at evaluation time, STA-MPJPE “$\beta$3” and PA-MPJPE “$\beta$4.” The ablation “w/o RGB-D Cross Attention” is worse at STA-MPJPE “$\beta$5” and PA-MPJPE “$\beta$6.”

The paper also specifies its own limits. Scaling studies show diminishing returns beyond roughly 2–4M synthetic samples, although HO-Cap continues to improve with more synthetic data. Appendix ratio studies show that “training on AnyHand alone is insufficient,” so real data remains necessary and the optimal mixing ratio is “an open question.” The authors release the generation pipeline, but licensing, exact access instructions, file schemas, hardware, and training compute are not reported.

4. Hand-agnostic dexterous grasping across robotic hands

In "AnyDexGrasp: General Dexterous Grasping for Different Hands with Human-level Learning Efficiency," AnyHand refers to adaptation across robotic hands rather than visual hand estimation (Fang et al., 23 Feb 2025). The core claim is that dexterous grasping can be decomposed into a universal, hand-agnostic perception stage and a lightweight per-hand decision stage. Stage 1 learns a mapping from partial scene geometry $\beta$7, represented as a single-view partial point cloud $\beta$8, to a contact-centric grasp representation (CGR). Stage 2 learns, for each robotic hand $\beta$9, a compact decision model that scores hand-compatible grasp candidates derived from that CGR.

The CGR is defined over local polar slices:

$30^\circ$0

Here, $30^\circ$1 is radial distance to the surface at angle $30^\circ$2, and $30^\circ$3 is the angle between the surface normal and the inward approach direction. A derived antipodal metric is

$30^\circ$4

The universal mapping is written as $30^\circ$5, and the per-hand decision model instantiates $30^\circ$6 for each grasp type $30^\circ$7 of hand $30^\circ$8.

The implementation is correspondingly modular. The universal model uses a sparse 3D Minkowski Engine backbone, a point-wise graspness MLP that selects 1024 seeds, a view-wise graspness MLP over 300 approach directions per seed, cylinder grouping, and a final MLP that outputs CGR with $30^\circ$9 in-plane angles and $40^\circ$0 depths at $40^\circ$1 m. Training data are re-annotated GraspNet-1Billion scenes from 40 objects, but with dense CGR labels totaling over a billion CGRs. The per-hand decision model is a seven-layer MLP with 1024 hidden units per layer, batch normalization, ReLU, and one skip connection from the 2nd to the 5th layer; its input is the flattened CGR, $40^\circ$2 dimensions.

At inference time, the system captures RGB-D with an Intel RealSense D415 mounted on a UR5 end-effector about 60 cm above the table, forms a partial point cloud, predicts CGRs guided by graspness, keeps the top-$40^\circ$3 CGRs by antipodal scores, generates hand-dependent candidates, scores them, rejects collisions using Open3D voxel intersection, and executes the highest-scoring collision-free grasp with a waypoint 10 cm behind the approach direction. The reported latency is about 0.5 s to generate 200 grasps, while CPU collision checking takes about 20 s.

Empirically, the system is evaluated on three hands: DH-3, Allegro, and Inspire. With full training data from 144 objects, average success on daily objects is 97% for DH-3, 78% for Allegro, and 83% for Inspire, versus 66%, 51%, and 58% for a parallel-alignment-plus-collision baseline. On adversarial objects, the corresponding success rates are 99%, 82%, and 79%, versus 72%, 54%, and 59%. Across 150 novel objects, average success ranges from roughly 80% to 98% depending on the hand. Under the reduced regime of 40 objects and 100 attempts per type, DH-3 reaches 94.5%, Allegro 75%, and Inspire 77%; with only 50 trials per type, DH-3 still reaches 93.1%. The accompanying geometry-coverage analysis argues that dense local geometry sampling per object matters more than simply increasing the number of training objects at a fixed label budget. A plausible implication is that the AnyHand idea in robotics is not universality through one end-to-end policy, but universality through a geometry representation that is independent of hand kinematics.

5. Single-stage reconstruction of any number of hands

In "End-to-end Weakly-supervised Single-stage Multiple 3D Hand Mesh Reconstruction from a Single RGB Image," the AnyHand concept denotes support for any number of hands from a single forward pass (Ren et al., 2022). The method, often referred to as SMHR, replaces the conventional detect-crop-regress pipeline with a single-stage encoder-decoder. A ResNet-50 encoder produces a shared stride-8 feature map of shape $40^\circ$4, and a multi-head auto-encoder predicts, at every spatial cell, a code vector $40^\circ$5 decomposed into center confidence $40^\circ$6, hand type $40^\circ$7, keypoint heatmaps $40^\circ$8, MANO parameters $40^\circ$9, texture $0.6/0.7/1.0$0, and lighting $0.6/0.7/1.0$1. Center peaks are extracted by local-max detection and optional radius-based NMS; each retained peak indexes a complete hand instance, so association is implicit rather than solved with Hungarian matching.

The 3D hand model is MANO, with

$0.6/0.7/1.0$2

and the camera model uses shared perspective intrinsics

$0.6/0.7/1.0$3

with $0.6/0.7/1.0$4 and $0.6/0.7/1.0$5, $0.6/0.7/1.0$6 in the multi-hand setting. Weak supervision is made feasible through stage-wise training: first, single-hand pretraining of the reconstruction head; then joint multi-hand training on full images. The total objective is

$0.6/0.7/1.0$7

combining focal-style localization losses, reprojection and bone-direction terms, masked photometric loss rendered through PyTorch3D, pose and shape regularization, and a global-local consistency term that forces agreement between keypoints estimated from local heatmaps and reprojected joints from global features.

Since large real multi-hand 3D datasets are scarce, the paper constructs a synthetic multi-hand dataset from FreiHAND, HO3D, and RHD. A full single-hand image provides scene context, while up to nine additional hand patches are pasted using masks and affine transforms; FreiHAND and HO3D are horizontally flipped to create left hands. The network thus learns occlusion, hand count variation, and left-right classification without requiring native multi-hand 3D capture. In practice, the system processes the top-$0.6/0.7/1.0$8 peaks, with examples such as $0.6/0.7/1.0$9.

Quantitatively, the weakly supervised model reaches on FreiHAND MPJPE 1.07 cm, AUC_J 0.788, MPVPE 1.10 cm, AUC_V 0.782, F_5 0.500, and F_15 0.937; the fully supervised version reaches MPJPE 0.80 cm, AUC_J 0.840, MPVPE 0.81 cm, AUC_V 0.839, F_5 0.649, and F_15 0.966. On HO3D, the weakly supervised model reaches MPJPE 1.03 cm and MPVPE 1.01 cm, while the fully supervised model reaches MPJPE 1.01 cm and MPVPE 0.97 cm. On RHD, it reports EPE 2.07 cm without ground-truth scale or hand type and left/right classification accuracy 97.65%. On the synthesized multi-hand benchmark, it achieves 2D error 7.41 px and MPJPE 0.95 cm, outperforming S2HAND plus detector and S2HAND with GT boxes. Runtime is 11.8 ms per image at $0.1$0 and 36.5 ms at $0.1$1, with cost described as independent of hand count because there is no per-hand re-encoding. The stated limitations are ambiguity in depth under 2D-only supervision, extreme occlusions, motion blur, unusual textures or lighting, and slight absolute-scale error when true camera intrinsics are unavailable.

6. Joint-driven 3DGS hand avatars

In "JGHand: Joint-Driven Animatable Hand Avater via 3D Gaussian Splatting," the AnyHand idea is a rendering and retargeting capability rather than a recognition capability (Sun et al., 31 Jan 2025). JGHand defines a canonical hand from MANO mean pose and shape, then represents appearance and geometry with 3D Gaussians parameterized by center position $0.1$2, opacity $0.1$3, color $0.1$4, and isotropic scale $0.1$5. Identity-dependent variation comes from a trainable triplane queried at canonical $0.1$6 coordinates, while pose-dependent variation comes from a non-rigid offset driven by encoded abduction and flexion angles. The system is driven only by $0.1$7 keypoints.

Its key mathematical contribution is a differentiable, zero-error skeleton transformation from a canonical skeleton $0.1$8 to a target skeleton $0.1$9:

$7.6_{76}$0

Gaussian centers are then transformed by linear blend skinning with Fast-SNARF weights,

$7.6_{76}$1

Rendering follows 3DGS front-to-back compositing,

$7.6_{76}$2

and JGHand also blends a differentiable depth map to support a per-pixel, screen-space self-shadow mask with $7.6_{76}$3 samples. The paper emphasizes that this shadowing layer is lightweight, GPU-friendly, and compatible with real-time operation.

Training uses

$7.6_{76}$4

with $7.6_{76}$5, $7.6_{76}$6, $7.6_{76}$7, $7.6_{76}$8, $7.6_{76}$9, and $7.3_{55}$0. Experiments use InterHand2.6M and HandCo, Fast-SNARF-based weights, canonical Gaussian sampling inside the MANO mesh at $7.3_{55}$1 samples per bone, and training for 30 epochs on a single RTX 3090.

The reported skeleton transformation accuracy is exact in MPJPE terms: MPJPE 0 on InterHand2.6M test/Capture0 and test/Capture1, versus HALO at 0.0103 and 0.0120, with Chamfer 0.92 and 0.97 versus 5.14 and 5.97. Rendering quality on InterHand2.6M test/Capture0 is SSIM 0.966, PSNR 33.44, and LPIPS 0.032; on test/Capture1, 0.966, 32.44, and 0.032; on val/Capture0, 0.969, 33.41, and 0.031; and on HandCo/0191, 0.974, 33.89, and 0.018. Inference speed is 0.040 s per image, approximately 25 FPS, compared with 0.087 s for LiveHand and 4.317 s for HandAvatar. Ablations show that removing the transformation drops performance to 0.923 / 27.04 / 0.097 on InterHand test/Capture0, and removing the shadow module drops it to 0.954 / 30.92 / 0.043. The paper’s AnyHand interpretation is therefore retargetable, real-time, photorealistic hand rendering from 3D joints alone, without requiring morphable-model parameters at inference.

Taken together, these strands give AnyHand a broad but coherent technical meaning. In hand pose estimation, it denotes synthetic RGB-D scale with explicit arm context and occlusion coverage. In robotic manipulation, it denotes decoupling hand-agnostic perception from hand-specific decision models. In single-image reconstruction, it denotes one-pass processing of variable hand count. In neural rendering, it denotes joint-driven retargetability across subjects and poses. The common pattern is explicit geometry combined with a mechanism for removing a previously task-specific constraint.