WiLoR for MANO Hand Parameters

• The paper presents a unified framework that leverages MANO parameter regression combined with transformer-based refinement for accurate, jitter-free 3D hand tracking.
• It integrates an anchor-free detection branch with a Vision Transformer encoder and multi-scale refinement, ensuring robust performance in multi-hand scenarios.
• The methodology is validated on large-scale, in-the-wild datasets, achieving state-of-the-art results in both 2D and 3D hand benchmarks.

WiLoR constitutes a unified, end-to-end framework for 3D hand localization and reconstruction from images in unconstrained settings, relying on the regression of MANO model parameters. Distinct from prior approaches, WiLoR integrates a high-performance anchor-free hand detector with a transformer-based parameter regressor, enabling real-time accurate inference for multi-hand scenarios under a wide range of real-world imaging conditions. The pipeline was designed to address persistent challenges in robust detection and jitter-free 3D hand tracking, leveraging a large-scale in-the-wild dataset and an integrated architecture to deliver state-of-the-art results (Potamias et al., 2024).

1. MANO Hand Model Representation

At the core of WiLoR is the MANO hand model, a parametric mesh representation with $N=778$ vertices defined as a differentiable function of low-dimensional shape $\beta\in\mathbb{R}^{10}$ and pose $\theta\in\mathbb{R}^{45}$, denoted $V = M(\beta, \theta)$. The generative sequence follows:

• Shape Deformation: The rest-pose template $\bar{T}\in\mathbb{R}^{N\times 3}$ is augmented by linear shape blend shapes $S\in\mathbb{R}^{N\times 3\times 10}$, generating $$T_{\mathrm{shaped}}=\bar{T}+\sum_{i=1}^{10}S_i\beta_i$$.
• Joint Computation: The joint locations at rest-pose $J(\beta)=J_{\mathrm{reg}}T_{\mathrm{shaped}}$ utilize a linear regressor $J_{\mathrm{reg}}\in\mathbb{R}^{K\times N}$ (with $K=15$).
• Pose-Dependent Deformation and Skinned Mesh Assembly: Pose blend shapes $\beta\in\mathbb{R}^{10}$0 produce $\beta\in\mathbb{R}^{10}$1, with final mesh vertices via linear blend skinning: $\beta\in\mathbb{R}^{10}$2, where $\beta\in\mathbb{R}^{10}$3 encodes blending weights. This results in a mesh-based encoding of diverse hand shapes and poses suitable for differentiable learning pipelines.

2. Detection and Localization Branch Design

The initial phase of WiLoR adopts a fully convolutional, anchor-free detector for hand localization on full-frame images, producing bounding boxes $\beta\in\mathbb{R}^{10}$4 and handedness labels $\beta\in\mathbb{R}^{10}$5. Box extraction includes padding and resizing to a canonical $\beta\in\mathbb{R}^{10}$6 patch $\beta\in\mathbb{R}^{10}$7, with internal handedness canonicalization by flipping (all inputs standardized as right hands). Detection utilizes a DarkNet-53 backbone, a PANet FPN neck, and three detection heads. Supervisory signals comprise binary cross-entropy, distribution focal, complete-IoU, and keypoint losses: $\beta\in\mathbb{R}^{10}$8 Domain-randomized augmentations (rotations, translations, mosaic, mixup, masking) and large-scale training (WHIM dataset, 2M+ hand images) support robustness to occlusions, lighting, and scale.

3. Transformer-Based Parameter Regression and Multi-Scale Refinement

Each detected hand crop undergoes parameter regression via a Vision Transformer (ViT) encoder, mirroring the ViTPose architecture. The processing steps include:

• Patch Embedding: $\beta\in\mathbb{R}^{10}$9 patches are flattened and linearly embedded to produce token sequences $\theta\in\mathbb{R}^{45}$0.
• Positional and Slot Tokens: 2D positional embeddings are added; three learned slot tokens—$\theta\in\mathbb{R}^{45}$1, $\theta\in\mathbb{R}^{45}$2, $\theta\in\mathbb{R}^{45}$3—initialize regression for camera, pose, and shape, respectively.
• Transformer Encoding: All tokens are processed by 12 transformer encoder layers (multi-head self-attention).
• Coarse Parameter Regression: Updated token vectors are mapped by two-layer MLPs to produce coarse MANO parameters: $\theta\in\mathbb{R}^{45}$4, $\theta\in\mathbb{R}^{45}$5, and weak-perspective camera parameters $\theta\in\mathbb{R}^{45}$6, $\theta\in\mathbb{R}^{45}$7.

For improved alignment between mesh and image evidence, a multi-scale refinement module is deployed. It reshapes output token embeddings to a feature map $\theta\in\mathbb{R}^{45}$8, upscales hierarchically, and samples per-vertex features at image-projected mesh locations. Pooled feature vectors $\theta\in\mathbb{R}^{45}$9 inform MLP regression of residuals $V = M(\beta, \theta)$0, refining coarse parameters: $V = M(\beta, \theta)$1, $V = M(\beta, \theta)$2.

4. Training Losses and Supervision Protocols

WiLoR’s regression of MANO parameters is supervised using a sum of per-vertex mesh, keypoint, direct parameter, and adversarial losses: $V = M(\beta, \theta)$3 where:

• $V = M(\beta, \theta)$4, $V = M(\beta, \theta)$5 are ground-truth mesh and 2D keypoints,
• $V = M(\beta, \theta)$6 are ground-truth MANO parameters (when available),
• $V = M(\beta, \theta)$7 is a discriminator enforcing plausibility of predicted hand shapes and poses.

Supervision draws from a broad pool of datasets (FreiHAND, HO3D, RHD, MTC, InterHand2.6M, DEX-YCB, COCO WholeBody, ARCTIC, Hot3D, BEDLAM), with augmentations encompassing rotation, scaling, flipping, color jitter, and blur to maximize generalization.

5. Integration of Detection and Reconstruction Streams

WiLoR’s architecture links the full-image detection and parameter regression via a shared flow:

• The image $V = M(\beta, \theta)$8 is processed by the detector to yield hand hypotheses $V = M(\beta, \theta)$9.
• Each hand crop $\bar{T}\in\mathbb{R}^{N\times 3}$0 is standardized and passed through the transformer-based regressor independently (batched).
• Outputs $\bar{T}\in\mathbb{R}^{N\times 3}$1 specify the MANO mesh and camera for each hand.
• Meshes are reprojected to the original image frame using the camera’s weak perspective and mirrored to restore true handedness if required.
• The pipeline is designed for jitter-free and stable 4D reconstructions in monocular videos, without temporal postprocessing.

6. Implementation Details and Benchmarks

Detection branch training involves 300k iterations with a batch size of 64, SGD (momentum 0.9, weight-decay $\bar{T}\in\mathbb{R}^{N\times 3}$2, learning rate $\bar{T}\in\mathbb{R}^{N\times 3}$3 decayed by 0.1x every 100k steps). The reconstruction branch is initialized from ViTPose-B/16 pretraining, uses AdamW ($\bar{T}\in\mathbb{R}^{N\times 3}$4 weight-decay $\bar{T}\in\mathbb{R}^{N\times 3}$5), and is trained for 30 epochs with linear warmup and cosine decay. The combined pipeline outperforms previous methods on diverse 2D and 3D hand benchmarks, attributed to the explicit regression and refinement of MANO parameters from large scale, multi-source annotated data. WiLoR achieves stable, high-fidelity hand mesh reconstructions in challenging environments, emphasizing robustness and practical deployability (Potamias et al., 2024).