---
title: 'ViDiHand: 4D Two-Hand Pose Reconstruction'
url: https://www.emergentmind.com/topics/vidihand
type: topic
---

# ViDiHand: 4D Two-Hand Pose Reconstruction

ViDiHand is a feed-forward pipeline for reconstructing metric-scale 4D two-hand pose directly from egocentric video, leveraging representations from a pretrained video diffusion model. It addresses key limitations of previous hand motion reconstruction approaches—namely, the reliance on unreliable per-frame hand detectors and the narrow supervision provided by scarce hand-pose annotations in video-based methods. ViDiHand offers a detector-free framework that encodes temporal dynamics, occlusion reasoning, and hand-object interaction by adapting large-scale video generative priors through a specialized hand-overlay rendering objective and decodes temporally coherent bimanual mesh pose via a dual-branch architecture [2606.30308].

## 1. Backbone Architecture and Feature Adaptation

ViDiHand builds on Wan2.1-VACE, a 1.3B-parameter video diffusion transformer (DiT) equipped with a flow-matching VACE branch. The core differentiator is the targeted adaptation of VACE: rather than freezing the diffusion model as a generic feature extractor, only the flow-matching branch is fine-tuned to reconstruct hand motion, allowing the model to retain complex priors over world, motion, and occlusion structure.

Adaptation proceeds via a two-stage "hand-overlay" rendering strategy over video:

- **Stage 1a (Joint-skeleton overlays):** In each frame, semi-transparent 2D hand joint sticks (derived from the MANO mesh) are rendered onto egocentric video. VACE is trained to reproduce the entire overlaid clip under a flow-matching loss, with no explicit supervision of MANO parameters.
- **Stage 1b (Full MANO-mesh overlays):** The procedure is repeated with full MANO hand mesh overlays, aligning the adapted diffusion features with the space from which the hand pose will later be decoded.

For a noisy video clip $x_0$ and a partially denoised latent at time $\tau$, $x_\tau$, the VACE branch $f_{\rm VACE}$ is optimized as:

\[
\mathcal{L}_{\rm VACE} = \mathbb{E}_{\tau,x_0,\epsilon}\,\left\|\epsilon - f_{\rm VACE}(x_\tau,\tau)\right\|^2,
\]

with the ground-truth $x_0$ replaced by the hand-overlay composition. The base DiT backbone remains frozen throughout adaptation.

After adaptation, mid-level transformer activations at block $L^*=15$ (of 30), at denoising step $\tau^*\approx0.7$, are extracted as latent-frame features $\mathbf F = \{F_\ell\}_{\ell=1}^{F_{\rm lat}}$ with $F_\ell\in\mathbb R^{C\times H_{\rm pat}\times W_{\rm pat}}$, where $F_{\rm lat}=21$ and $C=1536$.

## 2. Dual-Branch Decoder for Two-Hand Pose

ViDiHand employs a dual-branch decoder to map the adapted diffusion features to metric-scale 4D hand pose:

- **Spatio-temporal Tokenization:** Each $F_\ell$ is flattened and linearly projected to $h=512$, with three positional embeddings added: spatial $P^{\rm sp}$, temporal $P^{\rm tmp}_\ell$, and a ray-space encoding $g_{\rm ray}(\Gamma(K))$ driven by per-clip intrinsics $K$:

\[
X_\ell = \mathrm{LN}(W_F\,\mathrm{vec}(F_\ell)) + P^{\rm sp} + P^{\rm tmp}_\ell + g_{\rm ray}(\Gamma(K)).
\]

- **Hand-token Branch:** Two slot queries $Q\in\mathbb{R}^{2\times h}$ are cross-attended to all $X_\ell$ to generate per-hand embeddings $Q^{\rm hand}_\ell$.
- **Joint-Heatmap Branch:** A $1\times1$ convolution predicts $J=21$ per-hand heatmaps $\mathcal H_\ell$, which are spatially softmaxed to yield per-joint attention maps $A_\ell$ and initial 2D joint anchors $\widehat{\mathbf P}_\ell^{\rm init}$.
- **Mutual Cross-Attention Fusion:** A layer of bidirectional cross-attention fuses information between branches:

\[
\begin{aligned}
\widetilde Q^{\rm hand}_\ell
  &= \mathrm{LN}(Q^{\rm hand}_\ell + \mathrm{MHA}(Q^{\rm hand}_\ell,Q^{\rm joint}_\ell,Q^{\rm joint}_\ell)),\\
\widetilde Q^{\rm joint}_\ell
  &= \mathrm{LN}(Q^{\rm joint}_\ell + \mathrm{MHA}(Q^{\rm joint}_\ell,Q^{\rm hand}_\ell,Q^{\rm hand}_\ell)).
\end{aligned}
\]

- **Mixed-Projection Head:** From $\widetilde Q^{\rm hand}$, an MLP regresses MANO parameters $(\widehat R,\,\widehat\Theta,\,\widehat B,\,\hat\zeta,\,\hat e)$, with $\hat t^z = \exp(\hat\zeta)$. From $\widetilde Q^{\rm joint}$, 2D offsets $\Delta\mathbf p$ refine the joint anchors. The MANO forward process recovers canonical joints, to which $t^z$ is added, and in-plane translation $(\hat t^x,\hat t^y)$ is solved by closed-form weighted least-squares via pinhole projection:

\[
\hat u_j = f_x\frac{X^{\rm can}_j+\hat t^x}{Z^{\rm can}_j+\hat t^z} + c_x, \quad
\hat v_j = f_y\frac{Y^{\rm can}_j+\hat t^y}{Z^{\rm can}_j+\hat t^z} + c_y.
\]

Final output is full camera translation $\widehat{\mathbf t}$ and 3D joints $\widehat{\mathbf J}$.

## 3. Training Procedure and Data

ViDiHand is trained in three stages with different datasets and objectives:

- **Stage 1a:**
  - Dataset: 913,000 EgoDex clips (joint-only egocentric data).
  - Task: flow-matching loss on joint-skeleton overlays.
  - Optimizer: AdamW, learning rate $1\times10^{-4}$ with cosine decay, 500-step warmup.
  - Steps: 25,000 steps on 32 A100 GPUs, batch size 1, bfloat16.

- **Stage 1b:**
  - Dataset: ARCTIC (2,145 clips) and HOT3D (5,432 clips), fitted MANO mesh overlays.
  - Task: flow-matching loss on full MANO mesh overlays.
  - Same optimizer and schedule as Stage 1a.
  - Steps: 10,000 steps on 8 GPUs.

- **Stage 2:**
  - Training the decoder on cached Stage 1b features from ARCTIC and HOT3D.
  - Batch size: 16.
  - Optimizer: AdamW, learning rate $2\times10^{-4}$ with 200-step warmup.
  - Loss: summed over 10 terms (orientation, pose, shape, trans, 3D-joint L1, 2D reprojection, heatmap-2D, visibility, acceleration smoothness, shape consistency).
  - Steps: 30,000 steps on 8 GPUs.

HOI4D is reserved for cross-dataset evaluation.

## 4. Quantitative Evaluation

ViDiHand establishes superior performance across bimanual, single-hand, and challenging in-the-wild benchmarks under a penalty protocol that incorporates false negatives.

| Benchmark | FAcc | MPJPE-p (mm) | F1 | Jitter (mm/frame²) |
|-----------|------|--------------|----|--------------------|
| ARCTIC    | 0.997 (WiLoR: 0.919) | 21.67 (WildHands 25.70; OmniHands 29.67) | 0.999 | 3.18 |
| HOT3D     | 0.948 (WiLoR: 0.827) | 21.51 (WiLoR 44.83; WildHands 52.79) | 0.983 | 3.74 |
| HOI4D     | 0.984 (WiLoR: 0.962) | 30.09 (WiLoR 41.60) | -- | 4.01 |

Further ablations highlight the criticality of mid-level diffusion features ($L=15$, $\tau=0.7$), backbone adaptation (joint+mesh overlays), and dual-branch decoder components for low error and jitter. For example, omitting the Heatmap Branch increases EPE-p by 2.98px; removing the mixed-projection head raises EPE-p by 4.33px.

## 5. Robustness, Temporal Coherence, and Generalization

ViDiHand demonstrates several qualitative advantages:

- **Occlusion Handling:** Tracks fully occluded hands (e.g., behind objects) by hallucinating plausible finger poses, overcoming the failure modes of detector-based or frame-centric approaches.
- **Motion Dynamics:** The inherited video prior and acceleration smoothness promote temporally coherent, low-jitter trajectories; reported jitter is ~3 mm/frame², contrasting with ~40 mm/frame² for framewise methods.
- **Hand-Object Interaction:** Maintains accurate handedness and suppresses spurious detections; e.g., for HOI4D with predominantly single-hand activity, baseline F1 for the absent side drops to ~0.4 while ViDiHand remains at ≥0.98.
- **In-the-Wild Generalization:** Retains robustness to fisheye distortion, high dynamic range, extreme viewpoints, and unconstrained internet videos. The adapted diffusion features produce plausible hand meshes where discriminative baselines may fail.

## 6. Significance and Applicability

By harnessing video diffusion representations trained at internet scale and specializing them through a simple overlay pretext, ViDiHand unifies detector-free hand tracking, temporal smoothing, occlusion inference, and calibrated metric reconstruction within a single pipeline. Its consistent state-of-the-art performance on ARCTIC, HOT3D, and HOI4D indicates that adapted video diffusion models can serve as a powerful foundation for data-efficient, scalable 4D hand motion capture. This suggests broad applicability to in-the-wild embodied AI data collection and downstream manipulation tasks, where robust hand-object interaction reasoning under occlusion remains a central challenge [2606.30308].

Source: https://www.emergentmind.com/topics/vidihand