ViDiHand: 4D Two-Hand Pose Reconstruction

• The paper introduces ViDiHand, a novel detector-free pipeline that reconstructs temporally coherent, metric-scale 4D two-hand poses from egocentric video using adapted video diffusion priors.
• A dual-branch decoder fuses spatio-temporal features to regress MANO parameters and joint heatmaps, achieving low error and minimal jitter across multiple benchmarks.
• The approach effectively handles occlusions and hand-object interactions through a two-stage hand-overlay rendering strategy and a flow-matching loss during training.

ViDiHand is a feed-forward pipeline for reconstructing metric-scale 4D two-hand pose directly from egocentric video, leveraging representations from a pretrained video diffusion model. It addresses key limitations of previous hand motion reconstruction approaches—namely, the reliance on unreliable per-frame hand detectors and the narrow supervision provided by scarce hand-pose annotations in video-based methods. ViDiHand offers a detector-free framework that encodes temporal dynamics, occlusion reasoning, and hand-object interaction by adapting large-scale video generative priors through a specialized hand-overlay rendering objective and decodes temporally coherent bimanual mesh pose via a dual-branch architecture (Wang et al., 29 Jun 2026).

1. Backbone Architecture and Feature Adaptation

ViDiHand builds on Wan2.1-VACE, a 1.3B-parameter video diffusion transformer (DiT) equipped with a flow-matching VACE branch. The core differentiator is the targeted adaptation of VACE: rather than freezing the diffusion model as a generic feature extractor, only the flow-matching branch is fine-tuned to reconstruct hand motion, allowing the model to retain complex priors over world, motion, and occlusion structure.

Adaptation proceeds via a two-stage "hand-overlay" rendering strategy over video:

• Stage 1a (Joint-skeleton overlays): In each frame, semi-transparent 2D hand joint sticks (derived from the MANO mesh) are rendered onto egocentric video. VACE is trained to reproduce the entire overlaid clip under a flow-matching loss, with no explicit supervision of MANO parameters.
• Stage 1b (Full MANO-mesh overlays): The procedure is repeated with full MANO hand mesh overlays, aligning the adapted diffusion features with the space from which the hand pose will later be decoded.

For a noisy video clip $x_0$ and a partially denoised latent at time $\tau$, $x_\tau$, the VACE branch $f_{\rm VACE}$ is optimized as:

$$\mathcal{L}_{\rm VACE} = \mathbb{E}_{\tau,x_0,\epsilon}\,\left\|\epsilon - f_{\rm VACE}(x_\tau,\tau)\right\|^2,$$

with the ground-truth $x_0$ replaced by the hand-overlay composition. The base DiT backbone remains frozen throughout adaptation.

After adaptation, mid-level transformer activations at block $L^*=15$ (of 30), at denoising step $\tau^*\approx0.7$, are extracted as latent-frame features $\mathbf F = \{F_\ell\}_{\ell=1}^{F_{\rm lat}}$ with $$F_\ell\in\mathbb R^{C\times H_{\rm pat}\times W_{\rm pat}}$$, where $\tau$0 and $\tau$1.

2. Dual-Branch Decoder for Two-Hand Pose

ViDiHand employs a dual-branch decoder to map the adapted diffusion features to metric-scale 4D hand pose:

• Spatio-temporal Tokenization: Each $\tau$2 is flattened and linearly projected to $\tau$3, with three positional embeddings added: spatial $\tau$4, temporal $\tau$5, and a ray-space encoding $\tau$6 driven by per-clip intrinsics $\tau$7:

$\tau$8

• Hand-token Branch: Two slot queries $\tau$9 are cross-attended to all $x_\tau$0 to generate per-hand embeddings $x_\tau$1.
• Joint-Heatmap Branch: A $x_\tau$2 convolution predicts $x_\tau$3 per-hand heatmaps $x_\tau$4, which are spatially softmaxed to yield per-joint attention maps $x_\tau$5 and initial 2D joint anchors $x_\tau$6.
• Mutual Cross-Attention Fusion: A layer of bidirectional cross-attention fuses information between branches:

$x_\tau$7

• Mixed-Projection Head: From $x_\tau$8, an MLP regresses MANO parameters $x_\tau$9, with $f_{\rm VACE}$0. From $f_{\rm VACE}$1, 2D offsets $f_{\rm VACE}$2 refine the joint anchors. The MANO forward process recovers canonical joints, to which $f_{\rm VACE}$3 is added, and in-plane translation $f_{\rm VACE}$4 is solved by closed-form weighted least-squares via pinhole projection:

$f_{\rm VACE}$5

Final output is full camera translation $f_{\rm VACE}$6 and 3D joints $f_{\rm VACE}$7.

3. Training Procedure and Data

ViDiHand is trained in three stages with different datasets and objectives:

• Stage 1a:
  • Dataset: 913,000 EgoDex clips (joint-only egocentric data).
  • Task: flow-matching loss on joint-skeleton overlays.
  • Optimizer: AdamW, learning rate $f_{\rm VACE}$8 with cosine decay, 500-step warmup.
  • Steps: 25,000 steps on 32 A100 GPUs, batch size 1, bfloat16.
• Stage 1b:
  • Dataset: ARCTIC (2,145 clips) and HOT3D (5,432 clips), fitted MANO mesh overlays.
  • Task: flow-matching loss on full MANO mesh overlays.
  • Same optimizer and schedule as Stage 1a.
  • Steps: 10,000 steps on 8 GPUs.
• Stage 2:
  • Training the decoder on cached Stage 1b features from ARCTIC and HOT3D.
  • Batch size: 16.
  • Optimizer: AdamW, learning rate $f_{\rm VACE}$9 with 200-step warmup.
  • Loss: summed over 10 terms (orientation, pose, shape, trans, 3D-joint L1, 2D reprojection, heatmap-2D, visibility, acceleration smoothness, shape consistency).
  • Steps: 30,000 steps on 8 GPUs.

HOI4D is reserved for cross-dataset evaluation.

4. Quantitative Evaluation

ViDiHand establishes superior performance across bimanual, single-hand, and challenging in-the-wild benchmarks under a penalty protocol that incorporates false negatives.

Benchmark	FAcc	MPJPE-p (mm)	F1	Jitter (mm/frame²)
ARCTIC	0.997 (WiLoR: 0.919)	21.67 (WildHands 25.70; OmniHands 29.67)	0.999	3.18
HOT3D	0.948 (WiLoR: 0.827)	21.51 (WiLoR 44.83; WildHands 52.79)	0.983	3.74
HOI4D	0.984 (WiLoR: 0.962)	30.09 (WiLoR 41.60)	--	4.01

Further ablations highlight the criticality of mid-level diffusion features ($$\mathcal{L}_{\rm VACE} = \mathbb{E}_{\tau,x_0,\epsilon}\,\left\|\epsilon - f_{\rm VACE}(x_\tau,\tau)\right\|^2,$$0, $$\mathcal{L}_{\rm VACE} = \mathbb{E}_{\tau,x_0,\epsilon}\,\left\|\epsilon - f_{\rm VACE}(x_\tau,\tau)\right\|^2,$$1), backbone adaptation (joint+mesh overlays), and dual-branch decoder components for low error and jitter. For example, omitting the Heatmap Branch increases EPE-p by 2.98px; removing the mixed-projection head raises EPE-p by 4.33px.

5. Robustness, Temporal Coherence, and Generalization

ViDiHand demonstrates several qualitative advantages:

• Occlusion Handling: Tracks fully occluded hands (e.g., behind objects) by hallucinating plausible finger poses, overcoming the failure modes of detector-based or frame-centric approaches.
• Motion Dynamics: The inherited video prior and acceleration smoothness promote temporally coherent, low-jitter trajectories; reported jitter is ~3 mm/frame², contrasting with ~40 mm/frame² for framewise methods.
• Hand-Object Interaction: Maintains accurate handedness and suppresses spurious detections; e.g., for HOI4D with predominantly single-hand activity, baseline F1 for the absent side drops to ~0.4 while ViDiHand remains at ≥0.98.
• In-the-Wild Generalization: Retains robustness to fisheye distortion, high dynamic range, extreme viewpoints, and unconstrained internet videos. The adapted diffusion features produce plausible hand meshes where discriminative baselines may fail.

6. Significance and Applicability

By harnessing video diffusion representations trained at internet scale and specializing them through a simple overlay pretext, ViDiHand unifies detector-free hand tracking, temporal smoothing, occlusion inference, and calibrated metric reconstruction within a single pipeline. Its consistent state-of-the-art performance on ARCTIC, HOT3D, and HOI4D indicates that adapted video diffusion models can serve as a powerful foundation for data-efficient, scalable 4D hand motion capture. This suggests broad applicability to in-the-wild embodied AI data collection and downstream manipulation tasks, where robust hand-object interaction reasoning under occlusion remains a central challenge (Wang et al., 29 Jun 2026).