DreamHand: Repurposing Video Diffusion Models for Occlusion-Robust Egocentric 3D Hand Motion Recovery

Abstract: Egocentric video offers scalable manipulation data for embodied AI, yet recovering metric 3D hand trajectories remains challenging due to severe object occlusion and frequent out-of-sight gaps. Existing single-frame and windowed temporal regressors fail when hand shortly leaves the frame, while recent video diffusion models (VDMs) rely on heavy, stochastic multi-step sampling as pixel-space renderers. We instead repurpose VDM into a deterministic geometry encoder. A single forward pass over the clean latent exposes scene content beyond current observations, including occluded and out-of-sight hands. We introduce DreamHand, an offline clip-level framework that extracts features via a Deterministic Clean-Latent Encoder and decodes them with a Bidirectional Spatiotemporal Decoder. DreamHand recovers continuous bimanual trajectories with metric placement and no external detector, while a Ray-Based Camera Solver supports a second configuration that needs no test-time camera intrinsics. Across five egocentric benchmarks, DreamHand sets a new state of the art, cutting MPJPE-p by 30% on occlusion-heavy ARCTIC and 40% on HOT3D. These gains reach 46%-61% once out-of-sight hands are included in the evaluation, offering a scalable path from everyday human video to robot manipulation data.

• The paper introduces DreamHand, a deterministic video-diffusion encoder with bidirectional spatiotemporal decoding that recovers metric 3D bimanual hand motion despite occlusion and out-of-sight intervals.
• DreamHand achieves state-of-the-art results on 45 of 48 primary comparisons, including 15.26 mm MPJPE-p on ARCTIC, 12.89 mm on HOT3D, and up to 61.1% lower out-of-sight error than ViDiHand.
• The method combines clean-latent diffusion features, LoRA adaptation, learned viewing rays, and mixed-PnP placement to reach 63.1 fps—about 33 times faster than the strongest prior method—while remaining offline and limited in absolute out-of-sight localization.

Overview

DreamHand addresses metric 3D bimanual hand motion recovery from egocentric RGB video, targeting the two failure modes that dominate this setting: hand–object interaction occlusion and hands leaving the field of view entirely. The paper's central methodological claim is that a pretrained video diffusion model (VDM) is more useful as a deterministic geometry encoder than as a generative renderer. Rather than running multi-step sampling to synthesize frames before estimating pose — the route taken by ViDiHand (Wang et al., 29 Jun 2026) — DreamHand performs a single noise-free forward pass over the clean VAE latent of a Wan 2.2 DiT backbone (Wan et al., 26 Mar 2025), taps intermediate features at block 15 of 30, and decodes them into frame-wise MANO parameters with a lightweight transformer decoder. Across five egocentric benchmarks, the framework reports state-of-the-art results on 45 of 48 primary metric comparisons, reducing MPJPE-p by 30% on ARCTIC and 40% on HOT3D relative to ViDiHand, while running at 63.1 fps versus 1.91 fps for the strongest prior method — a 33× throughput gap.

From generator to encoder

The architecture repurposes a rectified-flow-pretrained DiT strictly as an encoder. A frozen VAE compresses an 81-frame clip into 21 latent frames (4× temporal, 16× spatial compression), and the DiT runs once at $\sigma=0$, truncated at block $L^\star=15$. Bypassing the remaining blocks and the generation head halves per-pass computation; a tap-depth sweep shows accuracy saturates by mid-stack, with deeper taps buying at most 0.17 mm MPJPE-p at substantially higher cost. Only the patch embedding, LoRA adapters (rank 64, ~161M parameters across all blocks, though only those in executed blocks receive gradients), a Ray Head, and the decoder are trained; the 30 transformer blocks remain frozen.

Two design choices are validated by controlled ablations. First, reading the clean latent at $\sigma=0$ outperforms reading a half-noised latent on every metric, improving MPJPE-p by 2.40 mm and translation error by 39% — evidence that injected noise degrades geometric information without any compensating benefit, since no generative pass is run. Second, LoRA adaptation is essential: removing it raises MPJPE-p from 16.95 to 23.10 mm on ARCTIC.

Bidirectional spatiotemporal decoding

The decoder processes 48 queries per latent frame: two fixed left/right hand tokens, 42 joint tokens, and four register tokens. Patch tokens combine a learned spatial PE with a ray PE derived from the predicted viewing-ray field, Fourier-encoded and mapped through a zero-initialized MLP. Four layers alternate spatial cross-attention (per-frame detail extraction) with temporal self-attention across all latent frames, using rotary relative position encodings so no absolute clip length is baked into the weights. Attention runs at the downsampled latent rate and is interpolated back to video rate, reducing whole-clip attention cost to roughly 1/15 of full-resolution cost. No causal mask is applied: each frame conditions on both past and future context, which is what enables reconstruction rather than extrapolation during out-of-sight intervals.

Readout heads include a Joint Head producing 2D anchors via differentiable soft-argmax over cross-attention heatmaps, a Pose Head regressing Gram–Schmidt-orthogonalized 6D rotations, a Camera Head predicting log-depth $\hat t_z = \exp(\hat\zeta)$, and a Shape Head predicting one shape vector $\hat\beta$ per hand per clip by temporal pooling — enforcing the physical invariant that hand shape is constant within a recording. Existence and visibility scores are predicted directly, avoiding Hungarian matching.

Ray-based camera solving

Rather than taking intrinsics as input, a zero-initialized 1×1 convolution predicts a per-cell unit viewing-ray field supervised by a cosine-distance loss against rays unprojected under training calibration. This supports two configurations: standard DreamHand computes projection bearings from supplied intrinsics, while the $K$-free variant samples bearings from the predicted ray field, enabling test-time inference without calibration. Metric placement uses a mixed-PnP scheme: only optical depth is regressed, and in-plane translation $(t_x, t_y)$ is solved in closed-form weighted least squares against predicted 2D anchors, with a fallback to inverse-projected wrist placement when too few joints vote or residuals are excessive.

Notably, the $K$-free variant matches the standard configuration's wrist-aligned pose almost exactly (15.264 vs. 15.256 mm MPJPE-p on ARCTIC) but loses coverage at the image periphery: in the outermost radial bin its translation error rises to 0.165 m versus 0.048 m on ARCTIC and 0.190 m versus 0.035 m on HOT3D, degrading Frame Accuracy from 0.986 to 0.752 on wide-angle HOT3D. On matched detections the two configurations tie, confirming the gap reflects lost peripheral coverage rather than degraded articulation.

Main results

Under a unified evaluation protocol with a false-negative penalization scheme (missed hands charged a canonical MANO mesh), DreamHand leads on 27 of 29 metrics across ARCTIC, HOT3D, and held-out HOI4D. Relative to ViDiHand, MPJPE-p drops from 21.67 to 15.26 mm on ARCTIC, 21.51 to 12.89 mm on HOT3D, and 30.09 to 23.03 mm on zero-shot HOI4D. Detection is near saturation (F1 = 1.000 on ARCTIC), whereas seven baselines fall below 0.70 FAcc on HOT3D.

The most consequential result concerns out-of-sight hands. Once out-of-sight ground truth enters evaluation, DreamHand achieves 16.8 and 17.3 mm MPJPE\textsuperscript{+OOS} on ARCTIC and HOT3D, reductions of 45.9% and 61.1% over ViDiHand. Restricted to the out-of-sight stratum alone, the margin is larger still: 35.2 and 38.6 mm versus 126–152 mm for every baseline, which collapse into a narrow band regardless of their in-view ranking. The paper is careful about what this certifies: both OOS metrics are wrist-aligned, so they establish articulation and orientation continuity through visual gaps, not absolute placement, which depends on regressed depth and bearings unconstrained by image evidence. No benchmark in the paper scores absolute out-of-sight localization.

On H2O and OakInk2 (appendix results), DreamHand cuts best-baseline MPJPE-p by 43% and 66% respectively, though the authors flag that these are in-domain comparisons against baselines trained elsewhere, and H2O labels are partially pseudo-ground-truth that the model itself trained on.

Ablations and analysis

The feature-source ablation isolates the value of generative VDM representations. Substituting V-JEPA 2 or VideoMAE features yields reasonable in-view accuracy (V-JEPA 2 at 17.81 mm MPJPE-p) but sharply worse out-of-sight error (66.1 mm versus 41.5 mm for DreamHand). A matched frozen-to-frozen comparison (V-JEPA 2 at 66.1 mm versus Wan without LoRA at 50.1 mm) points the same way, supporting the claim that generative pretraining contributes a temporal-geometric prior — plausibly object permanence — that discriminative self-supervision does not. Raw RGB and VAE latents fare far worse (46.5 and 32.5 mm).

Decoder ablations show rotary relative PE and joint-level queries are pivotal: swapping to absolute PE costs 2.52 mm MPJPE-p, and pooling joint queries adds 1.88 mm. Solver ablations confirm mixed-PnP stabilizes metric placement relative to point-wise inverse projection (CT-p 0.020 vs. 0.030 m), though direct regression returns slightly lower Jitter at the cost of image-space placement — a trade-off the authors decline.

A length sweep shows accuracy is flat past the 81-frame training window and degrades gently to 17.99 mm when entire recordings (7–13× the window) are decoded in one pass, with Jitter improving 4% — rotary PE genuinely removes sequence-length constraints, though long single passes trade accuracy for smoothness.

Limitations and open questions

The paper states several limitations plainly. The predicted ray field generalizes best within camera families seen during training, and the $K$-free configuration's border failure follows directly. Two of the five benchmarks rely partially on pseudo-ground-truth annotations, and split protocols are uneven: only ARCTIC and H2O are subject-disjoint, so subject overlap cannot be excluded on HOT3D and OakInk2. The out-of-sight gain is wrist-aligned and does not certify metric localization during gaps. Finally, the method is offline by construction — latency is bounded below by clip length, making it suitable for data curation rather than closed-loop control. Three specific questions remain open: how much of the accuracy depends on future-frame context (answerable by distilling a causal variant), whether identity and shape can be carried across multi-minute recordings beyond a single pass, and whether recovered trajectories improve downstream robot policy learning, which the retargeting demonstrations do not establish.

Conclusion

DreamHand demonstrates that a VDM read deterministically at the clean latent, adapted end-to-end with 3D supervision, provides features whose advantage concentrates precisely where egocentric hand recovery is hardest — under occlusion and complete absence from view — at a fraction of the cost of generative sampling. The combination of clip-level bidirectional decoding, a learned ray field enabling intrinsics-free operation, and mixed-PnP metric placement yields consistent state-of-the-art results across five benchmarks with a 33× speed advantage over the strongest prior method. The principal open issues are streaming deployment, out-of-sight metric localization, and validation of downstream utility for manipulation data curation.