HandDreamer: Zero-Shot Text to 3D Hand Model Generation using Corrective Hand Shape Guidance

Abstract: The emergence of virtual reality has necessitated the generation of detailed and customizable 3D hand models for interaction in the virtual world. However, the current methods for 3D hand model generation are both expensive and cumbersome, offering very little customizability to the users. While recent advancements in zero-shot text-to-3D synthesis have enabled the generation of diverse and customizable 3D models using Score Distillation Sampling (SDS), they do not generalize very well to 3D hand model generation, resulting in unnatural hand structures, view-inconsistencies and loss of details. To address these limitations, we introduce HandDreamer, the first method for zero-shot 3D hand model generation from text prompts. Our findings suggest that view-inconsistencies in SDS is primarily caused due to the ambiguity in the probability landscape described by the text prompt, resulting in similar views converging to different modes of the distribution. This is particularly aggravated for hands due to the large variations in articulations and poses. To alleviate this, we propose to use MANO hand model based initialization and a hand skeleton guided diffusion process to provide a strong prior for the hand structure and to ensure view and pose consistency. Further, we propose a novel corrective hand shape guidance loss to ensure that all the views of the 3D hand model converges to view-consistent modes, without leading to geometric distortions. Extensive evaluations demonstrate the superiority of our method over the state-of-the-art methods, paving a new way forward in 3D hand model generation.

• The paper presents a novel pipeline that integrates a MANO-based initialization with text-driven synthesis to generate anatomically plausible 3D hand models.
• It introduces a Corrective Hand Shape (CHS) loss to dynamically enforce geometric consistency and reduce view artifacts during diffusion optimization.
• Experimental results show that HandDreamer outperforms existing methods in terms of CLIP alignment, FID scores, and user preferences for realistic and detailed hand renderings.

HandDreamer: Zero-Shot Text to 3D Hand Model Generation using Corrective Hand Shape Guidance

Introduction

The proliferation of VR/AR applications and digital avatars has led to an increasing demand for automated and detailed 3D hand model generation—an open challenge, given the geometric complexity and high articulation inherent to hands. Conventional methods require extensive multi-view captures and skilled artists, making hand model generation costly and inflexible. While advances in zero-shot text-to-3D synthesis, especially via Score Distillation Sampling (SDS), have enabled customizable 3D asset creation, existing solutions do not generalize well to hands, frequently producing "Janus artifacts", view-inconsistencies, and loss of geometric fidelity.

HandDreamer addresses these limitations by introducing a pipeline explicitly tailored to the unique challenges of text-conditioned 3D hand modeling. This framework leverages strong geometric priors (MANO model), skeleton-conditioned diffusion, and a novel Corrective Hand Shape (CHS) guidance loss, thereby mitigating the mode ambiguity associated with SDS and achieving superior detail and view consistency.

Limitations of Existing Text-to-3D Methods for Hands

Standard text-to-3D generative pipelines, typically based on SDS and 2D diffusion models, are troubled by the multi-modality of the score landscape—especially in highly-articulated, self-similar object classes such as hands. Empirical analysis reveals that random initialization within SDS for hands leads similar viewpoints to converge to divergent local minima—manifesting as severe view-inconsistencies (Janus effect) and physiologically implausible geometry.

Figure 1: Gradient behavior analysis for view-consistency—random initialization versus MANO prior—demonstrates how random starting points lead to inconsistent modes and artifacts for hand geometry.

The challenge is further compounded by self-occlusion and the combinatorial complexity of plausible finger poses, which produces ambiguous gradients and distributed optimization basins in the SDS trajectory. Previous efforts targeting mode collapse (e.g., ESD [modeCollapse]), and per-view conditioning, do not adequately resolve this class of artifacts for hands.

Methodology

Overview

HandDreamer decouples the 3D hand generation process into two primary stages: (1) geometric prior initialization via the MANO hand model, and (2) text-driven model synthesis with skeleton conditioning and CHS loss.

Figure 2: Schematic of the HandDreamer pipeline. Stage (a) employs MANO-driven opacity map initialization; stage (b) integrates skeleton and CHS-guided SDS for text-consistent geometry.

Stage 1: MANO-Based Hand Shape Initialization

Initialization is conducted by aligning the NeRF volume’s opacity maps with those from a precomputed MANO mesh in the target pose. This enforces semantic proximity between the learned volume and plausible hand shape distributions, drastically reducing the space of plausible geometric modes for subsequent optimization. The distance between the NeRF and MANO silhouette is explicitly minimized.

Stage 2: Skeleton-Conditioned Score Distillation and Corrective Hand Shape Guidance

The optimization proceeds by integrating the 2D projections of the 3D hand skeleton as conditioning signals into the diffusion process via ControlNet. This embeds both pose and camera viewpoint information, minimizing inter-view ambiguity. The SDS trajectory is then further regularized with the CHS loss, which dynamically enforces geometric consistency between the synthesized hand and the anatomically plausible MANO initialization, especially critical during the higher-noise (early) optimization iterations to combat mode drift.

The total loss is a weighted combination of:

• SDS loss, conditioned by text and hand skeleton;
• CHS loss, with annealed weighting to prioritize geometry early and texture late in training;
• Image and latent-space regularization losses to enforce high-frequency fidelity and sharp surface boundaries.

Experimental Results

Qualitative Assessment

Examples generated by HandDreamer exhibit natural hand structure, detailed skin texture, and marked geometric consistency across viewpoints, with the absence of Janus or other anatomical artifacts.

Figure 3: Qualitative samples from HandDreamer demonstrating high-fidelity, view-consistent hand geometry and intricate texture synthesis.

Comparative analysis against canonical text-to-3D models (e.g., ProlificDreamer, ESD, CFD) and text-to-human (e.g., DreamAvatar, HumanNorm) frameworks demonstrates those models' tendencies toward artifacted geometry, missing detail, or implausible finger counts, highlighting the efficacy of HandDreamer's geometric priors and CHS correction.

Figure 4: Visual comparison with SOTA; HandDreamer eliminates Janus artifacts and inconsistency present in alternatives and achieves hand detail loss mitigation.

When tested against the one-shot image-based OHTA pipeline, HandDreamer is shown to generalize robustly to arbitrary, fictional, and fine-grained hand models, whereas OHTA is limited by texture coverage and template blending.

Figure 5: OHTA fails to capture textural or shape diversity in arbitrary prompts, compared to HandDreamer's robust synthesis.

Quantitative Metrics

Objectively, HandDreamer outperforms all baselines across semantic alignment (CLIP L14), visual realism (FID), and human preference (HPSv2):

Method	CLIP L14 ↑	FID ↓	HPSv2 ↑
DreamFusion'22	25.12	344.19	0.187
Fantasia3D'23	20.93	329.31	0.198
DreamWaltz'23	23.96	265.11	0.222
HumanNorm'24	23.01	327.42	0.177
CFD'25	26.62	262.83	0.223
HandDreamer	28.63	254.62	0.241

This is further supported by user preference studies showing HandDreamer is consistently rated best for geometry, texture, and view consistency across 8 methods and 30 prompts.

Figure 6: User preference study indicates that HandDreamer is preferred substantially over all baselines for fidelity and realism.

Articulation and Downstream Usability

HandDreamer-generated meshes are fully compatible with downstream pipelines for animation and AR/VR integration. The mesh quality supports both direct parametric pose manipulation (via MANO parameters) and high-precision rigging for complex poses.

Figure 7: Articulation examples demonstrating the mesh and skeleton expressivity enabled by HandDreamer’s outputs.

Ablation Studies

Component analysis establishes the necessity of each architectural element. The combination of skeleton conditioning, MANO initialization, and CHS loss is required for robust, high-fidelity, and artifact-free synthesis; omitting any component degrades CLIP scores and reintroduces anatomical errors.

Figure 8: Ablation study—CHS loss and MANO prior are functionally critical for geometric integrity, especially in occluded or side viewpoints.

Theoretical Implications

This work provides a rigorous characterization of SDS mode ambiguity in the high-articulation domain and presents an explicit methodology for reducing mode multiplicity in the probability landscape through geometric priors and skeleton conditioning. The introduction of the CHS loss as a dynamically scheduled geometric regularizer offers a principled direction for future control of 3D diffusion optimization in underdetermined object classes.

Conclusion

HandDreamer establishes a new upper bound for zero-shot text-to-3D hand model synthesis by directly addressing the root causes of artifact formation in SDS-based pipelines. The integration of strong geometric priors, explicit skeletal conditioning, and a corrective shape regularizer enables the robust generation of anatomically plausible, highly detailed, and expressive hand models. This research opens future directions in shape-constrained generative modeling, automated avatar creation, and adaptive textual control for high-DOF objects. The potential to extend this framework to other complex, articulated assets—such as whole-body avatars, or even non-anthropomorphic articulated objects—remains substantial.

Limitations of the presented approach involve residual dependence on the underlying diffusion model priors and limited automation in downstream mesh articulation. Addressing these areas will further streamline text-to-asset workflows for immersive and interactive applications.

---

Reference: "HandDreamer: Zero-Shot Text to 3D Hand Model Generation using Corrective Hand Shape Guidance" (2604.04425)