JointHOI: Jointly Generating Contact Maps Enhances Hand Object Interaction Generation

Abstract: Text driven hand object interaction (HOI) generation is gaining attention for immersive applications and robotics, yet producing physically plausible interactions remains challenging. Even when individual motions appear natural, small contact errors can cause conspicuous artifacts such as floating and interpenetration. Prior methods mitigate these issues using explicit contact cues or implicit grasp priors, but typically rely on multi stage pipelines and fail to model temporally evolving contact. We present JointHOI, a single stage diffusion framework that jointly generates 3D hand object motion and dynamic, distance based contact maps from text. By treating contact as an auxiliary inner modality, joint generation enables the model to learn contact motion coupling during training. At inference, contact guided sampling enforces consistency between generated contact maps and motion implied geometry, improving temporal stability and reducing penetration and floating. Experiments on GRAB and ARCTIC demonstrate consistent improvements in text adherence and physical plausibility over prior methods.

• The paper introduces a single-stage diffusion model that jointly generates bimanual trajectories and dynamic contact maps for realistic hand-object interactions.
• It leverages a transformer-based denoiser with shared self-attention to learn motion-contact dependencies, significantly reducing artifacts like floating and interpenetration.
• Quantitative evaluations on GRAB and ARCTIC datasets demonstrate improved accuracy and physical consistency, making JointHOI suitable for AR/VR and robotics applications.

JointHOI: Joint Generation of Contact Maps and Hand–Object Interactions

Motivation and Problem Statement

Text-driven hand–object interaction (HOI) generation remains a critical yet unsolved problem for immersive AR/VR and robotics, where the spatiotemporal fidelity of contact between hands and objects is essential for both realism and functional manipulation. Prior models have alleviated artifacts such as floating or interpenetration through either explicit, static contact conditioning or implicit modeling in latent spaces, but these commonly employ multi-stage and often complex pipelines, lacking a physically grounded and temporally resolved representation of contact.

JointHOI Architecture and Key Innovations

JointHOI introduces a single-stage diffusion-based generative framework that concurrently synthesizes bimanual hand and object pose trajectories from text, while also co-generating a dense, time-varying (dynamic) contact map encoding per-hand distances to the object surface. This explicit contact stream constitutes an auxiliary inner modality, allowing direct learning of motion–contact dependencies with shared cross-modal attention in a transformer denoiser.

The approach parameterizes contact as a proximity field over fixed canonical object surface anchors, enabling fine-grained, temporally continuous contact supervision and guidance during both training and inference.

Figure 1: JointHOI overview, showing joint diffusion of motion and per-hand contact maps, with transformer-based self-attention on conditioned text and object geometry, and inference-time contact-guided sampling.

Dynamic Contact Map Representation

JointHOI employs a farthest-point sampling scheme to define $N_c=1024$ stable object surface anchors. At each timestep, it computes for each hand a proximity vector as the minimum Euclidean distance between each anchor and all hand mesh vertices. This continuous field transcends coarse, binary or static contact cues, supporting robust gradient-based contact supervision and highly interpretable contact visualization.

Figure 2: Dynamic contact map visualizations — anchor-to-hand distances as heatmaps over time, showcasing emergence, movement, and dissipation of contact regions during interaction.

Joint Diffusion Training Paradigm

The model represents a full HOI sequence as concatenated per-frame tokens including hand pose (left/right), object state, and dynamic contact maps. The transformer-based diffusion denoiser jointly predicts all modalities under the constraints of text and object geometry conditioning, optimizing a composite MSE objective aggregating both motion and contact reconstruction. This shared self-attention facilitates context-sensitive cross-modal learning, such as hand reorientation upon proximity or coordinated bimanual manipulation.

At generation time, sampling proceeds by iterative reversal of the learned forward noising process, reconstructing both motion trajectories and contact maps.

Contact Inner Guidance: Inference-Time Consistency Enforcement

To enforce physical plausibility during synthesis and further suppress artifacts, JointHOI introduces Contact Inner Guidance (CIG), a classifier-based, training-free gradient guidance scheme. CIG constructs a log-difference energy between predicted and geometry-implied contact maps; its gradient with respect to the noisy diffusion variable is then used to steer generation toward consistency between explicit contact predictions and those derived from hand/object kinematics. This process is computationally efficient and fully differentiable.

Figure 3: Ablation of CIG variants and impact — showing that log-$\ell_1$ proximity emphasizes near-contact fidelity, optimizing both action accuracy and reducing penetration depth.

Experimental Evaluation

Quantitative Results

JointHOI is evaluated on GRAB (rigid objects) and ARCTIC (articulated objects) datasets using text prompts and object models as input. Metrics include action classification accuracy, Fréchet distance in a learned motion embedding space (FID), mesh penetration volume/depth (IV/ID), and contact ratio (CR) for valid contact frames.

Strong numerical results are reported:

• ARCTIC: Top-1/Top-3 accuracy: 0.948/0.983; FID: 0.033 (best); IV: 4.41; ID: 0.43; CR: 93.7%.
• GRAB: Top-1/Top-3 accuracy: 0.663/0.847; FID: 0.031 (best); IV: 3.42; ID: 0.53; CR: 93%.

Notably, penetration volume is reduced by over 50% relative to best prior work on GRAB, while maintaining strong text-to-action alignment (2607.01768).

Figure 4: Qualitative comparisons on text-conditioned generation, demonstrating improved bimanual coordination and reduced artifacts (floating, interpenetration) compared to baselines.

Incremental ablation studies demonstrate:

• Transitioning from two-stage to joint training drives a substantial improvement in action accuracy.
• Per-hand dynamic contact maps (vs. combined) yield additional FID gains, highlighting the asymmetric and loosely correlated dynamics of bimanual interactions.
• CIG delivers a large reduction in penetration with minimal inference-time overhead, outperforming even oracle-assisted, multi-stage alternatives.

Contact Consistency Analysis

A strong correspondence between explicit contact maps generated by the model and maps analytically computed from synthesized geometry evidences successful learning of motion–contact coupling.

Figure 5: Visualization of generated vs. analytically derived contact maps, confirming tight agreement and validating inner-modal training.

Efficiency and Practical Implications

JointHOI achieves inference times comparable to the fastest single-stage baselines, requiring less than 11s for a 196-frame sequence, with minimal computational overhead for CIG. This positions JointHOI as suitable for real-time or interactive use cases in simulation, AR/VR, and physically plausible robotics manipulation pipelines.

The explicit, temporally resolved contact signal enables further downstream integration, such as automatic error detection, post hoc adjustment, or more precise physical constraints in simulation environments.

Figure 6: End-to-end JointHOI pipeline — text-to-bimanual HOI sequence generation with dynamic contact maps, reducing artifacts such as floating fingers and object interpenetration.

Implications and Future Directions

JointHOI's methodology of modeling inner physical modalities sets a precedent for future generative models in physically grounded animation, affordance learning, and humanoid robotics. The transformer-based diffusion structure is extensible to whole-body manipulations, multi-step task hierarchies, or integration with visual scene understanding.

The dynamic, continuous contact representation lends itself to fine-grained analysis of affordance evolution and enables direct integration with differentiable physics or mocap-based refinement. Potential future advances include:

• Extension to scene-level interactions with multiple agents or deformable objects;
• Supramodal generative modeling via additional inner modalities, such as force, torque, or tactile cues;
• Scaling architectures with larger context windows and richer object interactions for human-centric learning and simulation.

Conclusion

JointHOI presents a unified, single-stage framework for contact-aware hand–object interaction generation from text, yielding significant improvements in both realism and physical plausibility via explicit dynamic contact co-generation and inference-time contact-guided sampling. These architectural and algorithmic choices advance the field toward robust, physically accurate generative models for interactive and embodied AI applications.

(2607.01768)