Asset-Conditioned HOI Synthesis

Updated 5 June 2026

The paper introduces a mapping framework from human and object assets to realistic HOI outputs, leveraging diffusion models, geometric conditioning, and physics-based simulation.
It details multimodal conditioning approaches by integrating image features, 3D geometries, and text priors to generate semantically accurate interactions.
The work also addresses challenges such as scaling to multi-agent scenarios and enforcing physical realism, pointing toward promising future research directions.

Asset-conditioned human-object interaction (HOI) synthesis refers to the generation of physically plausible and semantically meaningful images, motions, or videos depicting humans interacting with specific object assets, where the explicit identity, geometry, or appearance of each asset is provided as an input condition. This approach spans the full spectrum of modalities—ranging from single images to 4D motion sequences—and leverages advances in generative modeling, geometric conditioning, vision-language reasoning, and physics-based simulation for both rigid and articulated objects.

1. Formal Problem Definition and Scope

The asset-conditioned HOI synthesis problem is generally formulated as learning a mapping

$\mathcal{G}: (\mathcal{A}_H, \mathcal{A}_O, \mathcal{C}) \mapsto \mathcal{Y}$

where $\mathcal{A}_H$ and $\mathcal{A}_O$ respectively denote the provided human and object assets (e.g., RGB images, 3D meshes, signed distance functions), and $\mathcal{C}$ encodes the interaction specification (e.g., a text prompt or part-level contact graph). $\mathcal{Y}$ is the output—an image, video, or parameterized motion sequence in which the input assets engage in a realistic interaction defined by $\mathcal{C}$ . Models may support additional optional controls such as background images, spatial locations, or partial interaction constraints.

This field encompasses image-based synthesis with explicit appearance conditioning (Xu et al., 27 Aug 2025, Hu et al., 2022, Liang et al., 22 Jul 2025), low-level motion or pose trajectory generation (Diller et al., 2023, Li et al., 2023, Liu et al., 17 Nov 2025, Cai et al., 25 Mar 2026, Benishu et al., 28 May 2026, Ye et al., 2024), articulated or part-aware dynamics (Huang et al., 4 Mar 2026, Li et al., 8 Jun 2025), and multi-modal compositional tasks including video reenactment (Tong et al., 16 Mar 2026) and zero-shot open-vocabulary asset handling (Zhang et al., 30 May 2025).

2. Conditioning Paradigms and Asset Representations

Conditioning on asset-specific information is central. Various schemes have been introduced:

Pixel / Image Conditioning: For image-level synthesis, identity features are extracted from provided RGB crops of specific humans and objects, using models such as DINOv2 and Sobel-filter for fine detail, or patchwise feature fusion (Xu et al., 27 Aug 2025, Hu et al., 2022, Liang et al., 22 Jul 2025). Mask-guided or region-based pose guidance modules may delineate interaction configurations.
3D Geometry Conditioning: For physical or pose-level HOI, assets are processed as 3D meshes, point clouds, SDFs, or 3D Gaussian splats. Object assets are voxelized into unified interactive volumes (Liu et al., 17 Nov 2025), latent SDF codes (Ye et al., 2024), or part masks (Li et al., 8 Jun 2025).
Text and Visual Priors: For generalization and open-set object support, object and action semantics are encoded as text prompts (CLIP features, vision-LLM outputs), which may be fused with geometric encodings via cross-attention (Diller et al., 2023, Cai et al., 25 Mar 2026, Zhang et al., 30 May 2025).
Physics-based Representations: For physically grounded generative models, assets are mapped to differentiable simulation-ready forms (e.g., Material Point Method points, kinematic skeletons, or articulated Gaussian splats) supporting two-way physical feedback (Benishu et al., 28 May 2026, Huang et al., 4 Mar 2026).

3. Model Architectures and Synthesis Pipelines

Several architectural paradigms have been established:

Diffusion-based Frameworks: Most state-of-the-art approaches employ diffusion models—conditional DDPMs or DiTs—to iteratively refine noisy representations of interactions, enabling robust synthesis and stochasticity (Diller et al., 2023, Liu et al., 17 Nov 2025, Cai et al., 25 Mar 2026, Xu et al., 27 Aug 2025, Benishu et al., 28 May 2026, Li et al., 8 Jun 2025).
Two-stage and Hierarchical Pipelines: Many pipelines decompose the task. For instance, Interact-Custom splits into mask synthesis and detail-conditioned image generation (Xu et al., 27 Aug 2025); OMOMO denoises first the hand–object contacts, then completes body pose (Li et al., 2023); HOComp employs region-based pose layout followed by detail-consistent image assembly (Liang et al., 22 Jul 2025); ArtHOI decouples object articulation reconstruction from human motion synthesis (Huang et al., 4 Mar 2026).
Cross-Attention and Multi-Modal Fusion: Conditioning information—image, geometry, language, and control signals—is integrated using cross-attention at various UNet or transformer blocks, sometimes with hybrid fusion (e.g., three-way human/object/contact attention (Diller et al., 2023); visual/text Q-Formers (Cai et al., 25 Mar 2026)).
Physics Coupling and Simulation: Physically plausible motion is achieved by coupling generative diffusion models with explicit simulation or constraint stages: e.g., PhyGenHOI aligns human and object dynamics via MDM + MPM coupling and momentum transfer (Benishu et al., 28 May 2026); HOI-PAGE introduces part-level contact graph constraints in optimization (Li et al., 8 Jun 2025).

4. Datasets, Training Objectives, and Evaluation

Training and evaluation are driven by comprehensive, annotation-rich datasets and multi-criteria loss landscapes:

Dataset Construction: Large-scale datasets are curated to enable asset-level learning—Interact-Custom constructs a million-sample set of paired identity–pose HOI images (Xu et al., 27 Aug 2025); OMOMO and COUCH build extensive motion capture datasets with synchronized object geometry, kinematics, and contact labels (Li et al., 2023, Zhang et al., 2022); UV-aligned meshes support fine-grained ground truth in hand–object scenarios (Hu et al., 2022).
Loss Functions: Diffusion objectives dominate, with reconstruction objectives at every denoising step. Structure- and identity-aware losses include pose/vertex/part alignment (MPJPE, ADD, Chamfer), CLIP/DINO similarity, region-specific KL, and semantic mask losses. Physics and contact are enforced via windowed attraction, force-closure, contact-resampling, or SDF-based penalties (Benishu et al., 28 May 2026, Zhang et al., 30 May 2025, Li et al., 8 Jun 2025, Diller et al., 2023).
Metrics and Validation: Standard image and video synthesis metrics—FID, LPIPS, SSIM, CLIP-score—quantify realism and appearance. Motion realism is quantified with contact recall/precision, foot sliding, collision ratios, root translation/orientation error, and end-effector accuracy. Benchmark datasets (IHOC, HO3Dv3, DexYCB, BEHAVE, Sketchfab) and user studies are used for holistic evaluation (Liang et al., 22 Jul 2025, Hu et al., 2022, Diller et al., 2023, Li et al., 8 Jun 2025).
Ablation and Generalization: Ablations demonstrate the necessity of joint identity-interaction conditioning, multi-stage pipelines, physics-inspired loss terms, and the value of both real and interaction-aware synthetic data (Xu et al., 27 Aug 2025, Benishu et al., 28 May 2026, Liang et al., 22 Jul 2025). Several systems provide results for zero-shot or OOD (out-of-domain) assets—see InteractAnything (Zhang et al., 30 May 2025), HOI-PAGE (Li et al., 8 Jun 2025), ViHOI (Cai et al., 25 Mar 2026).

5. Controlling, Generalizing, and Interpreting Asset-Conditioned HOI Synthesis

Key control and generalization mechanisms include the following:

Explicit Asset, Location, and Interaction Control: Most frameworks enable asset, region, and interaction-level customization. For example, Interact-Custom allows explicit human, object, and background selection together with bounding box and action text (Xu et al., 27 Aug 2025); COUCH enables user- or model-specified contacts for varied chair affordances (Zhang et al., 2022); HOComp uses MLLMs for dynamic pose region guidance (Liang et al., 22 Jul 2025).
Part-Level and Semantic Guidance: Part Affordance Graphs in HOI-PAGE enable fine-grained human–object contact reasoning and compositionality, supporting multi-object/person interactions (Li et al., 8 Jun 2025). LLMs and VLMs are leveraged for affordance parsing, semantic prior extraction, and generalized “reasoning” about unseen assets (Zhang et al., 30 May 2025, Cai et al., 25 Mar 2026).
Physics and Contact Enforcement: Contact-rectification, score distillation, and explicit simulation play a central role in achieving spatial and physical coherence of synthesized interactions. Notably, inference-time contact guidance as in CG-HOI enhances physical plausibility without costly outer-loop optimization (Diller et al., 2023).
Zero-Shot and Open-Set Generalization: Systems such as InteractAnything (Zhang et al., 30 May 2025), HOI-PAGE (Li et al., 8 Jun 2025), and ViHOI (Cai et al., 25 Mar 2026) specifically demonstrate zero-shot synthesis on novel object assets, utilizing compositional reasoning, learned visual priors, and part or affordance abstraction to operate outside the distributions encountered at training.

6. Limitations, Open Problems, and Future Directions

Despite rapid progress, open challenges remain:

Fine-Grained Dexterity and Hand-Object Modeling: Accurate hand articulation, finger–object contact, and dynamic grasp synthesis—even in specialized frameworks like G-HOP—remain open for higher-fidelity interaction and are bottlenecked by asset annotation and mesh registration precision (Ye et al., 2024).
Multi-Object, Multi-Agent, and Long-Horizon Scenarios: Many frameworks are currently limited to single-human, single-object settings; scaling to complex scenes with temporal chains of interaction is an active area for future research (Diller et al., 2023, Li et al., 8 Jun 2025, Tong et al., 16 Mar 2026).
Physical Realism: Extensions to soft-body coupling, dynamic feedback, and fully physically simulated humans are underexplored; most current systems treat the human as a kinematic agent and do not support two-way force transfer except by proxy (Benishu et al., 28 May 2026, Huang et al., 4 Mar 2026).
Data Requirements and Modality Bridging: The reliance on corresponding ground-truth mesh parameters and large-scale annotation can be a limiting factor for generalization. Zero-shot and self-supervised synthesis, such as via 4D inverse rendering from only generated videos (Huang et al., 4 Mar 2026), are promising but not yet fully mature.
Evaluation Benchmarks: Although new datasets and metrics have emerged (IHOC, HOIBench), the field still lacks universally adopted, high-coverage benchmarks across all task types and output modalities (Liang et al., 22 Jul 2025).

Asset-conditioned HOI synthesis thus remains an active and expanding domain at the intersection of generative modeling, computer vision, geometric learning, and physical simulation, with considerable potential for advances spanning zero-shot interaction, controllable compositionality, and generalizable human–AI/robotic collaboration.