HOComp: Human–Object Interaction Synthesis

• HOComp is a framework for interaction-aware human–object composition that integrates region-based multimodal reasoning to ensure accurate pose guidance and spatial consistency.
• It employs a coarse-to-fine pipeline using GPT-4o for interaction description and conditional diffusion, effectively refining object placement and human pose alignment.
• HOComp introduces Detail-Consistent Appearance Preservation (DCAP) to maintain artifact-free composites, validated on the IHOC dataset with superior realism and interaction fidelity.

HOComp is a framework for interaction-aware human–object visual composition that addresses the technical challenges of synthesizing semantically and physically consistent human–object interaction (HOI) when compositing a foreground object into a human-centric background image. Unlike standard image-guided composition approaches, which often fail to harmonize detailed body pose and interaction context, HOComp integrates region-level multimodal reasoning and fine-grained appearance control to generate compositions exhibiting correct spatial relations, anatomically plausible poses, and faithful visual attributes for both human and object (Liang et al., 22 Jul 2025).

1. Region-Based Pose Guidance by Multimodal LLMs

The heart of HOComp’s interaction-awareness is its Multimodal LLM-driven Region-based Pose Guidance (MRPG). MRPG forms a “coarse-to-fine” control pipeline: a multimodal LLM (specifically GPT-4o) is queried to extract a structured interaction description, localize the foreground object’s placement, and identify the relevant interaction region on the human subject.

• Coarse-stage: The MLLM generates (1) a natural-language interaction prompt $C$ (e.g., “A woman is holding a hat”), (2) a bounding box for foreground placement $B_o$, and (3) a predicted interaction region box $B_r$ highlighting the active body part(s). $B_r$ is encoded as a binary mask $M_r$ over a 512×512 grid.
• Conditional diffusion: The mask $M_r$ and the embedded prompt $E_c$ condition the base Diffusion Transformer (DiT).
• Fine-stage: DWPose, a pre-trained 2D keypoint extractor, identifies keypoints within $B_r$ for both the generated and ground-truth images. A pose loss

$$\mathcal{L}_p = \frac{1}{n} \sum_{i:\,p^i \in B_r} \|p^i_{\mathrm{GT}} - p^i_{\mathrm{pred}}\|_2^2$$

ensures alignment specifically in the interaction region, reliably guiding fingers, limbs, and joints relevant to the intended action while preserving the uninvolved body’s structure.

MRPG thereby confers both semantic (what and where) and geometric (how) priors that dramatically increase the fidelity of human–object interactions.

2. Detail-Consistent Appearance Preservation (DCAP)

To further guarantee naturalistic and artifact-minimal composites, HOComp introduces DCAP, a three-pronged mechanism focused on both spatial and appearance-level consistency:

• Shape-aware Attention Modulation (SAAM): Certain cross-attention maps in DiT exhibit strong selectivity for the object silhouette. A residual-based modulation, parameterized by a ground-truth shape mask and controlled by a learnable parameter α, gently nudges attention higher inside the object region and suppresses it outside, preserving attention focus and maintaining pretrained weight stability:

$$A' = A + \alpha \left[M_{\mathrm{shape}} \odot (A_{\max} - A) - (1-M_{\mathrm{shape}}) \odot (A - A_{\min})\right]$$

• Multi-View Appearance Loss: To ensure semantic and texture consistency under arbitrary pose and viewpoint perturbations, multi-view renderings of both the predicted and ground-truth object (via Zero123+) are compared via a CLIP-based feature loss over $B_o$0 views:

$B_o$1

• Background Consistency Loss: A binary mask $B_o$2 (the complement of the interaction region mask) enforces per-pixel fidelity outside of the interaction zone:

$B_o$3

The total training objective combines the standard diffusion loss and the auxiliary losses with empirically-tuned weights ($B_o$4, α=1.0), allowing robust learning of the desired spatial and textural constraints (Liang et al., 22 Jul 2025).

3. The IHOC Dataset: Construction and Statistical Properties

HOComp’s development is underpinned by the Interaction-aware Human-Object Composition (IHOC) dataset, the first large-scale collection of paired pre- and post-interaction images suitable for this compositional paradigm.

• Composition: 11,700 total samples (5,850 real, 5,850 synthetic), built as follows:
  • Real: HICO-DET human–object images, filtered to single-person, unoccluded, clearly visible object scenes across 117 interaction types.
  • Synthetic: GPT-4o prompts drive sample generation using FLUX.1, guided by explicit textual and spatial conditioning.
• Foreground extraction: Segmentation is performed by SAM; GPT-4o is used to fill occluded regions, recovering canonical object shapes.
• Scene parameters: Diversity in interaction type (117 total), object category (342), body part (hand/arm: 54.3%), pose, object size (small/medium/large), image style, and scene complexity are systematically included.
• Region and prompt annotation: Each sample has a corresponding prompt $B_o$5, region mask $B_o$6, and “unchanged” background mask, directly supporting training and evaluation of MRPG and DCAP.

The IHOC dataset systematically addresses compositional, semantic, and geometric variety, and reflects real deployment scenarios for broad HOI support (Liang et al., 22 Jul 2025).

4. Experimental Setup, Quantitative Benchmarks, and Ablation Results

HOComp is evaluated on the HOIBench benchmark (30 humans × 20 sampled (object, interaction) pairs = 600 images) using both standard and custom metrics:

• FID: Frechet Inception Distance (image realism).
• CLIP-Score: Semantic image–prompt congruence.
• HOI-Score: UPT detector accuracy for generated HOI predictions.
• DINO-Score: Consistency of generated foreground identity.
• SSIM(BG): Structural similarity on the unchanged background.
• User study: 75 humans rate Image Quality (IQ), interaction harmonization (IH), and appearance preservation (AP).

Relative to nine SOTA baselines (AnyDoor, PbE, FreeCompose, GPT-4o, etc.), HOComp yields the best scores across all measured dimensions:

Metric	Best Baseline	HOComp (Ours)
FID ↓	9.98 (GPT-4o)	9.27
CLIP-Score ↑	29.77 (OmniGen)	30.29
HOI-Score ↑	75.22 (GPT-4o)	87.39
DINO-Score ↑	65.23 (GPT-4o)	78.21
SSIM(BG) ↑	90.71 (AnyDoor)	96.57

Ablation experiments confirm that each major component (MRPG, SAAM, multi-view loss, background loss) significantly improves its targeted metric, with the combined system providing the strongest interaction harmonization, identity preservation, and artifact-free background (Liang et al., 22 Jul 2025).

5. Qualitative Evaluation and Human-Object Harmonization

HOComp produces visually natural composites in which the object is placed in anatomically plausible poses with naturalistic physical contact and occlusion, detailed logos/textures are retained, and background regions remain unaffected. Figure 1 in (Liang et al., 22 Jul 2025) highlights HOComp’s ability to avoid prevalent errors of baselines, such as implausible joint articulation, object floatation, and visible warping of human features outside of the interaction area.

User studies reinforce these findings, with HOComp ranked best in both photorealism and interaction plausibility by large margins.

6. Limitations and Prospective Directions

The accuracy of the region inference via MLLM (notably GPT-4o) currently determines the upper limit of the system's HOI fidelity; in HOIBench, B_r is predicted correctly 91.33% of the time, with failures decreasing pose control precision. The reliance on 2D keypoint estimators constrains the modeling of complex 3D or occluded interactions. Future iterations could incorporate SMPL-based parametric body models for richer geometric control, fuse pose priors directly at the MLLM conditioning stage, and generalize HOComp to video or temporally coherent settings (Liang et al., 22 Jul 2025).

7. Significance and Contributions

HOComp represents the first fully-integrated, interaction-aware composition pipeline that jointly leverages the semantic expressiveness of MLLMs and fine-grained spatial/appearance constraints to harmonize human pose, object placement, and global scene consistency. The methodological combination of MRPG and DCAP, validated on the domain-specialized IHOC dataset, establishes a new standard both quantitatively and via human perceptual judgment for compositional image synthesis involving complex human–object interactions (Liang et al., 22 Jul 2025).