CoInteract: Physically-Consistent Human-Object Interaction Video Synthesis via Spatially-Structured Co-Generation

Abstract: Synthesizing human--object interaction (HOI) videos has broad practical value in e-commerce, digital advertising, and virtual marketing. However, current diffusion models, despite their photorealistic rendering capability, still frequently fail on (i) the structural stability of sensitive regions such as hands and faces and (ii) physically plausible contact (e.g., avoiding hand--object interpenetration). We present CoInteract, an end-to-end framework for HOI video synthesis conditioned on a person reference image, a product reference image, text prompts, and speech audio. CoInteract introduces two complementary designs embedded into a Diffusion Transformer (DiT) backbone. First, we propose a Human-Aware Mixture-of-Experts (MoE) that routes tokens to lightweight, region-specialized experts via spatially supervised routing, improving fine-grained structural fidelity with minimal parameter overhead. Second, we propose Spatially-Structured Co-Generation, a dual-stream training paradigm that jointly models an RGB appearance stream and an auxiliary HOI structure stream to inject interaction geometry priors. During training, the HOI stream attends to RGB tokens and its supervision regularizes shared backbone weights; at inference, the HOI branch is removed for zero-overhead RGB generation. Experimental results demonstrate that CoInteract significantly outperforms existing methods in structural stability, logical consistency, and interaction realism.

• The paper introduces CoInteract, an end-to-end diffusion transformer integrating human-aware MoE and dual-stream co-generation for structurally-consistent HOI video synthesis.
• It leverages spatial supervision and asymmetric co-attention to preserve detailed hand and face features while ensuring realistic human-object interactions.
• Quantitative evaluations demonstrate enhanced interaction plausibility and temporal coherence, validating the framework's efficiency in complex HOI scenarios.

Physically-Consistent HOI Video Synthesis via Spatially-Structured Co-Generation: An Analysis of CoInteract (2604.19636)

Motivation and Challenges in HOI Video Generation

Human-object interaction (HOI) video synthesis, particularly for scenarios requiring active product demonstration, demands high structural stability in sensitive regions (hands, faces) and physically plausible interactions (avoiding hand-object interpenetration). Existing diffusion-based generative models, while proficient in photorealistic rendering, are predominantly RGB-centric and lack explicit mechanisms for enforcing 3D spatial relationships or robust structural consistency. Prior approaches either rely on domain-specific external signals through heavy per-frame pose or object preprocessors, or composite references without embedding interaction structure, ultimately resulting in frequent failures of hand articulation, facial integrity, or interaction realism.

CoInteract Framework Overview

CoInteract introduces an end-to-end framework for HOI video synthesis, jointly conditioned on person/product reference images, text prompts, and speech audio. Two architectural innovations are embedded within a Diffusion Transformer (DiT) backbone: a Human-Aware Mixture-of-Experts (MoE) and Spatially-Structured Co-Generation.

Figure 1: CoInteract generates high-fidelity HOI videos from multimodal inputs, contrasting its unified, interaction-aware generation paradigm with inference-heavy/structure-deficient alternatives.

Figure 2: The dual-stream co-generation architecture realizes RGB and HOI streams within a shared DiT backbone; asymmetric co-attention masks structure learning and permit efficient inference.

Human-Aware Mixture-of-Experts (MoE)

CoInteract deploys region-specialized experts within DiT blocks. Using spatial supervision from face and hand bounding boxes, MoE routes tokens to lightweight, dedicated FFNs (Head, Hand) while the rest are processed by a base expert. Spatially supervised routing relies on a cross-entropy loss, and a stop-gradient operation prevents router interference with backbone learning. This enables high-frequency detail preservation in hands/faces and minimizes parameter overhead.

Spatially-Structured Co-Generation

A dual-stream paradigm is implemented: the RGB stream handles appearance, while an auxiliary HOI structure stream provides geometry priors by projecting human mesh and object masks to silhouette-like spatial representations. Both streams are tokenized and processed in a shared backbone with stream-specific modulations. Training utilizes a joint flow-matching objective; inference discards the HOI branch for zero computational overhead.

Figure 3: Two-stage training strategy with bidirectional attention followed by asymmetric co-attention; only RGB queries attend to RGB tokens in Stage 2, ensuring efficient inference.

Architectural Details and Data Curation

To effectively integrate multimodal inputs, 3D Rotary Positional Encoding (3D RoPE) assigns distinct spatial and temporal coordinates to all tokens. Reference frames are spatially separated and temporally anchored, historical motion tokens have negative indices, and RGB/HOI streams are concatenated but spatially shifted. The asymmetric co-attention mask in Stage 2 training ensures appearance and structure streams are coupled during learning but decoupled for inference.

Figure 4: Preprocessing pipeline for paired RGB–HOI-structure data, utilizing entity separation, mesh recovery, mask projection, and spatial encoding.

Quantitative and Qualitative Evaluation

Extensive experiments compare CoInteract against AnchorCrafter, Phantom, Humo, InteractAvatar, SkyReels-V3, and VACE. Evaluation covers perceptual metrics (AES, IQ, Smooth), interaction plausibility (Gemini VLM-QA, HQ with DWPose), reference consistency (DINOv2, ArcFace), and audio alignment (SyncNet).

CoInteract achieves the highest scores in VLM-QA (0.72) and HQ (0.724), with strong performance in identity preservation (DINO$_\text{id}$: 0.671, FaceSim: 0.696), and temporal coherence (Smooth: 0.9951). Marginal trade-offs in aesthetic scores are noted, attributable to strict reference scene adherence.

Figure 5: CoInteract produces videos with superior HOI fidelity, structural stability, and prompt compliance across diverse test cases.

User studies corroborate objective metrics, showing CoInteract's lowest mean rank in object consistency, background/identity stability, and interaction plausibility.

Mechanism Visualization and Ablation Analysis

Detailed visualization of dual-stream generation and MoE routing demonstrates precise spatio-temporal alignment between RGB and HOI streams, robust mitigation of hand-object interpenetration, and accurate expert token assignment during complex manipulation tasks.

Figure 6: Dual-stream co-generation and MoE routing heatmaps reveal coordinated geometric alignment and expert specialization during intricate manipulations.

Ablation studies systematically confirm the necessity of core components: removing MoE degrades hand quality and face identity; excluding the HOI stream results in a catastrophic drop in interaction plausibility (VLM-QA from 0.72 to 0.48); symmetric attention inflates inference cost but yields slight metric improvements; full model maintains optimal trade-offs with minimal overhead.

Figure 7: Removing unified HOI co-generation yields physical implausibility in interaction; removing MoE induces collapse and artifacts in high-frequency regions.

Implications and Future Directions

Practically, CoInteract establishes an architecture for HOI synthesis that is both structure-aware and computationally efficient, enabling broader application in e-commerce, digital advertising, and virtual assistants. Theoretically, it introduces an effective methodology for embedding geometric priors directly into diffusion backbones, coupling appearance and structure within transformer architectures. The streamlined inference enabled by asymmetric co-attention can generalize to other tasks requiring auxiliary supervision during training and efficiency at deployment.

Future research may focus on extending spatially-structured co-generation to broader multimodal synthesis domains, integrating richer geometric representations (e.g., 3D meshes or point clouds), improving temporal modeling for dynamic scenes, and scaling region-specific MoE modules for additional anatomical complexity.

Conclusion

CoInteract advances HOI video synthesis by embedding structural priors and interaction geometry within a diffusion transformer framework, enhanced via spatially-supervised expert routing. The architecture achieves superior physical plausibility, structural integrity, and computational efficiency, with robust quantitative and qualitative validation. Its dual innovations—MoE for hands/faces and asymmetric dual-stream co-generation—remarkably reduce common failure modes and set a foundation for future interactive video generation systems.