---
title: 'CoInteract: HOI Video Synthesis'
url: https://www.emergentmind.com/topics/cointeract
type: topic
---

# CoInteract: HOI Video Synthesis

to=arxiv_search.search 派奖中  北京赛车开 json
{"query":"ti:\"CoInteract: Physically-Consistent Human-Object Interaction Video Synthesis via Spatially-Structured Co-Generation\"","max_results":5,"sort_by":"submittedDate","sort_order":"descending"}【อ่านข้อความเต็มanalysis to=arxiv_search.search code  重庆时时彩彩ేషis truncated?
to=arxiv_search.search ￣亚洲 ԥсы  天天中彩票网 json
{"query":"ti:\"Interaction-aware Representation Modeling with Co-occurrence Consistency for Egocentric Hand-Object Parsing\"","max_results":5,"sort_by":"submittedDate","sort_order":"descending"}
to=arxiv_search.search  天天中彩票开奖 րոպე json
{"query":"ti:\"Interaction-Centered Intelligence: Toward Interaction as the Primary Unit of Analysis in Co-Creative AI and Human-AI Systems\"","max_results":5,"sort_by":"submittedDate","sort_order":"descending"}
CoInteract is an end-to-end framework for human–object interaction (HOI) video synthesis in an e-commerce and product-demo setting. It is conditioned on a person reference image, a product reference image, text prompts, and speech audio, and is designed to address two recurrent failure modes of RGB-centric diffusion video models: structural instability in sensitive regions such as hands and faces, and physically implausible contact such as hand–object interpenetration. Its core design combines a Human-Aware Mixture-of-Experts (MoE) with Spatially-Structured Co-Generation inside a Diffusion Transformer (DiT) backbone; during training, an auxiliary HOI structure stream regularizes shared weights, while at inference the HOI branch is removed for zero-overhead RGB generation [2604.19636].

## 1. Problem formulation and operating regime

CoInteract addresses speech-driven HOI video synthesis for product presentation. The inputs per clip are a person reference image \(\mathcal{I}_{ref}\), a product reference image \(\mathcal{I}_{prod}\), speech audio, a text prompt, and motion frames \(\mathcal{V}_{mot}\) used as previous-frame context for temporal continuity. The output is a speech-driven HOI video in which the person matches \(\mathcal{I}_{ref}\), the product matches \(\mathcal{I}_{prod}\), hands and body interact with the product according to the text and audio, interaction remains physically plausible, hands and faces stay structurally stable over time, and background and appearance remain consistent with the references [2604.19636].

The framework is motivated by limitations that the source paper characterizes as typical of standard video diffusion, talking-avatar, and HOI models. These models are described as RGB-centric and prone to structural instability in fine regions, including hand collapse, finger merging, disappearing fingers, twisting, face blurring, and lip misalignment. They also frequently produce physically implausible interaction, including hand–object interpenetration, hands floating near but not contacting objects, and failure to model object boundaries or occlusion ordering explicitly. The paper attributes these errors to the absence of explicit internal structure or geometry modeling: pixel-level RGB supervision alone forces the model to infer body topology and hand–object spatial relations indirectly, which is particularly brittle under challenging views and occlusions [2604.19636].

Within this formulation, CoInteract is not merely a renderer conditioned on multiple modalities. It is organized so that human/body priors and HOI geometry are embedded into the diffusion backbone itself. A plausible implication is that the model is intended to function as an interaction-aware generative system rather than as a purely appearance-driven video prior.

## 2. Backbone design and system organization

CoInteract is built on a DiT backbone initialized from WanS2V and extended with an HOI structure stream, 3D Rotary Positional Encoding (3D RoPE), and a Human-Aware MoE. The model jointly generates an RGB appearance stream \(\mathbf{z}_r\) and an auxiliary HOI structure stream \(\mathbf{z}_h\), while sharing attention weights and MoE-augmented FFNs across the two streams. Multimodal conditioning includes text embeddings, audio embeddings, and reference-image embeddings; motion-context and reference-image latents are tokenized alongside the main video latents [2604.19636].

| Component | Function | Inference status |
|---|---|---|
| Human-Aware MoE | Routes tokens to head, hand, base, and shared experts | Retained |
| Spatially-Structured Co-Generation | Jointly models RGB and HOI structure streams | HOI branch removed |
| 3D RoPE | Encodes \((h,w,t)\) across streams, context, and references | Retained |
| Asymmetric co-attention | Lets HOI supervise shared weights while preserving RGB-only inference | Training mechanism |

The pipeline proceeds through VAE encoding, tokenization, multimodal conditioning, shared DiT processing, dual-stream co-generation, and VAE decoding. During training, both RGB video frames \(\mathbf{V}_r\) and HOI structure frames \(\mathbf{V}_h\) are encoded into latent tensors; during inference, only RGB latents are sampled and decoded. The design goal is to preserve the geometric benefit of HOI supervision without incurring the cost of auxiliary generation at test time [2604.19636].

A notable architectural property is the treatment of structure as an auxiliary generative stream rather than as an external control signal. This differs from pipelines that append pose, depth, or segmentation conditions to a generic video generator. In CoInteract, the HOI signal regularizes the shared backbone directly, which the paper presents as a means of internalizing interaction geometry [2604.19636].

## 3. Human-Aware Mixture-of-Experts

The Human-Aware MoE targets small, high-frequency, highly articulated regions—especially hands and faces—that are disproportionately sensitive to generation errors. Instead of a uniform FFN applied to all tokens, CoInteract replaces the original DiT FFN with a router \(\mathcal{R}\) and four experts: a shared expert that reuses the original FFN, plus lightweight head, hand, and base experts. The lightweight experts use hidden dimension \(256\), and the router is a 2-layer MLP [2604.19636].

For a token \(x_i\) with hidden representation \(\mathbf{h}_i\), routing uses a stop-gradient operation:
$$
G(x_i) = \text{Softmax}\left(\mathcal{W}_g \cdot sg[\mathbf{h}_i]\right).
$$
Target region labels \(y_i \in \{\text{head}, \text{hand}, \text{base}\}\) are assigned using supervision from face and hand bounding boxes, with all remaining tokens labeled base. The routing specialization objective is:
$$
\mathcal{L}_{route} = - \sum_{i} \sum_{k \in \{\text{head}, \text{hand}, \text{base}\}} \mathbb{1}(y_i = k) \log(G(x_i)_k).
$$
The total loss adds this term to the flow objective:
$$
\mathcal{L}_{total} = \mathcal{L}_{flow} + \eta \mathcal{L}_{route},
$$
with \(\eta = 1\) in experiments [2604.19636].

The supervision signal comes from external detectors: face and hand bounding boxes are derived from Mediapipe and hand-contact detectors. Tokens inside a face bounding box are routed toward the head expert, tokens inside a hand bounding box toward the hand expert, and other tokens toward the base expert; the shared expert acts as a generalist residual path [2604.19636].

The ablation evidence assigns this component a specifically local role. Enabling MoE improves HQ from \(0.658\) to \(0.724\) and FaceSim from \(0.662\) to \(0.696\), while inference cost is reported as only \(1.04\times\) that of the non-MoE baseline. Visualizations are described as showing router heatmaps focusing accurately on hands and faces, with sharper and more stable structures under motion [2604.19636]. This suggests that CoInteract decomposes structural fidelity into region-specialized capacity allocation rather than relying on a monolithic backbone to learn all fine-grained anatomy uniformly.

## 4. Spatially-Structured Co-Generation and HOI priors

The auxiliary HOI structure stream is constructed to represent geometry while suppressing appearance shortcuts. The paper specifies a four-step procedure: SAM3D-body is used to recover a human mesh for each frame, SAM3 is used to segment the object mask, the human mesh is projected to the image plane, and the projected human silhouette is fused with the object mask to create a 3-channel silhouette-style image \(\mathbf{V}_h\). This representation is low-texture and geometry-focused, highlighting human–object boundaries and contact regions while discarding appearance cues [2604.19636].

The RGB and HOI streams are both encoded by the same VAE and patch-embedded into tokens. They share the same DiT attention weights and MoE-augmented FFNs, but differ in input embeddings and conditioning prompts: the RGB stream receives the full text/audio prompt, whereas the HOI stream uses a fixed descriptive template to stabilize semantics. Stream-specific modulation in mechanisms such as AdaLN allows statistics to differ per stream without duplicating the full backbone [2604.19636].

CoInteract adopts a flow-matching formulation for both streams. The RGB and HOI objectives are:
$$
\mathcal{L}_{r} = \mathbb{E}_{t,\mathbf{z}_0,\mathbf{z}_1} \left[ \left\|\mathbf{v}_r - \mathbf{v}_\theta(\mathbf{z}_{r,t}, t, \mathbf{c}) \right\|_2^2 \right],
$$
$$
\mathcal{L}_{h} = \mathbb{E}_{t,\mathbf{z}_0,\mathbf{z}_1} \left[ \left\|\mathbf{v}_h - \mathbf{v}_\theta(\mathbf{z}_{h,t}, t, \mathbf{c}) \right\|_2^2 \right],
$$
with combined flow loss
$$
\mathcal{L}_{flow} = \mathcal{L}_{r} + \lambda_h \mathcal{L}_{h},
$$
where \(\lambda_h = 1\) [2604.19636].

A central alignment mechanism is 3D RoPE over coordinates \((h,w,t)\). The paper places RGB and HOI streams in a virtual concatenation along width, assigns them the same height and time indices, and uses shifted width coordinates so that corresponding RGB and HOI tokens remain close in relative positional distance. Past motion context is assigned negative time indices, the current generation window occupies \(t \in [0, T-1]\), and reference images are placed at large future time indices such as \(30\) and \(31\), so that they behave as global anchors rather than local temporal neighbors [2604.19636].

Training is divided into two attention stages. Stage 1 uses full bidirectional co-attention for the first \(5\text{K}\) iterations so that RGB and HOI streams attend to one another symmetrically. Stage 2 imposes an asymmetric mask for \(2\text{K}\) iterations:
$$
\mathbf{M}_{i,j} =
\begin{cases}
1, & \text{if } i \in \mathcal{T}_r,\; j \in \mathcal{T}_r, \\
1, & \text{if } i \in \mathcal{T}_h,\; j \in \mathcal{T}_r \cup \mathcal{T}_h, \\
0, & \text{otherwise.}
\end{cases}
$$
Under this mask, RGB queries attend only to RGB keys, while HOI queries attend to both RGB and HOI tokens. The result is that \(\mathcal{L}_{h}\) can backpropagate through HOI \(\leftarrow\) RGB attention into shared DiT parameters, but the RGB stream is trained to operate independently at inference [2604.19636].

The paper explicitly notes that CoInteract does not use analytic contact or penetration penalties. Physical plausibility is instead enforced implicitly through HOI silhouettes encoding correct human–object contact and relative layout, plus the shared-weight regularization induced by the HOI branch [2604.19636]. Related egocentric parsing work provides a conceptually adjacent formulation: InterFormer treats a “CoInteract” system as one that builds interaction-aware representations and enforces co-occurrence consistency between hands and active objects, explicitly reducing “interaction illusion” in hand–object parsing [2602.20597]. The two systems operate in different modalities, but both treat interaction structure as a first-class prior rather than as a by-product of semantic appearance modeling.

## 5. Training corpus, evaluation protocol, and empirical behavior

The training corpus is derived from \(40\) hours of HOI, product demonstration, and live-streaming videos. A curation pipeline performs entity decoupling with Qwen-Edit and Qwen-Image, generates geometric supervision with SAM3 and SAM3D-body, extracts face and hand bounding boxes with Mediapipe and hand-contact detectors, and encodes paired RGB and HOI streams using a shared VAE. After filtering, \(12\text{K}\) clips are retained, each with paired RGB/HOI streams, face and hand bounding boxes, and silhouette masks. The held-out test set contains \(50\) clips with diverse products and unseen identities [2604.19636].

Optimization uses AdamW with learning rate \(1 \times 10^{-4}\) and cosine annealing. The training schedule comprises Stage 1 with full bidirectional self-attention for \(5\text{K}\) iterations and Stage 2 with the asymmetric co-attention mask for \(2\text{K}\) iterations. Inference uses classifier-free guidance scale \(5\), \(40\) diffusion steps, and \(480p\) output resolution; the HOI branch is removed, and only the RGB stream is sampled [2604.19636].

The evaluation suite spans video quality, HOI quality, reference consistency, and audio–visual alignment. Video quality uses AES, IQ, and Smooth. HOI quality uses VLM-QA, based on \(50\) binary QA questions about HOI plausibility answered by Gemini-3-Pro, and HQ, the mean confidence of hand keypoints detected by DWPose across frames. Reference consistency uses DINO\(_{id}\), DINO\(_{obj}\), and FaceSim. Audio alignment uses Sync\(_{conf}\) from SyncNet. Baselines are AnchorCrafter, Phantom, Humo, VACE, InteractAvatar, and SkyReels-V3 [2604.19636].

| Metric | CoInteract | Note |
|---|---:|---|
| AES | 0.554 | Near best |
| IQ | 0.749 | Best |
| Smooth | 0.9951 | Best |
| VLM-QA | 0.72 | Best |
| HQ | 0.724 | Best |
| DINO\(_{id}\) | 0.671 | Best |
| DINO\(_{obj}\) | 0.624 | Very close to best 0.635 |
| FaceSim | 0.696 | Best |
| Sync\(_{conf}\) | 5.87 | Slightly better than others that report it |

The quantitative results are strongest on HOI and structural metrics. VLM-QA reaches \(0.72\), compared with \(0.62\) for the second-best InteractAvatar, and HQ reaches \(0.724\), compared with \(0.696\) for the second-best InteractAvatar. FaceSim is \(0.696\), DINO\(_{id}\) is \(0.671\), and DINO\(_{obj}\) is \(0.624\), the latter described as very close to the best value of \(0.635\). The paper notes that AES is slightly below Phantom and Humo because those baselines hallucinate more stylish backgrounds, whereas CoInteract preserves reference scenes faithfully [2604.19636].

The user study involved \(24\) participants, each judging \(10\) random cases in random blind order. Ranking criteria were object consistency, human/background consistency, and interaction plausibility, with lower mean rank indicating better performance. CoInteract achieves the best mean rank on all three, including interaction plausibility at approximately \(1.79\), human/background consistency at \(1.92\), and object consistency at \(2.17\) [2604.19636].

Ablations separate the contributions of local anatomical specialization and global interaction geometry. Removing MoE decreases HQ to \(0.658\), FaceSim to \(0.662\), and VLM-QA to \(0.66\). Removing Co-Gen, and thus the HOI stream, causes a particularly large decline in interaction plausibility, with VLM-QA falling to \(0.48\) while AES and IQ remain similar. Removing the asymmetric mask but keeping HOI at inference yields slightly better HOI metrics—VLM-QA \(0.76\), HQ \(0.738\)—but raises inference cost to \(4.13\times\). The full model is reported as the best quality–efficiency balance, retaining VLM-QA \(0.72\), HQ \(0.724\), and inference cost \(1.04\times\) [2604.19636].

## 6. Position within interaction-aware generative research and stated limitations

CoInteract belongs to a broader 2026 movement in which interaction structure becomes an explicit modeling target. In a different domain, InterCMDM uses a block-causal latent diffusion framework for autoregressive two-person interaction generation, with mask-controlled coordination modes such as simultaneous, reactive, leader–follower, and independent motion [2607.01743]. In a more general theoretical register, Interaction-Centered Intelligence proposes interaction itself as the primary unit of analysis for co-creative AI and human–AI systems, emphasizing interaction trajectories, coordination patterns, participatory engagement, adaptive regulation, and interactional drift over static output-based evaluation [2606.00807]. CoInteract does not adopt those frameworks directly, but it is consistent with their common premise that interaction quality cannot be reduced to isolated appearance fidelity or single-frame correctness.

Within video synthesis specifically, the paper positions CoInteract against three families of prior systems. Relative to multi-condition HOI generators such as AnchorCrafter, ByteLoom, and DreamActor, it internalizes HOI constraints via dual-stream DiT training instead of depending on external per-frame pose and object signals at inference. Relative to multi-reference video generators such as Humo, VACE, SkyReels-V3, and InteractAvatar, it introduces HOI-specific structure supervision and an internal human-aware expert mechanism rather than focusing mainly on subject consistency. Relative to structure-aware or multi-stream diffusion systems such as UnityVideo, VideoJam, and Geometry-Forcing, it uses an HOI-specific structure representation and asymmetric co-attention so that the auxiliary branch is needed only during training [2604.19636].

The limitations identified in the source material are primarily geometric and domain-specific. Geometry remains implicit because the HOI stream represents 2D silhouettes and masks rather than full 3D contact; some interpenetrations may therefore persist in complex 3D layouts. The domain is focused on product demonstrations and may be less suitable for highly non-rigid or unusual objects, large-scale multi-person interaction, or rare contact types. The evaluated clips are relatively short and generated at \(480p\), so scaling to higher resolution and longer sequences remains unaddressed. The system also depends on external preprocessing pipelines, including SAM3, SAM3D-body, Qwen-Image, and detectors, so training labels can inherit upstream failures [2604.19636].

A plausible synthesis is that CoInteract defines a specific architectural answer to physically consistent HOI video generation: region-specialized token processing for fragile anatomy, dual-stream geometry supervision for contact realism, and an inference pathway that retains most of the structural gain without carrying auxiliary generation cost. In adjacent literature, the same naming ecology is associated with interaction-aware representation learning, co-occurrence consistency, and interaction-centered system analysis, which suggests that “CoInteract” has become a useful label for methods that treat interaction structure as an organizing principle rather than an incidental consequence of generative modeling [2602.20597].

Source: https://www.emergentmind.com/topics/cointeract