Layout-Aligned Modality Matching

• Layout-Aligned Modality Matching is a technique that synchronizes audio, visual, and appearance cues using spatial masks for precise, region-specific binding.
• The InterActHuman framework implements localized modality matching by injecting each subject's cues within designated regions, enabling independent control over multiple identities.
• Its iterative, mask-guided diffusion process improves lip-sync accuracy and visual identity retention, fostering realistic human-human and human-object interactions.

Layout-aligned modality matching refers to the precise, region- and identity-specific synchronization of multiple modalities—such as audio and visual appearance—over both space and time during human-centric video generation. The InterActHuman framework advances this concept by enabling explicit, per-person, per-region alignment of visual, audio, and appearance cues for generating highly controllable multi-concept videos with human-human and human-object interactions.

1. Framework Architecture and Rationale

InterActHuman introduces a new architecture for multi-modal human video generation that discards the single-entity assumption found in previous methods. Instead of fusing all input conditions (e.g., audio, text, image references) globally across the scene, InterActHuman enforces explicit layout-wise association between each modality and its corresponding visual region—most notably, binding each localized audio stream to the spatial region of its corresponding person. This is achieved through:

• A mask prediction module that dynamically infers spatiotemporal locations (“footprints”) for each concept (identity) at every denoising iteration.
• Local injection of conditions: E.g., each speaker’s audio is applied only within their spatial mask, permitting multiple identities to speak, move, and interact naturally and independently.
• A recursive, iterative process: Masks are predicted and updated at every step, directly guiding the modality injection as generation progresses.

This paradigm overcomes the limitations of previous global or attention-based approaches, which cannot deliver precise speaker or action control in multi-concept scenarios.

2. Region-Specific Binding and Mask-Guided Injection

Region-specific binding in InterActHuman is realized via explicit per-person masks, which denote where in space-time each subject is present and hence where to inject that subject's modality cues. The process is as follows:

• Reference cues (person images, audio samples) are provided for each subject.
• The model predicts, at each block and for each concept $i$, a soft mask $m_i \in [0,1]^T$ across video tokens using cross-attention between the current video features and the per-concept reference features. The attention output is processed by a multilayer perceptron and sigmoid activation.
• Local audio injection: For each identity $i$, audio features $a_i$ from a backbone such as wav2vec2 are injected only into video regions where mask $m_i$ is high, with all other regions receiving a "muted" signal.
• The process is formulated as:

$$\mathbf{h}^v \leftarrow \mathbf{h}^v + m_i \odot \mathbf{p}_i + (1-m_i) \odot \mathbf{p}_i^{\mathrm{mute}}$$

where $\odot$ denotes elementwise multiplication, $\mathbf{p}_i$ is the cross-attention result (audio), and $\mathbf{p}_i^{\mathrm{mute}}$ is the muted feature for non-speaking regions.

This explicit layout-based approach ensures that each subject’s lip movements, facial actions, and audio are synchronized only in their respective zones, enabling compositional video generation (i.e., multiple separate speakers or actors in the same scene).

3. Iterative Layout Alignment During Diffusion

During video generation via diffusion, InterActHuman predicts and updates the layout masks for each concept at every denoising step $k$:

• At time $k$, each DiT (Diffusion Transformer) block outputs updated hidden states for video and reference cues.
• The mask $m_i^{(l)}$ for concept $i$ in block $l$ is derived as

$$a_i^{(l)} = \operatorname{softmax}\left( \frac{\mathbf{Q}^v (\mathbf{K}_i^r)^{\top}}{\sqrt{d}} \right)\mathbf{V}_i^r$$

$$m_i^{(l)} = \operatorname{sigmoid}(\mathrm{MLP}(a_i^{(l)}))$$

• Across the final layers, masks are averaged to produce a consensus mask per concept per step. At step $k+1$, the model uses these masks to inject local audio/appearance features.
• Early diffusion steps use only global or no injection (allowing coarse layout emergence), with fine-grained, mask-based local injection activated in later steps.

This recursive process co-evolves the video layout and the assignment of modalities, progressively improving both the spatial alignment and the realism of the generated output.

4. Empirical Validation and Comparative Performance

Empirical results demonstrate the advantage of layout-aligned modality matching using region-specific, iterative binding:

• Lip-sync metrics (Sync-D↓, Sync-C↑): InterActHuman achieves superior synchronization between speakers’ lip motion and their own audio, especially in multi-person settings. Quantitatively, Sync-D reaches 6.67 versus 9.48 for OmniHuman without masks, and FVD drops to 22.88 compared to 33.90 or more for alternatives.
• Visual identity retention metrics (CLIP-I, DINO-I, face similarity): Scores are uniformly higher, indicating that explicit masking preserves subject appearances better when multiple people are present, avoiding cross-contamination and swaps.
• Ablation studies: Removing dynamic mask prediction or reverting to global audio injection leads to significant degradation—e.g., non-speaking subjects erroneously exhibit lip movement, or mask artifacts arise with fixed assignments.
• Subjective user preference: Large gains in user ratings for both lip sync and subject consistency, confirming improved controllability and realism.

5. Significance and Distinction from Previous Approaches

The InterActHuman layout-aligned modality matching framework marks a substantial step beyond earlier attention-based or global-conditioning models:

• Global approaches cannot restrict modality influence to particular spatial regions, leading to unnatural or entangled outputs (e.g., all faces speaking at once).
• Previous ID embedding approaches rely on global associations and are less reliable with complex, shifting layouts or dynamic scenes.
• Explicit, iterative binding with mask-predicted local injection supports dynamic scenes with moving speakers, turn-taking, and transient interactions.

This enables applications such as multi-party dialogue animation, collaborative action video synthesis, and controllable generation for virtual humans in entertainment, telepresence, and robotics.

6. Technical Implementation Details

Key implementation elements include:

• Mask prediction module recursively trained/supervised (using ground-truth masks, e.g., from Grounding-SAM).
• Backbone: DiT-based video diffusion transformer operating with both visual and reference tokens at each step.
• Audio/text/image modalities: All are fused through attention and per-mask injection operators.
• Training objective:

$$\mathcal{L} = \mathbb{E}_{t, z_0, \epsilon} \left\| v_{\Theta}(z_t, t, c_{img}, c_{audio}) - (z_1 - z_0) \right\|_2^2$$

with auxiliary mask supervision losses as needed.

7. Limitations and Future Directions

While the mask-based, layout-aligned matching framework demonstrates major advances in controllable, multi-modal video generation, some challenges remain:

• Effective mask prediction depends on robust dataset curation and may be sensitive to severe occlusions or fast motion.
• Extremely granular alignment (e.g., finger movements or small accessory regions) could require more sophisticated spatial modeling.
• Extending region-specific multimodal matching to 3D or more complex multi-object, multi-action scenarios is an open direction.

A plausible implication is that future systems will extend these alignment mechanisms to even more elaborate, open-ended, and dynamic multi-modal interaction settings, potentially enabling autonomous synthesis of highly complex, agent-rich scenes while preserving control and identity for each participant.