AlignDiT: Multimodal Diffusion Transformer

• The paper demonstrates that integrating cross-attention in DiT blocks yields synchronized and speaker-faithful speech synthesis.
• AlignDiT is a multimodal framework that fuses text, video, and audio cues to generate mel-spectrograms through iterative diffusion and tailored guidance.
• Empirical results show improved MOS, reduced WER, and enhanced audiovisual alignment, proving efficacy in tasks like ADR and video-to-speech synthesis.

AlignDiT refers to a multimodal Aligned Diffusion Transformer architecture for synchronized speech generation, designed to produce temporally precise, natural, and speaker-accurate speech from multiple input modalities—text, video, and reference audio. This approach systematically addresses core challenges in multimodal-to-speech synthesis, including audio-video synchronization, speech intelligibility, voice similarity to reference speakers, and overall naturalness, by leveraging adaptable DiT-style blocks, tailored multimodal fusion strategies, and a novel classifier-free guidance system for balancing conditional influence from heterogeneous modalities.

1. Architectural Framework

AlignDiT builds upon the Diffusion Transformer (DiT) paradigm, where the model generates mel-spectrograms by denoising a sequence of corrupted latents through iterative transformer blocks controlled by diffusion timestep embeddings. Each block is extended to accept conditioning from fused representations derived from text, video, and speaker reference audio, formalized as:

$$h_{\text{av}} = [ (1-M) \odot h_{\text{audio}};~ M \odot h_{\text{video}} ] \in \mathbb{R}^{T \times 2D}$$

where $h_{\text{audio}}$ and $h_{\text{video}}$ are projected to compatible frame rates, $M$ is a binary mask for inpainting, $T$ is the temporal dimension, and $D$ is the channel width. Text features $h_{\text{text}}$ are encoded separately and refined via convolutional encoding before integration.

The fused multimodal context is injected into each DiT block, enabling precise temporal and semantic alignment across modalities.

2. Multimodal Fusion and Alignment Strategies

Three fusion strategies were systematically evaluated:

• Early Fusion Self-Attention: Concatenates all modality representations and applies standard self-attention, with sequence length normalization via padding (E2 TTS-style). Input tensor: $h' = [h_{\text{av}}; h_{\text{text}}]$.
• Prefix Self-Attention: Attaches text features as a prefix to audio-video features. The concatenated result is then processed, with the prefix later discarded. Input tensor: $$h' = \text{Concat}(h_{\text{text}}, h_{\text{av}}) \in \mathbb{R}^{(L+T) \times 2D}$$, with $L$ the text sequence length.
• Multimodal Cross-Attention: The core method in AlignDiT, introduces a multi-head cross-attention (MHCA) module in each block, letting $h_{\text{av}}$ act as query and $h_{\text{text}}$ as key/value:

$$h = \text{MHCA}( h_{\text{av}} W_Q, h_{\text{text}} W_K, h_{\text{text}} W_V ) \in \mathbb{R}^{T \times D}$$

Alignment is achieved by explicitly allowing transformer blocks to learn how to combine linguistic and audiovisual features at each diffusion timestep, preserving both the temporal and semantic structure.

Empirical analysis demonstrated that cross-attention yields the highest performance for synchronized, accurate, and speaker-faithful speech synthesis.

3. Multimodal Classifier-Free Guidance (CFG)

Classifier-free guidance is extended from traditional conditional/unconditional blending to a multimodal setting, enabling adaptive balancing of text, video, and reference audio signals. The output at each diffusion step is:

$$v_{t,\text{CFG}} = v_t(x_t, h) + s_t \cdot [ v_t(x_t, h_{\text{text}}) - v_t(x_t, \varnothing)] + s_v \cdot [ v_t(x_t, h) - v_t(x_t, h_{\text{text}}) ]$$

where $v_t$ estimates the generative vector field, $s_t$ and $s_v$ control the influence of text and non-text (video/audio) modalities, and $h$ is the full multimodal conditioning. Modality dropout is employed during training for robustness.

This mechanism allows real-time trade-off between intelligibility of generated speech (text-driven), synchronization with lip movements (video-driven), and speaker similarity (reference audio-driven), supporting diverse use cases and data availability scenarios.

4. Experimental Results and Evaluation

Benchmark evaluations of AlignDiT revealed:

• Subjective quality: Highest MOS (Mean Opinion Score) for naturalness and speaker similarity versus earlier models (HPMDubbing, StyleDubber).
• Objective metrics: Lowest word error rate (WER), highest AVSync, and spkSIM scores, indicating accurate temporal synchronization and speaker matching.
• Video-to-speech performance: Maintained intelligibility and lip synchronization even when only silent video was available, outperforming dedicated audiovisual synthesis pipelines.
• Visual forced alignment (VFA): Produced word- or phoneme-level timestamps that exceeded performance of prior forced alignment models.

Results indicate superior performance in both conventional and challenging scenarios, with marked improvements in synchrony and multimodal integration.

5. Generalization to Related Tasks

AlignDiT demonstrates effective generalization across multiple speech-related multimodal applications:

• ADR (Automated Dialogue Replacement): Generates dubbed speech synchronized to pre-existing lip movements and audio-visual cues.
• Video-to-Speech Synthesis: Capable of producing natural speech aligned to video alone or in combination with pseudo-transcripts from lip reading systems.
• Visual Forced Alignment: Generates temporally-aligned speech facilitating high-precision forced alignment for ASR or linguistic annotation tasks.

The unified conditional framework and in-context learning support migration between task protocols without architectural modifications or additional duration prediction modules.

6. Real-World Applications and Implications

AlignDiT's design and performance suitability extend to:

• Film production and dubbing: Automates high-quality speech synthesis synchronized with on-screen actors, reducing manual post-processing.
• Virtual avatars and digital assistants: Enables lifelike multimodal characters that integrate speech with facial movements and contextual text or audio cues.
• Accessibility and archival enhancement: Generates synchronized narration for silent historical footage or multimodal educational material.
• General speech research: Serves as a framework for multimodal representation learning, signal fusion, and weakly supervised alignment tasks.

The avoidance of external duration predictors or aligners, combined with end-to-end trainability and modality-agnostic conditioning, mark a significant advancement for both research and deployment of multimodal speech synthesis.

7. Open Challenges and Future Directions

Despite empirical success, further research is encouraged in:

• Scaling to higher-dimensional and non-standard modalities (e.g., gesture, multilingual input).
• Exploring sparsity, pruning, and efficiency improvements for real-time use in resource-constrained environments.
• Investigating self-supervised alignment objectives and domain adaptation for cross-domain downstream applications (e.g., medical, legal, conversational AI).
• Fine-grained controllability in CFG at sub-word or sub-frame temporal resolutions.

Continued work may refine alignment and synchronization mechanisms, improve generalization, and deepen the integration between separate multimodal data sources. Potential implications include streamlined pipelines for ADR, scalable video-to-speech tools, and robust avatars for cross-lingual dialogue and accessibility technologies.