CineDub-CN: Neural Video Dubbing

• CineDub-CN is a neural automatic video dubbing system that integrates audiovisual synchronization, prosody transfer, and speaker timbre adaptation.
• It employs a multi-modal non-autoregressive TTS model that fuses phoneme sequences, video frames, and face images to generate synchronized speech.
• The system uses advanced alignment techniques and end-to-end loss optimization to achieve high-quality dubbing with precise lip movements and natural prosody.

CineDub-CN is a system conceptually informed by neural automatic video dubbing (AVD) research, characterized by components that jointly optimize audiovisual synchronization, prosody transfer, and speaker timbre adaptation. The architecture and methodology derive from the framework introduced in Neural Dubber (Hu et al., 2021), integrating advances in multi-modal text-to-speech (TTS) with face-driven speaker modeling to address the specific constraints of high-quality video dubbing.

1. Core Model Architecture

The foundation of CineDub-CN as inferred from Neural Dubber is a multi-modal, non-autoregressive TTS model designed for fast, inference-parallel generation of speech tightly synchronized to video input. The system ingests three primary modalities:

• Phoneme Sequence ($S_p = \{P_1, ..., P_{T_p}\}$): Encoded using an embedding layer followed by $N=4$ Feed-Forward Transformer (FFT) blocks, mapping to $\mathcal{H}_{pho} \in \mathbb{R}^{T_p \times d}$ with $d=256$.
• Video Frame Sequence ($S_v = \{I_1, ..., I_{T_v}\}$): Each $I_j$ is a 96x96 mouth crop, processed via a ResNet18 backbone (with an initial 3D convolution) and $K=2$ FFT blocks, producing $\mathcal{H}_{vid} \in \mathbb{R}^{T_v \times d}$.
• Face Image ($I^f$; multi-speaker only): 224x224, used for timbre modeling via the Image-based Speaker Embedding (ISE) module.

A text-video aligner applies scaled-dot-product attention with $\mathcal{H}_{vid}$ as queries and $\mathcal{H}_{pho}$ as keys/values, outputting $\mathcal{H}_{con} \in \mathbb{R}^{T_v \times d}$, which is upsampled to match target mel-frame rates. Downstream, a variance adaptor predicts pitch and energy (akin to FastSpeech 2), and a 4-layer FFT mel-spectrogram decoder maps $$\mathbb{R}^{T_m\times d} \rightarrow Y \in \mathbb{R}^{T_m\times80}$$.

2. Multi-modal Fusion and Prosody Control

CineDub-CN implements multi-modal fusion for both prosody and speaker identity. Prosody control exploits an attention-based fusion:

$$\text{Attention}(\mathcal{H}_{vid}, \mathcal{H}_{pho}, \mathcal{H}_{pho}) = \text{Softmax}\left(\frac{\mathcal{H}_{vid}\mathcal{H}_{pho}^T}{\sqrt{d}}\right) \cdot \mathcal{H}_{pho}$$

This mechanism allows video-derived temporal dynamics—specifically, lip movement sequences—to modulate generated speech rhythm and duration, surpassing approaches that rely solely on textual phoneme timing. Speaker-face fusion (for timbre) is achieved by broadcasting the speaker vector $e_{spk}$ to $\mathcal{H}_{mel}$, enabling seamless, automatic adaptation to new identities based on facial appearance.

Variance fusion introduces predicted pitch and energy values directly into the representation:

$$\mathcal{H}_{out} = \mathcal{H}_{mel} + \text{PitchPredictor}(\mathcal{H}_{mel}) + \text{EnergyPredictor}(\mathcal{H}_{mel})$$

3. Synchronization and Alignment Mechanisms

Temporal synchronization between dubbed audio and visual frames is realized through monotonic attention and diagonal constraint loss. The alignment rate $r$ is defined as:

$$r = \frac{1}{T_v} \sum_{s=1}^{T_v} \sum_{t = \max(ks - b, 1)}^{\min(ks + b, T_p)} A_{s,t}$$

where $k = T_p / T_v$ and $b$ is the window width. The diagonal constraint loss ($L_{DC} = -r$) encourages the cross-modal attention matrix $A$ to match the monotonic progression typical of human speech and lip movement. An upsampling factor $n = 4$ (derived from $$(\text{audio\_sr}/\text{hop\_size}) / \text{video\_FPS}$$) ensures temporal alignment to mel-frame rates.

4. Speaker Timbre Adaptation

The ISE module learns an embedding $e_{spk}$ unique to a speaker's face, without reliance on explicit speaker-ID or contrastive loss. The face is processed by a fixed ResNet-50 backbone yielding a 4096-D feature $\phi$, followed by an MLP to produce $e_{spk} \in \mathbb{R}^{256}$. The embedding is optimized end-to-end via the mel-reconstruction loss, ensuring that timbral variation in the synthesized speech is congruent with the visual speaker identity.

5. Training Objective and Loss Functions

Loss is structured as the sum of:

• Mel-spectrogram L1 loss: $L_{mel} = \| Y - \hat{Y} \|_1$
• Pitch prediction loss: $L_{pitch} = \| P - \hat{P} \|_1$
• Energy prediction loss: $L_{energy} = \| E - \hat{E} \|_1$
• Diagonal constraint loss: $L_{DC} = -r$
• Total loss: $$L = L_{mel} + \alpha_1 L_{pitch} + \alpha_2 L_{energy} + \alpha_3 L_{DC}$$, with $\alpha_1, \alpha_2$ as in FastSpeech 2 and $\alpha_3$ set empirically.

This compound objective enables end-to-end optimization of content fidelity, prosodic naturalness, and AV alignment.

6. Evaluation Methodologies and Results

Experimental protocols employ both subjective and objective measures:

• Subjective: Mean Opinion Score (MOS, 1–5) for audio quality and AV sync (≥20 raters × 30 clips).
• Objective: LSE-D (lower is better), LSE-C (higher is better) computed via SyncNet for synchronization; STOI, ESTOI, PESQ, and WER for intelligibility and perceptual quality.

Table: Example Results for Neural Dubber (Mel+PWG) (Hu et al., 2021)

Dataset	Audio Quality	AV Sync	LSE-D	LSE-C
Chem (Single spk)	3.74±0.08	3.91±0.07	7.212	7.037
LRS2 (Multi spk)	3.58±0.13	3.62±0.09	7.201	6.861

Qualitative analysis shows that generated mel-spectrograms closely approximate target prosody, and t-SNE visualization of speaker embeddings verifies clustering by face-derived identity. Lip2Wav baselines are substantially outperformed in STOI, ESTOI, PESQ, and WER.

7. Implications for the Design of CineDub-CN

The architectural choices, fusion strategies, and training/evaluation pipelines established in Neural Dubber provide a direct blueprint for CineDub-CN. Notable contributions include:

• Video-driven attention: Enables fine-grained alignment of speech with visual cues, critical for faithful prosody and sync.
• Automatic speaker adaptation: Face-derived embedding obviates need for speaker-specific data or manual lookup, supporting large-scale, diverse dubbing applications.
• Parallel, non-autoregressive generation: Ensures scalability for production environments.
• Stable monotonic alignment: Diagonal constraint provides a trainable, in-model alternative to post-processing alignment strategies, enhancing reliability.
• End-to-end loss: Simplifies optimization without decomposing into disparate sub-tasks.
• Comprehensive evaluation: The same suite of MOS, LSE-D/C, WER, and related metrics forms a robust benchmark for AVD system assessment.

By integrating these mechanisms, CineDub-CN can achieve improved lip–audio synchronization, more faithful prosody transfer, and automatic speaker timbre adaptation, all within an efficient, production-ready framework (Hu et al., 2021).