DualDub: Joint Video-to-Soundtrack Generation

• DualDub is a unified framework for video-to-soundtrack generation that jointly synthesizes coherent speech and background audio from silent videos, transcripts, and reference prompts.
• It employs a multimodal encoder, cross-modal aligner, and dual-decoding language model to ensure synchronized timing, loudness balance, and acoustic harmony.
• Through a three-stage curriculum learning and a novel benchmark (DualBench), DualDub demonstrates superior performance in speech intelligibility and audio-speech harmony compared to independent synthesis approaches.

DualDub is a unified framework for video-to-soundtrack (V2ST) generation that jointly produces synchronized background audio and speech from a silent video $V$, a transcript $T$, and a reference speech prompt $R$. Introduced in “DualDub: Video-to-Soundtrack Generation via Joint Speech and Background Audio Synthesis” (Tian et al., 14 Jul 2025), it extends earlier video-to-audio formulations by treating the soundtrack as a coordinated acoustic scene rather than as two independently synthesized streams. In the paper’s formulation,

$$\hat{W}^{\text{speech}}, \hat{W}^{\text{audio}} = \text{DualDub}(R, T, V),$$

with the central claim that speech and background audio should be generated within one model so that synchronization, loudness balance, and acoustic harmony are learned jointly rather than imposed after mixing (Tian et al., 14 Jul 2025).

1. Task formulation and conceptual scope

DualDub defines V2ST as a distinct task from conventional V2A. In V2A, the objective is to generate background or event audio from visual content. In V2ST, the objective is to generate the entire soundtrack, including both background audio and dubbed speech. The paper argues that prior V2A systems may contain speech examples in training data, but because they do not explicitly model transcript or speaker information, any speech they emit is typically “unintelligible,” “rumbling-like,” and “uncontrollable” (Tian et al., 14 Jul 2025).

The model is motivated by three deficiencies in prior pipelines. First, prior V2A largely ignores structured speech generation. Second, generating background audio and speech independently and then mixing them often yields timing conflicts, inconsistent volume balance, and acoustic mismatch. Third, fully paired video–text–speech–audio data are scarce, making direct end-to-end training difficult (Tian et al., 14 Jul 2025). DualDub therefore frames soundtrack generation as a joint conditional autoregressive process over speech and background-audio token streams: $\begin{split} & p(A_t^{\text{speech}}, A_t^{\text{audio}} \mid A^{\text{speech}}_{<t}, A^{\text{audio}}_{<t}, V, T, R) \ &= \prod_{t=0}^{T} p(A_t^{\text{speech}}, A_t^{\text{audio}} \mid A^{\text{speech}}_{<t}, A^{\text{audio}}_{<t}, V, T, R; \theta^{LM}, \theta^{Align}). \end{split}$ This formulation makes synchronization and cross-stream compatibility part of the generative model itself rather than a downstream composition step (Tian et al., 14 Jul 2025).

This scope distinguishes DualDub from speech-dubbing systems whose sole target is dubbed speech aligned to video. For example, VoiceCraft-Dub generates speech from text and facial cues (Sung-Bin et al., 3 Apr 2025), while MM-MovieDubber and DeepDubber-V1 condition speech generation on video understanding conclusions such as scene type, speaker age, gender, and emotion (Zheng et al., 22 May 2025, Zheng et al., 31 Mar 2025). DualDub instead addresses the broader soundtrack problem by modeling speech and background audio together (Tian et al., 14 Jul 2025).

2. Architecture

DualDub comprises three principal components: a multimodal encoder, a cross-modal aligner, and a multimodal LLM with dual decoding heads, followed by a waveform synthesis stage (Tian et al., 14 Jul 2025). The multimodal encoder processes four input types. For text, it uses Byte Pair Encoding, implemented with tiktoken BPE. For speaker identity, it takes a 3-second random crop of target speech during training, extracts a mel-spectrogram, passes it through a speaker encoder, and average-pools to a global speaker embedding. For acoustic tokenization, both background audio and speech are tokenized with WavTokenizer at 24 kHz, hop size 600, and 40 tokens per second, but the model uses two separate lookup tables for background-audio tokens and speech tokens. For video, frames are encoded by CLIP ViT-B/32, sampled at one image every three frames, then resampled with nearest-neighbor interpolation to match the acoustic token rate (Tian et al., 14 Jul 2025).

The most distinctive architectural element is the cross-modal aligner. It uses two types of cross-attention. Intra-modal causal cross-attention models interactions between the generated speech and background-audio streams. There are two modules—audio querying speech and speech querying audio—and a causal mask prevents either stream from accessing the other stream’s future tokens. Inter-modal non-causal cross-attention links both acoustic streams to video: audio queries attend to video keys/values, and speech queries attend to video keys/values, with non-causal access because the full video is available at inference time. The outputs of the four cross-attention modules are summed into the final fused representation (Tian et al., 14 Jul 2025). This design encodes a specific asymmetry: audio and speech must remain causal with respect to each other, but both may exploit full visual context.

The fused sequence is processed by a shared LLaMA-based multimodal LLM with 24 layers, 16 attention heads, and hidden size 2048. The paper states that this LM autoregressively processes the speaker prompt, text sequence, and fused multimodal sequence, then projects the hidden output through two LM heads to produce separate speech and background-audio token streams (Tian et al., 14 Jul 2025). This dual-head design allows joint generation within a shared autoregressive backbone while preserving modality-specific output spaces.

The final waveform stage replaces the standard WavTokenizer decoder with a two-stage token-to-latent-to-wave decoder. A pretrained StableAudio2 VAE provides target latent representations $Z_l$, and a DiT-based flow-matching model with 8 transformer layers, 16 attention heads, and hidden size 2048 maps token embeddings into that latent space. The paper defines linear interpolation

$Z_t = (1-t)\cdot Z_0 + t \cdot Z_l$

and trains with

$$\mathcal{L}_{FM} = \mathbb{E}_{Z_t, t} \left[ \left| v_{\theta}(Z_t, t) - \frac{Z_{l} - Z_0}{1} \right|_2^2 \right].$$

Inference solves

$$\hat{Z}_{l} = Z_0 + \int_{0}^{1} v_{\theta}(Z_t, t)\, dt, \qquad \hat{W}^{wave} = \theta^{VAE}(\hat{Z}_{l}),$$

with the VAE decoder frozen (Tian et al., 14 Jul 2025). This decoder is intended to improve perceptual quality beyond what the original codec decoder can provide.

3. Training methodology and data

DualDub addresses paired-data scarcity through a three-stage curriculum learning strategy. In Stage 1, the model is trained for video-to-audio using video $V$ and background-audio tokens $T$0, with the fixed text prompt “Generate audio for the video.” and an all-zero speaker prompt: $T$1 In Stage 2, the model adds text-to-speech while retaining V2A as a multitask objective: $T$2 Here the video input is replaced with all-zero vectors for TTS, and the same shared backbone is optimized through separate audio and speech heads. In Stage 3, the full V2ST objective is introduced, using paired supervision for joint speech-and-audio generation conditioned on $T$3, $T$4, and $T$5 (Tian et al., 14 Jul 2025). The paper states that all three stages use cross-entropy loss for token prediction, while the waveform decoder is trained separately with the flow-matching loss.

The training corpora are heterogeneous by construction. The V2A stage uses VGGSound at about 500 h. The TTS stage uses LibriTTS at about 550 h. Fully paired multimodal supervision in the final stage uses Anim at about 100 h, V2C-Animation at about 5 h, and private multimodal pairs at about 100 h. For the harmony metric model used in evaluation, the paper also uses private audio-speech pairs at about 1500 h (Tian et al., 14 Jul 2025). All audio-speech streams are separated according to the DualBench preprocessing pipeline to avoid speech leakage into the background track.

Optimization is performed on 8 NVIDIA A800 GPUs with Adam, $T$6, $T$7, and $T$8. The schedule is cosine with 4000 warm-up steps, global batch size 192, and 40 epochs per stage. Learning-rate ranges are $T$9 to $R$0 in Stage 1, $R$1 to $R$2 in Stage 2, and $R$3 to $R$4 in Stage 3. The CLIP encoder and WavTokenizer are frozen throughout; the optimized modules are the multimodal aligner and LLM (Tian et al., 14 Jul 2025). The ablation study reports that removing the curriculum worsens FD from 23.08 to 31.61, FAD from 5.86 to 9.61, AV-Align from 0.25 to 0.21, WER from 12.74 to 15.40, UTMOS from 2.70 to 2.58, and DualScore from 0.59 to 0.56, and also causes unstable and slow convergence (Tian et al., 14 Jul 2025).

4. DualBench and evaluation methodology

Because V2ST had no existing benchmark, the paper introduces DualBench, described as the first benchmark dedicated to joint evaluation of background audio, speech, synchronization, and cross-stream harmony (Tian et al., 14 Jul 2025). The test set is curated from V2C-Animation by first merging multi-channel audio into mono, then using Mel-RoFormer to separate human speech and background audio. Clips are discarded if either separated stream has energy below $R$5. This reduces the original 2,793 clips to 1,319 curated clips, with average duration 2.65 seconds (Tian et al., 14 Jul 2025).

DualBench evaluates four dimensions. For background audio quality, it reports FD, FAD, IS, and KLD using audioldm-eval. For speech quality, it reports SIM using a WavLM-based speaker verification model, WER using Whisper-large-v3, and UTMOS using SpeechMOS. For audio-video synchronization, it defines AV-Align, which detects audio and video peaks and computes their intersection-over-union. For audio-speech harmony, the paper introduces DualScore, computed as the cosine similarity between generated audio and speech embeddings from a contrastive model called CASP (Tian et al., 14 Jul 2025).

CASP is a dual-branch encoder with an audio branch initialized from BEATs, a speech branch of the same architecture trained from scratch, and attention pooling for aggregation. It is trained on random 5-second segments from paired audio-speech data. To validate DualScore, the paper reports retrieval performance on the 1,319-pair test set: Top-1 70%, Top-3 90%, and Top-5 95% (Tian et al., 14 Jul 2025). This is used as evidence that the learned embedding space meaningfully captures soundtrack-level correspondence between speech and background audio.

Subjective evaluation uses 10 non-expert listeners and 10 listeners with professional audio knowledge, who rate 16 sample pairs with 95% confidence intervals for OAQ (overall audio quality), OSQ (overall speech quality), and ASCH (audio-speech correspondence and harmony) (Tian et al., 14 Jul 2025). The benchmark is thus designed around the claim that a soundtrack should be evaluated as a composite acoustic scene rather than as isolated components.

5. Empirical performance

On DualBench speech metrics, DualDub reports WER 12.74, SIM 0.84, and UTMOS 2.70. The corresponding baseline numbers are WER 20.96, SIM 0.74, UTMOS 1.31 for HPMDubber; WER 24.92, SIM 0.82, UTMOS 2.28 for Speaker2Dubber; and WER 30.57, SIM 0.82, UTMOS 1.97 for StyleDubber (Tian et al., 14 Jul 2025). The paper attributes this margin primarily to the multimodal LM-based speech generation backbone and its stronger zero-shot TTS capability.

For the full V2ST task, the principal reported gain is in DualScore. Concatenation baselines achieve 0.19 for Speaker2Dubber + MMAudio, 0.23 for StyleDubber + MMAudio, 0.23 for Speaker2Dubber + FoleyCrafter, and 0.30 for StyleDubber + FoleyCrafter, whereas DualDub reaches 0.59 (Tian et al., 14 Jul 2025). In subjective evaluation, DualDub achieves OAQ 2.75 ± 0.08, OSQ 3.06 ± 0.07, and ASCH 3.44 ± 0.10, while concatenation baselines remain at about 1.94–2.12 for OAQ, 2.62–2.81 for OSQ, and 1.44–1.69 for ASCH (Tian et al., 14 Jul 2025). The paper emphasizes that the largest margin is in ASCH, consistent with the claim that joint generation improves acoustic harmony beyond what separate speech and audio models can provide.

On the V2A side, DualDub is described as competitive but not uniformly best on raw V2A metrics. On VGGSound, it reports FD 11.95, FAD 2.29, KLD 2.91, IS 11.50, and AV-Align 0.24. On DualBench, it reports FD 23.80, FAD 5.86, KLD 1.79, IS 2.72, and AV-Align 0.25 (Tian et al., 14 Jul 2025). The paper notes that MMAudio remains best on several pure audio metrics on VGGSound, and attributes part of DualDub’s relative weakness on some raw V2A scores to WavTokenizer limitations (Tian et al., 14 Jul 2025).

The ablation study isolates the role of the aligner and decoder. Removing the cross-modal aligner lowers AV-Align from 0.25 to 0.18 and DualScore from 0.59 to 0.36, while not uniformly hurting token-level speech metrics (Tian et al., 14 Jul 2025). Replacing the flow-matching decoder with the original WavTokenizer decoder causes a larger collapse: FD 42.61, FAD 13.61, KLD 6.74, IS 1.90, AV-Align 0.19, and DualScore 0.41 (Tian et al., 14 Jul 2025). This supports the paper’s decomposition of responsibilities: the aligner primarily improves synchronization and harmony, while the flow-matching decoder primarily improves acoustic fidelity.

6. Relation to adjacent dubbing research, limitations, and outlook

DualDub occupies a distinct position within the broader dubbing literature. It is not primarily a speech translation model such as Dub-S2ST, which targets textless speech-to-speech translation for seamless dubbing with explicit duration control and speaker-preserving synthesis (Choi et al., 27 May 2025). It is not primarily a speech-from-text-and-video model such as VoiceCraft-Dub, which generates speech from target text, source speech, and target video with lip-synchronization and facial expressiveness (Sung-Bin et al., 3 Apr 2025). It also differs from movie-dubbing systems such as MM-MovieDubber and DeepDubber-V1, which first infer scene type and speaker attributes from video, then generate dubbed speech conditioned on those semantic controls (Zheng et al., 22 May 2025, Zheng et al., 31 Mar 2025). DualDub instead makes the entire soundtrack—speech plus background audio—the central output object (Tian et al., 14 Jul 2025).

The paper’s design also contrasts with visual-dubbing and visual-editing work. EdiDub reformulates visual dubbing as content-aware editing of the original video to preserve occlusions, lighting, and local realism (Manela et al., 29 May 2025), while the self-bootstrapping editing framework “From Inpainting to Editing: A Self-Bootstrapping Framework for Context-Rich Visual Dubbing” argues that full-frame editing with paired synthetic companions is better conditioned than mask-based inpainting (He et al., 31 Dec 2025). Those systems solve the visual side of dubbing; DualDub instead addresses acoustic scene generation. A plausible implication is that the two lines of work are complementary rather than competing: one could pair a V2ST generator such as DualDub with an editing-based visual dubbing backend to obtain a fuller audiovisual dubbing system.

The limitations acknowledged by the paper are specific and consequential. First, WavTokenizer’s discrete tokens lack fine-grained acoustic detail, which limits naturalness for both speech and background audio (Tian et al., 14 Jul 2025). Second, Mel-RoFormer separation is imperfect, sometimes yielding empty speech or audio segments and forcing substantial data filtering (Tian et al., 14 Jul 2025). The paper does not specify a hard maximum clip length, analyze long-form generation stability, or report inference latency; this suggests that the reported setting is best understood as short-clip soundtrack generation rather than fully characterized long-form dubbing (Tian et al., 14 Jul 2025). The main future direction named in the paper is to replace discrete tokenization with continuous representations, following recent speech-language-model trends, to improve generation quality (Tian et al., 14 Jul 2025).

Within dubbing research more broadly, DualDub therefore marks a shift from speech-only dubbing to soundtrack-aware dubbing. Its central contribution is not merely a better speech model or a better V2A model, but the proposition that soundtrack generation should be learned as a joint multimodal problem in which speech, background audio, and video constrain one another throughout generation (Tian et al., 14 Jul 2025).