---
title: 'DualDub: Joint Video-to-Soundtrack Generation'
url: https://www.emergentmind.com/topics/dualdub
type: topic
---

# DualDub: Joint Video-to-Soundtrack Generation

DualDub is a unified framework for **video-to-soundtrack (V2ST) generation** that jointly produces synchronized **background audio** and **speech** from a **silent video** \(V\), a **transcript** \(T\), and a **reference speech prompt** \(R\). Introduced in “DualDub: Video-to-Soundtrack Generation via Joint Speech and Background Audio Synthesis” [2507.10109], it extends earlier video-to-audio formulations by treating the soundtrack as a coordinated acoustic scene rather than as two independently synthesized streams. In the paper’s formulation,
\[
\hat{W}^{\text{speech}}, \hat{W}^{\text{audio}} = \text{DualDub}(R, T, V),
\]
with the central claim that speech and background audio should be generated within one model so that synchronization, loudness balance, and acoustic harmony are learned jointly rather than imposed after mixing [2507.10109].

## 1. Task formulation and conceptual scope

DualDub defines **V2ST** as a distinct task from conventional **V2A**. In V2A, the objective is to generate background or event audio from visual content. In V2ST, the objective is to generate the **entire soundtrack**, including both **background audio** and **dubbed speech**. The paper argues that prior V2A systems may contain speech examples in training data, but because they do not explicitly model transcript or speaker information, any speech they emit is typically **“unintelligible,” “rumbling-like,” and “uncontrollable”** [2507.10109].

The model is motivated by three deficiencies in prior pipelines. First, prior V2A largely ignores structured speech generation. Second, generating background audio and speech independently and then mixing them often yields timing conflicts, inconsistent volume balance, and acoustic mismatch. Third, fully paired **video–text–speech–audio** data are scarce, making direct end-to-end training difficult [2507.10109]. DualDub therefore frames soundtrack generation as a **joint conditional autoregressive process** over speech and background-audio token streams:
\[
\begin{split}
& p(A_t^{\text{speech}}, A_t^{\text{audio}} \mid A^{\text{speech}}_{<t}, A^{\text{audio}}_{<t}, V, T, R) \\
&= \prod_{t=0}^{T} p(A_t^{\text{speech}}, A_t^{\text{audio}} \mid A^{\text{speech}}_{<t}, A^{\text{audio}}_{<t}, V, T, R; \theta^{LM}, \theta^{Align}).
\end{split}
\]
This formulation makes synchronization and cross-stream compatibility part of the generative model itself rather than a downstream composition step [2507.10109].

This scope distinguishes DualDub from speech-dubbing systems whose sole target is dubbed speech aligned to video. For example, VoiceCraft-Dub generates speech from **text and facial cues** [2504.02386], while MM-MovieDubber and DeepDubber-V1 condition speech generation on **video understanding conclusions** such as scene type, speaker age, gender, and emotion [2505.16279], [2503.23660]. DualDub instead addresses the broader soundtrack problem by modeling speech and background audio together [2507.10109].

## 2. Architecture

DualDub comprises three principal components: a **multimodal encoder**, a **cross-modal aligner**, and a **multimodal language model with dual decoding heads**, followed by a waveform synthesis stage [2507.10109]. The multimodal encoder processes four input types. For text, it uses **Byte Pair Encoding**, implemented with **tiktoken BPE**. For speaker identity, it takes a **3-second random crop** of target speech during training, extracts a mel-spectrogram, passes it through a speaker encoder, and average-pools to a global speaker embedding. For acoustic tokenization, both background audio and speech are tokenized with **WavTokenizer** at **24 kHz**, **hop size 600**, and **40 tokens per second**, but the model uses **two separate lookup tables** for background-audio tokens and speech tokens. For video, frames are encoded by **CLIP ViT-B/32**, sampled at **one image every three frames**, then resampled with nearest-neighbor interpolation to match the acoustic token rate [2507.10109].

The most distinctive architectural element is the **cross-modal aligner**. It uses two types of cross-attention. **Intra-modal causal cross-attention** models interactions between the generated speech and background-audio streams. There are two modules—audio querying speech and speech querying audio—and a causal mask prevents either stream from accessing the other stream’s future tokens. **Inter-modal non-causal cross-attention** links both acoustic streams to video: audio queries attend to video keys/values, and speech queries attend to video keys/values, with non-causal access because the full video is available at inference time. The outputs of the four cross-attention modules are summed into the final fused representation [2507.10109]. This design encodes a specific asymmetry: audio and speech must remain causal with respect to each other, but both may exploit full visual context.

The fused sequence is processed by a shared **LLaMA-based multimodal language model** with **24 layers**, **16 attention heads**, and **hidden size 2048**. The paper states that this LM autoregressively processes the **speaker prompt**, **text sequence**, and **fused multimodal sequence**, then projects the hidden output through **two LM heads** to produce separate **speech** and **background-audio** token streams [2507.10109]. This dual-head design allows joint generation within a shared autoregressive backbone while preserving modality-specific output spaces.

The final waveform stage replaces the standard WavTokenizer decoder with a **two-stage token-to-latent-to-wave decoder**. A pretrained **StableAudio2 VAE** provides target latent representations \(Z_l\), and a **DiT-based flow-matching model** with **8 transformer layers**, **16 attention heads**, and **hidden size 2048** maps token embeddings into that latent space. The paper defines linear interpolation
\[
Z_t = (1-t)\cdot Z_0 + t \cdot Z_l
\]
and trains with
\[
\mathcal{L}_{FM} = \mathbb{E}_{Z_t, t} \left[ \left| v_{\theta}(Z_t, t) - \frac{Z_{l} - Z_0}{1} \right|_2^2 \right].
\]
Inference solves
\[
\hat{Z}_{l} = Z_0 + \int_{0}^{1} v_{\theta}(Z_t, t)\, dt,
\qquad
\hat{W}^{wave} = \theta^{VAE}(\hat{Z}_{l}),
\]
with the VAE decoder frozen [2507.10109]. This decoder is intended to improve perceptual quality beyond what the original codec decoder can provide.

## 3. Training methodology and data

DualDub addresses paired-data scarcity through a **three-stage curriculum learning strategy**. In **Stage 1**, the model is trained for **video-to-audio** using video \(V\) and background-audio tokens \(A^{\text{audio}}\), with the fixed text prompt **“Generate audio for the video.”** and an all-zero speaker prompt:
\[
p(A_t^{\text{audio}} \mid A_{<t}^{\text{audio}}, V).
\]
In **Stage 2**, the model adds **text-to-speech** while retaining V2A as a multitask objective:
\[
p(A_t^{\text{speech}} \mid A_{<t}^{\text{speech}}, T, R).
\]
Here the video input is replaced with all-zero vectors for TTS, and the same shared backbone is optimized through separate audio and speech heads. In **Stage 3**, the full V2ST objective is introduced, using paired supervision for joint speech-and-audio generation conditioned on \(V\), \(T\), and \(R\) [2507.10109]. The paper states that all three stages use **cross-entropy loss** for token prediction, while the waveform decoder is trained separately with the flow-matching loss.

The training corpora are heterogeneous by construction. The V2A stage uses **VGGSound** at about **500 h**. The TTS stage uses **LibriTTS** at about **550 h**. Fully paired multimodal supervision in the final stage uses **Anim** at about **100 h**, **V2C-Animation** at about **5 h**, and **private multimodal pairs** at about **100 h**. For the harmony metric model used in evaluation, the paper also uses **private audio-speech pairs** at about **1500 h** [2507.10109]. All audio-speech streams are separated according to the DualBench preprocessing pipeline to avoid speech leakage into the background track.

Optimization is performed on **8 NVIDIA A800 GPUs** with **Adam**, \(\beta_1 = 0.9\), \(\beta_2 = 0.98\), and \(\epsilon = 10^{-9}\). The schedule is cosine with **4000 warm-up steps**, global batch size **192**, and **40 epochs per stage**. Learning-rate ranges are **\(2\times10^{-6}\) to \(2\times10^{-4}\)** in Stage 1, **\(2\times10^{-7}\) to \(2\times10^{-4}\)** in Stage 2, and **\(2\times10^{-7}\) to \(2\times10^{-5}\)** in Stage 3. The **CLIP encoder** and **WavTokenizer** are frozen throughout; the optimized modules are the **multimodal aligner** and **language model** [2507.10109]. The ablation study reports that removing the curriculum worsens **FD** from **23.08** to **31.61**, **FAD** from **5.86** to **9.61**, **AV-Align** from **0.25** to **0.21**, **WER** from **12.74** to **15.40**, **UTMOS** from **2.70** to **2.58**, and **DualScore** from **0.59** to **0.56**, and also causes unstable and slow convergence [2507.10109].

## 4. DualBench and evaluation methodology

Because V2ST had no existing benchmark, the paper introduces **DualBench**, described as the first benchmark dedicated to joint evaluation of background audio, speech, synchronization, and cross-stream harmony [2507.10109]. The test set is curated from **V2C-Animation** by first merging multi-channel audio into mono, then using **Mel-RoFormer** to separate **human speech** and **background audio**. Clips are discarded if either separated stream has energy below **\(-40\ \mathrm{dB}\)**. This reduces the original **2,793** clips to **1,319 curated clips**, with average duration **2.65 seconds** [2507.10109].

DualBench evaluates four dimensions. For **background audio quality**, it reports **FD**, **FAD**, **IS**, and **KLD** using audioldm-eval. For **speech quality**, it reports **SIM** using a WavLM-based speaker verification model, **WER** using Whisper-large-v3, and **UTMOS** using SpeechMOS. For **audio-video synchronization**, it defines **AV-Align**, which detects audio and video peaks and computes their intersection-over-union. For **audio-speech harmony**, the paper introduces **DualScore**, computed as the cosine similarity between generated audio and speech embeddings from a contrastive model called **CASP** [2507.10109].

CASP is a dual-branch encoder with an **audio branch initialized from BEATs**, a speech branch of the same architecture trained from scratch, and attention pooling for aggregation. It is trained on random **5-second segments** from paired audio-speech data. To validate DualScore, the paper reports retrieval performance on the 1,319-pair test set: **Top-1 70%**, **Top-3 90%**, and **Top-5 95%** [2507.10109]. This is used as evidence that the learned embedding space meaningfully captures soundtrack-level correspondence between speech and background audio.

Subjective evaluation uses **10 non-expert listeners** and **10 listeners with professional audio knowledge**, who rate **16 sample pairs** with **95% confidence intervals** for **OAQ** (overall audio quality), **OSQ** (overall speech quality), and **ASCH** (audio-speech correspondence and harmony) [2507.10109]. The benchmark is thus designed around the claim that a soundtrack should be evaluated as a composite acoustic scene rather than as isolated components.

## 5. Empirical performance

On **DualBench speech metrics**, DualDub reports **WER 12.74**, **SIM 0.84**, and **UTMOS 2.70**. The corresponding baseline numbers are **WER 20.96, SIM 0.74, UTMOS 1.31** for HPMDubber; **WER 24.92, SIM 0.82, UTMOS 2.28** for Speaker2Dubber; and **WER 30.57, SIM 0.82, UTMOS 1.97** for StyleDubber [2507.10109]. The paper attributes this margin primarily to the multimodal LM-based speech generation backbone and its stronger zero-shot TTS capability.

For the full V2ST task, the principal reported gain is in **DualScore**. Concatenation baselines achieve **0.19** for Speaker2Dubber + MMAudio, **0.23** for StyleDubber + MMAudio, **0.23** for Speaker2Dubber + FoleyCrafter, and **0.30** for StyleDubber + FoleyCrafter, whereas **DualDub reaches 0.59** [2507.10109]. In subjective evaluation, DualDub achieves **OAQ 2.75 ± 0.08**, **OSQ 3.06 ± 0.07**, and **ASCH 3.44 ± 0.10**, while concatenation baselines remain at about **1.94–2.12** for OAQ, **2.62–2.81** for OSQ, and **1.44–1.69** for ASCH [2507.10109]. The paper emphasizes that the largest margin is in **ASCH**, consistent with the claim that joint generation improves acoustic harmony beyond what separate speech and audio models can provide.

On the V2A side, DualDub is described as competitive but not uniformly best on raw V2A metrics. On **VGGSound**, it reports **FD 11.95**, **FAD 2.29**, **KLD 2.91**, **IS 11.50**, and **AV-Align 0.24**. On **DualBench**, it reports **FD 23.80**, **FAD 5.86**, **KLD 1.79**, **IS 2.72**, and **AV-Align 0.25** [2507.10109]. The paper notes that **MMAudio** remains best on several pure audio metrics on VGGSound, and attributes part of DualDub’s relative weakness on some raw V2A scores to **WavTokenizer limitations** [2507.10109].

The ablation study isolates the role of the aligner and decoder. Removing the **cross-modal aligner** lowers **AV-Align** from **0.25** to **0.18** and **DualScore** from **0.59** to **0.36**, while not uniformly hurting token-level speech metrics [2507.10109]. Replacing the **flow-matching decoder** with the original WavTokenizer decoder causes a larger collapse: **FD 42.61**, **FAD 13.61**, **KLD 6.74**, **IS 1.90**, **AV-Align 0.19**, and **DualScore 0.41** [2507.10109]. This supports the paper’s decomposition of responsibilities: the aligner primarily improves synchronization and harmony, while the flow-matching decoder primarily improves acoustic fidelity.

## 6. Relation to adjacent dubbing research, limitations, and outlook

DualDub occupies a distinct position within the broader dubbing literature. It is not primarily a **speech translation** model such as Dub-S2ST, which targets **textless speech-to-speech translation for seamless dubbing** with explicit duration control and speaker-preserving synthesis [2505.20899]. It is not primarily a **speech-from-text-and-video** model such as VoiceCraft-Dub, which generates speech from **target text**, **source speech**, and **target video** with lip-synchronization and facial expressiveness [2504.02386]. It also differs from movie-dubbing systems such as MM-MovieDubber and DeepDubber-V1, which first infer scene type and speaker attributes from video, then generate dubbed speech conditioned on those semantic controls [2505.16279], [2503.23660]. DualDub instead makes the entire **soundtrack**—speech plus background audio—the central output object [2507.10109].

The paper’s design also contrasts with visual-dubbing and visual-editing work. EdiDub reformulates visual dubbing as **content-aware editing** of the original video to preserve occlusions, lighting, and local realism [2505.23406], while the self-bootstrapping editing framework “From Inpainting to Editing: A Self-Bootstrapping Framework for Context-Rich Visual Dubbing” argues that full-frame editing with paired synthetic companions is better conditioned than mask-based inpainting [2512.25066]. Those systems solve the visual side of dubbing; DualDub instead addresses acoustic scene generation. A plausible implication is that the two lines of work are complementary rather than competing: one could pair a V2ST generator such as DualDub with an editing-based visual dubbing backend to obtain a fuller audiovisual dubbing system.

The limitations acknowledged by the paper are specific and consequential. First, **WavTokenizer’s discrete tokens lack fine-grained acoustic detail**, which limits naturalness for both speech and background audio [2507.10109]. Second, **Mel-RoFormer separation is imperfect**, sometimes yielding empty speech or audio segments and forcing substantial data filtering [2507.10109]. The paper does not specify a hard maximum clip length, analyze long-form generation stability, or report inference latency; this suggests that the reported setting is best understood as short-clip soundtrack generation rather than fully characterized long-form dubbing [2507.10109]. The main future direction named in the paper is to replace discrete tokenization with **continuous representations**, following recent speech-language-model trends, to improve generation quality [2507.10109].

Within dubbing research more broadly, DualDub therefore marks a shift from **speech-only dubbing** to **soundtrack-aware dubbing**. Its central contribution is not merely a better speech model or a better V2A model, but the proposition that soundtrack generation should be learned as a joint multimodal problem in which **speech**, **background audio**, and **video** constrain one another throughout generation [2507.10109].

Source: https://www.emergentmind.com/topics/dualdub