StableDub: Diffusion-Based Visual Dubbing

• StableDub is a diffusion-based visual dubbing framework that generates synchronized mouth movements from audio while retaining unique speaker lip habits.
• It employs a single U-Net latent diffusion model augmented with lip-habit and occlusion-aware conditioning to preserve identity, style, and background consistency.
• The framework improves training efficiency and robustness, outperforming prior methods on metrics such as FID, FVD, and LMD.

StableDub is a diffusion-based visual dubbing framework that generates mouth movements synchronized with driving audio on arbitrary avatar videos or images while preserving speaker-specific “lip habits” and remaining robust to mouth occlusions such as microphones and hands. It is formulated in a person-agnostic setting: the input is a target avatar video $$x^{1:N} \in \mathbb{R}^{N \times c \times h \times w}$$ or image together with a driving audio sequence $c_a^{1:N} \in \mathbb{R}^{N \times d}$, and the output is a dubbed video $\hat{x}^{1:N}$ in which the mouth region is modified so that lip motion follows the audio while identity, style, and background are largely preserved (Chen et al., 26 Sep 2025). The framework is motivated by two deficiencies in prior visual dubbing systems: audio-only driving paradigms that fail to capture speaker-specific orofacial dynamics, and blind inpainting approaches that erase occluding objects and therefore frequently produce artifacts under real-world occlusions (Chen et al., 26 Sep 2025).

1. Task definition and motivation

StableDub addresses visual dubbing as a generative editing problem in which only the speech-relevant facial region should change. In the formulation reported for the method, inputs may include human videos, 2D or 3D game characters, and anime characters; outputs are required to preserve identity, style, and background while producing lip motions that are synchronized with the new audio and resemble the target avatar’s idiosyncratic lip behavior (Chen et al., 26 Sep 2025).

The method is positioned against two recurrent failure modes in prior work. First, many audio-driven systems learn a largely speaker-agnostic mapping from audio to lip motion. The paper identifies this as insufficient for reproducing speaker-specific lip habits such as characteristic mouth opening, puckering, spreading, and coarticulation patterns. Second, many systems perform lower-face inpainting after masking the mouth region. Under mouth occlusions, this “blind inpainting” paradigm removes the occluding object itself from the condition and forces the model to hallucinate both lips and object appearance, which leads to implausible lip geometries, distorted microphones or hands, and temporal flicker (Chen et al., 26 Sep 2025).

The diffusion backbone is chosen because latent diffusion models provide strong image and video quality while operating in a lower-dimensional latent space, but the paper also emphasizes that such backbones are computationally expensive and are often made heavier by auxiliary priors such as ReferenceNet or SyncExpert. StableDub’s stated objective is therefore not only generalized visual dubbing, but generalized visual dubbing with improved training efficiency on a Stable-Diffusion-based backbone (Chen et al., 26 Sep 2025).

2. Core architecture and latent diffusion formulation

StableDub combines a VAE-based latent inpainting pipeline with audio conditioning, lip-habit conditioning, and a mask-aware training objective. A randomly selected appearance frame $$x_{\text{app}} \in \mathbb{R}^{c \times h \times w}$$ is encoded by a pretrained VAE encoder $\mathcal{E}$ into an appearance latent $z_{\text{app}} = \mathcal{E}(x_{\text{app}})$. A separate lip-habit segment $c_h \in \mathbb{R}^{K \times c \times h \times w}$ is sampled from the same video, cropped around the lips, and encoded into a global lip-habit feature. The input video $x^{1:N}$ is also encoded framewise into latents $z_0^{1:N} = \mathcal{E}(x^{1:N})$, after which the denoising model operates in latent space rather than RGB space (Chen et al., 26 Sep 2025).

Component	Input	Role
Appearance reference	$x_{\text{app}}$	Preserve identity and style
Lip-habit encoder	$c_a^{1:N} \in \mathbb{R}^{N \times d}$0	Extract global lip-habit feature
Audio encoder	$c_a^{1:N} \in \mathbb{R}^{N \times d}$1	Provide phonemic timing features
VAE encoder/decoder	$c_a^{1:N} \in \mathbb{R}^{N \times d}$2, $c_a^{1:N} \in \mathbb{R}^{N \times d}$3	Map video to and from latent space
Diffusion U-Net	$c_a^{1:N} \in \mathbb{R}^{N \times d}$4, conditions	Denoise and synthesize dubbed mouth motion

The latent input to the denoiser is constructed as

$c_a^{1:N} \in \mathbb{R}^{N \times d}$5

where $c_a^{1:N} \in \mathbb{R}^{N \times d}$6 is the union mask in latent space, $c_a^{1:N} \in \mathbb{R}^{N \times d}$7 denotes element-wise multiplication, and $c_a^{1:N} \in \mathbb{R}^{N \times d}$8 denotes frame-wise concatenation. This design provides the denoiser with a masked latent sequence together with a persistent appearance reference from the same target video (Chen et al., 26 Sep 2025).

The diffusion objective follows the standard latent diffusion formulation: $c_a^{1:N} \in \mathbb{R}^{N \times d}$9 with noise prediction loss

$\hat{x}^{1:N}$0

StableDub augments this with a latent coarse facial loss that emphasizes the lower-half facial region: $\hat{x}^{1:N}$1 where $\hat{x}^{1:N}$2 is the coarse mask. The reported total loss is

$\hat{x}^{1:N}$3

This formulation keeps synchronization supervision in latent space rather than introducing an additional RGB-space sync prior (Chen et al., 26 Sep 2025).

3. Lip-habit-modulated conditioning

A central concept in StableDub is “lip habits,” defined as speaker-specific orofacial dynamics such as characteristic mouth shapes, degree of opening, puckering or stretching patterns, and coarticulation behavior. The paper treats these dynamics as stable within a speaker or avatar but variable across identities, and therefore as information that should be extracted from the target avatar’s own video rather than inferred from audio alone (Chen et al., 26 Sep 2025).

Driving audio is encoded with Wav2Vec into a temporal feature sequence

$\hat{x}^{1:N}$4

while the lip-habit segment is encoded by a ViT-based lip encoder into a global feature

$\hat{x}^{1:N}$5

The paper then modulates the audio feature with the lip-habit feature through AdaLN: $\hat{x}^{1:N}$6 where $\hat{x}^{1:N}$7 and $\hat{x}^{1:N}$8 are channelwise statistics and $\hat{x}^{1:N}$9 map $$x_{\text{app}} \in \mathbb{R}^{c \times h \times w}$$0 into scale and shift terms. The resulting habit-aware audio condition is described as AdaLN output plus a residual term weighted by a learnable vector: $$x_{\text{app}} \in \mathbb{R}^{c \times h \times w}$$1 The paper characterizes this as a lightweight alternative to more expensive conditioning strategies while retaining habit resemblance (Chen et al., 26 Sep 2025).

These modulated audio features are injected into the denoising U-Net through cross-attention: $$x_{\text{app}} \in \mathbb{R}^{c \times h \times w}$$2 with

$$x_{\text{app}} \in \mathbb{R}^{c \times h \times w}$$3

Condition dropout with probability $$x_{\text{app}} \in \mathbb{R}^{c \times h \times w}$$4 is applied to audio and lip-habit features during training. At inference time, StableDub uses classifier-free guidance with separate audio and habit guidance scales: $$x_{\text{app}} \in \mathbb{R}^{c \times h \times w}$$5 This explicitly separates guidance for phonemic synchronization and guidance for speaker-specific lip style (Chen et al., 26 Sep 2025).

4. Occlusion-aware synthesis and mask exposure

StableDub’s occlusion handling begins with a reformulation of the masking procedure. Instead of using a per-frame lower-face box, the method constructs a detailed mask from face landmarks, then an eroded coarse mask $$x_{\text{app}} \in \mathbb{R}^{c \times h \times w}$$6, and finally a temporally aggregated union mask $$x_{\text{app}} \in \mathbb{R}^{c \times h \times w}$$7 obtained by taking the union of coarse masks across frames and blurring the boundaries. The stated purpose is to avoid per-frame bounding-box jitter and to provide a temporally consistent masked region that better accommodates static mouth-adjacent objects such as microphones (Chen et al., 26 Sep 2025).

The decisive addition is “mask exposure.” The union mask is partitioned into spatial patches, and a proportion of those patches is randomly unmasked during training so that parts of the occluding object become visible to the inpainting process. Early in training, the exposure rate is $$x_{\text{app}} \in \mathbb{R}^{c \times h \times w}$$8, so much of the occluded region is revealed; the rate then decreases progressively to $$x_{\text{app}} \in \mathbb{R}^{c \times h \times w}$$9, forcing the model to infer object appearance and lip-object interaction from less direct evidence. The paper describes this as explicitly exposing occlusion objects to the inpainting process, in contrast to conventional blind inpainting, which removes them entirely from the condition (Chen et al., 26 Sep 2025).

This design has two consequences in the paper’s interpretation. First, the network can learn object continuity because it sometimes sees the object itself inside the nominally masked region. Second, the fixed-area union mask, together with multi-frame conditioning, reduces temporal inconsistency that would otherwise come from moving masks and unstable object boundaries. The reported ablation indicates that removing mask exposure worsens FVD from $\mathcal{E}$0 to $\mathcal{E}$1 and LMD from $\mathcal{E}$2 to $\mathcal{E}$3, which the paper interprets as degradation in both video consistency and lip-related fidelity (Chen et al., 26 Sep 2025).

5. Hybrid Mamba-Transformer design and training efficiency

StableDub replaces spatial and temporal self-attention layers in the denoising U-Net with Mamba-based layers while retaining Transformer-style cross-attention for audio conditioning. For a feature tensor

$\mathcal{E}$4

Spatial Mamba operates over spatial dimensions and Temporal Mamba operates over the frame dimension $\mathcal{E}$5. The paper positions this as a hybrid Mamba-Transformer architecture: Mamba handles spatiotemporal dependencies with lower computational overhead, while attention is preserved where cross-modal fusion is most critical (Chen et al., 26 Sep 2025).

The reported efficiency gains are concrete. Compared with a full-attention variant, the hybrid design reduces memory consumption for longer video inputs by $\mathcal{E}$6 and increases supported batch size by $\mathcal{E}$7. The method also removes several modules commonly added to diffusion-based dubbing pipelines. ReferenceNet is eliminated, reducing a dual-U-Net design to a single U-Net; the text encoder is removed; and SyncExpert is not used. According to the paper, this reduces parameter count from $\mathcal{E}$8B to $\mathcal{E}$9M. In the efficiency comparison table, StableDub is reported at $z_{\text{app}} = \mathcal{E}(x_{\text{app}})$0 s/frame for 16 frames with $z_{\text{app}} = \mathcal{E}(x_{\text{app}})$1 GB inference memory, compared with $z_{\text{app}} = \mathcal{E}(x_{\text{app}})$2 s/frame and $z_{\text{app}} = \mathcal{E}(x_{\text{app}})$3 GB for Diff2Lip, and $z_{\text{app}} = \mathcal{E}(x_{\text{app}})$4 s/frame and $z_{\text{app}} = \mathcal{E}(x_{\text{app}})$5 GB for Hallo (Chen et al., 26 Sep 2025).

Training is described in two stages. Stage 1 uses videos without audio and optimizes spatial Mamba layers on roughly $z_{\text{app}} = \mathcal{E}(x_{\text{app}})$6 million images extracted from filtered videos to improve image quality and generalization; its reported duration is $z_{\text{app}} = \mathcal{E}(x_{\text{app}})$7 hours on $z_{\text{app}} = \mathcal{E}(x_{\text{app}})$8 NVIDIA V100 GPUs. Stage 2 uses audio-synchronized videos, freezes spatial Mamba layers, and trains audio cross-attention together with temporal Mamba layers on $z_{\text{app}} = \mathcal{E}(x_{\text{app}})$9-frame clips; its reported duration is $c_h \in \mathbb{R}^{K \times c \times h \times w}$0 hours on the same hardware. The data source is a filtered VoxCeleb2 subset resampled to $c_h \in \mathbb{R}^{K \times c \times h \times w}$1 fps and cropped to $c_h \in \mathbb{R}^{K \times c \times h \times w}$2, yielding about $c_h \in \mathbb{R}^{K \times c \times h \times w}$3 hours of paired audio-video data after removing clips with low clarity, multiple people, excessive head transitions, large head angles, small faces, poor audio-lip sync, or high background noise (Chen et al., 26 Sep 2025).

6. Empirical performance and ablation evidence

StableDub is evaluated on LRS2 and VFHQ-C, with quantitative emphasis on self-reenactment and cross-audio dubbing. On VFHQ-C self-reenactment, the paper reports LPIPS $c_h \in \mathbb{R}^{K \times c \times h \times w}$4, LSE-C $c_h \in \mathbb{R}^{K \times c \times h \times w}$5, LMD $c_h \in \mathbb{R}^{K \times c \times h \times w}$6, FID $c_h \in \mathbb{R}^{K \times c \times h \times w}$7, FVD $c_h \in \mathbb{R}^{K \times c \times h \times w}$8, and WER $c_h \in \mathbb{R}^{K \times c \times h \times w}$9. In the same setting, Diff2Lip reports LPIPS $x^{1:N}$0, LSE-C $x^{1:N}$1, LMD $x^{1:N}$2, FID $x^{1:N}$3, FVD $x^{1:N}$4, and WER $x^{1:N}$5, while IP-LAP reports LPIPS $x^{1:N}$6, LSE-C $x^{1:N}$7, LMD $x^{1:N}$8, FID $x^{1:N}$9, FVD $z_0^{1:N} = \mathcal{E}(x^{1:N})$0, and WER $z_0^{1:N} = \mathcal{E}(x^{1:N})$1. On VFHQ-C cross-audio dubbing, StableDub reports LSE-C $z_0^{1:N} = \mathcal{E}(x^{1:N})$2, LSE-D $z_0^{1:N} = \mathcal{E}(x^{1:N})$3, FID $z_0^{1:N} = \mathcal{E}(x^{1:N})$4, FVD $z_0^{1:N} = \mathcal{E}(x^{1:N})$5, and WER $z_0^{1:N} = \mathcal{E}(x^{1:N})$6, again outperforming the listed baselines on FID and FVD and surpassing them on multiple synchronization-related measures (Chen et al., 26 Sep 2025).

The user study uses three MOS dimensions: visual quality (VQ), lip habit resemblance plus audio-lip sync (RS), and occlusion robustness (OR). StableDub reports $z_0^{1:N} = \mathcal{E}(x^{1:N})$7 on VQ, $z_0^{1:N} = \mathcal{E}(x^{1:N})$8 on RS, and $z_0^{1:N} = \mathcal{E}(x^{1:N})$9 on OR, while ground truth is $x_{\text{app}}$0, $x_{\text{app}}$1, and $x_{\text{app}}$2, respectively. The best baseline scores listed in the table remain lower across all three categories; for example, Diff2Lip reports $x_{\text{app}}$3 on VQ, $x_{\text{app}}$4 on RS, and $x_{\text{app}}$5 on OR (Chen et al., 26 Sep 2025).

The ablation results are tightly aligned with the claimed contributions. Removing habit modulation changes LMD from $x_{\text{app}}$6 to $x_{\text{app}}$7, the largest LMD deterioration in the reported ablations, which the paper uses as direct evidence that lip-habit modulation is responsible for habit resemblance. Replacing Mamba with attention worsens FID from $x_{\text{app}}$8 to $x_{\text{app}}$9 and FVD from $c_a^{1:N} \in \mathbb{R}^{N \times d}$00 to $c_a^{1:N} \in \mathbb{R}^{N \times d}$01. Removing Temporal Mamba has an especially strong effect on FVD, worsening it to $c_a^{1:N} \in \mathbb{R}^{N \times d}$02, which the paper interprets as reduced temporal consistency. A dual-U-Net variant produces nearly identical FID and FVD but worse LMD at $c_a^{1:N} \in \mathbb{R}^{N \times d}$03, suggesting that the more concise single-U-Net formulation does not sacrifice image quality while retaining an advantage in lip-related fidelity (Chen et al., 26 Sep 2025).

7. Position within visual dubbing research, applications, and limitations

StableDub belongs to a broader technical trajectory in visual dubbing that increasingly treats conditioning quality, temporal consistency, and robustness as first-class design problems. STSA, for example, attributes instability to semantic ambiguity between spatial and temporal domains and addresses it through dual-path alignment, mutual-information-based Consistent Information Learning, and probabilistic heatmaps (Ding et al., 29 Mar 2025). A later self-bootstrapping framework reframes dubbing from mask inpainting to context-rich video-to-video editing through synthetic aligned pairs and timestep-adaptive multi-phase learning (He et al., 31 Dec 2025). StableDub occupies a distinct position within this landscape: it retains a concise Stable-Diffusion-based inpainting formulation, but augments it with lip-habit-aware conditioning, explicit occlusion-aware training, and a hybrid Mamba-Transformer architecture (Chen et al., 26 Sep 2025).

The applications explicitly emphasized for StableDub include movie and television dubbing, virtual human live streaming, game character and avatar animation, and social media video creation. The paper gives qualitative examples such as dubbing characters from “Frozen” with Chinese singing audio and improving motion-capture-related mouth artifacts for characters from “The Last of Us.” These examples are used to argue that the framework generalizes beyond photorealistic human faces to stylized and game-character domains (Chen et al., 26 Sep 2025).

The reported limitations are twofold. First, larger generating areas reduce the ability to preserve complex backgrounds perfectly, and the use of a single reference image can make detailed background maintenance harder; the paper suggests combining masks of different scales and using multiple reference images as future directions. Second, diffusion-based inference remains slower than GAN-based alternatives, even though StableDub is presented as more efficient than several diffusion baselines; acceleration methods such as Latent Consistency Models are suggested as a possible next step (Chen et al., 26 Sep 2025). A plausible implication is that StableDub’s main contribution is not the elimination of the quality-efficiency trade-off, but a substantial rebalancing of it: the framework preserves the visual benefits of latent diffusion while reducing auxiliary-model overhead and improving robustness in two practical regimes—speaker-specific lip style and mouth occlusion.