Authentic-Dubber: Emotion-Centric Movie Dubbing

• Authentic-Dubber is an automatic movie dubbing framework that models dubbing as a retrieval-augmented director–actor interaction process to generate expressive speech.
• Its Multimodal Reference Footage Library (MRFL) constructs emotion embeddings from scene, face, text, and audio modalities, enabling rich emotional context in dubbing.
• The Progressive Graph-based Speech Generation module incrementally fuses intrinsic and retrieved emotional cues, enhancing lip-sync and emotional authenticity.

Searching arXiv for the specified paper and closely related video dubbing work. Authentic-Dubber is an automatic movie dubbing framework that models dubbing as a retrieve-augmented director–actor interaction process rather than as a direct script-to-speech mapping. It is designed to generate vivid speech from scripts, replicate a speaker’s timbre from a brief timbre prompt, and ensure lip-sync with a silent video, while explicitly emphasizing emotional internalization through three modules: a Multimodal Reference Footage Library (MRFL), Emotion-Similarity-based Retrieval-Augmentation (ESRA), and Progressive Graph-based Speech Generation (PGSG) (Liu et al., 18 Nov 2025).

1. Conceptualization of “authentic” dubbing

Authentic-Dubber is motivated by a critique of prevailing automatic movie dubbing workflows. Existing approaches are described as simulating a simplified setting in which actors dub directly without preparation, whereas authentic workflows involve directors guiding actors to internalize contextual cues, specifically emotion, before performance (Liu et al., 18 Nov 2025). On this view, the core problem is not only synchronizing speech with lip motion or transferring timbre from a prompt, but also reconstructing a mediated emotional preparation process.

The system operationalizes this idea through a three-stage decomposition. First, it constructs a multimodal reference repository that plays the role of director-provided footage. Second, it retrieves emotionally relevant reference material for a target silent clip and script. Third, it incrementally fuses the target clip’s own emotion cues with retrieved indirect and direct evidence during speech generation. This suggests that “authentic” in this context is defined primarily in terms of emotional expressiveness grounded in a production-style workflow, rather than only in terms of pronunciation accuracy or duration matching (Liu et al., 18 Nov 2025).

A related misconception is to equate Authentic-Dubber with visual face re-animation systems. The paper instead addresses movie dubbing as speech generation conditioned on silent video, script, and timbre prompt. By contrast, works such as neural style-preserving visual dubbing and identity-preserving video dubbing using motion warping focus on synthesizing or editing video frames to preserve facial style or identity under new audio (Kim et al., 2019, Liu et al., 8 Jan 2025).

2. Multimodal Reference Footage Library

The MRFL is the system’s memory substrate. For each clip $i$ in the V2C-Animation dataset, which contains 10,217 clips and 153 characters, Authentic-Dubber extracts four emotion vectors: scene $S_i \in \mathbb{R}^d$, face $F_i \in \mathbb{R}^d$, text $T_i \in \mathbb{R}^{2d}$, and audio $A_i \in \mathbb{R}^d$ (Liu et al., 18 Nov 2025). The text representation is a concatenation of self- and react-emotion components, and the vectors are stored in modality-specific embedding spaces for scene, face, text, and audio.

The library is built with explicit use of LLMs for emotional interpretation. VideoLLaMA 2 is used to generate visual-to-text emotion captions, and COMET is used for commonsense reaction captions. These captions are then mapped into emotion embeddings through RTER. The construction is summarized by the following transformations: $$c_{\mathrm{scene}} = \mathrm{VideoLLaMA2}(\mathrm{frames}; \text{“Describe the scene’s emotion, mentioning hue, lightness, saturation.”}),$$

$$S = \mathrm{RTER}(c_{\mathrm{scene}}), \qquad F = \mathrm{RTER}\bigl(\mathrm{VideoLLaMA2}(\mathrm{face\ region})\bigr),$$

$$T^{\mathrm{self}} = \mathrm{RTER}(\mathrm{text}), \qquad c_{\mathrm{react}} = \mathrm{COMET}(\mathrm{text}), \qquad T^{\mathrm{react}} = \mathrm{RTER}(c_{\mathrm{react}}).$$

This design separates “indirect” emotional cues from “direct” audio evidence. Scene, face, and text embeddings serve as queryable descriptors of emotion-bearing context, while matched audio embeddings provide directly usable speech-side evidence after retrieval. A plausible implication is that the MRFL functions as a structured emotional prior over the training corpus, allowing dubbing to be conditioned on richer affective evidence than the target clip alone provides.

3. Emotion-Similarity-based Retrieval-Augmentation

ESRA is the mechanism that turns the MRFL into conditioning information for a new dubbing instance. Given a target silent video and script, Authentic-Dubber extracts basic emotion embeddings $S$, $F$, and $S_i \in \mathbb{R}^d$0. For each modality $S_i \in \mathbb{R}^d$1, it computes similarity scores against all library entries and retrieves the Top-$S_i \in \mathbb{R}^d$2 indirect embeddings $S_i \in \mathbb{R}^d$3 together with their matched direct audio embeddings $S_i \in \mathbb{R}^d$4 (Liu et al., 18 Nov 2025).

The similarity function is cosine similarity: $S_i \in \mathbb{R}^d$5 The paper also gives a softmax weighting form,

$S_i \in \mathbb{R}^d$6

but states that Authentic-Dubber uses hard Top-$S_i \in \mathbb{R}^d$7 selection rather than weighted aggregation (Liu et al., 18 Nov 2025).

The retrieval procedure is modality-specific. Scene queries retrieve scene-related emotional analogues, face queries retrieve visually similar facial affect, and text queries retrieve semantically and affectively related textual states. Because each retrieved indirect embedding is paired with direct audio from the same library item, the retrieved set contains both abstract emotion evidence and speech-proximal realization patterns. This suggests a two-hop conditioning strategy: indirect multimodal emotion similarity first, then direct audio realization second.

4. Progressive Graph-based Speech Generation

PGSG is the speech-generation core. It incrementally incorporates retrieved multimodal emotional knowledge through three successive graphs: the Basic Emotion Graph $S_i \in \mathbb{R}^d$8, the Indirect-Extended Graph $S_i \in \mathbb{R}^d$9, and the Direct-Extended Graph $F_i \in \mathbb{R}^d$0 (Liu et al., 18 Nov 2025).

The basic graph contains three nodes, corresponding to the target clip’s own scene, face, and text emotion embeddings: $F_i \in \mathbb{R}^d$1 with initial node features

$F_i \in \mathbb{R}^d$2

Its edge set is

$F_i \in \mathbb{R}^d$3

The indirect-extended graph augments this structure with retrieved indirect emotion nodes $F_i \in \mathbb{R}^d$4, each connected to the corresponding base modality node. The direct-extended graph then further adds the matched audio nodes $F_i \in \mathbb{R}^d$5. Each graph is encoded by a Graph Attention Encoder using message passing of the form

$F_i \in \mathbb{R}^d$6

with attention coefficients

$F_i \in \mathbb{R}^d$7

The three graph representations are then injected progressively into the decoder through cross-attention against the cross-modal aligned feature $F_i \in \mathbb{R}^d$8. The hierarchical aggregation equation is

$F_i \in \mathbb{R}^d$9

Here $T_i \in \mathbb{R}^{2d}$0 denotes channel-wise concatenation. The resulting $T_i \in \mathbb{R}^{2d}$1 is passed to a Mel-decoder and a GAN-based vocoder (Liu et al., 18 Nov 2025).

The progression from $T_i \in \mathbb{R}^{2d}$2 to $T_i \in \mathbb{R}^{2d}$3 is central. It formalizes a staged incorporation of emotion evidence: intrinsic target cues first, retrieved multimodal analogues second, and direct audio realizations last. The paper presents this as an algorithmic analogue of how actors internalize director-provided context before producing a final performance.

5. Objectives, optimization, and empirical behavior

Authentic-Dubber uses a mel-spectrogram reconstruction loss,

$T_i \in \mathbb{R}^{2d}$4

a duration/alignment loss described as being “as in StyleDubber,” and an optional adversarial GAN loss $T_i \in \mathbb{R}^{2d}$5 from the vocoder (Liu et al., 18 Nov 2025). Lip-sync alignment is enforced by the cross-modal aligner in StyleDubber via a frame-phoneme contrastive loss. Training on V2C-Animation uses a 60% train, 10% validation, and 30% test split, with 25 fps video, 22.05 kHz audio, STFT parameters of window length 1024 and hop 256, and 256-D projections for all emotion features. Optimization uses Adam with $T_i \in \mathbb{R}^{2d}$6, $T_i \in \mathbb{R}^{2d}$7, $T_i \in \mathbb{R}^{2d}$8, and learning rate $T_i \in \mathbb{R}^{2d}$9 (Liu et al., 18 Nov 2025).

Evaluation combines objective and subjective metrics. The objective metrics are Emotion Accuracy (EMO-ACC), WER via Whisper-large-v3, Speaker Encoder Cosine Similarity (SECS), and MCD-DTW-SL. Subjective evaluation uses MOS-Dubbing Emotion (MOS-DE) and MOS-Speech Emotion (MOS-SE), with 20 raters, 12 samples each, on a 1–5 scale; 95% confidence intervals are reported, and a paired $A_i \in \mathbb{R}^d$0-test indicates significant superiority over baselines at $A_i \in \mathbb{R}^d$1 (Liu et al., 18 Nov 2025).

The excerpted main comparison against StyleDubber is as follows.

Method	Objective metrics	Subjective metrics
StyleDubber	EMO-ACC 45.73, WER 24.70, SECS 83.46, MCD-DTW-SL 9.40	MOS-DE 3.676 ± 0.048, MOS-SE 3.738 ± 0.049
Authentic-Dubber	EMO-ACC 47.21, WER 25.95, SECS 84.40, MCD-DTW-SL 9.68	MOS-DE 3.792 ± 0.055, MOS-SE 3.889 ± 0.053

The gains emphasized by the paper are emotional. Authentic-Dubber improves EMO-ACC by 1.48 percentage points over the strongest baseline and raises MOS-DE by 0.116 (Liu et al., 18 Nov 2025). The ablation study attributes these gains to all three major components: removing scene-caption-based LLM features reduces EMO-ACC by 0.87 percentage points; ablating all retrievals reduces it by 1.98; removing direct audio nodes reduces it by 1.91; removing indirect nodes reduces it by 1.26; and replacing graph-based modeling with simple concatenation reduces it by 1.29. At the same time, the excerpted results also show that WER and MCD-DTW-SL are not uniformly improved over StyleDubber, which is important for interpreting “authenticity” here as primarily an emotion-centered improvement rather than a universal dominance across every reported metric.

Qualitative analysis in the paper aligns with this emphasis: mel-spectrogram comparisons show sharper high-frequency modulations for “angry” and more natural pitch contours for “happy” (Liu et al., 18 Nov 2025).

6. Relation to prior dubbing architectures

Authentic-Dubber is situated within a sequence of video-conditioned speech-generation systems that progressively enriched conditioning signals beyond text and lip motion. Neural Dubber introduced automatic video dubbing as a multi-modal TTS problem, using lip movement to control prosody and an image-based speaker embedding to infer timbre from the speaker’s face (Hu et al., 2021). StyleDubber shifted dubbing learning from frame level to phoneme level, combining a multimodal style adaptor, utterance-level style learning, and a phoneme-guided lip aligner (Cong et al., 2024). EmoDubber added explicit user emotion control through Lip-related Prosody Aligning, Pronunciation Enhancing, Speaker Identity Adapting, and Flow-based User Emotion Controlling (Cong et al., 2024).

Subsequent work broadened the conditioning regime. DeepDubber-V1 used multimodal chain-of-thought reasoning to infer scene type, gender, age, and emotion before DiT-based generation (Zheng et al., 31 Mar 2025). MM-MovieDubber likewise used a multi-modal VLM to recognize dubbing types and fine-grained attributes, then guided a diffusion-based speech generator with those conclusions (Zheng et al., 22 May 2025). FlowDubber combined Qwen2.5-based script/context encoding, semantic-aware phoneme learning, dual contrastive aligning, and flow-based voice enhancement (Cong et al., 2 May 2025), while DiFlowDubber introduced a two-stage transfer framework with a Face-to-Prosody Mapper and Synchronizer on top of a discrete flow matching backbone (Nguyen et al., 15 Mar 2026). FunCineForge extended the problem to diverse cinematic scenes using timestamp-speaker tokens, clue instructions, and speaker-switch support (Liu et al., 21 Jan 2026). HoliDubber moved beyond speech-only dubbing to joint synthesis of speech and sound effects from structured text prompts (Guan et al., 8 Jun 2026).

Against this background, Authentic-Dubber’s main distinction is not a new lip-sync backbone or a new flow-matching decoder alone, but a retrieval-centered formulation of emotional preparation (Liu et al., 18 Nov 2025). Its novelty lies in constructing a multimodal reference footage library, retrieving emotion-similar evidence, and injecting that evidence through progressive graph fusion. This suggests a shift in dubbing research from direct multimodal conditioning toward explicit external emotional memory and staged emotional reasoning.