Context-Edge Hybrid Adapter (CH-Adapter)

• Context-Edge Hybrid Adapter (CH-Adapter) is a hybrid module that combines global context with local edge features to enhance neural network representation learning.
• It efficiently integrates high-level semantic context with detailed edge information, improving performance in applications like object recognition and language processing.
• The architecture supports flexible integration with existing models, offering practical benefits in computational efficiency and scalable performance.

Searching arXiv for the specified work and closely related dubbing papers to ground the article. Authentic-Dubber is an automatic movie dubbing model that explicitly simulates an “authentic” director–actor workflow rather than treating dubbing as direct script-to-speech generation. Introduced as a Retrieve-Augmented Director-Actor Interaction Learning scheme, it is organized around three mechanisms: a Multimodal Reference Footage Library (MRFL), an Emotion-Similarity-based Retrieval-Augmentation strategy (ESRA), and a Progressive Graph-based Speech Generation module (PGSG). The stated objective is to let directors provide emotionally relevant reference footage, let actors internalize that footage, and then let the dubbing system synthesize speech that preserves lip-sync while improving emotional expressiveness (Liu et al., 18 Nov 2025).

1. Research context and problem framing

Automatic video and movie dubbing has developed from early multimodal text-to-speech systems toward increasingly structured models of style, identity, emotion, and scene understanding. Neural Dubber formulates automatic video dubbing as synthesizing human speech synchronized with a given video from text, using lip movement to control prosody and, in the multi-speaker setting, an image-based speaker embedding to infer timbre from the speaker’s face (Hu et al., 2021). StyleDubber shifts dubbing learning from the frame level to the phoneme level, with a multimodal style adaptor, utterance-level style learning, and a phoneme-guided lip aligner (Cong et al., 2024). EmoDubber adds user-specified emotion type and emotional intensity through Lip-related Prosody Aligning, Pronunciation Enhancing, Speaker Identity Adapting, and Flow-based User Emotion Controlling (Cong et al., 2024). DeepDubber-V1 further introduces multimodal Chain-of-Thought reasoning for scene type, gender, age, and emotion prediction before conditioned speech generation (Zheng et al., 31 Mar 2025). FlowDubber uses Qwen2.5, semantic-aware learning, dual contrastive aligning, and flow-based voice enhancement (Cong et al., 2 May 2025). MM-MovieDubber uses a multi-modal large vision-LLM to recognize dubbing types and fine-grained attributes, then conditions a speech generator on those multimodal conclusions (Zheng et al., 22 May 2025). DiFlowDubber adopts a two-stage transfer framework with a discrete flow matching backbone, a Face-to-Prosody Mapper, and a Synchronizer for precise speech-lip synchronization (Nguyen et al., 15 Mar 2026). HoliDubber extends the task beyond speech-only dubbing to joint speech-and-sound synthesis from structured text prompts (Guan et al., 8 Jun 2026).

Within that progression, Authentic-Dubber is distinctive because it does not merely add another conditioning signal. Instead, it redefines the dubbing process as a dynamic collaboration in which directors actively engage with actors and guide them to internalize context cues, specifically emotion, before performance (Liu et al., 18 Nov 2025). A plausible implication is that “authenticity” in this formulation refers less to a single objective metric than to a workflow prior: emotionally relevant preparation is treated as part of the generation problem.

2. System organization and operating assumptions

Authentic-Dubber comprises three tightly coupled modules: MRFL, ESRA, and PGSG. Its experimental setting is the V2C-Animation dataset, with 10,217 clips and 153 characters, split into 60% train, 10% validation, and 30% test. Video is processed at 25 fps; audio is represented at 22.05 kHz with STFT parameters win $=1024$ and hop $=256$; and all emotion features are projected to 256-D (Liu et al., 18 Nov 2025).

Module	Inputs	Function
MRFL	Scene, face, text, and audio emotion vectors from each reference clip	Builds modality-specific embedding spaces
ESRA	Target silent video and script	Retrieves Top-$K$ emotionally similar embeddings and matched audio
PGSG	Base embeddings, retrieved embeddings, retrieved audio, and cross-modal feature $H_{tvr}$	Progressively fuses emotion knowledge for speech generation

The model stores four emotion vectors for each clip $i$: scene $S_i\in\mathbb{R}^d$, face $F_i\in\mathbb{R}^d$, text $T_i\in\mathbb{R}^{2d}$, and audio $A_i\in\mathbb{R}^d$. These are placed in modality-specific embedding spaces $\mathcal{E}_{\mathrm{scene}}$, $=256$0, $=256$1, and $=256$2. Generation then proceeds by extracting basic emotion embeddings from the target, retrieving related evidence from the library, encoding that evidence as successive graphs, cross-attending those graph representations against the cross-modal aligned feature $=256$3, and finally decoding a mel-spectrogram followed by waveform generation with a GAN-based vocoder (Liu et al., 18 Nov 2025).

This modularization separates three functions that earlier dubbing systems often compress into a single conditioning path: emotion representation, retrieval of related evidence, and hierarchical fusion into the speech generator. This suggests that Authentic-Dubber treats emotional context as reusable external memory rather than as information that must always be inferred from the target clip alone.

3. Multimodal Reference Footage Library

The MRFL is the system’s reference memory. For each clip in V2C-Animation, it extracts four emotion vectors and stores them in distinct embedding spaces. The text emotion representation is structurally different from the others: $=256$4 is the concatenation of self-emotion and react-emotion, whereas scene, face, and audio are each $=256$5-dimensional (Liu et al., 18 Nov 2025).

The library’s embeddings are generated through an LLM-mediated emotion understanding pipeline. Authentic-Dubber leverages VideoLLaMA 2 for visual-to-text emotion captions and COMET for commonsense reaction captions. The scene representation is defined by

$=256$6

followed by

$=256$7

The face emotion vector is obtained as

$=256$8

while the text representation is split into

$=256$9

The paper describes this as a cascade of LLM prompts, captions, and a RoBERTa emotion extractor that yields “deeply aligned multimodal emotion embeddings” (Liu et al., 18 Nov 2025).

The significance of MRFL lies in how it changes supervision. Rather than only learning a direct mapping from silent video and script to speech, the model first constructs a searchable corpus of emotionally annotated exemplars. A plausible implication is that MRFL functions as a structured emotional prior over the dubbing domain, especially for affective patterns that may be sparse in direct paired training examples.

4. Emotion-Similarity-based Retrieval-Augmentation

ESRA operates on the target silent video and script. It first extracts their basic emotion embeddings $K$0, $K$1, and $K$2. For each modality $K$3, it computes similarities between the query embedding and all library entries, then retrieves Top-$K$4 indirect embeddings $K$5 together with their matched direct audio $K$6 (Liu et al., 18 Nov 2025).

The similarity function is cosine similarity: $K$7 The paper also notes an optional softmax weighting,

$K$8

with weighted aggregation $K$9, but states that Authentic-Dubber uses hard Top-$H_{tvr}$0 selection (Liu et al., 18 Nov 2025).

This retrieval design distinguishes between two kinds of retrieved evidence. Indirect embeddings $H_{tvr}$1 preserve modality-specific emotion representations, while the matched direct audio $H_{tvr}$2 provides acoustically grounded emotional exemplars. The paper does not introduce this as a separate theorem, but a plausible interpretation is that ESRA is intended to bridge semantic affect cues and speech-realization cues within one retrieval pass.

5. Progressive Graph-based Speech Generation

PGSG is the generation core. It builds three graphs in succession.

The Basic Emotion Graph $H_{tvr}$3 contains the target’s base nodes $H_{tvr}$4. The Indirect-Extended Graph $H_{tvr}$5 adds the retrieved indirect embeddings $H_{tvr}$6, each connected to its modality node. The Direct-Extended Graph $H_{tvr}$7 then adds the matched direct audio nodes $H_{tvr}$8 (Liu et al., 18 Nov 2025).

Formally, the basic graph begins with node set

$H_{tvr}$9

with initial node features

$i$0

and edges

$i$1

Graph encoding uses a Graph Attention Encoder: $i$2 where

$i$3

The three encoded graphs yield node representations $i$4, $i$5, and $i$6 (Liu et al., 18 Nov 2025).

These graph states are fused into generation by hierarchical cross-attention against the cross-modal aligned feature $i$7: $i$8 Here $i$9 denotes channel-wise concatenation and $S_i\in\mathbb{R}^d$0 denotes cross-attention. The resulting $S_i\in\mathbb{R}^d$1 is fed into a mel-decoder and a GAN vocoder (Liu et al., 18 Nov 2025).

The training objective includes mel-spectrogram $S_i\in\mathbb{R}^d$2 loss,

$S_i\in\mathbb{R}^d$3

a duration/alignment loss “as in StyleDubber,” and an optional adversarial loss $S_i\in\mathbb{R}^d$4 from the vocoder. Lip-sync alignment is enforced by the cross-modal aligner in StyleDubber via a frame-phoneme contrastive loss (Liu et al., 18 Nov 2025). This dependence is important: Authentic-Dubber’s main novelty is emotional retrieval and graph fusion, not a replacement of the underlying lip-sync mechanism.

6. Evaluation, ablations, and interpretation

Authentic-Dubber is evaluated on V2C-Animation with objective metrics and human judgments. The reported objective metrics are Emotion Accuracy (EMO-ACC), WER via Whisper-large-v3, Speaker Encoder Cosine Similarity (SECS), and MCD-DTW-SL. Subjective evaluation uses 20 raters, each scoring 12 samples on a 1–5 scale for MOS-Dubbing Emotion (MOS-DE) and MOS-Speech Emotion (MOS-SE), with 95% confidence intervals; a paired $S_i\in\mathbb{R}^d$5-test reports that Authentic-Dubber significantly outperforms baselines at $S_i\in\mathbb{R}^d$6 (Liu et al., 18 Nov 2025).

Method	EMO-ACC $S_i\in\mathbb{R}^d$7	WER $S_i\in\mathbb{R}^d$8	SECS $S_i\in\mathbb{R}^d$9	MCD-DTW-SL $F_i\in\mathbb{R}^d$0
StyleDubber	45.73	24.70	83.46	9.40
Authentic-Dubber	47.21	25.95	84.40	9.68

The same comparison reports MOS-DE of $F_i\in\mathbb{R}^d$1 for StyleDubber and $F_i\in\mathbb{R}^d$2 for Authentic-Dubber, and MOS-SE of $F_i\in\mathbb{R}^d$3 for StyleDubber and $F_i\in\mathbb{R}^d$4 for Authentic-Dubber (Liu et al., 18 Nov 2025).

The ablation study attributes emotional gains to each of the three main innovations. Removing LLM-based scene captions reduces EMO-ACC by 0.87 percentage points. Removing all retrievals in ESRA reduces EMO-ACC by 1.98 percentage points. In PGSG, removing direct audio nodes reduces EMO-ACC by 1.91 percentage points, removing indirect nodes reduces it by 1.26 percentage points, and replacing graph-based modeling with simple concatenation reduces it by 1.29 percentage points (Liu et al., 18 Nov 2025).

The results indicate a specific performance profile. Authentic-Dubber improves EMO-ACC, SECS, MOS-DE, and MOS-SE relative to the strongest baseline, but its reported WER and MCD-DTW-SL are 25.95 and 9.68, compared with 24.70 and 9.40 for StyleDubber (Liu et al., 18 Nov 2025). This suggests that the model’s principal gain is emotional expressiveness rather than uniformly better scores on all objective metrics. A common misconception would be to read “authentic” as synonymous with universally lower distortion or error rates; in the reported experiments, authenticity is operationalized primarily through director-guided emotional comprehension, emotion-aware retrieval, and improved perceived dubbing emotion.

In the broader dubbing literature, this emphasis places Authentic-Dubber alongside a line of systems that progressively externalize higher-level control variables. StyleDubber externalizes style across phoneme and utterance levels (Cong et al., 2024); EmoDubber externalizes user emotion prompts (Cong et al., 2024); DeepDubber-V1 and MM-MovieDubber externalize scene-type and attribute reasoning through multimodal large models (Zheng et al., 31 Mar 2025, Zheng et al., 22 May 2025); and FunCineForge extends this tendency to timestamp–speaker tuples, clue instructions, and multi-scene control in diverse cinematic settings (Liu et al., 21 Jan 2026). Authentic-Dubber’s specific contribution within that trajectory is to treat emotionally relevant reference footage as a retrievable, graph-structured intermediary between scene understanding and speech synthesis (Liu et al., 18 Nov 2025).