Face2VoiceSync: Face-Driven Speech & Video

• Face2VoiceSync is a text-driven system that produces synchronized talking-face videos and synthesized speech from a single face image and text input.
• It employs a lightweight VAE bridge (VoiceAdapter) to map facial embeddings to speaker embeddings, significantly reducing training costs.
• The system innovatively balances diversity and identity consistency using the novel DCTS metric to evaluate voice-face alignment.

Searching arXiv for the primary paper and closely related talking-face / face-to-voice work.

Face2VoiceSync is a text-driven talking-face generation system that, given a single static face image and an input text string, simultaneously produces a realistic talking-face video and a synthesized speech signal whose speaker characteristics match the input face. The framework extends speech-driven talking-face generation to a more challenging setting in which the system must generate both the driving speech and the facial animation, motivated by the observation that reliance on fixed-driven speech limits further applications such as face-voice mismatch. Its stated contributions are Voice-Face Alignment, Diversity Manipulation, Efficient Training through a lightweight VAE bridge between visual and audio large-pretrained models, and a new evaluation metric for the diversity–identity-consistency tradeoff (Kang et al., 25 Jul 2025).

1. Task definition and problem setting

Face2VoiceSync addresses a bimodal generation problem: from a single face image and text to speak, it generates both speech and video. In contrast to conventional speech-driven talking-face pipelines, the speech signal is not externally provided. The system therefore has to infer a speaker-compatible voice representation from facial appearance and then use the generated speech to drive a talking-face model (Kang et al., 25 Jul 2025).

The paper formulates this as a response to a specific limitation in prior talking-face work: many systems can animate a face from audio, but they do not resolve whether the generated or provided voice is consistent with the depicted identity. Face2VoiceSync makes that consistency an explicit modeling target. A plausible implication is that the framework sits at the intersection of face-conditioned speech synthesis and audio-driven talking-face generation, rather than belonging exclusively to either subfield.

A common misconception is to conflate Face2VoiceSync with silent-video-to-speech systems. The model does not reconstruct speech from lip motion or from a silent video. Its inputs are a static face image and text, and its outputs are generated speech and a generated talking-face video (Kang et al., 25 Jul 2025).

2. System architecture and inference pathway

The architecture consists of three frozen, large pre-trained backbones plus one lightweight trainable bridging module, VoiceAdapter. The frozen Text-to-Speech generator is CosyVoice, which takes a text sequence and a 192-dim speaker embedding and outputs a speech waveform. The frozen Audio-to-Video generator is Hallo, which takes a static face image plus driving audio and outputs a talking-face video sequence. VoiceAdapter is a lightweight VAE that maps 512-dim facial features from Hallo’s face encoder to the 192-dim identity embedding required by CosyVoice (Kang et al., 25 Jul 2025).

Component	Training status	Role
CosyVoice	Frozen	Text-to-Speech generator
Hallo	Frozen	Audio-to-Video generator
VoiceAdapter	Trainable	512-dim face feature to 192-dim voice identity embedding

At inference, the data flow is sequential. First, the input image is passed through Hallo’s face encoder to obtain a 512-dim embedding $v$. Second, the text and a sampled identity embedding $s$ from VoiceAdapter are passed into CosyVoice to generate waveform $a$. Third, the same facial embedding $v$ and the generated audio $a$ are passed into Hallo’s diffusion-based video generator to synthesize the final talking-face video. Because CosyVoice and Hallo remain frozen, only the few-hundred-thousand parameters of VoiceAdapter are learned, which the paper presents as a drastic reduction in training cost (Kang et al., 25 Jul 2025).

The bridging role of VoiceAdapter is central. It is not a generic latent mapper; it defines a conditional distribution $p(s\mid v)$ from facial features $v\in\mathbb{R}^{512}$ to identity embeddings $s\in\mathbb{R}^{192}$. This design makes the face-conditioned voice representation the only trainable interface between the speech and video backbones.

3. Voice–face alignment and the lightweight VAE bridge

VoiceAdapter is implemented as a two-layer-MLP encoder, latent sampling in $z$, and a two-layer-MLP decoder. The encoder maps $$\mathbb{R}^{512}\to\mathbb{R}^{256}\to(\mu_z,\sigma_z)$$, and the decoder maps $s$0. The paper reports approximately $s$1 trainable parameters in VoiceAdapter, in contrast to joint-training approaches that require hundreds of millions of parameters (Kang et al., 25 Jul 2025).

The VAE objective replaces the conventional KL regularizer with MMD:

$s$2

with $s$3. This choice directly supports the paper’s diversity formulation, because it regularizes the latent distribution toward a sampling-friendly prior while preserving identity structure (Kang et al., 25 Jul 2025).

Training is divided into two stages. In the embedding-learning stage, VoiceAdapter is aligned to target speaker embeddings $s$4 obtained from CosyVoice’s frozen speaker-encoder. The reported components are a reconstruction-style cosine term,

$s$5

a standard contrastive loss over a mini-batch of identity embeddings, and a center loss,

$s$6

In the generative pre-training stage, VoiceAdapter is connected to frozen CosyVoice and further to a frozen Wav2Vec 2.0 encoder to reduce the audio–video modality gap. The reported losses are

$s$7

and

$s$8

The first enforces speaker-embedding agreement between image-conditioned and reference speech; the second aligns Wav2Vec representations to improve lip-sync (Kang et al., 25 Jul 2025).

4. Diversity manipulation and the DCTS metric

A distinctive feature of Face2VoiceSync is that the latent space is used not only for identity transfer but also for controllable diversity. The paper states that multiple $s$9 can be sampled for the same face feature $a$0, producing diverse identity embeddings $a$1. Because the prior regularization is imposed with MMD, the framework can trade off diversity against identity consistency. The paper describes this as a many-to-many distribution-to-distribution mapping and links it to fine-grained paralinguistic control, including pitch and timbre, through latent sampling (Kang et al., 25 Jul 2025).

To evaluate that tradeoff, the paper introduces the Diversity–Consistency Tradeoff Score, or DCTS. It combines intra-speaker consistency and inter-speaker diversity in the generated audio embeddings. The metric is built from the Relative Independence Ratio (RIR), based on intra-class and inter-class independence, and the Relative Cosine Ratio (RCR), based on intra-class and inter-class cosine-geometric distances. The combined score is

$a$2

The paper further interprets $a$3 as strong identity consistency with tight intra-cluster structure, and $a$4 as higher diversity with weaker consistency (Kang et al., 25 Jul 2025).

This metric is specific to the paper’s face-conditioned speech setting. Unlike standard ASR or speaker-similarity metrics, DCTS is intended to measure whether stochastic sampling produces useful variability without collapsing speaker consistency.

5. Training protocol, datasets, and reported results

The reported datasets are LRS2, with 3,347 speakers for training and 318 speakers for testing, and HDTF, with 300+ speakers for video-quality evaluation. Training uses Adam with learning rate $a$5, samples 3 frames per video as 3 image inputs, and is performed on one 40 GB NVIDIA A100 GPU. The pipeline is staged: Stage 1 trains VoiceAdapter with speaker-oriented losses, MMD, and VAE reconstruction on the LRS2 training set; Stage 2 adds the generative pre-training losses; inference then samples $a$6, computes $a$7, runs CosyVoice to generate audio, and runs Hallo to generate video (Kang et al., 25 Jul 2025).

The paper reports separate visual and audio evaluations. On HDTF, the visual-quality table gives the following values: Hallo attains FID $a$8, FVD $a$9, Sync-C $v$0, and Sync-D $v$1; SadTalker attains FID $v$2, FVD $v$3, Sync-C $v$4, and Sync-D $v$5; Face2VoiceSync attains FID $v$6, FVD $v$7, Sync-C $v$8, and Sync-D $v$9. Real video is reported at Sync-C $a$0 and Sync-D $a$1. On LRS2 audio evaluation, CosyVoice records WER $a$2, SS $a$3, DCTS $a$4, RIR $a$5, and RCR $a$6; FaceTTS records WER $a$7, SS $a$8, DCTS $a$9, RIR $p(s\mid v)$0, and RCR $p(s\mid v)$1; Face2VoiceSync records WER $p(s\mid v)$2, SS $p(s\mid v)$3, DCTS $p(s\mid v)$4, RIR $p(s\mid v)$5, and RCR $p(s\mid v)$6 (Kang et al., 25 Jul 2025).

Evaluation	Method	Reported values
HDTF visual	Face2VoiceSync	FID 24.272; FVD 194.462; Sync-C 7.605; Sync-D 7.876
LRS2 audio	Face2VoiceSync	WER 3.669; SS 0.669; DCTS 0.659; RIR 1.231; RCR 3.554
LRS2 ablation	without VAE	WER 4.423; SS 0.672; DCTS 0.662; RIR 1.328; RCR 3.316

The ablation study on LRS2 reports that removing the VAE changes WER from $p(s\mid v)$7 to $p(s\mid v)$8, SS from $p(s\mid v)$9 to $v\in\mathbb{R}^{512}$0, DCTS from $v\in\mathbb{R}^{512}$1 to $v\in\mathbb{R}^{512}$2, RIR from $v\in\mathbb{R}^{512}$3 to $v\in\mathbb{R}^{512}$4, and RCR from $v\in\mathbb{R}^{512}$5 to $v\in\mathbb{R}^{512}$6. Removing $v\in\mathbb{R}^{512}$7 gives WER $v\in\mathbb{R}^{512}$8, SS $v\in\mathbb{R}^{512}$9, DCTS $s\in\mathbb{R}^{192}$0, RIR $s\in\mathbb{R}^{192}$1, and RCR $s\in\mathbb{R}^{192}$2. The paper also reports a KDE-versus-GMM variant for DCTS with DCTS $s\in\mathbb{R}^{192}$3, RIR $s\in\mathbb{R}^{192}$4, and RCR $s\in\mathbb{R}^{192}$5 (Kang et al., 25 Jul 2025).

The detailed numbers indicate that the method is especially strong on the audio side of the task, particularly in WER. A plausible implication is that the proposed decomposition is most immediately effective as a face-conditioned speaker-embedding interface for TTS, while the visual side inherits both the strengths and limits of the frozen audio-driven video generator.

6. Position within adjacent research and reported limitations

Face2VoiceSync belongs to a broader line of work on face-conditioned speech generation, lip-to-speech synthesis, and talking-face synchronization. Earlier face-conditioned speech systems include crossmodal voice conversion, which uses a face latent code as auxiliary input to a speech converter and enforces recoverability of that code from generated speech (Kameoka et al., 2019); Face-TTS, which conditions a diffusion TTS model on a 512-dim face embedding and introduces a speaker feature binding loss (Lee et al., 2023); and Face-StyleSpeech, which separates face-derived timbre from prosody through a dedicated prosody encoder and Prosody LLM (Kang et al., 2023). Silent-video-to-speech systems follow a different route: Lip2Speech conditions speech generation on lip motion and a speaker code derived from facial characteristics (Millerdurai et al., 2022), VisageSynTalk explicitly disentangles speech content and visage style for unseen-speaker video-to-speech synthesis (Hong et al., 2022), and ImaginTalk formulates vision-guided speech generation in a discrete diffusion framework with a lip aligner, an error detector, and a face-style adapter (Ye et al., 19 Mar 2025).

On the talking-face side, related systems focus on synchronization, motion realism, or emotional control rather than joint voice generation. ReSyncer rewires a StyleGAN-based generator around 3D facial dynamics predicted by a style-injected Transformer (Guan et al., 2024). SyncDiff improves diffusion-based talking heads with a bottlenecked temporal visual prior and AVHuBERT-derived audio features (Fan et al., 17 Mar 2025). SynchroRaMa combines text sentiment, speech emotion recognition, valence–arousal features, an audio-to-motion module, and LLM-generated scene descriptions for lip-synchronized and emotion-aware talking-face generation (Yee et al., 24 Sep 2025). SyncLipMAE, by contrast, is a self-supervised representation-learning framework that aligns per-frame vocal-motion prompt tokens with audio tokens for synchronization-sensitive downstream tasks (Ling et al., 11 Oct 2025). This suggests that Face2VoiceSync is best understood as an overview-time integration of face-conditioned TTS and audio-driven animation, rather than as a synchronization-only model.

The paper explicitly reports several limitations and future directions. It notes slightly higher FID and FVD compared to purely audio-driven methods, which it attributes to the greater difficulty of the text-driven setting. It also notes reliance on large frozen backbones and proposes future exploration of end-to-end fine-tuning with reduced compute. Further extensions mentioned in the paper are cross-lingual control, emotional prosody control through richer latent-space conditioning, and more rigorous user studies on perceived identity–voice alignment and naturalness (Kang et al., 25 Jul 2025).