Shared Latent Representation for Joint Text-to-Audio-Visual Synthesis (2511.05432v1)

Abstract: We propose a text-to-talking-face synthesis framework leveraging latent speech representations from HierSpeech++. A Text-to-Vec module generates Wav2Vec2 embeddings from text, which jointly condition speech and face generation. To handle distribution shifts between clean and TTS-predicted features, we adopt a two-stage training: pretraining on Wav2Vec2 embeddings and finetuning on TTS outputs. This enables tight audio-visual alignment, preserves speaker identity, and produces natural, expressive speech and synchronized facial motion without ground-truth audio at inference. Experiments show that conditioning on TTS-predicted latent features outperforms cascaded pipelines, improving both lip-sync and visual realism.

• The paper proposes a unified framework that conditions both TTS and talking-face generation on shared Wav2Vec2 latent features, achieving natural audio-lip synchronization.
• It employs a two-stage training paradigm to mitigate domain mismatch between real audio and TTS-predicted features, enhancing identity preservation and lip-sync accuracy.
• Empirical results on LRS2 demonstrate state-of-the-art video quality and speech intelligibility with improved WER and visual fidelity over prior models.

Shared Latent Representation for Joint Text-to-Audio-Visual Synthesis

Introduction and Motivation

The paper presents a unified framework for synthesizing talking-face videos and speech directly from input text by leveraging shared latent speech representations. This approach addresses major limitations in cascaded text-to-video pipelines, namely audio-visual misalignment and error compounding introduced by training talking-face generation (TFG) models on real audio then inferring with synthetic speech. Rather than relying on traditional mel-spectrograms or sequential systems, the framework utilizes HierSpeech++-derived Wav2Vec2 embeddings predicted from text as a conditioning signal for both speech synthesis and facial motion generation.

Figure 1: Joint text-to-audio-visual synthesis framework utilizing latent Wav2Vec2 features for tightly integrated speech and facial motion generation.

This architecture facilitates parallel generation of speech and video within a consistent latent space, ensuring natural audio-lip synchronization and coherent speaker identity transfer at inference—without needing ground-truth audio.

Technical Approach

Text-to-Speech and Latent Representation Construction

The backbone TTS model, HierSpeech++, performs hierarchical speech synthesis using linguistic, acoustic, and prosodic representations. Critically, the Text-to-Vec (TTV) module is trained as a variational autoencoder (VAE) to predict Wav2Vec2 (W2V2) embeddings and $F_0$ prosody contours from text. Training employs ground-truth W2V2 embeddings, reconstructing them through the VAE and duration predictors via monotonic alignment search. Inference involves generating predicted W2V2 features—for both speech synthesis and TFG—using only text and reference audio (for style conditioning), enabling zero-shot speaker adaptation.

GAN-based Talking Face Generation

The facial animation module builds on a GAN architecture that processes identity reference images and joint latent features. It omits a standalone audio encoder, directly concatenating the shared W2V2 embeddings from TTV with visual features. A modified preprocessing network produces a silent-face image from the identity reference for improved training stability and lip sync fidelity. The generation objective combines adversarial, perceptual (VGG-19), pixel-wise reconstruction, and stabilized synchronization losses—a strategy found empirically superior for audio-lip alignment.

Two-Stage Training Paradigm

Recognizing the statistical shift between clean pretrained W2V2 embeddings and TTS-predicted vectors, the authors introduce a two-stage training procedure. Stage one trains the TFG model with authentic W2V2 features; stage two adapts to features generated by the TTV module, fine-tuning for robust performance with synthetic speech. This directly mitigates domain-mismatch issues observed in standard cascaded approaches, ensuring lip movements are synchronized to the speech generated from text.

Experimental Results

Quantitative and Qualitative Performance

Extensive experiments on LRS2 demonstrate competitive, often state-of-the-art, results in both video quality and audio-visual synchronization. Metrics include SSIM, PSNR, FID for video; LSE-C, LSE-D, CSIM for lip-sync and identity. The framework matches or exceeds the best prior work (e.g., Diff2Lip, PLGAN) in identity consistency and perceptual similarity, especially when tested with TTS-generated audio.

Figure 2: Qualitative comparison illustrating accurate lip-sync and speaker alignment across various models. Joint training with predicted features enables coherent and realistic synthesis.

A strong empirical claim is evident in the WER results from the speech synthesis module, which achieves lower WER than ground-truth speech (1.51% vs. 4.47%), attributed to less environmental noise in synthesized audio. UTMOS and speaker embedding cosine similarity further confirm naturalness and speaker authenticity.

Ablation and Cross-Domain Robustness

Ablation studies highlight the essential nature of two-stage training for generalization. Models trained only on real audio features deteriorate in lip-sync metrics when exposed to TTS outputs. Removing explicit synchronization loss in stage two also reduces performance, confirming the necessity of carefully tuned loss compositions. Cross-pairing audio/video during testing supports the model’s robustness in mismatched scenarios.

Critical Analysis and Implications

The approach marks a substantial advance in tightly integrated text-to-audio-visual synthesis. By conditioning both TTS and TFG on shared, text-derived W2V2 latent features, the model avoids error accumulation and decouples synthesis from ground-truth audio. The two-stage training paradigm is particularly effective in closing the gap between clean speech feature statistics and those predicted by TTS, ensuring stable lip synchronization and identity preservation throughout.

Limitations include the reliance on high-quality latent feature prediction, which may limit performance in low-resource languages or degrade under noisy TTS outputs. The model largely focuses on lip movement synchronization, without explicitly modeling nuanced facial dynamics such as micro-expressions, emotional context, or head poses.

Future Prospects

Future work may target full emotional and gestural expressiveness via multi-modal latent representations, investigate robustness to out-of-domain or code-switched text, and integrate advanced diffusion models for personality and style transfer. Exploration of joint training strategies with multimodal datasets and more granular facial action units may further enhance cross-lingual generalization and video realism. Extension to NeRF- or radiance field-based avatars, as well as real-time applications, are compelling subsequent directions.

Conclusion

The proposed joint text-to-audio-visual synthesis framework demonstrates that conditioning both speech and facial animation modules on shared TTS-predicted latent features enables natural audio-lip alignment, preserves speaker identity, and achieves high perceptual realism. The two-stage training procedure is essential for robust performance with synthetic speech. While limitations in generalization and facial expressiveness persist, the paradigm is extensible to future work on multimodal, emotionally driven avatars and more generalized audio-visual synthesis.