LatentSync: Taming Audio-Conditioned Latent Diffusion Models for Lip Sync with SyncNet Supervision (2412.09262v2)

Abstract: End-to-end audio-conditioned latent diffusion models (LDMs) have been widely adopted for audio-driven portrait animation, demonstrating their effectiveness in generating lifelike and high-resolution talking videos. However, direct application of audio-conditioned LDMs to lip-synchronization (lip-sync) tasks results in suboptimal lip-sync accuracy. Through an in-depth analysis, we identified the underlying cause as the "shortcut learning problem", wherein the model predominantly learns visual-visual shortcuts while neglecting the critical audio-visual correlations. To address this issue, we explored different approaches for integrating SyncNet supervision into audio-conditioned LDMs to explicitly enforce the learning of audio-visual correlations. Since the performance of SyncNet directly influences the lip-sync accuracy of the supervised model, the training of a well-converged SyncNet becomes crucial. We conducted the first comprehensive empirical studies to identify key factors affecting SyncNet convergence. Based on our analysis, we introduce StableSyncNet, with an architecture designed for stable convergence. Our StableSyncNet achieved a significant improvement in accuracy, increasing from 91% to 94% on the HDTF test set. Additionally, we introduce a novel Temporal Representation Alignment (TREPA) mechanism to enhance temporal consistency in the generated videos. Experimental results show that our method surpasses state-of-the-art lip-sync approaches across various evaluation metrics on the HDTF and VoxCeleb2 datasets.

• The paper introduces a novel latent diffusion framework that directly synchronizes audio signals with lip movements, bypassing conventional pixel-space methods.
• It incorporates Temporal REPresentation Alignment (TREPA) to integrate self-supervised temporal features for enhanced frame consistency.
• SyncNet supervision in both latent and pixel spaces improves accuracy, raising HDTF test scores from 91% to 94% without altering the architecture.

Overview of LatentSync: Audio Conditioned Latent Diffusion Models for Lip Sync

The paper introduces LatentSync, a novel approach in the domain of lip synchronization, leveraging latent diffusion models conditioned on audio input. The researchers propose an end-to-end framework that bypasses traditional intermediate motion representations, a departure from prior diffusion-based lip sync methodologies that operate within pixel space or require two-stage generation processes. By utilizing Stable Diffusion's advanced capabilities, they achieve dynamic modeling of complex audio-visual correlations directly.

An identified issue in prior diffusion-based lip sync methods is the lack of temporal consistency due to variance in the diffusion process across frames. To address this, the authors present Temporal REPresentation Alignment (TREPA), a technique that employs temporal representations derived from large-scale self-supervised video models to synchronize generated frames with ground truth frames, thus enhancing temporal consistency without compromising lip-sync accuracy.

A core component studied within this paper is SyncNet's convergence challenge, which has been an impediment, with its accuracy assessed using the HDTF test set increasing from 91% to a commendable 94%. The improvements were achieved without altering SyncNet's framework, thus offering insights applicable to a broad spectrum of lip sync and audio-driven portrait animation methods utilizing SyncNet.

Framework and Methodology

1. LatentSync Framework: Instead of pixel space diffusion, the authors propose a latent diffusion model where audio and visual features are synchronized using a novel end-to-end approach. Unlike two-stage methods that might lose nuanced expression details, LatentSync directly learns intricate correlations between audio signals and lip movements.
2. Temporal Representation with TREPA: The TREPA method aligns temporal representations of video sequences to improve consistency. The authors employ VideoMAE-v2 for extracting rich temporal information to address frame inconsistency seen in other methods. Distinctively, TREPA improves not just temporal coherence but also lip-sync accuracy due to its unique manner of integrating temporal data without adding additional model parameters.
3. SyncNet Supervision: The paper details two methods for adding SyncNet supervision within latent diffusion spaces: pixel-space and latent-space supervision. It concludes that although latent space supervision offers certain advantages, decoded pixel space supervision yields better lip-sync accuracy and temporal coherence.
4. Two-Stage Training and Mixed Noise Model: The methodology also uses two training stages: a larger batch size first stage to understand visual features, followed by refined SyncNet supervision in the second stage. Additionally, the deployment of a mixed noise model improves temporal consistency, which stabilizes the learning process across frames.

Experimental Evidence

Comparative analyses demonstrate the superiority of LatentSync over existing GAN-based and two-stage methods. Significant metrics such as FID, SSIM, SyncNet confidence scores, and Fréchet Video Distance (FVD) are employed to contrast LatentSync's performance against contemporaneous techniques on datasets such as HDTF and VoxCeleb2, underlining improvements both in the visual fidelity and the synchronization accuracy of lip movements.

Implications and Future Work

The implications of this research extend to various applications requiring realistic audio-video synthesis, such as virtual avatars, film dubbing, and real-time conferencing, wherein accurate lip synchronization with audio is crucial. The integration of TREPA and robust training methodologies like the mixed noise model also suggest potential evolution in other audio-visual tasks beyond lip sync by fostering more coherent and contextually rich video generation techniques.

Concluding, this research contributes significantly to the digital media landscape by proposing a robust framework for addressing prevalent challenges in lip synchronization through the novel application of latent diffusion models and enhanced alignment techniques. Future explorations may leverage these insights to further refine AI-driven audio-visual synthesis, particularly in domains demanding high temporal and detail fidelity.