Efficient Long-duration Talking Video Synthesis with Linear Diffusion Transformer under Multimodal Guidance

Abstract: Portrait image animation using audio has rapidly advanced, but challenges remain in efficiently fusing multimodal inputs while ensuring temporal and portrait consistency with minimal computational cost. To address this, we present LetsTalk, a LinEar diffusion TranSformer for Talking video synthesis. LetsTalk incorporates a deep compression autoencoder to obtain efficient latent representations, and a spatio-temporal-aware transformer with efficient linear attention to effectively fuse multimodal information and enhance spatio-temporal consistency. We systematically explore and summarize three fusion schemes, ranging from shallow to deep fusion. We thoroughly analyze their characteristics, applicability, and trade-offs, thereby bridging critical gaps in multimodal conditional guidance. Based on modality differences of image, audio, and video generation, we adopt deep (Symbiotic Fusion) for portrait to ensure consistency, and shallow (Direct Fusion) for audio to align animation with speech while preserving motion diversity. To maintain temporal consistency in long-duration video generation, we propose a memory bank mechanism that preserves inter-clip dependencies, effectively preventing degradation across extended sequences. Furthermore, we develop a noise-regularized training strategy that explicitly compensates for DDPM sampling artifacts, significantly improving the model's robustness in continuous generation scenarios.Our extensive experiments demonstrate that our approach achieves state-of-the-art generation quality, producing temporally coherent and realistic videos with enhanced diversity and liveliness, while maintaining remarkable efficiency through its optimized model design with 8$\times$ fewer parameters.