MoCoTalk: Multi-Conditional Diffusion with Adaptive Router for Controllable Talking Head Generation

Abstract: Talking-head generation requires joint modeling of identity, head pose, facial expression, and mouth dynamics. Existing methods typically address only a subset of these factors, and rely on fixed-weight or heuristic fusion when multiple conditions are involved. We present MoCoTalk, a multi-conditional video diffusion framework that unifies four complementary control signals: a reference image, facial keypoints, 3DMM-rendered shading meshes, and the corresponding speech audio. To resolve destructive interference among heterogeneous conditions, we introduce an Adaptive Multi-Condition Router that computes channel-wise, timestep-aware gating over the four condition streams, allowing the fusion strategy to vary with both feature subspace and noise level. To better capture speech-related facial dynamics, we design a Mouth-Augmented Shading Mesh, a 3DMM-based representation that decouples head motion, mouth motion, expression, and lighting. This design provides a temporally consistent geometric prior and allows flexible recombination of these attributes at inference. We further introduce a lip consistency loss to tighten audio-visual alignment. Extensive experiments show that MoCoTalk achieves state-of-the-art performance on the majority of structural, motion, and perceptual metrics, while offering attribute-level controllability that single-condition methods do not provide.

• The paper introduces a multi-conditional diffusion framework that uses an adaptive router for dynamic fusion of diverse modalities.
• It integrates mouth-augmented mesh generation and a lip consistency loss to improve audio-visual alignment and temporal coherence.
• Experimental validation on benchmarks like HDTF and CelebV-HQ shows significant gains in PSNR, SSIM, and reduced FVD for video quality.

MoCoTalk: Multi-Conditional Diffusion with Adaptive Router for Controllable Talking Head Generation

Problem Formulation and Motivation

MoCoTalk addresses the challenge of controllable talking-head video generation, wherein the goal is to animate a static portrait according to multimodal drivers—such as motion cues, facial expressions, and speech—while preserving identity and achieving high temporal and perceptual fidelity. Prevailing approaches typically rely on either video-driven or audio-driven paradigms, but suffer from limitations in attribute-level controllability, modality fusion, and global consistency. Most prior works employ rigid fusion strategies and lack the architectural flexibility required to cohesively balance diverse signals, especially under cross-reenactment settings.

Architecture and Methodological Innovations

MoCoTalk introduces a unified diffusion model for multi-conditional talking-head synthesis, incorporating four heterogeneous control signals: reference appearance (portrait image), facial keypoints (motion), mouth-augmented 3DMM shading meshes (geometry and illumination), and audio speech tracks. Each signal is processed by a dedicated spatial adapter, enabling aligned latent feature extraction compatible with the U-Net backbone of Stable Video Diffusion (SVD).

Adaptive Multi-Condition Router

A core innovation is the Adaptive Router, which replaces static fusion with channel-wise, timestep-aware gating. The router leverages spatially pooled summaries from each modality—plus backbone and timestep embeddings—to compute dynamic softmax weights. This mechanism prevents destructive interference among modalities and allows adaptive subspace allocation, thereby balancing the fusion of appearance, expression, pose, and audio in accordance with the denoising trajectory. It supports both full multimodal and any-modal settings by masking missing modalities during training and inference, enhancing robustness.

Mouth-Augmented Mesh Generation

To address underfitting and temporal instability in lower-face geometries observed in prior 3DMM renderings (e.g., DECA), MoCoTalk integrates mouth tracking from SPECTRE. This fusion allows decoupling and recombination of identity, lighting, head motion, and speech-driven mouth dynamics at inference. The result is a temporally coherent mesh sequence that improves geometric priors for the lower face, facilitating customizable motion transfer and fine-grained articulation.

Lip Consistency Loss

The introduction of a lip consistency objective, computed via a frozen lip-reading encoder on mouth-region crops, ensures that audio–visual alignment and semantic lip motion are strongly supervised. This is particularly important for speech-driven animation, as it mitigates drift and stuttering and enhances synchronization fidelity.

Experimental Validation

MoCoTalk is evaluated on multiple benchmarks—including HDTF, CelebV-HQ, MEAD, and MultiTalk—using both self- and cross-reenactment protocols. Strong quantitative improvements are demonstrated across structural, motion, perceptual, and synchronization metrics:

• Structural Fidelity (PSNR, SSIM, LPIPS): MoCoTalk attains the highest scores on seven out of ten metrics under self-reenactment, with PSNR of 23.55 and SSIM of 0.7998.
• Temporal Coherence (FVD): The model reports a substantial reduction in Fréchet Video Distance (FVD), achieving 40.75 (a 37% improvement over prior baselines).
• Motion and Expression Accuracy (AED, APD, AKD): It excels in average expression and pose distance, confirming high precision in motion transfer.
• Attribute-Level Controllability: Unlike single-condition methods (e.g., SadTalker, Diff2Lip, Hallo2), MoCoTalk allows independent and arbitrary recombination of identity, lighting, head pose, and mouth dynamics at inference.

Ablation studies reveal that removal of the Adaptive Router causes catastrophic degradation in video quality and motion fidelity, underscoring its necessity for robust multi-modal fusion. The Mouth-Augmented Mesh further improves audio–visual consistency and temporal coherence, especially under cross-reenactment.

Limitations and Extensions

While MoCoTalk offers competitive performance and flexibility, it does not lead on identity preservation and lip-sync metrics compared to specialized architectures. The method currently generates only short video segments (eight frames at a time), which may result in discontinuities when concatenating longer sequences. Future directions include stronger identity and synchronization constraints and migration to autoregressive backbones for continuous long-form generation.

Theoretical and Practical Implications

The multi-conditional fusion paradigm introduced by MoCoTalk advances the state-of-the-art in joint modeling of heterogeneous cues—enabling a framework where each driver signal contributes dynamically to the denoising process. The channel-wise, timestep-aware gating mechanism sets a precedent for adaptive fusion in video synthesis and other multimodal generative tasks. Practically, this architecture facilitates highly customizable avatars and digital humans, with potential applicability in telepresence, dubbing, animation, and HCI scenarios.

On a theoretical level, the decoupling of spatial, temporal, and semantic signals in the latent space via adaptive routers and mesh augmentation informs future research in both diffusion-based video generative models and multimodal control systems. The robustness to missing modalities and ability to provide attribute-level manipulation mark significant progress towards practical deployment in real-world multimodal interfaces.

Conclusion

MoCoTalk demonstrates a multi-conditional diffusion framework for talking-head generation that unifies adaptive multimodal fusion, temporally coherent geometric priors, and audio–visual alignment. The adaptive router and mouth-augmented mesh constitute critical architectural innovations, validated by strong numerical performance and controllability across complementary facial attributes. Although further improvements are needed in identity and lip-sync fidelity, MoCoTalk lays the groundwork for robust, customizable talking-head synthesis and offers new directions for multimodal generative modeling in video domains.