MoCoTalk: Multi-Conditional Talking-head Diffusion

• MoCoTalk is a diffusion-based multi-conditional talking-head generator that enables controllable facial video synthesis by integrating four distinct modalities.
• The system employs an Adaptive Multi-Condition Router to fuse identity cues, head pose, 3DMM-based shading, and audio, ensuring channel-wise and timestep-aware modulation.
• It introduces a Mouth-Augmented Shading Mesh and lip consistency loss to enhance temporal stability and synchronization between visual and audio signals.

Searching arXiv for the cited papers to ground the article in current records. MoCoTalk is a diffusion-based, multi-conditional talking-head generator for controllable facial video synthesis. It is designed to jointly model identity, head pose, facial expression, mouth dynamics, and lighting within a single latent video diffusion framework, while avoiding the fixed-weight or heuristic condition fusion that characterizes many earlier systems. The method unifies four control signals—a reference image, facial keypoints, 3DMM-rendered shading meshes, and corresponding speech audio—and introduces an Adaptive Multi-Condition Router for channel-wise, timestep-aware fusion, a Mouth-Augmented Shading Mesh for geometry and lighting control, and a lip consistency loss for audio-visual alignment (Ye et al., 8 May 2026).

1. Problem formulation and control space

MoCoTalk targets controllable talking-head generation under a multi-factor view of facial dynamics. In this formulation, identity and appearance are tied to a reference portrait; head pose and coarse expression are represented by facial keypoints; geometry and lighting are conveyed through 3DMM-based shading meshes; and speech timing and mouth articulation are provided by audio. The central premise is that these factors are complementary but heterogeneous, and that naive multi-condition fusion can produce destructive interference rather than stable control (Ye et al., 8 May 2026).

The four condition streams have distinct roles.

Condition stream	Primary role
Reference image	Identity and appearance cues
Facial keypoints	Head pose and coarse facial expression
3DMM-rendered shading meshes	Detailed geometry and lighting
Speech audio	Mouth dynamics and timing

MoCoTalk is explicitly designed for attribute-level controllability. Identity is controlled by the reference image $I_{\text{ref}}$; lighting is controlled by a lighting reference image $I_{\text{light}}$; head pose is controlled by a head-motion video $V_{\text{head}}$; and mouth motion and expression are controlled by speech video or audio $V_{\text{speech}}$. This makes it possible to recombine sources at inference, such as identity from one image, lighting from another, head motion from one video, and speech-driven mouth motion from another (Ye et al., 8 May 2026).

A common misconception is to treat talking-head synthesis as primarily an audio-to-lips problem. MoCoTalk instead treats talking-head generation as a coupled control problem over multiple facial and scene attributes. This suggests that its main contribution is not only better lip synchronization, but a broader control interface over the latent factors that govern facial video realism.

2. Diffusion backbone and condition adapters

MoCoTalk is built on Stable Video Diffusion (SVD), a latent video diffusion model. A VAE encoder $\mathcal{E}(\cdot)$ maps RGB video $X_0$ to latents $z_0$, and the forward noising process is

$$z_t = \sqrt{\bar{\alpha}_t} z_0 + \sqrt{1 - \bar{\alpha}_t}\,\epsilon,\quad \epsilon \sim \mathcal{N}(0, I).$$

The denoising network $\epsilon_\theta$ is trained with the standard SVD noise-prediction objective

$$\mathcal{L}_{\text{SVD}} = \mathbb{E}_{z_0, c, \epsilon, t}\left[ \|\epsilon - \epsilon_\theta(z_t, t, c)\|_2^2 \right].$$

MoCoTalk retains this backbone but replaces single-condition conditioning with four parallel condition adapters aligned to the U-Net’s shallow layers (Ye et al., 8 May 2026).

The reference adapter $I_{\text{light}}$0 takes the reference portrait $I_{\text{light}}$1 and encodes identity and appearance cues such as texture, hair, and clothing. It also uses CLIP embeddings, as in SVD, for deeper cross-attention. The motion adapter $I_{\text{light}}$2 operates on per-frame 2D facial keypoint heatmaps and captures head pose and coarse expression. The shading adapter $I_{\text{light}}$3 processes per-frame rendered shading maps from the Mouth-Augmented Mesh pipeline and provides a 3D geometric and illumination prior. The audio adapter $I_{\text{light}}$4 begins with a pre-trained Wav2Vec 2.0 encoder, constructs a local temporal window $I_{\text{light}}$5 with $I_{\text{light}}$6 and $I_{\text{light}}$7, and maps the resulting audio features into spatial feature maps so that audio can be fused in the same spatial feature space as the visual conditions (Ye et al., 8 May 2026).

This spatialization of audio is architecturally important. Rather than routing audio through a separate token-level attention path, MoCoTalk projects all four conditions into a common shallow-feature space and then learns how to fuse them adaptively. A plausible implication is that this design makes audio a first-class control signal in the same representational regime as pose and geometry, rather than an auxiliary modulation source.

3. Adaptive Multi-Condition Router

The Adaptive Multi-Condition Router is MoCoTalk’s main fusion mechanism. Its purpose is to replace fixed additive fusion or fixed scalar weights with channel-wise, timestep-aware gates computed from the current U-Net feature, the diffusion timestep, and the pooled summaries of all condition branches (Ye et al., 8 May 2026).

Let $I_{\text{light}}$8 be the shallow U-Net feature after input convolution, and let each adapter output $I_{\text{light}}$9 have the same shape. The router first computes global pooled summaries

$V_{\text{head}}$0

where $V_{\text{head}}$1 is the diffusion timestep embedding and $V_{\text{head}}$2 projects it to channel dimension $V_{\text{head}}$3. These summaries are concatenated and passed through a small MLP $V_{\text{head}}$4, followed by a softmax over the four condition branches:

$V_{\text{head}}$5

The fused feature is then

$V_{\text{head}}$6

The gates $V_{\text{head}}$7 are channel-wise and branch-specific, with shape $V_{\text{head}}$8, so different channels can emphasize different modalities. Conditions that are absent or dropped are masked by assigning their logits a large negative constant before softmax, which effectively sets $V_{\text{head}}$9 (Ye et al., 8 May 2026).

The router’s empirical role is unusually strong. In ablation, removing it and reverting to naive additive fusion causes FVD to increase from 40.75 to 592.5, while identity and motion metrics also degrade substantially. The paper therefore treats adaptive gating not as a refinement but as a necessary mechanism for stable multi-conditional synthesis (Ye et al., 8 May 2026).

This result directly distinguishes MoCoTalk from earlier SVD-based talking-head systems that combine conditions by addition, concatenation, or static heuristics. Relative to a two-stage motion-disentangled design such as MoDiTalker, which separates audio-to-motion and motion-to-video and conditions the second stage with tri-plane motion, pose, and identity latents, MoCoTalk instead retains a single diffusion backbone and resolves condition interaction through adaptive routing (Kim et al., 2024).

4. Mouth-Augmented Shading Mesh and lip supervision

MoCoTalk’s geometric control is based on a 3DMM pipeline that combines DECA and SPECTRE. DECA provides FLAME-based estimates of shape $V_{\text{speech}}$0, expression $V_{\text{speech}}$1, pose $V_{\text{speech}}$2, camera $V_{\text{speech}}$3, and illumination $V_{\text{speech}}$4. However, frame-wise DECA fitting underfits the mouth region and introduces temporal jitter when applied directly to speech video. To address this, MoCoTalk augments DECA with SPECTRE, which provides temporally consistent expression and jaw estimates, especially for lips (Ye et al., 8 May 2026).

SPECTRE outputs a base expression $V_{\text{speech}}$5, an expression residual $V_{\text{speech}}$6, a jaw pose $V_{\text{speech}}$7, and a residual $V_{\text{speech}}$8. MoCoTalk combines them as

$V_{\text{speech}}$9

This yields a mouth-aware expression and jaw representation that is smoother and more detailed than plain DECA fits. For cross-source control, head rotation $\mathcal{E}(\cdot)$0 is taken from a head-motion video, while jaw pose and expression are taken from speech video, and the full pose vector is formed as

$\mathcal{E}(\cdot)$1

The renderer then produces grayscale shading maps

$\mathcal{E}(\cdot)$2

with identity shape $\mathcal{E}(\cdot)$3 and illumination $\mathcal{E}(\cdot)$4 held fixed for the chosen identity and lighting reference (Ye et al., 8 May 2026).

The resulting Mouth-Augmented Shading Mesh decouples identity and shape, lighting, head motion, and mouth motion and expression. At inference, this supports recombination across sources: identity from $\mathcal{E}(\cdot)$5, lighting from $\mathcal{E}(\cdot)$6, head rotation from $\mathcal{E}(\cdot)$7, and mouth dynamics from $\mathcal{E}(\cdot)$8 or audio. Ablation indicates that this mesh augmentation is especially beneficial in cross-reenactment, where it improves temporal stability and motion transfer even when some static image metrics remain similar (Ye et al., 8 May 2026).

MoCoTalk further adds a lip consistency loss. Predicted denoised latents are decoded to RGB frames, resized to $\mathcal{E}(\cdot)$9, and cropped with a single temporally stable mouth bounding box computed from ground-truth facial landmarks over the full clip. A frozen lip-reading encoder $X_0$0 from SPECTRE extracts features from mouth crops of both ground-truth and predicted frames:

$X_0$1

The lip loss is

$X_0$2

with only $X_0$3 frames sampled per step for memory efficiency. The full training objective is

$X_0$4

Removing $X_0$5 moderately worsens FVD and slightly degrades motion metrics, while slightly improving VQA, indicating that the loss specifically sharpens mouth dynamics rather than generic perceptual quality (Ye et al., 8 May 2026).

5. Training procedure, controllability, and empirical performance

MoCoTalk is trained on CelebV-HQ, MultiTalk, TFHP, and MEAD. Because SVD is sensitive to video quality, videos are filtered with FAST-VQA and those with quality score below 0.6 are removed, leaving a final training set of approximately 34,000 videos. Frames are resized and cropped to $X_0$6, training uses 8-frame clips, optimization runs for 30,000 iterations with batch size 8 and gradient accumulation, the optimizer is 8-bit Adam, the learning rate is $X_0$7, training is performed on a single NVIDIA RTX 6000 (32GB), and training time is approximately 7 days. Stochastic condition dropout is applied to all four branches so that the router learns to operate under partial conditioning (Ye et al., 8 May 2026).

At inference, MoCoTalk exposes a compositional control interface. One may fix identity and lighting while swapping head pose, or hold identity and head pose fixed while changing mouth motion by changing speech input. The paper describes the procedure as: select $X_0$8, $X_0$9, $z_0$0, and speech audio or $z_0$1; run DECA on the reference and lighting images; run SPECTRE+DECA on speech video for expression and jaw; obtain head rotation from the head-motion source; render shading meshes; compute facial keypoint maps; extract and window audio features with Wav2Vec; and then perform diffusion sampling through the routed SVD backbone (Ye et al., 8 May 2026).

Evaluation is conducted on HDTF using the first 100 frames of each test video in self-reenactment and cross-reenactment settings. Self-reenactment metrics include PSNR, SSIM, LPIPS, ID, AED, APD, AKD, FVD, VQA, and Sync-C. Cross-reenactment emphasizes AED, APD, and ID+ (Ye et al., 8 May 2026).

In self-reenactment, MoCoTalk reports SSIM $z_0$2, PSNR $z_0$3 dB, LPIPS $z_0$4, FVD $z_0$5, AED $z_0$6, APD $z_0$7, AKD $z_0$8, VQA $z_0$9, ID $$z_t = \sqrt{\bar{\alpha}_t} z_0 + \sqrt{1 - \bar{\alpha}_t}\,\epsilon,\quad \epsilon \sim \mathcal{N}(0, I).$$0, and Sync-C $$z_t = \sqrt{\bar{\alpha}_t} z_0 + \sqrt{1 - \bar{\alpha}_t}\,\epsilon,\quad \epsilon \sim \mathcal{N}(0, I).$$1. The paper characterizes these results as state-of-the-art or near state-of-the-art on most structural, motion, and perceptual metrics, with especially strong FVD and motion consistency. In cross-reenactment, MoCoTalk reports AED $$z_t = \sqrt{\bar{\alpha}_t} z_0 + \sqrt{1 - \bar{\alpha}_t}\,\epsilon,\quad \epsilon \sim \mathcal{N}(0, I).$$2, APD $$z_t = \sqrt{\bar{\alpha}_t} z_0 + \sqrt{1 - \bar{\alpha}_t}\,\epsilon,\quad \epsilon \sim \mathcal{N}(0, I).$$3, and ID+ $$z_t = \sqrt{\bar{\alpha}_t} z_0 + \sqrt{1 - \bar{\alpha}_t}\,\epsilon,\quad \epsilon \sim \mathcal{N}(0, I).$$4, with the paper noting that AED is best among methods except FLOAT and ID+ is second-best behind SadTalker (Ye et al., 8 May 2026).

The metrics also expose the method’s trade-offs. MoCoTalk does not maximize identity similarity or lip-sync in isolation: ID $$z_t = \sqrt{\bar{\alpha}_t} z_0 + \sqrt{1 - \bar{\alpha}_t}\,\epsilon,\quad \epsilon \sim \mathcal{N}(0, I).$$5 is below SadTalker’s $$z_t = \sqrt{\bar{\alpha}_t} z_0 + \sqrt{1 - \bar{\alpha}_t}\,\epsilon,\quad \epsilon \sim \mathcal{N}(0, I).$$6, VQA $$z_t = \sqrt{\bar{\alpha}_t} z_0 + \sqrt{1 - \bar{\alpha}_t}\,\epsilon,\quad \epsilon \sim \mathcal{N}(0, I).$$7 is below LCVD’s $$z_t = \sqrt{\bar{\alpha}_t} z_0 + \sqrt{1 - \bar{\alpha}_t}\,\epsilon,\quad \epsilon \sim \mathcal{N}(0, I).$$8, and some specialized lip-sync baselines report higher Sync-C. The paper explicitly frames this as a balance in favor of overall controllability and multi-factor fidelity rather than optimization for a single axis (Ye et al., 8 May 2026).

6. Position in the literature, limitations, and naming ambiguities

Within talking-head synthesis, MoCoTalk occupies a specific point between audio-only, video-driven, and 3DMM-guided diffusion systems. Audio-only models such as SadTalker, DiffTalk, and Hallo2 can produce strong lip synchronization but do not provide independent control over head pose or lighting. Video-driven methods based on landmarks or implicit motion can transfer pose and expression but often lack strong audio alignment or suffer from temporal instability. LCVD, an SVD-based relightable portrait animation model, uses a shading adapter and reference adapter, but MoCoTalk argues that LCVD still relies on fixed fusion between subspaces and uses DECA meshes that underfit lip dynamics and jitter frame-wise (Ye et al., 8 May 2026).

Compared with MoDiTalker, which tackles the same high-level task through an explicit audio-to-motion and motion-to-video decomposition, MoCoTalk is less stage-separated and more fusion-centric: it uses one diffusion backbone conditioned by four parallel modalities and resolves inter-condition conflict through an adaptive router rather than an explicit motion bottleneck (Kim et al., 2024). This suggests a broader design distinction in the field between motion-disentangled pipelines and unified multi-condition diffusion.

The method has several explicit limitations. The backbone operates on 8-frame clips, so longer sequences are generated by stitching overlapping segments; this can introduce subtle discontinuities at segment boundaries. The system remains bounded by the expressiveness of DECA, SPECTRE, and the underlying 3DMM, so extreme poses, occlusions, or unusual expressions remain difficult. Diffusion cost is nontrivial: 100-frame generation takes approximately 2 minutes on an RTX 6000. The paper also notes that identity preservation and lip-sync can still be improved relative to specialized baselines (Ye et al., 8 May 2026).

There is also a nomenclature issue. In the DeepTalk paper, “MoCoTalk” is used only as a hypothetical conversational, speech-centric multimodal system in a discussion of adaptive modality-specific MoE; that usage does not denote the controllable talking-head diffusion model described here (Shao et al., 27 Jun 2025). Likewise, “MoCo” refers to a separate motion-consistent human video generation framework based on structure-appearance decoupling, not to MoCoTalk’s talking-head architecture (Wang et al., 24 Aug 2025).

Overall, MoCoTalk is best understood as a multi-conditional controllable talking-head diffusion system whose defining innovations are adaptive router-based fusion, a mouth-aware 3DMM shading representation, and explicit lip-region supervision. Its contribution is not merely higher-fidelity facial animation, but a more factorized and manipulable control model for talking-head generation (Ye et al., 8 May 2026).