Mouth-Augmented Shading Mesh for Talking Heads

• Mouth-Augmented Shading Mesh is a 3DMM-rendered shading mesh sequence designed to correct the underfitting of mouth motion in standard facial models.
• It employs a four-source fusion pipeline that recombines identity, lighting, head motion, and speech-driven mouth dynamics for enhanced temporal stability.
• Empirical evaluations reveal that the method improves temporal coherence, motion robustness, and identity disentanglement compared to conventional 3DMM approaches.

Searching arXiv for the cited papers and closely related entries to ground the article. The Mouth-Augmented Shading Mesh is a 3DMM-rendered shading mesh sequence introduced in "MoCoTalk: Multi-Conditional Diffusion with Adaptive Router for Controllable Talking Head Generation" (Ye et al., 8 May 2026). It is designed for talking-head generation, where standard 3DMM-based rendered meshes are described as effective for global identity, pose, and illumination but weak in the mouth region. In MoCoTalk, the representation addresses two stated deficiencies of standard frame-wise 3DMM fitting—fine-grained lip articulation that is underfit and temporal jitter in mouth geometry—by combining DECA’s strengths for identity, coarse facial structure, and lighting with SPECTRE’s strengths for temporally consistent lip geometry. Its defining property is explicit decoupling and recombination of identity / shape, lighting, head motion, and mouth motion / jaw articulation / speech-driven expression, yielding what the paper describes as a temporally consistent geometric prior (Ye et al., 8 May 2026).

1. Problem setting and motivation

MoCoTalk situates the Mouth-Augmented Shading Mesh within the broader problem of controllable talking-head generation, which requires joint modeling of identity, head pose, facial expression, and mouth dynamics. The paper states that existing methods typically address only a subset of these factors and rely on fixed-weight or heuristic fusion when multiple conditions are involved. Within that setting, standard 3DMM-based shading meshes are said to capture global identity, pose, and illumination, yet to be weak in the mouth region (Ye et al., 8 May 2026).

The paper identifies two concrete failure modes. First, fine-grained lip articulation is underfit: the mesh does not accurately follow subtle mouth and jaw motion driven by speech. Second, temporal jitter appears frame-by-frame when a 3DMM fitter such as DECA is applied independently to each frame. The stated consequences are that standard meshes can “smear fine-grained lip articulation”, exhibit “temporal jitter”, and produce a structural mismatch between mesh geometry and actual lip motion.

These deficiencies are presented as harmful to audio-visual sync, lower-face appearance flow, temporal coherence, and cross-reenactment robustness. The Mouth-Augmented Shading Mesh is therefore not defined as a generic shading renderer improvement; it is a mouth-region-specific corrective mechanism inside a multi-condition generation system. A plausible implication is that MoCoTalk treats lower-face geometry as a bottleneck for both temporal stability and controllability, rather than only for local visual fidelity.

2. Construction, notation, and rendering pipeline

The representation is assembled through what the paper calls a four-source fusion pipeline. Rather than estimating all facial attributes from a single fitted frame or a single fitted video source, MoCoTalk recombines separate sources as follows (Ye et al., 8 May 2026):

• identity / shape: from a reference image
• lighting: from a chosen lighting reference image
• head motion: from an optional head-motion video
• mouth motion / jaw articulation / speech-driven expression: from the speech video, with SPECTRE providing temporally stable mouth tracking

The first step is identity and lighting extraction. DECA is fit to the reference image to obtain identity shape and camera parameters, and to the lighting reference image to obtain spherical-harmonic illumination:

• $s \in \mathbb{R}^{100}$: identity shape
• $c \in \mathbb{R}^{3}$: camera parameters
• $l \in \mathbb{R}^{27}$: spherical-harmonic illumination

These parameters are held constant across all output frames.

The second step is mouth-aware motion tracking. For each input video, SPECTRE predicts a base FLAME expression and DECA predicts a residual correction. MoCoTalk defines this fusion as:

$$e^{exp} = e^{SPECTRE} + \Delta e^{DECA}, \quad O^{jaw} = O^{SPECTRE}_{jaw} + \Delta O^{DECA}_{jaw}. \tag{7}$$

Here, $e^{SPECTRE} \in \mathbb{R}^{50}$ is the expression from SPECTRE, $\Delta e^{DECA} \in \mathbb{R}^{50}$ is the residual expression correction from DECA, $O^{SPECTRE}_{jaw}$ is the jaw pose from SPECTRE, and $\Delta O^{DECA}_{jaw}$ is the residual jaw correction from DECA. This is a hybrid construction: SPECTRE supplies temporally stable mouth-related tracking, while DECA is retained as a correction term rather than being discarded.

The third step is cross-source motion fusion. At frame $t$, MoCoTalk takes global head rotation $r_{head} \in \mathbb{R}^{3}$ from $c \in \mathbb{R}^{3}$0 and jaw pose $c \in \mathbb{R}^{3}$1 together with expression $c \in \mathbb{R}^{3}$2 from $c \in \mathbb{R}^{3}$3, then forms the fused pose vector

$c \in \mathbb{R}^{3}$4

The brackets denote concatenation. This explicitly separates global pose from speech-driven jaw articulation.

The fourth step is differentiable shading rendering. Using DECA’s renderer, the per-frame shading map is rendered as

$c \in \mathbb{R}^{3}$5

The output is a per-frame shading map $c \in \mathbb{R}^{3}$6, and the resulting sequence $c \in \mathbb{R}^{3}$7 is used as conditioning input for the shading branch.

3. Decoupling of facial factors

A central feature of the Mouth-Augmented Shading Mesh is explicit decoupling at both the source level and the parameter level (Ye et al., 8 May 2026).

At the source level, different attributes originate from different inputs:

• $c \in \mathbb{R}^{3}$8 provides identity and shape
• $c \in \mathbb{R}^{3}$9 provides lighting
• $l \in \mathbb{R}^{27}$0 provides head rotation
• $l \in \mathbb{R}^{27}$1 provides mouth motion and expression

At the parameter level, rendering is organized around separate variables for shape $l \in \mathbb{R}^{27}$2, expression $l \in \mathbb{R}^{27}$3, pose $l \in \mathbb{R}^{27}$4, camera $l \in \mathbb{R}^{27}$5, and lighting $l \in \mathbb{R}^{27}$6. The key fused pose relation,

$l \in \mathbb{R}^{27}$7

separates global head orientation $l \in \mathbb{R}^{27}$8 from speech-driven jaw articulation $l \in \mathbb{R}^{27}$9, while illumination remains isolated because $$e^{exp} = e^{SPECTRE} + \Delta e^{DECA}, \quad O^{jaw} = O^{SPECTRE}_{jaw} + \Delta O^{DECA}_{jaw}. \tag{7}$$0 is extracted from a separate lighting reference and then fixed over time.

The paper contrasts this with a standard 3DMM or DECA shading mesh, which is typically reconstructed per frame or from a single driving source and therefore entangles head pose, facial expression, jaw motion, identity, and lighting. The distinction can be summarized concisely.

Representation	Source structure	Main stated property
Standard 3DMM shading mesh	One fitted frame or one fitted video source	Often underfits mouth motion and can jitter over time
Mouth-Augmented Shading Mesh	Reference image, lighting image, speech video, optional head-motion video	Decouples and recombines identity, lighting, head motion, and mouth motion

This decomposition is the basis for MoCoTalk’s claim of attribute recombination at inference. The paper gives examples such as using a neutral frontal head motion but an expressive nodding sequence, preserving original lip dynamics while changing head motion, and specifying identity, head motion, mouth motion, and audio independently. This suggests that the representation is intended not only to stabilize the mouth region, but also to function as a controllable factorization of lower-face dynamics within a diffusion system.

4. Role inside MoCoTalk

Within MoCoTalk, the Mouth-Augmented Shading Mesh is one of four conditions passed to the generative model (Ye et al., 8 May 2026). The four conditions are:

1. reference portrait
2. facial keypoints
3. mouth-augmented shading mesh
4. audio

These conditions are processed by four adapters:

• Reference Adapter
• Motion Adapter
• Shading Adapter
• Audio Adapter

They are then fused inside the denoising U-Net by the Adaptive Multi-Condition Router, which computes channel-wise, timestep-aware gating over the four condition streams.

In this architecture, the Mouth-Augmented Shading Mesh serves as the geometry + lighting control stream. It is not the final output image sequence. Rather, it is a conditioning representation that guides the diffusion model. The paper states that the shading branch captures facial geometry and lighting, and that the mesh sequence “provides a richer appearance-flow signal around the lower face.”

The importance of sequence-level conditioning is emphasized in the implementation description. The mesh is used as a sequence, not by independently fitting frames in a naive manner, and training includes stochastic condition dropout so that the router can handle missing modalities. A plausible implication is that MoCoTalk treats the shading mesh as an intermediate control prior whose value lies in temporal regularity and factorized conditioning, rather than as a stand-alone geometric reconstruction product.

5. Empirical behavior and ablation evidence

MoCoTalk reports that the Mouth-Augmented Shading Mesh contributes primarily to temporal coherence, motion-related behavior, cross-reenactment robustness, and identity disentanglement, rather than to marginal gains on single-frame reconstruction metrics (Ye et al., 8 May 2026).

The ablation labeled “w/o Augmented Mesh” is described as achieving slightly better SSIM, PSNR, and LPIPS, but performing worse on temporal coherence, motion-related metrics, cross-reenactment robustness, and identity disentanglement. The paper explicitly interprets this as evidence that frame-level metrics can hide instability and that the absence of a temporally coherent geometric prior becomes more obvious under cross-reenactment.

The reported quantitative examples are as follows:

Metric	Full model	w/o Augmented Mesh
Self-reenactment FVD	40.75	41.51
Self-reenactment AED	0.1028	0.1030
Cross-reenactment AED	89.60	100.8
Cross-reenactment APD	0.2382	0.2655
Cross-reenactment ID	0.8293	0.8065

These numbers support the paper’s specific claim that the augmented mesh is especially valuable when the driving identity changes. The qualitative claims are aligned with that interpretation: the representation is said to help avoid smearing in the mouth region, jitter in lip articulation, and mismatch between rendered mesh and true speech motion. Figure 1 is described as comparing a source video, a DECA-only mesh, and the mouth-augmented fusion, with the augmented mesh visually better capturing lower-face geometry.

The stated contribution is therefore not primarily one of per-frame sharpness. It is the provision of a temporally stable geometric prior whose benefits emerge more clearly in sequence-level evaluation and in cross-identity transfer.

6. Relation to adjacent mouth-aware representations and limitations

The Mouth-Augmented Shading Mesh is a mouth-aware representation, but it should not be conflated with explicit inner-mouth geometry synthesis. In MoCoTalk, the representation is a 3DMM-rendered shading mesh sequence used as a conditioning signal. By contrast, "OmniFaceRig: Fully Automatic Inner-Mouth-Aware Face Rigging Across Diverse 3D Character Topologies" (Wang et al., 6 Jun 2026) describes a different form of mouth augmentation: a static, surface-only 3D character mesh is converted into an inner-mouth-aware FACS rig by synthesizing teeth, gums, and tongue, repacking UVs, and transferring blendshapes. That contrast indicates that “mouth augmentation” can refer either to a conditioning representation for video generation or to explicit oral-cavity construction for rigging and rendering.

MoCoTalk’s representation remains constrained by the broader limitations of the full system. The paper states that the framework is still constrained by the fitting accuracy of 3DMM-style reconstruction. It also notes that the model generates only 8 frames per forward pass, with longer videos stitched using overlap, which can create subtle boundary discontinuities. In addition, the paper identifies room for improvement in identity preservation and lip-sync accuracy overall.

These limitations matter directly for the Mouth-Augmented Shading Mesh because the representation is only as reliable as the quality of DECA and SPECTRE fitting, the availability of the source videos, and the consistency of the inferred motion decomposition. A plausible implication is that the representation is best understood not as a complete solution to mouth modeling, but as a structured and temporally stabilized control signal that addresses a specific weakness of standard DECA-style shading meshes in talking-head generation.