M2DAO-Talker: Audio-Driven Talking-Head Framework

• M2DAO-Talker is an audio-driven talking-head generation framework that reformulates synthesis into three stages: video preprocessing, motion representation, and rendering reconstruction.
• It decouples motion using multi-granular strategies by separating rigid head rotation, macro facial expressions, and fine oral articulations for improved realism.
• An alternating optimization approach with a motion consistency constraint enhances boundary fidelity, achieving superior PSNR improvements and real-time inference.

Searching arXiv for the specified paper and closely related talking-head generation work for accurate citation. M2DAO-Talker is an audio-driven talking-head generation framework that reformulates the synthesis problem into a unified pipeline with three stages: video preprocessing, motion representation, and rendering reconstruction. It is designed to address rendering artifacts reported for existing 3D methods, including motion blur, temporal jitter, and local penetration, which are attributed to limitations in representing stable, fine-grained motion fields. The framework combines a 2D portrait preprocessing pipeline, multi-granular motion decoupling, a motion consistency constraint, and an alternating optimization strategy. Experiments across multiple datasets report state-of-the-art performance, including a 2.43 dB PSNR improvement in generation quality and a 0.64 gain in user-evaluated video realness versus TalkingGaussian, while maintaining 150 FPS inference speed (Jiang et al., 11 Jul 2025).

1. Unified formulation of talking-head generation

M2DAO-Talker reformulates talking-head synthesis into a three-step framework. The first step, video preprocessing, extracts per-frame deformation controls, specifically semantic masks and camera parameters, in order to decouple rigid and non-rigid motions. The second step, motion representation, builds three separate motion branches: rigid head rotation through camera parameters, non-rigid facial expressions through a Face Branch, and fine oral articulations through an Inside Mouth Branch. The third step, rendering reconstruction, alternately optimizes facial and oral motion parameters under a motion consistency constraint to produce a final fused frame (Jiang et al., 11 Jul 2025).

This formulation is significant because it treats preprocessing, motion parameterization, and reconstruction as coupled stages rather than isolated subproblems. The paper’s stated premise is that richer priors from preprocessing enable more effective motion decoupling, and that alternating optimization subsequently refines the branches into a coherent final rendering. This suggests that the contribution is not limited to a new renderer or a new motion prior in isolation, but to a pipeline-level reorganization of the talking-head generation problem.

2. 2D portrait preprocessing and deformation controls

The preprocessing stage is intended to supply stable, disentangled motion priors. It consists of Motion Region Segmentation (MRS) and Head Pose Estimation (HPE) (Jiang et al., 11 Jul 2025).

For MRS, the method begins with sparse positional prompts $P_{le},P_{re},P_{m},\dots$ derived from facial landmarks, including eyes, mouth corners, and torso points. A position-guided encoder refines these prompts into a binary foreground mask and then into subregion masks for the face and mouth: $$M_R \;=\;\mathrm{Enc}_p\bigl(P_{le},P_{re},P_{m},P_{lt},P_{rt}\bigr), \quad \{M_{\rm f},M_{\rm m}\} = \mathrm{parse}(M_R).$$ The paper characterizes this as a two-stage, teeth-aware parsing procedure that alleviates edge blur and local “piercing” artifacts seen in prior bisenet-based methods (Jiang et al., 11 Jul 2025).

For HPE, the framework isolates rigid head motion by detecting temporally stable keypoints $K_t$, specifically ears and hairline points, using Laplacian-filtered optical flow. Camera scaling $T$, rotation $R$, and focal length $F$ are then estimated by minimizing

$$\mathcal{L}_{\rm flow} \;=\; \sum_{t=1}^T \bigl\|\,P_t(R,T,F)\;-\;K_t\|_2,$$

where $P_t(\cdot)$ denotes projected 3D facial landmarks. Frames with excessive flow error are excluded from the rigid-motion fit, but retained later in the full reconstruction (Jiang et al., 11 Jul 2025).

After segmentation and pose estimation, the portrait frame $I$ is partitioned into face, mouth, and background components: $$I_{\rm f}=I\odot M_{\rm f},\quad I_{\rm m}=I\odot M_{\rm m},\quad I_{\rm bg}=I - (I_{\rm f}+I_{\rm m}).$$ This decomposition operationalizes the later motion decoupling. A plausible implication is that the quality of these masks directly constrains the quality of branch-specific supervision and the stability of region-wise optimization.

3. Multi-granular motion decoupling

The central modeling choice in M2DAO-Talker is multi-granular motion decoupling. Rigid head rotation is fully captured by the camera parameters $$M_R \;=\;\mathrm{Enc}_p\bigl(P_{le},P_{re},P_{m},P_{lt},P_{rt}\bigr), \quad \{M_{\rm f},M_{\rm m}\} = \mathrm{parse}(M_R).$$0, while non-rigid motion is split into a Face Branch for macro facial expressions and an Inside Mouth Branch for fine oral articulations. Both branches deform a shared canonical 3D Gaussian splatting model through localized offsets $$M_R \;=\;\mathrm{Enc}_p\bigl(P_{le},P_{re},P_{m},P_{lt},P_{rt}\bigr), \quad \{M_{\rm f},M_{\rm m}\} = \mathrm{parse}(M_R).$$1 (Jiang et al., 11 Jul 2025).

Let $$M_R \;=\;\mathrm{Enc}_p\bigl(P_{le},P_{re},P_{m},P_{lt},P_{rt}\bigr), \quad \{M_{\rm f},M_{\rm m}\} = \mathrm{parse}(M_R).$$2 denote Gaussian centers in the face and mouth regions. The paper fuses phoneme-aware audio feature $$M_R \;=\;\mathrm{Enc}_p\bigl(P_{le},P_{re},P_{m},P_{lt},P_{rt}\bigr), \quad \{M_{\rm f},M_{\rm m}\} = \mathrm{parse}(M_R).$$3 and expression feature $$M_R \;=\;\mathrm{Enc}_p\bigl(P_{le},P_{re},P_{m},P_{lt},P_{rt}\bigr), \quad \{M_{\rm f},M_{\rm m}\} = \mathrm{parse}(M_R).$$4, described as six-dimensional FACS AUs, as follows: $$M_R \;=\;\mathrm{Enc}_p\bigl(P_{le},P_{re},P_{m},P_{lt},P_{rt}\bigr), \quad \{M_{\rm f},M_{\rm m}\} = \mathrm{parse}(M_R).$$5 where $$M_R \;=\;\mathrm{Enc}_p\bigl(P_{le},P_{re},P_{m},P_{lt},P_{rt}\bigr), \quad \{M_{\rm f},M_{\rm m}\} = \mathrm{parse}(M_R).$$6 is a tri-plane hash encoder and $$M_R \;=\;\mathrm{Enc}_p\bigl(P_{le},P_{re},P_{m},P_{lt},P_{rt}\bigr), \quad \{M_{\rm f},M_{\rm m}\} = \mathrm{parse}(M_R).$$7 denotes concatenation. The resulting offsets are applied to Gaussian position, scale, and orientation, while view-dependent color is retained through spherical harmonics (Jiang et al., 11 Jul 2025).

The resulting decomposition separates rigid motion from two distinct non-rigid scales of motion, one associated with broader facial deformation and the other with intraoral articulation. This suggests that the method’s notion of “multi-granular” refers not only to spatial localization but also to motion frequency and semantic specificity. In that sense, the framework treats lip, teeth, and oral cavity dynamics as insufficiently modeled by a single facial deformation field.

Component	Signal or parameterization	Role
Rigid head motion	$$M_R \;=\;\mathrm{Enc}_p\bigl(P_{le},P_{re},P_{m},P_{lt},P_{rt}\bigr), \quad \{M_{\rm f},M_{\rm m}\} = \mathrm{parse}(M_R).$$8	Captures head rotation through camera parameters
Face Branch	$$M_R \;=\;\mathrm{Enc}_p\bigl(P_{le},P_{re},P_{m},P_{lt},P_{rt}\bigr), \quad \{M_{\rm f},M_{\rm m}\} = \mathrm{parse}(M_R).$$9	Models macro facial expressions
Inside Mouth Branch	$K_t$0	Models fine oral articulations

4. Motion consistency constraint and kinematic coherence

To address “penetration” and kinematic discontinuity between moving face and torso, M2DAO-Talker introduces a motion consistency constraint (MCC) based on full-portrait supervision. The Face Branch renders color $K_t$1 with transparency $K_t$2, which is composited with the static background $K_t$3 to form

$K_t$4

The associated loss is

$K_t$5

with $K_t$6 (Jiang et al., 11 Jul 2025).

According to the paper, MCC forces facial Gaussians’ alpha and color to respect the torso background, thereby sharpening the face-torso boundary and eliminating ghosting. The stated target is not merely pixel-wise fidelity but head-torso kinematic consistency. This suggests that the method treats boundary quality as a manifestation of motion aliasing and compositional inconsistency rather than solely as a segmentation problem.

An important technical distinction is that the consistency term is defined at the full-portrait level rather than only within the facial crop. That design choice aligns the non-rigid facial motion field with the static or separately modeled torso context. The paper explicitly identifies this as a mechanism for mitigating penetration artifacts caused by motion aliasing (Jiang et al., 11 Jul 2025).

5. Alternating optimization and fused rendering

Joint optimization of facial and oral branches under MCC is reported to induce boundary artifacts such as lip-color leakage and tooth discoloration. M2DAO-Talker therefore adopts a two-stage alternating optimization scheme (Jiang et al., 11 Jul 2025).

In Phase 1, the Inside Mouth Branch is frozen and the Face Branch parameters $K_t$7 are updated by minimizing a region-specific loss,

$K_t$8

In Phase 2, both branches are unfrozen and the final fused image is optimized: $K_t$9 under the full reconstruction loss

$T$0

The paper provides the following pseudocode outline: $T$1

$T$2

$T$3

The stated rationale is that this alternating sequence quickly establishes a robust facial geometry before allowing the mouth branch to correct finer details. The paper interprets this as preventing overfitting interference (Jiang et al., 11 Jul 2025). A plausible implication is that branch interference is especially acute in the lip-tooth boundary regime, where region overlap and high-frequency appearance variation make simultaneous optimization unstable.

6. Quantitative findings, ablations, and scope

On seven standard talking-head clips, M2DAO-Talker reports self-reconstruction quality of PSNR $T$4 dB, LPIPS $T$5, and SSIM $T$6. Relative to TalkingGaussian, the paper reports a $T$7 dB improvement in PSNR. For motion accuracy, it reports Landmark Distance LMD $T$8, Lower-face AU error $T$9, and Sync-C $R$0. In a 20-user perceptual study, the reported realness gain is $R$1 on a 1–5 scale, reaching 87% of ground-truth realness. Runtime is reported as 0.6 h training per identity and 150 FPS inference on an NVIDIA RTX3090 (Jiang et al., 11 Jul 2025).

The ablation findings reported in the paper are structurally aligned with the model design. The hybrid MRS, compared with BiSeNet, and the proposed HPE, compared with 3DMM, both yield consistent PSNR and LMD gains. Disabling MCC or the alternating scheme markedly degrades boundary sharpness and lip-sync fidelity (Jiang et al., 11 Jul 2025).

Several common misunderstandings are clarified by these results. First, the method is not described as relying only on audio-conditioned deformation; expression feature $R$2 is explicitly incorporated in the Face Branch. Second, the principal artifact class addressed is not limited to blur or jitter; local penetration and face-torso discontinuity are treated as central failure modes. Third, the reported real-time behavior does not eliminate per-identity optimization, since the training cost is still given as 0.6 h per identity.

Taken together, the reported evidence positions M2DAO-Talker as a framework in which precise 2D preprocessing, disentangled motion modeling, full-portrait consistency supervision, and staged optimization are mutually dependent design elements rather than interchangeable modules. The paper’s own summary is that these components jointly establish a new state of the art in high-fidelity, real-time talking-head generation (Jiang et al., 11 Jul 2025).