X-NeMo: Diffusion Portrait Animation

• X-NeMo is a zero-shot, diffusion-based portrait-animation architecture that transfers driving facial motion while preserving the reference identity.
• It employs a cross-attention framework conditioning on a low-dimensional motion descriptor to bypass spatial alignment and capture extreme expressions.
• Benchmark results indicate that X-NeMo outperforms prior methods in identity similarity, expression accuracy, and perceptual quality across various tests.

X-NeMo (eXpressive Neural Motion reenactment via disentangled latent attention) is a zero-shot, diffusion-based portrait-animation architecture that synthesizes photorealistic face video from a single static portrait using the head poses and expressions from a driving video of potentially different identity. X-NeMo addresses core challenges in neural motion transfer, notably identity leakage and the inability of prior systems to capture extreme or subtle expressions. It achieves this through a cross-attention framework conditioned on an end-to-end learned, low-dimensional, identity-agnostic motion descriptor. This design paradigm eschews spatial-aligned structural motion guidance and instead injects driving motion via disentangled latent attention through the generative backbone, leading to improved preservation of reference identity and expressive fidelity (Zhao et al., 30 Jul 2025).

1. System Architecture

X-NeMo comprises several interlinked modules:

1. Static Portrait Encoder $\mathcal{R}$ Implements the reference net of MasaCtrl, producing multi-scale feature maps $c_{ref} = \{c_{ref}^l\}_{l=1}^L$ from the reference image $I_R$. These maps, used as keys/values in the UNet’s self-attention layers, encode appearance and style information robustly.
2. Latent Motion Encoder $E_{mot}$ Receives an augmented driving frame $I_{D_i}$ and emits a 1-D vector $f_{mot} \in \mathbb{R}^{512}$ capturing facial motion while discarding positional and high-frequency identity information.
3. Relative Translation-Scaling Triplet $f_{rts}$ Extracts and encodes face center displacement $(\Delta x, \Delta y)$ and scale ratio ($s_d/s_r$) via an MLP, concatenated to $f_{mot}$ to encapsulate pose changes.
4. Diffusion Backbone $c_{ref} = \{c_{ref}^l\}_{l=1}^L$0 Utilizes a pretrained Latent Diffusion UNet (e.g., Stable Diffusion v1.5) that receives conditioning from $c_{ref} = \{c_{ref}^l\}_{l=1}^L$1 (mutual/self-attention) and $c_{ref} = \{c_{ref}^l\}_{l=1}^L$2 (cross-attention), with temporal transformers for video coherence.

Motion is injected via cross-attention:

For spatial feature map $c_{ref} = \{c_{ref}^l\}_{l=1}^L$3 at each transformer layer, cross-motion attention is defined as

$c_{ref} = \{c_{ref}^l\}_{l=1}^L$4

where $c_{ref} = \{c_{ref}^l\}_{l=1}^L$5 are learnable projections and $c_{ref} = \{c_{ref}^l\}_{l=1}^L$6 is the attention dimension. This diverts from the ControlNet approach of using spatial-aligned motion controls.

2. Training Framework and Losses

Denoising-Diffusion Loss:

The model employs latent diffusion modeling,

$c_{ref} = \{c_{ref}^l\}_{l=1}^L$7

where the UNet predicts the noise $c_{ref} = \{c_{ref}^l\}_{l=1}^L$8 conditioned on both appearance and motion guidance.

Dual-Head GAN Supervision:

A StyleGAN-2 decoder reconstructs $c_{ref} = \{c_{ref}^l\}_{l=1}^L$9 from the appearance embedding $I_R$0 and motion embedding $I_R$1, with the following losses:

• $I_R$2 Reconstruction: $I_R$3
• VGG perceptual: $I_R$4, $I_R$5
• Adversarial and feature-matching: $I_R$6, $I_R$7

Overall,

$I_R$8

with $I_R$9.

Classifier-Free Guidance (CFG):

CFG is applied to interpolate between pure appearance and motion guidance at inference,

$E_{mot}$0

where $E_{mot}$1.

3. Motion-Identity Disentanglement

Disentanglement is accomplished by:

• Targeted Augmentations: Random color jitter, scaling ($E_{mot}$2), piecewise affine warps, and face-centered cropping are applied to $E_{mot}$3 before encoding. These disrupt correlated appearance and spatial structure while keeping motion cues.
• Low-Dimensional Bottleneck: $E_{mot}$4 is restricted to a 1D vector of length $E_{mot}$5, aggressively filtering static or background attributes.
• Reference-Feature Masking (RFM): $E_{mot}$6 of $E_{mot}$7 tokens are randomly masked before self-attention to prevent the network from copying motion cues when $E_{mot}$8 and $E_{mot}$9 show similar expressions.

Collectively, these lead to a motion descriptor with significantly reduced identity and structure leakage.

4. Training and Implementation Protocols

X-NeMo is trained on HDTF and VFHQ (talking head) as well as NerSemble (expressive facial) datasets, using 25 fps, $I_{D_i}$0 crops. The procedure and hyperparameters are:

• Pretrain $I_{D_i}$1 (appearance modeling) from Stable Diffusion.
• Joint training of $I_{D_i}$2, cross-attention layers, and GAN head on $I_{D_i}$3.
• Fine-tune temporal transformers for 24-frame sequences.
• AdamW optimizer with learning rate $I_{D_i}$4.
• Batch sizes: 64 (static appearance+motion), 16 (temporal clips).
• Motion latent dimensions assessed via ablation (128/512/1024).
• Inference with 25 DDIM steps.

5. Evaluation Benchmarks

Self-Reenactment:

For $I_{D_i}$5 self-reenactment, metrics are L1 (↓), SSIM (↑), and LPIPS (↓):

Method	L1	SSIM	LPIPS
PD-FGC	0.085	0.728	0.291
LivePortrait	0.074	0.770	0.236
X-Portrait	0.063	0.793	0.209
FYE	0.075	0.741	0.249
AniPortrait	0.057	0.812	0.198
X-NeMo	0.055	0.826	0.168

Cross-Reenactment (Zero-Shot):

ID-SIM (ArcFace cosine ↑), AED/APD (L1 blendshape/head-pose diff ↓), EMO-SIM ((CCC+Pearson) valence/arousal ↑):

Method	ID-SIM	AED/APD	EMO-SIM
PD-FGC	0.604	0.045/3.95	0.49
LivePortrait	0.702	0.055/6.61	0.48
X-Portrait	0.695	0.041/4.07	0.52
FYE	0.725	0.062/4.49	0.41
AniPortrait	0.713	0.043/4.14	0.46
X-NeMo	0.787	0.039/3.42	0.65

X-NeMo achieves the leading scores, especially in identity similarity, expression transfer accuracy, and perceptual metrics. Qualitative analysis confirms effective handling of large structural variations, subtle, and extreme facial actions.

6. Ablation Analysis and Component Significance

Component importance is assessed by ablating individual architectural choices:

Component removed	ID-SIM	AED/APD	EMO-SIM
w/o GAN head	0.789	0.045/4.64	0.43
w/o end-to-end	0.782	0.040/3.49	0.52
w/o RFM	0.791	0.039/3.41	0.62
w/o augmentations	0.724	0.042/3.63	0.50
w/o cross-attn	0.697	0.040/3.55	0.48
Full X-NeMo	0.787	0.039/3.42	0.65

Results indicate that the GAN head is crucial for expression accuracy (EMO-SIM), end-to-end training boosts identity consistency, and RFM and augmentations jointly suppress identity leakage. The transition from spatial to cross-attention-based control is critical; ControlNet-style spatial maps severely degrade ID preservation and expression transfer.

7. Summary and Implications

X-NeMo demonstrates an overview of architectural innovations: a bottlenecked latent motion encoder, motion injection via learned cross-attention, and rigorous disentanglement through augmentations and dual-branch GAN supervision. These features collectively enable state-of-the-art, zero-shot, expressive portrait animation with minimal identity drift or motion ambiguity, validated across multiple quantitative benchmarks and ablation protocols (Zhao et al., 30 Jul 2025). A plausible implication is that decoupling motion injection from spatial alignment paves the way for generalizable facial motion transfer with strong robustness to structural identity mismatches and challenging expressions.