---
title: 'X-NeMo: Diffusion Portrait Animation'
url: https://www.emergentmind.com/topics/x-nemo
type: topic
---

# X-NeMo: Diffusion Portrait Animation

X-NeMo (eXpressive Neural Motion reenactment via disentangled latent attention) is a zero-shot, diffusion-based portrait-animation architecture that synthesizes photorealistic face video from a single static portrait using the head poses and expressions from a driving video of potentially different identity. X-NeMo addresses core challenges in neural motion transfer, notably identity leakage and the inability of prior systems to capture extreme or subtle expressions. It achieves this through a cross-attention framework conditioned on an end-to-end learned, low-dimensional, identity-agnostic motion descriptor. This design paradigm eschews spatial-aligned structural motion guidance and instead injects driving motion via disentangled latent attention through the generative backbone, leading to improved preservation of reference identity and expressive fidelity [2507.23143].

## 1. System Architecture

X-NeMo comprises several interlinked modules:

1. **Static Portrait Encoder $\mathcal{R}$**  
   Implements the reference net of MasaCtrl, producing multi-scale feature maps $c_{ref} = \{c_{ref}^l\}_{l=1}^L$ from the reference image $I_R$. These maps, used as keys/values in the UNet’s self-attention layers, encode appearance and style information robustly.

2. **Latent Motion Encoder $E_{mot}$**  
   Receives an augmented driving frame $I_{D_i}$ and emits a 1-D vector $f_{mot} \in \mathbb{R}^{512}$ capturing facial motion while discarding positional and high-frequency identity information.

3. **Relative Translation-Scaling Triplet $f_{rts}$**  
   Extracts and encodes face center displacement $(\Delta x, \Delta y)$ and scale ratio ($s_d/s_r$) via an MLP, concatenated to $f_{mot}$ to encapsulate pose changes.

4. **Diffusion Backbone $\mathcal{D}$**  
   Utilizes a pretrained Latent Diffusion UNet (e.g., Stable Diffusion v1.5) that receives conditioning from $c_{ref}$ (mutual/self-attention) and $f_{mot}$ (cross-attention), with temporal transformers for video coherence.

**Motion is injected via cross-attention:**  
For spatial feature map $X$ at each transformer layer, cross-motion attention is defined as
\[
Q = W_q X, \quad K = W_k f_{mot}, \quad V = W_v f_{mot}, \quad \mathrm{Attn}(Q,K,V) = \mathrm{Softmax}\left(\frac{Q K^\top}{\sqrt{d}}\right)V,
\]
where $W_q, W_k, W_v$ are learnable projections and $d$ is the attention dimension. This diverts from the ControlNet approach of using spatial-aligned motion controls.

## 2. Training Framework and Losses

**Denoising-Diffusion Loss:**  
The model employs latent diffusion modeling,
\[
\mathcal{L}_{ldm} = \mathbb{E}_{z_t,\epsilon\sim\mathcal{N}(0,I),t}\left[\|\epsilon - \epsilon_\theta(z_t, c_{ref}, f_{mot})\|_2^2\right],
\]
where the UNet predicts the noise $\epsilon$ conditioned on both appearance and motion guidance.

**Dual-Head GAN Supervision:**  
A StyleGAN-2 decoder reconstructs $I_{D}$ from the appearance embedding $f_{app} = \mathrm{ResNet50}(I_R)$ and motion embedding $f_{mot}$, with the following losses:
- $L_1$ Reconstruction: $\mathcal{L}_{recon} = \| I_D - I_{R\to D} \|_1$
- VGG perceptual: $\mathcal{L}_{vgg}$, $\mathcal{L}_{vggf}$
- Adversarial and feature-matching: $\mathcal{L}_{adv}$, $\mathcal{L}_{fm}$

Overall,
\[
\mathcal{L}_{gan} = \mathcal{L}_{adv}
+ \lambda_r \mathcal{L}_{recon}
+ \lambda_{vgg}\mathcal{L}_{vgg}
+ \lambda_{vggf}\mathcal{L}_{vggf}
+ \lambda_{fm}\mathcal{L}_{fm},
\]
with $\{\lambda_r, \lambda_{vgg}, \lambda_{vggf}, \lambda_{fm}\} = \{1.0, 3\times10^{-2}, 6\times10^{-3}, 10\}$.

**Classifier-Free Guidance (CFG):**
CFG is applied to interpolate between pure appearance and motion guidance at inference,
\[
\tilde\epsilon_\theta = (1+w)\epsilon_\theta(z_t, c_{ref}, f_{mot}) - w\epsilon_\theta(z_t, \emptyset, f_{ref\_mot}), \quad w=3.5,
\]
where $f_{ref\_mot} = E_{mot}(I_R)$.

## 3. Motion-Identity Disentanglement

Disentanglement is accomplished by:

- **Targeted Augmentations:** Random color jitter, scaling ($\pm30\%$), piecewise affine warps, and face-centered cropping are applied to $I_D$ before encoding. These disrupt correlated appearance and spatial structure while keeping motion cues.

- **Low-Dimensional Bottleneck:** $f_{mot}$ is restricted to a 1D vector of length $d=512$, aggressively filtering static or background attributes.

- **Reference-Feature Masking (RFM):** $30\%$ of $c_{ref}$ tokens are randomly masked before self-attention to prevent the network from copying motion cues when $I_R$ and $I_D$ show similar expressions.

Collectively, these lead to a motion descriptor with significantly reduced identity and structure leakage.

## 4. Training and Implementation Protocols

X-NeMo is trained on HDTF and VFHQ (talking head) as well as NerSemble (expressive facial) datasets, using 25 fps, $512\times512$ crops. The procedure and hyperparameters are:

- Pretrain $\mathcal{D}+\mathcal{R}$ (appearance modeling) from Stable Diffusion.
- Joint training of $E_{mot}$, cross-attention layers, and GAN head on $\mathcal{L}_{ldm}+\mathcal{L}_{gan}$.
- Fine-tune temporal transformers for 24-frame sequences.
- AdamW optimizer with learning rate $1\times 10^{-5}$.
- Batch sizes: 64 (static appearance+motion), 16 (temporal clips).
- Motion latent dimensions assessed via ablation (128/512/1024).
- Inference with 25 DDIM steps.

## 5. Evaluation Benchmarks

**Self-Reenactment:**  
For $256 \times 256$ self-reenactment, metrics are L1 (↓), SSIM (↑), and LPIPS (↓):

| Method        | L1    | SSIM  | LPIPS  |
|-------------- |------ |-------|--------|
| PD-FGC        | 0.085 | 0.728 | 0.291  |
| LivePortrait  | 0.074 | 0.770 | 0.236  |
| X-Portrait    | 0.063 | 0.793 | 0.209  |
| FYE           | 0.075 | 0.741 | 0.249  |
| AniPortrait   | 0.057 | 0.812 | 0.198  |
| X-NeMo        | 0.055 | 0.826 | 0.168  |

**Cross-Reenactment (Zero-Shot):**
ID-SIM (ArcFace cosine ↑), AED/APD (L1 blendshape/head-pose diff ↓), EMO-SIM ((CCC+Pearson) valence/arousal ↑):

| Method        | ID-SIM | AED/APD   | EMO-SIM |
|---------------|--------|-----------|---------|
| PD-FGC        | 0.604  | 0.045/3.95| 0.49    |
| LivePortrait  | 0.702  | 0.055/6.61| 0.48    |
| X-Portrait    | 0.695  | 0.041/4.07| 0.52    |
| FYE           | 0.725  | 0.062/4.49| 0.41    |
| AniPortrait   | 0.713  | 0.043/4.14| 0.46    |
| X-NeMo        | 0.787  | 0.039/3.42| 0.65    |

X-NeMo achieves the leading scores, especially in identity similarity, expression transfer accuracy, and perceptual metrics. Qualitative analysis confirms effective handling of large structural variations, subtle, and extreme facial actions.

## 6. Ablation Analysis and Component Significance

Component importance is assessed by ablating individual architectural choices:

| Component removed      | ID-SIM | AED/APD   | EMO-SIM |
|-----------------------|--------|-----------|---------|
| w/o GAN head          | 0.789  | 0.045/4.64| 0.43    |
| w/o end-to-end        | 0.782  | 0.040/3.49| 0.52    |
| w/o RFM               | 0.791  | 0.039/3.41| 0.62    |
| w/o augmentations     | 0.724  | 0.042/3.63| 0.50    |
| w/o cross-attn        | 0.697  | 0.040/3.55| 0.48    |
| Full X-NeMo           | 0.787  | 0.039/3.42| 0.65    |

Results indicate that the GAN head is crucial for expression accuracy (EMO-SIM), end-to-end training boosts identity consistency, and RFM and augmentations jointly suppress identity leakage. The transition from spatial to cross-attention-based control is critical; ControlNet-style spatial maps severely degrade ID preservation and expression transfer.

## 7. Summary and Implications

X-NeMo demonstrates a synthesis of architectural innovations: a bottlenecked latent motion encoder, motion injection via learned cross-attention, and rigorous disentanglement through augmentations and dual-branch GAN supervision. These features collectively enable state-of-the-art, zero-shot, expressive portrait animation with minimal identity drift or motion ambiguity, validated across multiple quantitative benchmarks and ablation protocols [2507.23143]. A plausible implication is that decoupling motion injection from spatial alignment paves the way for generalizable facial motion transfer with strong robustness to structural identity mismatches and challenging expressions.

Source: https://www.emergentmind.com/topics/x-nemo