Motion- and Expression-Aware Reconstruction Loss

• MEAR Loss is a dynamic, per-pixel reweighting scheme that fuses motion and landmark maps to emphasize critical facial regions in video processing.
• It enhances the reconstruction of facial identity, pose, and expression by adaptively weighting pixels based on motion cues and landmark detections.
• Empirical evaluations show improved identity similarity and reduced pose and expression errors compared to standard diffusion loss methods.

Motion- and Expression-Aware Reconstruction (MEAR) loss is a dynamic, per-pixel reweighting scheme for diffusion model objectives in video head swapping. Developed in the context of mask-free, direct head swapping with the DirectSwap framework, MEAR loss exploits temporal and facial expression saliency to enhance the fidelity and temporal coherence of facial identity, pose, and articulation across frames. It induces an inductive bias for the generative diffusion process to focus on dynamically changing regions and critical expression loci—namely eyes, mouth, and related facial features—by deriving spatiotemporal attention maps from unsupervised motion cues and facial landmark detection (Wang et al., 10 Dec 2025).

1. Motivation and Rationale

In conventional video diffusion or denoising schemes, the pixelwise mean squared error (MSE) or $L_2$ loss is spatially uniform, assigning equal supervision to all pixels regardless of semantic or temporal relevance. However, talking-head videos are characterized by spatial heterogeneity: a small subset of regions (e.g., eyes and mouth) encode the semantic core of expressions and speech, while only certain areas—such as hair strands or turning head boundaries—manifest strong motion. Uniform losses therefore dilute optimization power, leading to less reliable recovery of high-frequency expression, mouth shapes, and subtle head motions, particularly in the presence of occlusions or background distractors.

MEAR loss addresses this gap by constructing a composite saliency prior per frame that fuses a motion heatmap (from frame differences) and a soft landmark attention map (from facial landmark detections), and applying these as pixelwise multipliers for the MSE term during diffusion model training. This design amplifies supervision on pixels that are most likely to influence temporal smoothness, expression consistency, and motion fidelity.

2. Mathematical Definition

Let $x_0 = \{x_{0t}\}_{t=1}^F$ be a clean video clip with $F$ frames, each $x_{0t} \in \mathbb{R}^{3 \times H \times W}$. For each frame (except the last), MEAR loss modifies the basic per-pixel diffusion denoising objective. For model noise prediction $\epsilon_\theta(x_t, c)$ and ground-truth noise $\epsilon$:

$$\mathcal{L}_{\mathrm{mse}} = \mathbb{E}_{x_0, \epsilon, t} \big[ \| \epsilon - \epsilon_\theta(x_t, c) \|_2^2 \big]$$

is replaced by

$$\mathcal{L}_{\mathrm{mear}} = \mathbb{E}_{x_0, \epsilon, t} \left[ \sum_{i, j} A_t(i, j) \cdot (\epsilon(i, j) - \epsilon_\theta(x_t, c)(i, j))^2 \right]$$

where $A_t \in \mathbb{R}^{H \times W}$ is the adaptive attention map for frame $t$, constructed by

1. Motion map $D_t$:

$G_t = \text{Grayscale}(x_{0t})$; $$D_t^{\mathrm{raw}}(i,j) = |G_{t+1}(i,j) - G_t(i,j)|$$ After morphological dilation (kernel $k_{\mathrm{motion}}$) and normalization:

$$D_t = \frac{ \text{Dilate}(D_t^{\mathrm{raw}}, k_{\mathrm{motion}}) }{ \max_{i,j} \text{Dilate}(D_t^{\mathrm{raw}}, k_{\mathrm{motion}}) }$$

2. Landmark map $L_t$:

Landmarks $S_t = \{ (x_k, y_k) \}$ (PIPNet, excluding jaw boundary) are rasterized: $M_t(i,j) = 1$ iff $(i,j)$ is within radius 2 pixels of some $(x_k,y_k)$. The map $$L_t = \frac{ \text{GaussianBlur}(M_t, \sigma_{\mathrm{landmark}}) }{ \max_{i,j} \text{GaussianBlur}(M_t, \sigma_{\mathrm{landmark}}) }$$, with $\sigma_{\mathrm{landmark}} \approx 3$.

3. Fusion: For scalar $\alpha > 0$ (default 0.7),

$$A_t(i, j) = D_t(i, j) + \alpha \cdot L_t(i, j) \cdot (1 - D_t(i, j))$$

This ensures that in regions of noticeable motion ($D_t \approx 1$), MEAR gives high loss weight regardless of expression saliency, while in static zones ($D_t \approx 0$), landmark-attached pixels gain weight proportional to $\alpha$.

3. Implementation Workflow

The MEAR mechanism is integrated into the DirectSwap training pipeline as follows:

• Sample video clips of fixed temporal length ($F=8$).
• For frames $t=1$ to $F-1$, compute $A_t$ on clean $x_0$ prior to the forward diffusion pass.
• Downsample $A_t$ to network resolution if model operates on latent feature grids.
• During per-pixel loss calculation, use elementwise multiplication of $A_t$ and the squared error.
• Normalize $A_t$ such that $\frac{1}{HW} \sum_{i,j} A_t(i,j) = 1.0$ to match loss scales with vanilla MSE objectives.
• Typical hyperparameters: $k_{\mathrm{motion}} = 3 \times 3$ dilation, $\sigma_{\mathrm{landmark}} = 3$, $\alpha=0.7$, batch size 2, $F=8$, $\sim 20$K iterations on H200-class GPU hardware.
• The MEAR loss is combined additively with standard diffusion loss without requiring auxiliary loss terms or adversarial penalties.

The approach is fully differentiable and lightweight, introducing negligible computational overhead. No additional norm-based loss scaling is needed beyond the per-map normalization.

4. Empirical Evidence and Quantitative Results

Comprehensive ablation on HeadSwapBench (cross-identity paired video test set) shows MEAR loss yields improvements over several baselines and partial variants. Key metrics:

Method	Sim_ID (↑)	Pose MAE (↓)	Expr. NME (↓)
Baseline	0.871	2.182	0.082
w/o motion	0.867	2.098	0.077
w/o expression	0.873	2.019	0.080
Full MEAR	0.880	1.859	0.075

Full MEAR delivers the best overall performance: highest identity similarity, lowest pose and expression errors. Qualitative analysis (Figure 1, (Wang et al., 10 Dec 2025)) evidences smoother and more accurate mouth and eye trajectories, with a marked reduction in flicker in both facial and hair regions.

A plausible implication is that temporal coherence and the perceptual quality of expression transfer in diffusion-based video editing pipelines can be substantially improved through adaptive, saliency-informed pixel weighting—without explicit temporal consistency loss terms or sequence-level discriminators.

5. Practical Guidelines and Extensions

• To generalize MEAR for object classes other than faces (e.g., hands or full bodies), the landmark map $L_t$ may be substituted with analogous keypoint heatmaps relevant to the target semantics.
• For scenarios with substantial camera movement or illumination shifts, robustness of $D_t$ can be improved by computing optical flow and warping $G_{t+1}$ prior to differencing against $G_t$.
• $\alpha$ offers a tradeoff: higher values emphasize static facial detail (expressivity), while lower values focus on aggregate motion cues.
• Always normalize $A_t$ to keep average loss magnitude consistent and prevent biasing optimization.
• Small kernel sizes for dilation ($k_{\mathrm{motion}}$) and moderate width for Gaussian smoothing ($\sigma_{\mathrm{landmark}}$) distribute supervision across both edge-near and fine expression regions.
• MEAR can also be applied to other per-pixel loss functions (e.g. LPIPS), by analogously reweighting them via $A_t$.
• The MEAR weighting is compatible with all video diffusion architectures where a spatial attention prior is desirable.

6. Context, Relation to Alternative Losses, and Significance

MEAR loss is situated among a taxonomy of video and expression-aware supervision strategies. Prior methods in related 3D talking-head or expression reconstruction domains have deployed perceptual losses derived from pre-trained lipreading networks ("lipread loss"; (Filntisis et al., 2022)), pairwise landmark distance constraints, or emotion perception networks to address mouth articulation and expression realism. However, MEAR’s key distinction is its unsupervised, differentiable fusion of motion and landmark saliency—operationalized without requiring audio, manual annotation, or complex sequence-level loss curves.

Whereas perceptual or lipread losses explicitly aim for semantic or phonetic matching (e.g., reducing Character Error Rate or Viseme Error Rate as in (Filntisis et al., 2022)), MEAR directly biases the generative process towards spatial-temporal loci of likely frame disagreement and facial action. This not only yields superior temporal smoothness and semantic alignment (as measured by expression NME and pose MAE), but can be flexibly ported to alternative video-generative tasks where unsupervised attention priors are effective.

7. Limitations and Prospective Extensions

While MEAR loss requires only basic landmark and grayscale operations, its efficacy hinges on the quality of motion estimation and landmark localization. Scenarios with significant occlusion, extreme viewpoint variation, or nonfacial deformation may degrade the quality of the $D_t$ and $L_t$ maps. Extensions might consider robustifying the motion map via learned optical flow, or introducing multi-modal attention features (combining audio cues as in lipread supervision (Filntisis et al., 2022)). The balance parameter $\alpha$ offers tunable control but must be chosen to avoid focus collapse onto either exclusively dynamic or static regions.

Overall, the MEAR loss framework is a salient instance of spatiotemporally aware, differentiable supervision for generative video models, offering a principled mechanism to bridge the gap between naïve pixelwise losses and highly engineered perceptual or adversarial objectives (Wang et al., 10 Dec 2025).