---
title: 'InterAnimate: Region-Aware Hand–Face Animation'
url: https://www.emergentmind.com/topics/interanimate-region-aware-hand-face-animation
type: topic
---

# InterAnimate: Region-Aware Hand–Face Animation

InterAnimate is a region-aware video diffusion model specifically designed for animating realistic human hand–face interactions. Addressing the gap in modeling complex interactive gestures in human animation, InterAnimate jointly learns spatio-temporal contact dynamics and biomechanically plausible deformation effects, facilitating natural interactions where hand motion induces anatomically realistic facial deformations while ensuring collision-free contact. The model was introduced alongside InterHF, a comprehensive dataset with 90,000 annotated hand–face interaction videos across 18 distinct interaction categories, establishing a new benchmark for both data and animation quality [2504.10905].

## 1. Large-Scale Interaction Data: InterHF

A cornerstone of the InterAnimate framework is the InterHF dataset, the first systematic and large-scale video corpus focused exclusively on hand–face interactions:

- **Participants:** 5,000 individuals, stratified across ten age brackets and balanced for gender (3,000 male, 2,000 female).
- **Capture Protocol:** Videos captured in eight diverse environments (combinations of natural/artificial, front/back-lit, indoor/outdoor, and across device types).
- **Annotations:** Each clip is annotated with 2D whole-body (DW-Pose) keypoints, from which binary masks for hands (M_h) and face (M_f) are computed. 
- **Interaction Taxonomy:** 18 subclasses within four main gesture categories:
    - Pinching (e.g., nose pinch left/right)
    - Stroking (forehead, eyebrows, chin, ear)
    - Poking (cheek, with different numbers of fingers and approach angles)
    - Swiping (cross-face palm swipes)
- **Scale:** 90,000 videos (each ≈3 s), amounting to ~75 hours.
- **Preprocessing:** All sequences resized to 384×640 pixels; training samples consist of a single reference frame and a 16-frame driving clip, with latent encoding via a VAE.

This extensive data resource enables robust training and evaluation of region-aware interaction models and supports advances in biometric anti-spoofing and broader interactive motion modeling [2504.10905].

## 2. Model Architecture and Region-Aware Processing

InterAnimate adopts a latent video diffusion architecture grounded on a Stable Video Diffusion 1.1 backbone:

- **Latent Encoding:** Input video $x$ is encoded as $z=E_{VAE}(x)$. The diffusion process operates in this latent space with standard Markovian forward and reverse processes.
- **Learnable Spatio-Temporal Latents:**
    - **Spatial Latents ($L^s \in \mathbb{R}^{n \times d}$):** Encode hand/face topology and appearance in the contact region.
    - **Temporal Latents ($L^t \in \mathbb{R}^{m \times d}$):** Model temporal patterns reflecting action class and local deformation dynamics.
    - Implementation: $n = m = 512$, $d = 512$.
- **Region-Aware Interaction Mechanism:** A core innovation lies in the injection of interaction priors into the denoising U-Net:
    - **Soft Quantization:** For each latent feature, compute soft assignments to spatio-temporal latents using squared Euclidean distance with a temperature $\tau=1.0$, yielding quantized spatial ($\bar{V}^s$) and temporal ($\bar{V}^t$) features.
    - **Cross-Attention and Mixing:** Standard cross-attention blends these with original latent queries and the learned priors. Linear fusion with $\alpha=0.5$ combines spatial and temporal information.
    - **Region Masking and Residual Injection:** Output is masked and injected only within hand and face binary regions $M_h$, $M_f$.
    - **ID Preserver:** An ArcFace-derived identity embedding is injected into the face region through an additional cross-attention module to maintain identity consistency.

The following summarizes the interaction block sequence:

| Stage                     | Operation                                              | Region Scope         |
|---------------------------|-------------------------------------------------------|---------------------|
| Soft Quantization         | $\bar{v}_i = \sum_j w_{ij} L_j$                       | All features        |
| Cross-Attention & Mixing  | $\hat{V}^s, \hat{V}^t$, $V^{\mathrm{latent}}$         | Spatial, Temporal   |
| Region Masking + Injection| $V^{\mathrm{latent}}_m = V^{\mathrm{latent}} \odot M_h \odot M_f$, $V^o_{att} = V^i + V^{\mathrm{latent}}_m$ | Hand, Face regions  |
| ID Preserver              | $V^{\mathrm{face}} = \mathrm{CrossAttn}(V^i, A_{emb}, A_{emb})$ | Face region         |

This mechanism ensures targeted injection of contact priors and identity cues where hand–face interaction physically occurs, differentiating InterAnimate from global-only or rigid modeling approaches.

## 3. Training Objectives and Supervision

InterAnimate’s training is driven by region-amplified losses and orthogonality-promoting regularization:

- **Region-Weighted Diffusion Loss:**
    \[
    \mathcal{L}_\mathrm{diff} = \mathbb{E}_z \left[ \| z - \hat{z} \|_2^2 \cdot W_{\mathrm{hand}} \cdot W_{\mathrm{face}} \right]
    \]
    Where $W_r = \lambda_r$ in the relevant mask ($\lambda_{\mathrm{hand}}=5.0$, $\lambda_{\mathrm{face}}=2.0$), biasing the model to penalize errors in interactive regions.
- **Orthogonality Loss:**
    \[
    \mathcal{L}_\mathrm{ortho}^L = \mathrm{MSE}(S_{ij}, \delta_{ij}),\, (i \ne j); \quad \mathcal{L}_\mathrm{ortho} = \mathcal{L}_\mathrm{ortho}^{L^s} + \mathcal{L}_\mathrm{ortho}^{L^t}
    \]
    With $\beta=10^{-4}$, encouraging decorrelation of the learned latent priors.
- **Overall Objective:**
    \[
    \mathcal{L}_\mathrm{total} = \mathcal{L}_\mathrm{diff} + \beta \mathcal{L}_\mathrm{ortho}
    \]
- **Ablations:** Removing any region-aware injection, quantization, identity preservation, or loss weighting leads to significant degradation (e.g., FID increases from 23.22 to over 29, FVD to above 174), confirming the necessity of each module [2504.10905].

## 4. Quantitative and Qualitative Performance

InterAnimate demonstrates superior region-aware hand–face animation fidelity as evidenced by:

- **Image-Level Metrics:** FID=23.22, SSIM=0.785, PSNR=23.36, LPIPS=0.141, L1=$2.27 \times 10^{-5}$
- **Video-Level Metrics:** FID-VID=10.79, FVD=122.90
- **Comparison Baselines:** Outperforms prior works retrained on InterHF, including Disco, AnimateAnyone, MagicAnimate, MimicMotion, DisPose, and StableAnimator (e.g., StableAnimator: FID=24.95, FVD=176.20).

In qualitative studies, InterAnimate uniquely produces realistic deformations in complex interactive gestures such as nose pinches, cheek pokes, and cross-face swipes, showing anatomically consistent

Source: https://www.emergentmind.com/topics/interanimate-region-aware-hand-face-animation