InterAnimate: Region-Aware Hand–Face Animation

• The paper introduces InterAnimate, a diffusion model that targets hand–face regions to animate natural and anatomically consistent interactive gestures.
• It employs a latent video diffusion architecture with region-specific injection, soft quantization, and an ID-preserving module to ensure realistic deformations.
• Quantitative results on the InterHF dataset show superior performance with FID=23.22 and FVD=122.90, establishing a new benchmark in interactive animation.

InterAnimate is a region-aware video diffusion model specifically designed for animating realistic human hand–face interactions. Addressing the gap in modeling complex interactive gestures in human animation, InterAnimate jointly learns spatio-temporal contact dynamics and biomechanically plausible deformation effects, facilitating natural interactions where hand motion induces anatomically realistic facial deformations while ensuring collision-free contact. The model was introduced alongside InterHF, a comprehensive dataset with 90,000 annotated hand–face interaction videos across 18 distinct interaction categories, establishing a new benchmark for both data and animation quality (Lin et al., 15 Apr 2025).

1. Large-Scale Interaction Data: InterHF

A cornerstone of the InterAnimate framework is the InterHF dataset, the first systematic and large-scale video corpus focused exclusively on hand–face interactions:

• Participants: 5,000 individuals, stratified across ten age brackets and balanced for gender (3,000 male, 2,000 female).
• Capture Protocol: Videos captured in eight diverse environments (combinations of natural/artificial, front/back-lit, indoor/outdoor, and across device types).
• Annotations: Each clip is annotated with 2D whole-body (DW-Pose) keypoints, from which binary masks for hands (M_h) and face (M_f) are computed.
• Interaction Taxonomy: 18 subclasses within four main gesture categories:
  • Pinching (e.g., nose pinch left/right)
  • Stroking (forehead, eyebrows, chin, ear)
  • Poking (cheek, with different numbers of fingers and approach angles)
  • Swiping (cross-face palm swipes)
• Scale: 90,000 videos (each ≈3 s), amounting to ~75 hours.
• Preprocessing: All sequences resized to 384×640 pixels; training samples consist of a single reference frame and a 16-frame driving clip, with latent encoding via a VAE.

This extensive data resource enables robust training and evaluation of region-aware interaction models and supports advances in biometric anti-spoofing and broader interactive motion modeling (Lin et al., 15 Apr 2025).

2. Model Architecture and Region-Aware Processing

InterAnimate adopts a latent video diffusion architecture grounded on a Stable Video Diffusion 1.1 backbone:

• Latent Encoding: Input video $x$ is encoded as $z=E_{VAE}(x)$. The diffusion process operates in this latent space with standard Markovian forward and reverse processes.
• Learnable Spatio-Temporal Latents:
  • Spatial Latents ($L^s \in \mathbb{R}^{n \times d}$): Encode hand/face topology and appearance in the contact region.
  • Temporal Latents ($L^t \in \mathbb{R}^{m \times d}$): Model temporal patterns reflecting action class and local deformation dynamics.
  • Implementation: $n = m = 512$, $d = 512$.
• Region-Aware Interaction Mechanism: A core innovation lies in the injection of interaction priors into the denoising U-Net:
  • Soft Quantization: For each latent feature, compute soft assignments to spatio-temporal latents using squared Euclidean distance with a temperature $\tau=1.0$, yielding quantized spatial ($\bar{V}^s$) and temporal ($\bar{V}^t$) features.
  • Cross-Attention and Mixing: Standard cross-attention blends these with original latent queries and the learned priors. Linear fusion with $\alpha=0.5$ combines spatial and temporal information.
  • Region Masking and Residual Injection: Output is masked and injected only within hand and face binary regions $z=E_{VAE}(x)$0, $z=E_{VAE}(x)$1.
  • ID Preserver: An ArcFace-derived identity embedding is injected into the face region through an additional cross-attention module to maintain identity consistency.

The following summarizes the interaction block sequence:

Stage	Operation	Region Scope
Soft Quantization	$z=E_{VAE}(x)$2	All features
Cross-Attention & Mixing	$z=E_{VAE}(x)$3, $z=E_{VAE}(x)$4	Spatial, Temporal
Region Masking + Injection	$z=E_{VAE}(x)$5, $z=E_{VAE}(x)$6	Hand, Face regions
ID Preserver	$z=E_{VAE}(x)$7	Face region

This mechanism ensures targeted injection of contact priors and identity cues where hand–face interaction physically occurs, differentiating InterAnimate from global-only or rigid modeling approaches.

3. Training Objectives and Supervision

InterAnimate’s training is driven by region-amplified losses and orthogonality-promoting regularization:

• Region-Weighted Diffusion Loss:

  $z=E_{VAE}(x)$8

  Where $z=E_{VAE}(x)$9 in the relevant mask ($L^s \in \mathbb{R}^{n \times d}$0, $L^s \in \mathbb{R}^{n \times d}$1), biasing the model to penalize errors in interactive regions.

• Orthogonality Loss:

  $L^s \in \mathbb{R}^{n \times d}$2

  With $L^s \in \mathbb{R}^{n \times d}$3, encouraging decorrelation of the learned latent priors.

• Overall Objective:

  $L^s \in \mathbb{R}^{n \times d}$4

• Ablations: Removing any region-aware injection, quantization, identity preservation, or loss weighting leads to significant degradation (e.g., FID increases from 23.22 to over 29, FVD to above 174), confirming the necessity of each module (Lin et al., 15 Apr 2025).

4. Quantitative and Qualitative Performance

InterAnimate demonstrates superior region-aware hand–face animation fidelity as evidenced by:

• Image-Level Metrics: FID=23.22, SSIM=0.785, PSNR=23.36, LPIPS=0.141, L1=$L^s \in \mathbb{R}^{n \times d}$5
• Video-Level Metrics: FID-VID=10.79, FVD=122.90
• Comparison Baselines: Outperforms prior works retrained on InterHF, including Disco, AnimateAnyone, MagicAnimate, MimicMotion, DisPose, and StableAnimator (e.g., StableAnimator: FID=24.95, FVD=176.20).

In qualitative studies, InterAnimate uniquely produces realistic deformations in complex interactive gestures such as nose pinches, cheek pokes, and cross-face swipes, showing anatomically consistent