Implicit Motion Transformation

• Implicit Motion Transformation (IMT) is a set of methods that implicitly represents and manipulates motion at the feature or latent level, bypassing explicit pixel-wise displacement.
• It offers robust, end-to-end solutions for video tasks like generative human video coding, frame interpolation, and super-resolution by integrating motion estimation directly with synthesis.
• IMT pipelines use feature extraction, transformer-based attention, and dynamic filter networks to overcome limitations of traditional warping, achieving higher visual fidelity in complex scenes.

Implicit Motion Transformation (IMT) refers to a family of methods that learn to represent and manipulate motion implicitly—at the feature, token, or latent level—rather than via explicit, pixel-wise displacement fields such as optical flow. IMT provides robust and compact mechanisms for propagating and synthesizing temporal dynamics in video tasks, especially where explicit motion modeling is brittle or hard to supervise. IMT paradigms have demonstrated significant advances in generative human video coding, frame interpolation, video super-resolution, character animation, and multi-object motion transfer, effectively addressing complex, large-scale, or compositional motion patterns.

1. Core Principles and Motivation

Traditional explicit motion guidance relies on estimating and applying pixel-wise or part-wise transformations (e.g., dense optical flow, keypoints, transformation grids) for warping or reconstructing frames. While effective in constrained settings (such as facial videos), explicit methods are prone to artifacts under large, non-rigid, or multi-object motion due to imperfect flow estimation, warping errors, and difficulty handling occlusions or rapid articulations. By contrast, IMT replaces explicit, spatially-local displacement fields with learned, global or semi-global feature-level transformations.

IMT typically operates by:

• Encoding input videos (or frames) into compact feature representations or tokenized embeddings.
• Using neural architectures (e.g., transformers, dynamic filter networks, coordinate-based MLPs) to infer motion "intention" or transformation implicitly from these features.
• Synthesizing or reconstructing video frames based on motion-aware latent features, without intermediate explicit warping or pose representations.

This shift enables:

• Robustness to spatially complex, high-dimensional, or non-rigid motions.
• Tighter integration between motion estimation and downstream synthesis or compression tasks.
• More efficient, end-to-end trainable pipelines that generalize across identity and varying object arrangements.

IMT's success has been demonstrated across tasks such as Generative Human Video Coding (GHVC) (Chen et al., 12 Jun 2025), Video Frame Interpolation (VFI) (Guo et al., 2024), Video Super-Resolution (VSR) (Liu et al., 2020), Identity-Decoupled Character Animation (Xu et al., 7 Feb 2026), and Multi-Object Motion Transfer (Li et al., 1 Mar 2026).

2. Mathematical Formulations and Algorithmic Models

IMT instantiations vary by task, but share the characteristic of learning motion transformations indirectly through neural networks. Key representative models are summarized in the table:

Domain	Input Representation	IMT Mechanism	Output
GHVC (Chen et al., 12 Jun 2025)	Compressed U-Net features	Cross-attention transformer fusion	GAN-synthesized frames
VFI (Guo et al., 2024)	Pre-trained flow, CNN latent	SIREN coordinate-based MLP with latent	Bilateral interpolated flows
VSR (Liu et al., 2020)	Stacked LR frames	Dynamic Local Filter Network (LC layers)	Compensated SR frames
Animation (Xu et al., 7 Feb 2026)	1D motion tokens (VQ)	Transformer + mask bottleneck	Decoupled motion tokens
Motion Transfer (Li et al., 1 Mar 2026)	Motion tokens per object	Masked transformer attention (MDMA)	Multi-object motion control

Example: Generative Human Video Coding (Chen et al., 12 Jun 2025)

• Extract a highly compressed feature map $\theta_{\rm comp}$ for each frame via a downsampled U-Net with conv+GDN bottleneck:

$$\theta_{\rm comp} = \varrho_{\rm (conv,GDN)} \bigl(f_{\rm U\text{-}Net}(\phi(X,s))\bigr).$$

• At the decoder, upsample features and fuse appearance and motion using a cross-attention transformer:

$$f^{I_l}_Q, f^{I_l}_K, f^{I_l}_V = \rho_Q(f^{I_l}_{\rm motion}), \rho_K(f_{\rm app} \parallel f^{\hat K}_{\rm motion}), \rho_V(f_{\rm app} \parallel f^{\hat K}_{\rm motion})$$

$$f^{I_l}_{\rm atten} = \operatorname{Softmax}(f^{I_l}_Q (f^{I_l}_K)^T) f^{I_l}_V + f^{I_l}_{\rm motion}$$

$$f^{I_l}_{\rm trans} = \tau(\operatorname{LN}(f^{I_l}_{\rm atten}), \operatorname{LN}(f^{I_l}_{\rm motion}))$$

• Synthesize reconstructed frame using a GAN generator:

$\hat I_l = G_{\rm frame}(f^{I_l}_{\rm trans})$

Example: Video Frame Interpolation (Guo et al., 2024)

• Predict flow at arbitrary time $t$ with coordinate-based SIREN MLP:

$\hat V_t(x,y) = f(x,y,t; L_t(x,y))$

where $L_t$ is a locally extracted motion latent, and reconstruction/enhancement follows warping and blending steps.

Example: Video Super-Resolution (Liu et al., 2020)

• Dynamic local filters $\Theta_{(i,j),l}$ are generated per pixel and channel location, encoding motion implicitly in their weights:

$$\hat Y_{l}^{(i,j)} = \sum_{u=-d}^d \sum_{v=-d}^d \sum_{k=1}^C \Theta_{(i,j,l)}^{(u+d,v+d,k)} Y_{t-T+k-1}^{(i+u,j+v)}$$

No explicit flow or warping is required, and filter generation is conditioned on the entire frame stack.

3. Architectures and Network Modules

IMT pipelines employ diverse architectural modules depending on context:

• Feature Extraction: U-Net encoders, CNN stacks, linear patchification, vector quantizers. Feature bottlenecking (e.g., $6\times6\times1$ maps or $N_\mathrm{motion}$ tokens) enables bitrate reduction or abstraction.
• Transformation Modules: Cross-attention transformers (GHVC), coordinate-based MLPs (VFI), dynamic locally-connected filters (VSR), Vision Transformers with token bottlenecks (animation), and mask-structured attention (multi-object transfer).
• Decoding/Synthesis: GANs, frame-refinement CNNs, VAE decoders, and diffusion-transformer backbones reconstruct video frames or sequences based on transformed feature representations.

An illustrative summary for generative coding (Chen et al., 12 Jun 2025):

Module (Side)	Key Operations	Output
Encoder	U-Net encoder, conv+GDN, quantization, entropy coding	Compact features
Decoder	VVC decoding, convolutional appearance feature extraction, upsampling, cross-attention, GAN	Reconstructed frames

For multi-object motion transfer (Li et al., 1 Mar 2026), the pipeline involves segment-specific motion tokens, mask attention constraints, and progressive mask propagation to maintain object-wise disentanglement.

4. Losses, Training Strategies, and Optimization

IMT-based methods are trained with tailored loss functions coordinating motion, appearance, and synthesis quality. Key formulations include:

• Perceptual, Adversarial, Texture Losses: For high-fidelity synthesis, e.g., in GHVC:

$$\mathcal L_{\rm total} = \lambda_{\rm per} \mathcal L_{\rm per} + \lambda_{\rm adv} \mathcal L_{\rm adv} + \lambda_{\rm tex} \mathcal L_{\rm tex}$$

with $\mathcal L_{\rm per}$ (VGG-based), $\mathcal L_{\rm adv}$ (PatchGAN), and $\mathcal L_{\rm tex}$ (texture fidelity).

• Motion-Specific Reconstruction Losses: For motion token/latent learning,

$$L_{\rm motion} = \frac{1}{T} \sum_{t=1}^T \| H_{\rm motion, t} - H_{\rm gt, t} \|_2^2\quad\text{[2602.07498]}$$

for keypoint-based supervision.

• Diffusion/Noise Prediction Losses: For token-conditional video synthesis and motion transfer,

$$\mathcal L_{\rm diff} = \mathbb E_{t,\epsilon,y,\mathbf{I}_0} \| \epsilon_\theta(\mathbf{z}_t, y, t, \mathbf{I}_0, T_m) - \epsilon \|_2^2\quad\text{[2603.01000]}$$

• Coordinate/Flow Regression Losses: In VFI, e.g., mean squared error on (normalized) flow predictions (Guo et al., 2024).

Training strategies include:

• Pretraining motion encoder/quantizer on motion-only tasks.
• Mask-token bottlenecks or mask attention to enforce separation between identity and motion signal.
• Multi-stage or joint curriculum (motion, retargeting, then full synthesis).
• Ablative validation to confirm contribution of implicit modules.

5. Comparative Evaluation and Empirical Performance

IMT methods provide strong empirical performance versus explicit motion or warping-based approaches:

• GHVC (Chen et al., 12 Jun 2025): IMT yields BD-rate gains up to $-70.56\%$ relative to explicit motion methods across Rate-LPIPS, Rate-DISTS, and Rate-FVD. Visual fidelity surpasses explicit approaches, especially in clothing topology and pose articulation.
• VFI (Guo et al., 2024): GIMM (implicit) outperforms linear and flow-based VFI on Vimeo-Triplet and Vimeo-Septuplet, delivering higher PSNR and lower EPE, particularly in multi-frame, nonlinear scenarios.
• VSR (Liu et al., 2020): Dynamic local filter IMT surpasses state-of-the-art DUF-52L on Vid4 ($+0.13$ dB PSNR), with marked reduction in flicker and improved sharpness.
• Animation (Xu et al., 7 Feb 2026): IM-Animation matches or exceeds explicit skeleton/DensePose baselines in cross-ID and self-reenactment, with superior artifact avoidance in large pose mismatches.
• Multi-Object Transfer (Li et al., 1 Mar 2026): FlexiMMT supports accurate per-object motion assignment and achieves robust compositional video generation not possible with previous single-motion pipelines.

A representative comparison table:

Task	Explicit Baseline	IMT Variant	Primary Metric Gain
GHVC	CFTE, MTTF	IMT (cross-attn)	BD-rate $-70.56\%$ LPIPS (Chen et al., 12 Jun 2025)
VFI	Linear/End-to-End	GIMM	+2.53 dB PSNR, -0.10 EPE (Guo et al., 2024)
VSR	DUF-52L	DLFN+GRN	$+0.13$ dB PSNR (Liu et al., 2020)
Animation	Wan-Animate	IM-Animation	Lower FID/LPIPS (Xu et al., 7 Feb 2026)
Motion Trans	n/a (single-object)	FlexiMMT	Multi-motion compositionality (Li et al., 1 Mar 2026)

Qualitative analysis across these domains consistently highlights improved temporal coherence, avoidance of flow-warp artifacts, and better adaptation to diverse motions.

6. Limitations and Prospective Directions

Notwithstanding empirical advances, current IMT schemes face several challenges:

• Bitrate Trade-offs: Marginal performance reduction at very high bitrates where explicit residuals may better capture finer details (Chen et al., 12 Jun 2025).
• Pretrained Module Dependency: Some models (e.g., GIMM (Guo et al., 2024)) depend on the reliability of initial flow estimators or segmenters for mask propagation (Li et al., 1 Mar 2026).
• Scene Complexity: Many pipelines are specialized for single-person or frontal scenarios and may require adaptation for multi-person, multi-view, or scene-level dynamics (Chen et al., 12 Jun 2025).
• Long-Horizon Temporal Dependencies: Current architectures often operate over moderate frame-horizons; extending IMT principles to longer, globally consistent temporal synthesis remains an active research frontier.
• Motion-Identity Disentanglement: While mask-token and attention-constraint mechanisms demonstrate efficacy, perfect identity-motion separation remains challenging, particularly in unconstrained Internet data (Xu et al., 7 Feb 2026).

Future work includes:

• Integrating explicit motion priors with implicit modules for hybrid performance at low bitrates.
• Adapting mask-based and token-based IMT to general video domains with complex object interactions (Li et al., 1 Mar 2026).
• Research on richer tokenization and more expressive latent spaces to further decouple and recombine motion and identity for controllable synthesis (Xu et al., 7 Feb 2026).
• Leveraging temporal transformers or sequence models for enhanced long-range motion modeling (Chen et al., 12 Jun 2025).

7. Context, Impact, and Theoretical Significance

IMT defines a paradigm shift in video modeling, prioritizing learned, data-driven motion abstraction over classical geometric or part-wise displacement. This transition has broad ramifications:

• Compression: By intertwining compact feature learning with demotionization, IMT achieves state-of-the-art bitrate reductions without sacrificing perceptual quality (Chen et al., 12 Jun 2025).
• Compositionality: Feature- or token-level motion abstraction enables unprecedented flexibility, including cross-person, cross-object, and multimodal transfer unattainable with explicit flows (Li et al., 1 Mar 2026).
• Generalization: IMT orthogonalizes motion pattern inference from scene structure, permitting transfer across identities, scales, and scene layouts (Xu et al., 7 Feb 2026).
• Neural Representation: IMT approaches extend the notion of “implicit neural representation” (INR) from static geometry to spatiotemporal dynamics, leveraging transformers, vector quantization, dynamic filters, and coordinate MLPs.

The IMT framework continues to influence generative modeling, video understanding, and neural compression, rapidly evolving towards more robust, general, and scalable representations for high-dimensional video data.