Generative Motion Infilling

• Generative motion infilling is a technique that synthesizes coherent, plausible motion trajectories from sparse constraints using advanced conditional generative models.
• It employs diffusion, flow-matching, GANs, and autoencoder architectures to ensure temporal coherence, physical plausibility, and efficient frame synthesis.
• Applications include human pose recovery, 3D animation, video inpainting, and facial dynamics while addressing challenges like latency, boundary artifacts, and style control.

Generative motion infilling refers to the data-driven synthesis of temporally coherent, semantically plausible, and visually realistic motion trajectories given sparse constraints (e.g., keyframes, context, semantics, or partial observations). The research field encompasses methods targeting human pose sequences, full-body animation, video inpainting, 3D/4D scene reconstruction, and facial dynamics, employing generative modeling—particularly diffusion, flow-matching, GAN, and autoencoder-based architectures. This article reviews foundational models, conditioning mechanisms, algorithmic advances, evaluation criteria, and open challenges across the field.

1. Mathematical Formulations and Conditioning Principles

Generative motion infilling is typically formalized as a conditional generative modeling problem over sequences. Given observed subsets of frames or poses (contexts, $C$) and unknown/missing ones (gaps, $U$), the goal is to sample from $p(\mathbf{x}_U \mid \mathbf{x}_C)$. For pose or video sequences, $\mathbf{x}_t$ may encode 2D/3D joint positions, quaternion rotations, or video frames.

Key conditioning scenarios include:

• Boundary-constrained infilling: Start/end frames fixed, synthesize plausible transitions (Li et al., 2023).
• Anchor/semantic-constrained infilling: Some frames or semantic labels fixed throughout (Kim et al., 2022).
• Imprecisely timed or loosely placed constraints: Keyframes located with temporal uncertainty, requiring the model to retime as part of synthesis (Goel et al., 2 Mar 2025).
• Mask-based or inpainting setups: Arbitrary mask $M$ designates which frames to synthesize, allowing for deletion, insertion, or substitution edits (Sung-Bin et al., 16 Dec 2025, Höppe et al., 2022).

Generative models for this purpose must jointly ensure temporal coherence, context consistency, physical/kinematic plausibility, and, in extended cases, style compliance or diversity.

2. Diffusion, ODE, and Flow-Matching Architectures

Diffusion and flow-based generative models have established the state-of-the-art in motion infilling for video, pose, and structured motion data.

• Diffusion Dynamics: Standard approaches (e.g., DDPM, DDIM) transform clean data sequences into progressive noisy states and optimize noise-removal (score or noise prediction), with the reverse process reconstructing plausible samples step-wise (typically hundreds of steps) (Okupnik et al., 16 Oct 2025, Goel et al., 2 Mar 2025, Höppe et al., 2022).
• Conditional Infilling via Masking: Random mask diffusion schemes (e.g., RaMViD) enable training a single 3D-UNet where masked frames are generated conditionally, and unmasked frames are clamped during reverse steps, unifying infilling, prediction, and unconditional synthesis (Höppe et al., 2022).
• Iterative Latent Variable Refinement (ILVR): Low-high frequency decomposition in latent space, allowing for controlled fusion of motion style from a reference within a diffusion process, supports flexible mimicry versus creative deviation (Okupnik et al., 16 Oct 2025).
• Flow Matching/ODE Sampling: Replacing stochastic SDE paths with straight-line ODEs between prior and data (Motion Flow Matching), with neural vector fields trained to realize these flows, drastically reduces necessary steps (10–30) and facilitates deterministic endpoint enforcement (Hu et al., 2023). Trajectory rewriting ensures known constraints are strictly honored in the sampled motion.
• Inference-Time Bidirectional Constraint Alignment: Techniques such as Motion Prior Distillation transfer the dynamic residuals from forward (start-conditioned) diffusion paths to backward (end-conditioned) paths, overcoming latent misalignments that typically cause artifacts in classic bidirectional strategies (Jeon et al., 13 Feb 2026).

3. Alternative and Hybrid Generative Infilling Paradigms

While diffusion and flow-matching dominate recent advances, other generative strategies remain relevant:

• Stochastic Latent Dynamics (SDVI): A bi-directional constraint propagation mechanism (Residual Bi-ConvLSTMs) propagates reference momentum across long video intervals. Sampling from learned latent inference priors—with complementary KL divergences and $L_1$ loss—enables flexible, coherent frame synthesis between distant references (Xu et al., 2018).
• Convolutional Autoencoders: Framing skeletal infilling as 2D image inpainting, these models substitute missing time-blocks in pose–time matrices and reconstruct spatial-temporal consistency via deep CAE architectures, supporting arbitrary gaps and partial observations (Kaufmann et al., 2020).
• Generative Motion Matching: An exemplar-based method mines and recombines motion patches using a bidirectional similarity cost extended to multi-stage (coarse-to-fine) refinement, enforcing both plausibility (coherence) and diversity (completeness) with direct boundary constraining (Li et al., 2023).
• GAN-based Long-Range Inbetweening: GAN architectures with two-stage generators (local joint rotation, then global trajectory) conditioned on sparse keyframes and optionally on "Motion DNA" (seed motion patches) provide user-control and style diversity for long-horizon synthesis, though with weaker endpoint guarantees (Zhou et al., 2020).
• Motion-aware Generative Interpolation: Hybrid approaches inject optical flow and occlusion maps as both latent and feature-level guidance into diffusion frameworks, allowing the generative process to exploit correspondence information for improved stability and realism in non-rigid scene interpolation (Zhang et al., 7 Jan 2025).

4. Specializations: Style-Control, Editability, and High-Dimensional Domains

Motion infilling research has evolved beyond simple gap-filling to allow fine-grained control, higher-level semantics, and applicability to complex data modalities.

• Part-Wise Phase-Editable Models: Periodic autoencoders decompose motion into limb-wise phase–amplitude representations, allowing localized editing (swap, scale, retime at limb level), with a mixture-of-experts and LSTM-based sampler enforcing coherence across parts (Dai et al., 11 Mar 2025).
• Semantics, Conditioning, and Style: Transformer-based non-autoregressive infillers optionally prepend semantic tokens and use Gaussian random path augmentation for natural variation and improved anchor accuracy (Kim et al., 2022). ILVR further enables continuous control over style transfer versus creativity (Okupnik et al., 16 Oct 2025).
• Complex 4D Motion Interpolation: For object or scene-level 3D/4D infilling (e.g., In-2-4D), hierarchical keyframe decomposition, 3D Gaussian splatting, deformation fields, temporal attention, and segment-wise rigid regularization yield temporally consistent, spatially detailed 3D motion between single-view image endpoints (Nag et al., 11 Apr 2025).
• Talking Face/Facial Motion Infilling: Models like FacEDiT cast speech-conditioned facial editing/generation as masked motion infilling over LivePortrait latents, leveraging conditional flow-matching loss, Diffusion Transformer architectures with cross-attention, and strict temporal coherence at edit boundaries (Sung-Bin et al., 16 Dec 2025).

5. Evaluation Metrics and Benchmarks

Empirical assessment uses both general-purpose and domain-specific metrics:

Metric Type	Examples	Description
Physical/Temporal	L2/Jerk/Acceleration; NPSS; Foot Skate	Kinematic plausibility
Distributional	FID, FVD (Fréchet Inception/Video Distance)	Real vs. generated statistics, temporal structure (Hu et al., 2023, Zhang et al., 7 Jan 2025)
Perceptual	LPIPS, SSIM, PSNR, LMS	Perceptual and flow similarity (Xu et al., 2018)
Style/Diversity	Diversity (pairwise distance), Patch Coverage	Motion variance, exemplar coverage (Okupnik et al., 16 Oct 2025, Li et al., 2023)
Application-Specific	Lip Sync Error, Identity Similarity	Talking face infilling (Sung-Bin et al., 16 Dec 2025)

Datasets include AIST++, LAFAN1, HumanEva, BAIR, Kinetics-600, UCF101, DAVIS, and I4D-15, spanning kinematic, video, and 4D infilling.

6. Limitations and Open Directions

Despite advances, generative motion infilling faces several challenges:

• Latency and Efficiency: Many diffusion models require 100–1000 denoising steps. Flow-matching and ODE-based variants address this but may compromise realism at high compression (Hu et al., 2023).
• Boundary Artifacts: Classic bidirectional strategies can induce ghosting and discontinuities unless explicit trajectory alignment is performed (Jeon et al., 13 Feb 2026).
• Style and Realism: Ill-posed infilling under sparse constraints may generate mean-like or semantically drifted transitions. Improvements demand richer conditional priors and semantic modeling (Okupnik et al., 16 Oct 2025, Zhou et al., 2020).
• Variable-Length and Retiming: Correctly infilling under variable interval lengths or loose anchor timing remains an area of active development (Goel et al., 2 Mar 2025).
• Editability and Interactivity: Full support for real-time, fine-grained editability (e.g. per-limb, partial spans, style transfer) is only recently practical with phase-based and transformer samplers (Dai et al., 11 Mar 2025).
• Generalization Across Domains: Transfer to unseen actions, objects, domains, or multi-modal infilling (complex 3D, faces, non-humanoid) is limited, often requiring special representation or architectural adaptations (Nag et al., 11 Apr 2025, Sung-Bin et al., 16 Dec 2025).

7. Synthesis and Future Prospects

Generative motion infilling now constitutes a mature, multidimensional field integrating advances from diffusion models, flow-matching, GANs, and conditional autoencoding. It enables applications from real-time animation, video restoration, and artistic stylization, to editable 3D/4D scene synthesis and facial editing. Prospective trends include accelerated ODE/consistency models (Hu et al., 2023), hybrid architectures fusing explicit motion constraints with powerful generative refinement (Zhang et al., 7 Jan 2025), advanced style/semanitic control (Okupnik et al., 16 Oct 2025, Dai et al., 11 Mar 2025), and generalization across structured modalities (e.g., high-dimensional articulated objects, complex environmental contexts) (Nag et al., 11 Apr 2025). Embedded physical correctness, robust domain transfer, and seamless user interaction will be central to next-generation motion infilling systems.