Disentangled Motion Modeling for Video Frame Interpolation (2406.17256v2)

Abstract: Video Frame Interpolation (VFI) aims to synthesize intermediate frames between existing frames to enhance visual smoothness and quality. Beyond the conventional methods based on the reconstruction loss, recent works have employed generative models for improved perceptual quality. However, they require complex training and large computational costs for pixel space modeling. In this paper, we introduce disentangled Motion Modeling (MoMo), a diffusion-based approach for VFI that enhances visual quality by focusing on intermediate motion modeling. We propose a disentangled two-stage training process. In the initial stage, frame synthesis and flow models are trained to generate accurate frames and flows optimal for synthesis. In the subsequent stage, we introduce a motion diffusion model, which incorporates our novel U-Net architecture specifically designed for optical flow, to generate bi-directional flows between frames. By learning the simpler low-frequency representation of motions, MoMo achieves superior perceptual quality with reduced computational demands compared to the generative modeling methods on the pixel space. MoMo surpasses state-of-the-art methods in perceptual metrics across various benchmarks, demonstrating its efficacy and efficiency in VFI.

• The paper introduces a novel two-stage training process that disentangles motion modeling from frame synthesis using a diffusion-based approach.
• The method leverages a motion diffusion model to predict bi-directional optical flows, yielding improved performance on perceptual metrics such as LPIPS and DISTS.
• Architectural innovations, including a U-Net design with convex upsampling at reduced resolution, significantly boost computational efficiency while enhancing video quality.

Disentangled Motion Modeling for Video Frame Interpolation: An Expert Overview

The paper "Disentangled Motion Modeling for Video Frame Interpolation" by J. Lew et al. presents a novel approach to enhance video frame interpolation (VFI), a critical task in computer vision aimed at synthesizing intermediate frames between existing ones to improve video smoothness and quality. Traditional methods have predominantly relied on $L_1$ or $L_2$ reconstruction loss, often resulting in high PSNR scores but underwhelming perceptual quality. To address these shortcomings, recent advancements have included deep feature spaces and generative models, albeit with substantial computational demands.

The authors propose Disentangled Motion Modeling (MoMo), a diffusion-based VFI approach that shifts focus toward intermediate motion modeling rather than directly synthesizing frames. MoMo introduces a two-stage training process that decouples frame synthesis and motion modeling to enhance efficiency and perceptual outcomes.

Methodological Contributions

1. Two-Stage Training Process: The methodology involves initially training an overview network ($\mathcal{S}$) and an optical flow model ($\mathcal{F}$) independently. $\mathcal{S}$ generates frames from neighboring input frames and optical flows, while $\mathcal{F}$ improves flow accuracy via fine-tuning. The enhanced $\mathcal{F}$ serves as a teacher to subsequently train a motion diffusion model.
2. Motion Diffusion Model: The novel motion diffusion model $\mathcal{M}$ is trained to predict bi-directional optical flow maps using ground-truth labels provided by the fine-tuned flow estimator. This marks a departure from existing VFI techniques that focus on direct RGB frame generation, instead leveraging the diffusion model to generate the intermediate motions crucial for high-quality video interpolation.
3. Architectural Innovations: A key innovation is the diffusion-based U-Net architecture designed to operate on low-frequency representations of optical flow at 1/8 the original resolution, incorporating a convex upsampling mechanism for final flow map refinement. This design choice enhances computational efficiency and performance, aligning with the intrinsic properties of optical flows.

Empirical Evaluation

Experiments conducted on benchmarks such as SNU-FILM, Middlebury, and Xiph datasets indicate MoMo's superior performance in perceptual metrics like LPIPS and DISTS, compared to both traditional and contemporary generative methods for VFI. MoMo demonstrates enhanced perceptual quality across various motion complexities and resolutions, thus underscoring the benefits of focused motion modeling within the VFI paradigm.

Implications and Future Directions

The paper presents significant implications for both the theoretical underpinnings of VFI and its practical applications. By focusing on disentangled motion modeling through diffusion processes, this approach not only achieves improved perceptual outcomes but also reduces the computational overhead associated with high-frequency generative tasks. Future developments may explore further optimization of the diffusion model's architecture, particularly for handling high-resolution inputs directly, and extending this framework to broader video restoration tasks.

In conclusion, the approach detailed in "Disentangled Motion Modeling for Video Frame Interpolation" represents a pivotal step toward refining the VFI process by emphasizing intermediate motion as a primary target of modeling. This focus has yielded notable improvements in visual quality, balancing computational efficiency with the sophisticated demands of perceptually oriented video applications.