Multiple Video Frame Interpolation via Enhanced Deformable Separable Convolution

Abstract: Generating non-existing frames from a consecutive video sequence has been an interesting and challenging problem in the video processing field. Typical kernel-based interpolation methods predict pixels with a single convolution process that convolves source frames with spatially adaptive local kernels, which circumvents the time-consuming, explicit motion estimation in the form of optical flow. However, when scene motion is larger than the pre-defined kernel size, these methods are prone to yield less plausible results. In addition, they cannot directly generate a frame at an arbitrary temporal position because the learned kernels are tied to the midpoint in time between the input frames. In this paper, we try to solve these problems and propose a novel non-flow kernel-based approach that we refer to as enhanced deformable separable convolution (EDSC) to estimate not only adaptive kernels, but also offsets, masks and biases to make the network obtain information from non-local neighborhood. During the learning process, different intermediate time step can be involved as a control variable by means of an extension of coord-conv trick, allowing the estimated components to vary with different input temporal information. This makes our method capable to produce multiple in-between frames. Furthermore, we investigate the relationships between our method and other typical kernel- and flow-based methods. Experimental results show that our method performs favorably against the state-of-the-art methods across a broad range of datasets. Code will be publicly available on URL: \url{https://github.com/Xianhang/EDSC-pytorch}.

• The paper proposes Enhanced Deformable Separable Convolution (EDSC), a novel kernel-based method extending deformable convolution and CoordConv to handle large motion and interpolate frames at arbitrary temporal positions.
• Experiments show EDSC achieves competitive or superior performance on standard datasets like UCF101 and Vimeo90K in terms of PSNR and SSIM, while being computationally efficient.
• The research demonstrates that EDSC effectively handles challenging cases like occlusions and large motion, providing a framework that relates kernel-based and flow-based interpolation methods.

Enhanced Deformable Separable Convolution for Video Frame Interpolation

The paper "Multiple Video Frame Interpolation via Enhanced Deformable Separable Convolution" introduces an approach to video frame interpolation that innovatively extends non-flow kernel-based methods with improved adaptive convolutional components. This work advances the synthesis of non-existing video frames, which is a foundational challenge in video processing, impacting applications such as frame rate up-conversion, slow motion generation, and novel view synthesis.

The authors propose a model called Enhanced Deformable Separable Convolution (EDSC), which addresses several key limitations in prior interpolation approaches. Conventional kernel-based methods typically estimate pixel values using adaptive convolution processes, but struggle with scenes exhibiting large motion given their fixed kernel sizes. Such methods also limit the generation of frames to a fixed temporal midpoint between input frames. EDSC circumvents these constraints by introducing learnable offsets, masks, and biases, drawing on the success of deformable convolution networks. Furthermore, through an extension of the coord-conv trick, this model integrates intermediate time steps as control variables, empowering the generation of frames at any arbitrary temporal position.

The research delineates the theoretical relationship between EDSC and traditional flow-based methods, showing that flow-based approaches can be seen as special cases of their model. This is supported by experimental analysis throughout, demonstrating that EDSC handles large-motion interpolation effectively without reliance on flow estimation components, offering both computational efficiency and improved interpolation quality. With a kernel size reduced to 5x5, EDSC achieves comparable or superior performance relative to other kernel-based methods utilizing significantly larger kernels.

Key numerical results exhibit EDSC's competitive performance against state-of-the-art methods across a variety of widely recognized datasets, such as UCF101, Vimeo90K, and Middlebury. On these datasets, the proposed approach shows strong quantitative improvements in terms of PSNR and structural similarity, and effectively minimizes interpolation errors in both occluded and boundary regions. The reduction in FLOPs and model parameters, achieved through the employment of HetConv, underscores EDSC's efficiency.

The practical implications of this research are substantial, enabling more robust frame interpolation even in challenging conditions that involve occlusions or extensive motion. Theoretical implications extend to the unification of kernel-based and flow-based approaches under a single framework, paving the way for future developments in optimizing video interpolation techniques.

Moving forward, this work points to the potential integration of auxiliary information, such as additional temporal data or enhanced pre-trained models, to further refine the quality of interpolated frames. The notion of joint video enhancement tasks, expanding from interpolation to issues such as video resolution enhancement or stabilization, presents an intriguing horizon for the continued evolution of semantic video processing capabilities.