Quadratic video interpolation (1911.00627v1)

Abstract: Video interpolation is an important problem in computer vision, which helps overcome the temporal limitation of camera sensors. Existing video interpolation methods usually assume uniform motion between consecutive frames and use linear models for interpolation, which cannot well approximate the complex motion in the real world. To address these issues, we propose a quadratic video interpolation method which exploits the acceleration information in videos. This method allows prediction with curvilinear trajectory and variable velocity, and generates more accurate interpolation results. For high-quality frame synthesis, we develop a flow reversal layer to estimate flow fields starting from the unknown target frame to the source frame. In addition, we present techniques for flow refinement. Extensive experiments demonstrate that our approach performs favorably against the existing linear models on a wide variety of video datasets.

• The paper presents a quadratic model for motion estimation that incorporates acceleration to predict non-linear motion accurately.
• It introduces a flow reversal layer that converts forward flow into backward flow for enhanced frame synthesis quality.
• The approach reduces frame prediction errors and outperforms state-of-the-art methods on benchmarks like GOPRO and DAVIS.

Overview of Quadratic Video Interpolation

The paper "Quadratic Video Interpolation" introduces a novel method in the field of computer vision aimed at enhancing the accuracy of video interpolation by accounting for the complex, non-linear motion that objects often undergo in videos. Traditional interpolation methods generally rely on the assumption of uniform motion between frames, often using linear models that predict intermediate frames based simply on a straight-line trajectory assumption. This poses significant challenges for accurately capturing the true motion of objects, which can vary in velocity and path. This paper proposes a quadratic video interpolation model that incorporates acceleration information permitting predictions of motion with curvilinear trajectories and varying velocities.

Core Contributions

1. Quadratic Model for Motion Estimation: The primary contribution of this research is the formulation of a quadratic model to estimate motion. This method utilizes additional input frames to predict more accurate intermediate frames by considering acceleration, thereby accounting for non-linear movement common in realistic scenarios.
2. Flow Reversal Layer: To estimate flows from target to source frames, the authors introduce a flow reversal layer. This layer is designed to more effectively convert forward flow (predicted using the quadratic model) into backward flow, which is essential for higher-quality frame synthesis.
3. Flow Refinement Techniques: To further improve the interpolation results, the paper presents novel techniques for refining flow fields to address artifacts and enhance accuracy.

Numerical Results

The proposed approach was benchmarked against state-of-the-art video interpolation techniques on a variety of datasets, including those with high frame rates such as GOPRO and Adobe240, as well as datasets featuring videos with complex, dynamic scenes such as UCF101 and DAVIS. Strong performance improvements were reported in both single and multiple frame interpolation tasks. Quantitatively, the algorithm achieved higher PSNR and SSIM metrics indicating better frame synthesis compared to existing methods. Notably, the acceleration-aware interpolation enabled a significant reduction in frame prediction errors, demonstrating its capability in approximating non-linear motion trajectories more accurately than linear models.

Implications and Future Directions

The implications of this research are significant both practically and theoretically. Practically, the quadratic interpolation method can be applied to various applications, such as motion deblurring, video editing, and virtual reality, where accurate motion interpolation is crucial. Theoretically, the incorporation of acceleration information introduces a higher-order method of understanding motion dynamics in videos, offering new avenues for exploring even more complex movement predictions in video data.

Future developments might explore expanding this framework to incorporate cubic or even higher-order models to further refine motion prediction capabilities in extremely non-linear or complex environments. There's also potential to apply similar quadratic frameworks to related problems beyond interpolation, such as in multi-frame optical flow estimation or novel view synthesis tasks.

This paper contributes to the broader discourse on improving motion modeling in video data, demonstrating the utility of moving beyond linear assumptions and providing a template for exploiting higher-order information in visual data processing.