Video Frame Synthesis using Deep Voxel Flow (1702.02463v2)

Abstract: We address the problem of synthesizing new video frames in an existing video, either in-between existing frames (interpolation), or subsequent to them (extrapolation). This problem is challenging because video appearance and motion can be highly complex. Traditional optical-flow-based solutions often fail where flow estimation is challenging, while newer neural-network-based methods that hallucinate pixel values directly often produce blurry results. We combine the advantages of these two methods by training a deep network that learns to synthesize video frames by flowing pixel values from existing ones, which we call deep voxel flow. Our method requires no human supervision, and any video can be used as training data by dropping, and then learning to predict, existing frames. The technique is efficient, and can be applied at any video resolution. We demonstrate that our method produces results that both quantitatively and qualitatively improve upon the state-of-the-art.

• The paper's main contribution is the DVF method, which synthesizes novel video frames by learning 3D optical flow.
• It employs a fully convolutional encoder-decoder with skip connections and TV regularization to ensure spatial and temporal coherence.
• DVF outperforms state-of-the-art methods by ~1.6 dB on benchmarks like UCF-101, proving its practical efficacy.

Video Frame Synthesis using Deep Voxel Flow

The paper, "Video Frame Synthesis using Deep Voxel Flow," introduces an innovative approach to synthesizing new video frames through a method named Deep Voxel Flow (DVF). The primary objective of this research is to enhance frame interpolation (synthesizing video frames between existing ones) and extrapolation (predicting future frames). The cornerstone of this method is a convolutional neural network (CNN) that learns to generate new frames by flowing pixel values from existing frames, thereby mitigating the typical challenges encountered in both traditional optical-flow-based and direct pixel synthesis approaches.

Overview

The DVF method integrates aspects from both optical flow techniques and generative deep learning methods. Traditional optical flow approaches, while effective in scenarios where flow estimation is precise, often introduce artifacts if the flow computation fails. Conversely, recent generative CNN methods, which directly predict pixel values, tend to produce blurry results due to the complexities of directly hallucinating pixel values. The proposed DVF method addresses these deficiencies by leveraging a deep network to learn a flow-based pixel interpolation mechanism, utilizing existing video frames to predict missing frames more accurately.

Methodology

DVF sets itself apart by employing a fully convolutional encoder-decoder network architecture. The network is trained in a self-supervised manner, where any video can serve as training data by discarding and then predicting certain frames. The novel aspect of this method is the introduction of a voxel flow layer, which predicts a 3D optical flow vector for each pixel across space and time. This predicted voxel flow is then used to generate new frames by trilinear interpolation of pixel values within the video volume.

The DVF employs total variation (TV) regularizations to maintain spatial and temporal coherence, which significantly reduces visual artifacts. The network's architecture contains multiple convolution and deconvolution layers, coupled with skip connections that preserve spatial details.

Results

The paper demonstrates that DVF achieves superior performance over state-of-the-art methods across various benchmarks, including the UCF-101 and THUMOS-15 datasets. The results are evaluated using PSNR and SSIM metrics, with DVF outperforming both conventional optical flow methods and generative CNN approaches. Quantitatively, DVF improves by approximately 1.6 dB over traditional methods on video interpolation tasks.

In addition to single-step prediction, DVF can extend to multi-step prediction, showing consistent qualitative and quantitative improvements. The network's ability to effectively handle large motions is also enhanced by a multi-scale approach, which processes video frames from coarse to fine scales and fuses the information from different resolutions.

Implications and Future Work

The implications of this research are multifaceted. Practically, DVF can be integrated into applications involving video re-timing, slow-motion effects in film production, and potentially in video editing to upscale frame rates. Theoretically, the incorporation of voxel flow within deep learning frameworks underscores a significant step in leveraging unsupervised learning for complex spatiotemporal tasks.

Furthermore, the research indicates that DVF can generalize to tasks beyond video frame interpolation, such as reconstructing novel views in view synthesis. This generalization capability is tested and verified on the KITTI dataset, with DVF showing superior performance, even without fine-tuning.

Future research could explore integrating flow layers with pixel synthesis layers to predict pixels that cannot be adequately copied from existing frames. Additionally, refining the multi-frame prediction mechanisms and optimizing the network for deployment on resource-constrained mobile devices are promising directions.

Conclusion

The introduction of Deep Voxel Flow presents a compelling advancement in video frame synthesis, effectively merging the precision of optical flow methods with the generative capabilities of modern CNNs. Through rigorous evaluation on benchmark datasets and practical applications, the DVF method establishes a new standard in frame interpolation and extrapolation, showcasing broader potential in video-related tasks. This work opens avenues for further research in leveraging deep learning techniques for more sophisticated and higher-quality video frame synthesis.