Sparsity-guided Network Design for Frame Interpolation (2209.04551v1)

Abstract: DNN-based frame interpolation, which generates intermediate frames from two consecutive frames, is often dependent on model architectures with a large number of features, preventing their deployment on systems with limited resources, such as mobile devices. We present a compression-driven network design for frame interpolation that leverages model pruning through sparsity-inducing optimization to greatly reduce the model size while attaining higher performance. Concretely, we begin by compressing the recently proposed AdaCoF model and demonstrating that a 10 times compressed AdaCoF performs similarly to its original counterpart, where different strategies for using layerwise sparsity information as a guide are comprehensively investigated under a variety of hyperparameter settings. We then enhance this compressed model by introducing a multi-resolution warping module, which improves visual consistency with multi-level details. As a result, we achieve a considerable performance gain with a quarter of the size of the original AdaCoF. In addition, our model performs favorably against other state-of-the-art approaches on a wide variety of datasets. We note that the suggested compression-driven framework is generic and can be easily transferred to other DNN-based frame interpolation algorithms. The source code is available at https://github.com/tding1/CDFI.

• The paper presents a sparsity-guided framework that compresses the AdaCoF model by tenfold using ℓ1-norm regularization without performance loss.
• It introduces a multi-resolution warping module with a U-Net feature pyramid to improve visual quality and feature consistency.
• The optimized model outperforms state-of-the-art methods, achieving over 1 dB higher PSNR on the Middlebury dataset while reducing computational demands.

Sparsity-Guided Network Design for Frame Interpolation: A Summary

The paper discusses a novel approach for designing deep neural network (DNN) architectures for the task of video frame interpolation. This approach leverages model compression grounded in sparsity-inducing optimization to create a more efficient model, retaining performance while significantly reducing computational overhead. The authors use the recently proposed AdaCoF framework as a baseline for their experiments, demonstrating that through strategic compression and enhancement steps, the models can be drastically optimized.

The problem of video frame interpolation, where intermediate frames are generated from two consecutive frames, is computationally demanding and often relies on resource-heavy models. This demand makes it challenging to deploy such models on devices with limited resources, like mobile devices. The authors address this challenge by first compressing the AdaCoF model, reducing its size by a factor of ten without a loss of performance. This compression is achieved through model pruning with an $\ell_1$-norm sparsity regularizer, effectively discarding superfluous parameters in the model. The compression yields a model that performs comparably to the original, indicating the redundancy in the original architecture.

Beyond compression, the paper further optimizes the network by introducing a multi-resolution warping module that enhances the visual quality of interpolated frames through improved feature consistency. This module utilizes a feature pyramid derived from a U-Net encoder, conducting multi-scale feature warping to facilitate improved output synthesis. This, together with an enhanced synthesis network, leads to a substantial boost in performance while minimizing model complexity. The resulting model is shown to outperform AdaCoF as well as state-of-the-art methods on various datasets, achieving over 1 dB higher Peak Signal-to-Noise Ratio (PSNR) on the Middlebury dataset, while maintaining a quarter of the original model size.

From a theoretical perspective, the framework provides insights into network architecture design by identifying essential model components and eliminating redundancy. Practically, the reduced model size and computational load enable deployment on resource-constrained systems, broadening the applicability of frame interpolation technology. This methodology is also posited to be extendable to other DNN-based frame interpolation algorithms, facilitating advancements in model efficiency across various contexts.

Looking to the future, one intriguing direction suggested by the authors is tightening the integration between compression and design processes, potentially iterating to arrive at an optimal architecture more effectively. This investigation into the underpinnings of neural network efficiency through structured reduction could pave the way for significant improvements in how ai models are developed and applied across diverse technological fields.