Fast and Memory-Efficient Network Towards Efficient Image Super-Resolution (2204.08397v1)

Abstract: Runtime and memory consumption are two important aspects for efficient image super-resolution (EISR) models to be deployed on resource-constrained devices. Recent advances in EISR exploit distillation and aggregation strategies with plenty of channel split and concatenation operations to make full use of limited hierarchical features. In contrast, sequential network operations avoid frequently accessing preceding states and extra nodes, and thus are beneficial to reducing the memory consumption and runtime overhead. Following this idea, we design our lightweight network backbone by mainly stacking multiple highly optimized convolution and activation layers and decreasing the usage of feature fusion. We propose a novel sequential attention branch, where every pixel is assigned an important factor according to local and global contexts, to enhance high-frequency details. In addition, we tailor the residual block for EISR and propose an enhanced residual block (ERB) to further accelerate the network inference. Finally, combining all the above techniques, we construct a fast and memory-efficient network (FMEN) and its small version FMEN-S, which runs 33% faster and reduces 74% memory consumption compared with the state-of-the-art EISR model: E-RFDN, the champion in AIM 2020 efficient super-resolution challenge. Besides, FMEN-S achieves the lowest memory consumption and the second shortest runtime in NTIRE 2022 challenge on efficient super-resolution. Code is available at https://github.com/NJU-Jet/FMEN.

Fast and Memory-Efficient Network Towards Efficient Image Super-Resolution

The paper "Fast and Memory-Efficient Network Towards Efficient Image Super-Resolution" addresses the challenges of deploying image super-resolution (EISR) models on resource-constrained devices, particularly focusing on runtime efficiency and memory consumption. The authors propose a novel architecture aimed at optimizing these parameters without compromising the quality of the enhanced images.

Key Contributions and Methodologies

The paper introduces a lightweight network architecture that achieves efficiency through sequential network operations, notably limiting the usage of feature fusion techniques typically employed in recent EISR models. The network design centers around two building blocks:

1. Enhanced Residual Block (ERB): This block is crafted to optimize network inference speed and memory usage. By leveraging re-parameterizable techniques, the ERB maintains the benefits of residual learning while eschewing the memory overhead introduced by traditional residual blocks. This is achieved by leveraging the structure of ResNet-style architectures but eliminating the skip connections during inference, which otherwise add to inference time and memory use.
2. High-Frequency Attention Block (HFAB): The HFAB is engineered to emphasize high-frequency image details, crucial for super-resolution tasks. This attention mechanism calculates pixel importance based on local and global contexts, using sequential operations like $3 \times 3$ convolutions and efficient activations. By eschewing complex multi-branch topologies, HFAB focuses computational resources more effectively, resulting in a lower computational burden while still enhancing detail.

These key innovations culminate in the assembly of a Fast and Memory-Efficient Network (FMEN), along with its reduced variant, FMEN-S. FMEN-S notably achieves a 33% increase in speed and a 74% reduction in memory consumption compared to contemporary state-of-the-art models.

Experimental Results and Implications

The FMEN and FMEN-S networks are assessed in terms of performance and resource efficiency. FMEN-S stands out by recording the lowest memory usage and the second shortest runtime in the NTIRE 2022 efficient super-resolution challenge, providing competitive image restoration quality.

The use of sequential operations instead of feature fusion addresses typical bottlenecks in EISR model inference — memory consumption from multiple feature maps and slow processing due to additional memory access and operation dependencies. The practical advantages are demonstrated through empirical evaluations where FMEN-S executes 33% quicker and consumes significantly less memory than models like E-RFDN, the prior winner of the AIM 2020 challenge.

Theoretical and Practical Implications

Practically, the proposed methodologies in FMEN have critical implications for deploying super-resolution models on devices with constrained resources such as mobile phones or embedded systems. The reduction in runtime and memory usage translates into lower energy consumption and potentially wider application scenarios.

Theoretically, the paper sets a precedent in the EISR field for prioritizing resource efficiency alongside restoration quality. By circumventing the traditional emphasis on feature fusion, it opens avenues for further exploration into lightweight attention mechanisms and optimized convolutional structures — areas that could drive future advancements in efficient model design.

Future Directions

This research lays the groundwork for future endeavors focusing on novel block designs and re-parameterization techniques that further streamline inference efficiency. Moreover, exploring other attention mechanisms which exploit frequency domain insights or adaptive on-the-fly re-parameterization could yield even more pronounced resource efficiency. As AI and computer vision applications continue to grow, the need for such resource-conscious approaches will undoubtedly become more pronounced. The architectures presented in this paper are well-positioned to influence the next wave of developments in efficient image enhancement methods.