Wide Activation for Efficient and Accurate Image Super-Resolution (1808.08718v2)

Abstract: In this report we demonstrate that with same parameters and computational budgets, models with wider features before ReLU activation have significantly better performance for single image super-resolution (SISR). The resulted SR residual network has a slim identity mapping pathway with wider ((2\times) to (4\times)) channels before activation in each residual block. To further widen activation ((6\times) to (9\times)) without computational overhead, we introduce linear low-rank convolution into SR networks and achieve even better accuracy-efficiency tradeoffs. In addition, compared with batch normalization or no normalization, we find training with weight normalization leads to better accuracy for deep super-resolution networks. Our proposed SR network \textit{WDSR} achieves better results on large-scale DIV2K image super-resolution benchmark in terms of PSNR with same or lower computational complexity. Based on WDSR, our method also won 1st places in NTIRE 2018 Challenge on Single Image Super-Resolution in all three realistic tracks. Experiments and ablation studies support the importance of wide activation for image super-resolution. Code is released at: https://github.com/JiahuiYu/wdsr_ntire2018

• The paper introduces a novel wide activation approach in residual blocks that improves image super-resolution accuracy while maintaining efficiency.
• The paper details two models—WDSR-A and WDSR-B—that implement wide activation to balance feature expansion and computational resources.
• The paper demonstrates that weight normalization accelerates convergence and boosts PSNR, outperforming traditional SISR methods.

An Evaluation of Wide Activation Techniques in Image Super-Resolution

The paper entitled "Wide Activation for Efficient and Accurate Image Super-Resolution" presents significant advancements in the domain of Single Image Super-Resolution (SISR) through innovative neural network architectures. The primary focus is on enhancing the effectiveness of residual networks in SISR by leveraging wide activation, which involves expanding the features width before ReLU activation. This approach is shown to substantially improve accuracy without increasing parameters or computational load.

Key Contributions

1. Wide Activation in Residual Networks: The central thesis is that widening features before ReLU activation in residual blocks results in significant improvements in SISR tasks. This modification allows more information to flow through the network by permitting low-level feature propagation to deeper layers, thereby enhancing dense pixel value predictions.
2. Efficient Methods for Feature Expansion: The paper introduces two models, WDSR-A and WDSR-B, each incorporating wide activation differently to balance efficiency with accuracy. WDSR-A achieves wider activation by slimming identity mapping pathways and expanding features before activation. WDSR-B introduces linear low-rank convolution allowing activation widening without additional parameters or computational cost.
3. Weight Normalization: Weight normalization overcomes the inherent limitations of batch normalization in SISR networks. By decoupling weight vector length from its direction, it facilitates faster convergence with higher learning rates, leading to improved performance metrics, including PSNR.

Numerical Results and Implications

The experimental results provide compelling evidence for the superiority of wide activation techniques. The WDSR models consistently surpass traditional models like EDSR in PSNR scores under equivalent parameter budgets. For instance, under similar settings, WDSR-B achieves a PSNR of 34.409 compared to EDSR's 34.284 in the DIV2K validation dataset for x2 super-resolution tasks.

These results not only confirm the efficacy of wide activation in improving image reconstruction quality but also align with contemporary trends in deep learning where model architecture innovations directly correlate with performance improvements.

Theoretical and Practical Implications

Theoretically, the use of wide activation advocates for a paradigm shift in neural network design for SISR, emphasizing the importance of non-linear transformation with broad feature representations. Practically, these innovations make wide-activation networks viable candidates for real-time applications requiring high-resolution image interpolations, such as satellite imagery analysis, medical imaging diagnostics, and enhanced video streaming services.

Future Work

Future research may explore the potential of wide activation in other image restoration tasks including denoising and deblurring, and its application beyond convolutional networks, such as in transformers. Additionally, integrating these methods with other state-of-the-art techniques, like attention mechanisms, could further push the boundaries of image resolution quality.

In summary, this paper provides a comprehensive blueprint for achieving efficient and accurate image super-resolution using wide activation. It signals a progression towards more resource-efficient neural networks, expanding the prospects for deploying advanced deep learning models in computationally constrained environments.