MemNet: A Persistent Memory Network for Image Restoration (1708.02209v1)

Abstract: Recently, very deep convolutional neural networks (CNNs) have been attracting considerable attention in image restoration. However, as the depth grows, the long-term dependency problem is rarely realized for these very deep models, which results in the prior states/layers having little influence on the subsequent ones. Motivated by the fact that human thoughts have persistency, we propose a very deep persistent memory network (MemNet) that introduces a memory block, consisting of a recursive unit and a gate unit, to explicitly mine persistent memory through an adaptive learning process. The recursive unit learns multi-level representations of the current state under different receptive fields. The representations and the outputs from the previous memory blocks are concatenated and sent to the gate unit, which adaptively controls how much of the previous states should be reserved, and decides how much of the current state should be stored. We apply MemNet to three image restoration tasks, i.e., image denosing, super-resolution and JPEG deblocking. Comprehensive experiments demonstrate the necessity of the MemNet and its unanimous superiority on all three tasks over the state of the arts. Code is available at https://github.com/tyshiwo/MemNet.

• The paper presents a novel deep CNN architecture with persistent memory blocks that capture both short- and long-term dependencies in image restoration tasks.
• It integrates recursive units and adaptive gate mechanisms to merge features effectively, achieving superior performance such as a PSNR of 34.09 on the Set5 dataset.
• The network’s versatile design across denoising, super-resolution, and JPEG deblocking tasks opens avenues for broader applications in advanced image processing.

MemNet: A Persistent Memory Network for Image Restoration

Introduction

The paper "MemNet: A Persistent Memory Network for Image Restoration" by Ying Tai et al. introduces a novel convolutional neural network (CNN) architecture designed to address limitations in current image restoration algorithms. The proposed network, MemNet, incorporates persistent memory blocks to better handle long-term dependencies, which conventional deep CNN architectures struggle with as their depth increases. The paper comprehensively evaluates MemNet across three image restoration tasks: image denoising, single-image super-resolution (SISR), and JPEG deblocking. Numerical results demonstrate MemNet's superior performance over state-of-the-art methods.

MemNet Architecture

MemNet comprises three key components: Feature Extraction Network (FENet), memory blocks, and the Reconstruction Network (ReconNet). FENet extracts features from the input degraded image. Multiple memory blocks are then stacked to progressively refine the extracted features. Finally, ReconNet reconstructs the high-quality image by learning the residual between the input and output images.

A distinctive feature of MemNet is its memory blocks, which consist of recursive and gate units to realize persistent memory. The recursive unit captures multi-level features under different receptive fields, functioning as short-term memory, while the gate unit adaptively decides the contribution of short-term and long-term (i.e., from previous memory blocks) memories.

Key Contributions

1. Memory Block Design: The proposed memory block comprises a recursive unit to capture short-term dependencies and a gate unit to maintain long-term memory. This design conceptually resembles concepts from neuroscience about memory persistence in the neocortex.
2. Deep End-to-End Network: MemNet is notably deep, with up to 80 convolutional layers in its base configuration, representing one of the deepest architectures for image restoration tasks to date.
3. Versatility Across Tasks: MemNet's structure is adept at various image restoration tasks including denoising, super-resolution, and JPEG deblocking, using the same architecture.

Experimental Results

The paper reports comprehensive evaluations on MemNet's performance across multiple image restoration benchmarks. For instance, on the Set5 dataset with a super-resolution factor of 3, MemNet achieves a PSNR of 34.09, outperforming other methods such as VDSR (33.66 PSNR) and DRCN (33.82 PSNR). In image denoising tasks, MemNet consistently outperforms methods such as RED and BM3D across different noise levels.

The ablation studies confirm that both long-term dense connections and short-term recursive connections contribute significantly to performance improvements. Specifically, removing long-term dense connections (MemNet_NL) resulted in noticeable performance degradation.

Implications and Future Work

MemNet's ability to maintain long-term dependencies introduces significant improvements in image restoration tasks, suggesting potential broader applications in other domains requiring deep learning-based denoising or enhancement. Future research may explore optimizing memory block configurations for specific tasks or extending the architecture to handle more complex degradation models. Another promising area could be investigating the integration of MemNet with other network structures or using it as a backbone for other advanced image processing algorithms.

Conclusion

The paper underscores the importance of long-term memory in deep CNN architectures for image restoration. By introducing memory blocks combining recursive units with gate units, MemNet effectively maintains and utilizes both short-term and long-term dependencies, setting a new performance benchmark in image restoration tasks. These findings open avenues for further research into optimizing deep networks for diverse image processing applications.