Image and Video Compression with Neural Networks: A Review (1904.03567v2)

Abstract: In recent years, the image and video coding technologies have advanced by leaps and bounds. However, due to the popularization of image and video acquisition devices, the growth rate of image and video data is far beyond the improvement of the compression ratio. In particular, it has been widely recognized that there are increasing challenges of pursuing further coding performance improvement within the traditional hybrid coding framework. Deep convolution neural network (CNN) which makes the neural network resurge in recent years and has achieved great success in both artificial intelligent and signal processing fields, also provides a novel and promising solution for image and video compression. In this paper, we provide a systematic, comprehensive and up-to-date review of neural network based image and video compression techniques. The evolution and development of neural network based compression methodologies are introduced for images and video respectively. More specifically, the cutting-edge video coding techniques by leveraging deep learning and HEVC framework are presented and discussed, which promote the state-of-the-art video coding performance substantially. Moreover, the end-to-end image and video coding frameworks based on neural networks are also reviewed, revealing interesting explorations on next generation image and video coding frameworks/standards. The most significant research works on the image and video coding related topics using neural networks are highlighted, and future trends are also envisioned. In particular, the joint compression on semantic and visual information is tentatively explored to formulate high efficiency signal representation structure for both human vision and machine vision, which are the two dominant signal receptor in the age of artificial intelligence.

Image and Video Compression with Neural Networks: A Review

This paper provides an extensive review of neural network-based methodologies for image and video compression, highlighting the transition from traditional hybrid coding frameworks to advanced deep learning techniques. The authors, Siwei Ma et al., meticulously document the evolution of compression technologies, emphasizing the growing challenges in improving coding performance within conventional frameworks. By leveraging the success of Convolutional Neural Networks (CNNs) in artificial intelligence and signal processing, this review outlines innovative solutions that CNNs provide to image and video compression.

The core contribution is a structured analysis of deep learning's impact on enhancing the compression of visual signals. The paper dissects the integration of neural networks into both image and video compression, offering a distinct summary of methodologies ranging from deep learning-enhanced transform coding to advanced prediction strategies. Notable sections include discussions on end-to-end coding frameworks and adaptation of CNNs within the High Efficiency Video Coding (HEVC) protocols.

Key Insights and Contributions

1. Deep Learning in Image Compression:
   • The paper explores how neural networks, specifically CNNs, revolutionize image compression by embedding end-to-end architectures that replace traditional entropy and transform-based methods. The authors reference significant results where neural networks have surpassed traditional compression schemes like JPEG and JPEG2000 in compressing images both efficiently and effectively. Additive noise models facilitate training neural networks for lossy compression tasks, overcoming the gradient propagation challenges posed by quantization.
2. Video Compression Advancements:
   • The use of CNNs in improving intra-prediction within the HEVC standard is a focal point. Proposals such as IPCNN and IPFCN yield substantial bitrate savings through refined prediction capabilities. Methods also extend to fractional-pixel interpolation using Fractional-pixel Reference generation CNN (FRCNN), evidencing marked improvements in coding efficiency through CNN-powered motion prediction strategies.
3. Optimization and Entropy Coding:
   • The authors discuss the integration of CNNs in entropy coding processes, where these networks predict probability distributions for syntax elements, leading to enhanced compression. Additionally, neural networks are optimized for quantization tasks, reflecting potential advances in preserving visual quality while achieving compression.
4. Loop Filtering and Post-processing:
   • Contributions in neural network-based loop filtering showcase the potential to significantly reduce compression artifacts, improving visual quality post-decoding. This section highlights approaches such as Residual Highway CNN (RHCNN) and content-aware filtering that forecast trends toward more adaptive and context-sensitive video restoration technologies.

Implications and Future Directions

The paper realizes the robustness of neural networks in fostering new paradigms for video compression, particularly in handling complexities where traditional methodologies plateau. The adaptability of neural networks proposed allows for optimization paths uncharted in signal processing, providing a stronghold for future research and exploration.

Research implications suggest a trajectory focusing on semantically enriched compression mechanisms and rate-distortion optimized neural models. As visual data consumption escalates, the shift toward neural network-integrated frameworks promises a future where compression serves both efficient data transmission and advanced computational vision tasks.

The authors articulate areas for further exploration, including the design of memory and computationally efficient codec structures, which remains a nascent yet vital inquiry in practical applications. The proposed multi-network adaptive approaches and semantic fidelity-oriented compression methods offer fertile ground for continued advancements in neural compression efficiency and perceptual quality improvement.

Overall, this comprehensive review underscores the transformative role of neural networks in redefining image and video compression standards, laying a foundation for subsequent phases in visual data compression research. With further research into computationally feasible network structures, the potential to revolutionize real-time multimedia applications is increasingly tangible.