Toward DNN of LUTs: Learning Efficient Image Restoration with Multiple Look-Up Tables (2303.14506v1)

Abstract: The widespread usage of high-definition screens on edge devices stimulates a strong demand for efficient image restoration algorithms. The way of caching deep learning models in a look-up table (LUT) is recently introduced to respond to this demand. However, the size of a single LUT grows exponentially with the increase of its indexing capacity, which restricts its receptive field and thus the performance. To overcome this intrinsic limitation of the single-LUT solution, we propose a universal method to construct multiple LUTs like a neural network, termed MuLUT. Firstly, we devise novel complementary indexing patterns, as well as a general implementation for arbitrary patterns, to construct multiple LUTs in parallel. Secondly, we propose a re-indexing mechanism to enable hierarchical indexing between cascaded LUTs. Finally, we introduce channel indexing to allow cross-channel interaction, enabling LUTs to process color channels jointly. In these principled ways, the total size of MuLUT is linear to its indexing capacity, yielding a practical solution to obtain superior performance with the enlarged receptive field. We examine the advantage of MuLUT on various image restoration tasks, including super-resolution, demosaicing, denoising, and deblocking. MuLUT achieves a significant improvement over the single-LUT solution, e.g., up to 1.1dB PSNR for super-resolution and up to 2.8dB PSNR for grayscale denoising, while preserving its efficiency, which is 100$\times$ less in energy cost compared with lightweight deep neural networks. Our code and trained models are publicly available at https://github.com/ddlee-cn/MuLUT.

• The paper presents MuLUT, a multi-LUT architecture that extends single LUT limitations by expanding the receptive field for superior image restoration.
• It employs complementary, hierarchical, and channel indexing to capture richer spatial and color details, significantly enhancing tasks like super-resolution and demosaicing.
• The approach achieves measurable gains—up to 1.1dB PSNR in super-resolution and 6dB in demosaicing—while consuming up to 100x less computational energy than comparable DNNs.

Learning Efficient Image Restoration with Multiple Look-Up Tables

The paper "Toward DNN of LUTs: Learning Efficient Image Restoration with Multiple Look-Up Tables" introduces a novel method for leveraging multiple look-up tables (LUTs) to efficiently perform image restoration tasks, offering a practical and computationally feasible solution to accommodate high-definition displays on resource-constrained edge devices. The proposed approach, "MuLUT," seeks to address major limitations inherent in single LUT solutions by expanding the receptive field of LUTs, thereby significantly enhancing their performance in tasks such as super-resolution, demosaicing, denoising, and deblocking.

Methodology

MuLUT extends the capacity and capability of traditional LUTs by defining an architecture composed of multiple LUTs, drawing inspiration from the structural organization seen in deep neural networks (DNNs). The core ideas are the following:

1. Complementary Indexing: By introducing diverse indexing patterns across multiple LUTs operating in parallel, MuLUT effectively captures more spatial information, addressing the receptive field constraints of single LUT solutions. This method is demonstrated to aggregate richer contextual data from the image, vital for restoration tasks.
2. Hierarchical Indexing: MujLT also leverages a method of cascading LUTs to form hierarchies that emulate multi-layer architectures in DNNs. Through a re-indexing mechanism, intermediate outputs can be further processed across LUT layers, thus expanding the effective receptive field linearly as opposed to exponentially increasing memory requirements.
3. Channel Indexing: Cross-channel interactions are enabled through channel-wise indexing, which provides an avenue for cohesive processing of color channels, allowing MuLUT to deal efficiently with color image data.
4. LUT-aware Finetuning Strategy: The paper introduces a finetuning strategy that optimizes the LUT-stored values according to the performance loss arising from uniform sampling and interpolation processes. This adaptation ensures that MuLUT remains highly performant despite size reductions for feasible deployment.

Results and Performance

The experiments conducted in the paper exhibit MuLUT's strong performance on several image restoration tasks, demonstrating particular efficacy in scenarios with greater receptive field requirements:

• Super-Resolution: MuLUT achieves a performance increase of up to 1.1dB PSNR compared to single LUT implementations. The method approaches the efficiency and PSNR benchmarks of light DNN solutions like FSRCNN but requires significantly lower computational energy—up to 100 times less than comparable DNN models with quantization and AdderNet techniques.
• Demosaicing: In demosaicing tasks on Bayer-patterned images, MuLUT significantly outperforms baseline LUT architectures by incorporating both hierarchical and channel indexing, resulting in improved color reconstructions with up to 6dB gains in comparative PSNR metrics.
• Denoising and Deblocking: MuLUT demonstrates substantial gains in restoring high-frequency details in grayscale and color images, particularly evidencing its flexibility and adeptness at leveraging a wider receptive field through novel LUT construction methods.

Implications and Future Prospects

MuLUT represents a significant advancement in the practical use of LUTs for efficient image restoration on edge devices. Its design principles effectively mimic DNN-like hierarchical and parallel processing capabilities while eliminating the prohibitive computational costs and memory constraints that DNNs typically pose. As edge devices increasingly adopt high-resolution displays, the need for energy-efficient, high-performance image restoration methods like MuLUT will become more pronounced.

MuLUT's general framework also opens opportunities for further investigation into integrating non-local operations and the application of attention mechanisms within the LUT paradigm, potentially expanding its domain into more complex video restoration tasks. These future steps could further potentiate MuLUT's capabilities and demonstrate its broader applicability in real-time image processing applications.