Unprocessing Images for Learned Raw Denoising (1811.11127v1)

Abstract: Machine learning techniques work best when the data used for training resembles the data used for evaluation. This holds true for learned single-image denoising algorithms, which are applied to real raw camera sensor readings but, due to practical constraints, are often trained on synthetic image data. Though it is understood that generalizing from synthetic to real data requires careful consideration of the noise properties of image sensors, the other aspects of a camera's image processing pipeline (gain, color correction, tone mapping, etc) are often overlooked, despite their significant effect on how raw measurements are transformed into finished images. To address this, we present a technique to "unprocess" images by inverting each step of an image processing pipeline, thereby allowing us to synthesize realistic raw sensor measurements from commonly available internet photos. We additionally model the relevant components of an image processing pipeline when evaluating our loss function, which allows training to be aware of all relevant photometric processing that will occur after denoising. By processing and unprocessing model outputs and training data in this way, we are able to train a simple convolutional neural network that has 14%-38% lower error rates and is 9x-18x faster than the previous state of the art on the Darmstadt Noise Dataset, and generalizes to sensors outside of that dataset as well.

Unprocessing Images for Learned Raw Denoising: An Expert Perspective

The paper "Unprocessing Images for Learned Raw Denoising" by Brooks et al. presents an innovative approach to enhance the performance of machine learning-based image denoising methods, specifically for images captured by camera sensors in their raw format. The authors address a pertinent issue in the domain of image processing where the mismatch between synthetic data used for training deep learning models and real raw images used for evaluation leads to suboptimal performance. They propose a novel methodology to synthesize realistic raw data from commonly available internet photos by systematically 'unprocessing' these images, thereby closely aligning the training data with the real-world noise characteristics and processing pipelines of camera sensors.

The paper underlines the necessity to consider the comprehensive image processing pipeline, which includes gain, color correction, and tone mapping in addition to noise properties, to better prepare neural networks for handling real sensor data. This systematic unprocessing framework enables the synthesis of realistic raw sensor measurements, thus improving the generalization of the denoising algorithms from synthetic to real-world data.

Key findings from this research include a significant improvement in error rates where the proposed model achieves 14%-38% lower error rates compared to the previous state of the art when evaluated on the Darmstadt Noise Dataset. Moreover, the authors highlight computational efficiency, noting that the proposed model is 9x-18x faster than existing methods, demonstrating not only enhanced accuracy but also improved runtime performance.

For the experimental setup, the authors design a sophisticated pipeline that inverts the conventional processing steps like tone mapping, gamma compression, white balance adjustment, color correction, and applies a realistic noise model considering raw sensor attributes such as shot and read noise. This comprehensive approach is pivotal in reducing the domain gap between the synthetic training data and the real evaluation data by leveraging physical and statistical models of noise and sensor characteristics.

Furthermore, the authors adopt a U-Net architecture with modifications suited for raw images, enabling the effective regression from noisy to clean images. The inclusion of noise parameters as input aids the network in adapting to varying noise levels typical of real-world scenarios, enhancing the robustness of the proposed solution.

From a theoretical perspective, this work offers a significant contribution by incorporating traditional image processing knowledge into the neural network training paradigm, thus advancing the integration of algorithmic insight with learned models. Practically, the methodology presents a feasible solution to reduce the labor-intensive process of capturing extensive paired datasets for different sensors, allowing rapid adaptation to new devices.

Future developments could explore further refinement of the unprocessing methodology to include more sophisticated models of sensor-specific nonlinearities and the incorporation of additional components of image pipelines such as denoising engines or specialized demosaicking processes. Investigation into generalization across diverse device categories, including mobile and specialized imaging devices, could expand the applicability of the proposed model.

In summary, this paper makes a commendable advancement in the field of image denoising by bridging the gap between synthetic data training and real-world application. The enhanced performance metrics and computational efficiency underscore its practical relevance, making it a valuable reference for researchers and practitioners working on sensor-based image enhancement and restoration tasks.