Replacing Mobile Camera ISP with a Single Deep Learning Model (2002.05509v1)

Abstract: As the popularity of mobile photography is growing constantly, lots of efforts are being invested now into building complex hand-crafted camera ISP solutions. In this work, we demonstrate that even the most sophisticated ISP pipelines can be replaced with a single end-to-end deep learning model trained without any prior knowledge about the sensor and optics used in a particular device. For this, we present PyNET, a novel pyramidal CNN architecture designed for fine-grained image restoration that implicitly learns to perform all ISP steps such as image demosaicing, denoising, white balancing, color and contrast correction, demoireing, etc. The model is trained to convert RAW Bayer data obtained directly from mobile camera sensor into photos captured with a professional high-end DSLR camera, making the solution independent of any particular mobile ISP implementation. To validate the proposed approach on the real data, we collected a large-scale dataset consisting of 10 thousand full-resolution RAW-RGB image pairs captured in the wild with the Huawei P20 cameraphone (12.3 MP Sony Exmor IMX380 sensor) and Canon 5D Mark IV DSLR. The experiments demonstrate that the proposed solution can easily get to the level of the embedded P20's ISP pipeline that, unlike our approach, is combining the data from two (RGB + B/W) camera sensors. The dataset, pre-trained models and codes used in this paper are available on the project website.

• The paper introduces PyNET, a single deep learning model that integrates multiple ISP tasks to enhance RAW-to-RGB conversion.
• The authors use a pyramidal CNN architecture with hierarchical loss functions to progressively optimize both global corrections and fine-detail enhancements.
• Experimental results show improved PSNR and MS-SSIM scores, with images exhibiting superior color vibrancy and detail relative to native ISP outputs.

Replacing Mobile Camera ISP with a Single Deep Learning Model

The paper introduces a novel approach to mobile photography by proposing the replacement of traditional Image Signal Processing (ISP) pipelines in mobile cameras with a single deep learning model. Traditional ISPs are intricate systems designed to enhance image quality through sequential processing steps tailored to specific camera hardware. Despite their complexity, these systems often struggle with the inherent limitations of mobile camera hardware, such as small sensors and limited optics, leading to image artifacts like noise and poor color rendition.

PyNET: An End-to-End Deep Learning Solution

The authors present PyNET, a new convolutional neural network (CNN) architecture, to address the RAW to RGB image conversion problem in mobile photography. PyNET is structured as a pyramidal network, which processes images at multiple scales, merging global and local correction processes. This architecture is designed to map RAW Bayer data directly from a camera sensor to an RGB image of professional quality—specifically targeting the quality of images produced by high-end DSLR cameras.

Distinct from existing approaches which separately handle ISP tasks such as demosaicing and denoising, PyNET offers an integrated model that encapsulates all these processes. In doing so, it circumvents the need for device-specific ISP pipelines, achieving flexibility and uniformity across different hardware.

Data and Methodology

The researchers validate PyNET through extensive experimentation using a large-scale dataset comprising 10,000 paired RAW and RGB images. These images were captured with a Huawei P20 smartphone and a Canon 5D Mark IV DSLR. For training, they leverage diverse conditions and settings to ensure robustness in their model. The training process for PyNET is progressive, with the model first optimized at lower image resolutions before gradually increasing to full scale, effectively bridging coarse global modifications with fine-detail enhancements.

Loss functions are carefully tailored for different pyramid levels—employing mean squared error (MSE) at the lowest levels to address global characteristics, and incorporating perceptual losses at higher levels to hone fine details. This hierarchical learning process enables PyNET to balance between maintaining global consistency and enhancing local details.

Experimental Results and Implications

Quantitatively, PyNET demonstrates superior performance over a range of contemporary deep learning frameworks typically deployed for image enhancement tasks. This is confirmed by higher PSNR and MS-SSIM scores. Qualitatively, the reconstructed images are observed to have improved color vibrancy and texture quality.

User studies further bolster these findings, indicating that images processed with PyNET are perceptually closer to DSLR quality compared to those processed by the native ISP of the Huawei P20. Moreover, PyNET exhibits potential for generalizability, displaying competent results when applied to images captured on a different smartphone model not included in its training set, albeit with some degradation in certain image aspects.

Conclusion and Future Prospects

This work underscores the viability of adopting a unified deep learning strategy to replace conventional ISP processes in mobile cameras. By training a neural network to handle the intricacies of image processing in an end-to-end manner, the authors pave the way for potentially lowering the dependency on bespoke hardware-specific solutions, thereby making high-end image quality more universally attainable on mobile devices. Future work should aim to further optimize model efficiency for deployment and explore its adaptability across a broader range of devices and sensors, which could accelerate industry adoption and set a new standard for computational photography.