WaterGAN: Unsupervised Generative Network to Enable Real-time Color Correction of Monocular Underwater Images (1702.07392v3)

Abstract: This paper reports on WaterGAN, a generative adversarial network (GAN) for generating realistic underwater images from in-air image and depth pairings in an unsupervised pipeline used for color correction of monocular underwater images. Cameras onboard autonomous and remotely operated vehicles can capture high resolution images to map the seafloor, however, underwater image formation is subject to the complex process of light propagation through the water column. The raw images retrieved are characteristically different than images taken in air due to effects such as absorption and scattering, which cause attenuation of light at different rates for different wavelengths. While this physical process is well described theoretically, the model depends on many parameters intrinsic to the water column as well as the objects in the scene. These factors make recovery of these parameters difficult without simplifying assumptions or field calibration, hence, restoration of underwater images is a non-trivial problem. Deep learning has demonstrated great success in modeling complex nonlinear systems but requires a large amount of training data, which is difficult to compile in deep sea environments. Using WaterGAN, we generate a large training dataset of paired imagery, both raw underwater and true color in-air, as well as depth data. This data serves as input to a novel end-to-end network for color correction of monocular underwater images. Due to the depth-dependent water column effects inherent to underwater environments, we show that our end-to-end network implicitly learns a coarse depth estimate of the underwater scene from monocular underwater images. Our proposed pipeline is validated with testing on real data collected from both a pure water tank and from underwater surveys in field testing. Source code is made publicly available with sample datasets and pretrained models.

• The paper introduces an unsupervised GAN that simulates underwater light effects with a modular generator design for attenuation, scattering, and camera modeling.
• It employs a two-stage restoration pipeline that demonstrates improved color accuracy and consistency in varied depth conditions.
• The approach achieves real-time processing efficiency, enabling advanced applications in marine imaging such as object detection and 3D reconstruction.

Overview of WaterGAN: Unsupervised Generative Network for Color Correction of Underwater Images

This paper introduces WaterGAN, an unsupervised generative adversarial network developed to address the complex problem of color correction in monocular underwater images. The process of light propagation underwater introduces unique challenges such as absorption and scattering that affect image quality. The proposed methodology offers a novel approach to enhancing underwater vision systems, crucial for fields ranging from marine archaeology to coral reef monitoring.

Technical Contributions

WaterGAN's architecture is designed to generate realistic underwater images from in-air images coupled with depth information, leveraging advancements in deep learning models. The network addresses the difficulty of acquiring large datasets for supervised learning by using unsupervised learning mechanisms. The primary contributions include:

• Modular Generator Design: WaterGAN comprises three main stages - attenuation, scattering, and camera modeling - each reflecting real-world underwater image formation processes. The attenuation module simulates light decay, the scattering module uses depth information and noise inputs to simulate backscattering, and the camera model applies vignetting effects.
• Image Restoration Pipeline: Beyond data generation, the paper introduces a two-stage network for image restoration. This network corrects for underwater image distortions in real time, using the synthetic data produced by WaterGAN as training inputs.

Results

The proposed approach is validated using both controlled (pure water tank) and real-world field data (Port Royal, Jamaica, and Lizard Island, Australia). Experimental results demonstrate:

• Color Accuracy: The method outperforms existing strategies like histogram equalization and physical model-based corrections, especially in cases with varying depth-induced color changes.
• Color Consistency: WaterGAN notably improves the consistency of color across images from different perspectives, an essential quality for coherent underwater imaging.
• Processing Efficiency: The restoration network operates with computational efficiency suitable for real-time applications, processing images in milliseconds.

Discussion and Implications

The introduction of WaterGAN has significant implications for underwater imaging. The ability to generate realistic training data for neural networks can overcome the scarcity of labeled underwater images, circumventing a major bottleneck in deep learning applications under the sea. Moreover, the model demonstrates potential beyond restoration, including tasks like object detection and 3D reconstruction.

Theoretical implications relate to the effectiveness of augmented generators in GANs, where the task-specific modeling of image formation processes enhances the realism of generated samples. The paper also explores limitations, such as assumptions in the vignetting model, and suggests opportunities for enhancements through additional parameterization.

Future Developments

Potential enhancements to WaterGAN include joint training of WaterGAN and the restoration network for more cohesive model optimization. Further, exploring the application of WaterGAN in different environmental conditions and adapting the approach for domain transfer present intriguing prospects. These developments could expand the usability and robustness of underwater imaging systems across diverse marine environments.

In conclusion, WaterGAN presents a refined, technically nuanced approach to underwater image restoration, offering both practical advancements and a foundation for future research in this challenging domain.