Learning to Generate Realistic Noisy Images via Pixel-level Noise-aware Adversarial Training (2204.02844v2)

Abstract: Existing deep learning real denoising methods require a large amount of noisy-clean image pairs for supervision. Nonetheless, capturing a real noisy-clean dataset is an unacceptable expensive and cumbersome procedure. To alleviate this problem, this work investigates how to generate realistic noisy images. Firstly, we formulate a simple yet reasonable noise model that treats each real noisy pixel as a random variable. This model splits the noisy image generation problem into two sub-problems: image domain alignment and noise domain alignment. Subsequently, we propose a novel framework, namely Pixel-level Noise-aware Generative Adversarial Network (PNGAN). PNGAN employs a pre-trained real denoiser to map the fake and real noisy images into a nearly noise-free solution space to perform image domain alignment. Simultaneously, PNGAN establishes a pixel-level adversarial training to conduct noise domain alignment. Additionally, for better noise fitting, we present an efficient architecture Simple Multi-scale Network (SMNet) as the generator. Qualitative validation shows that noise generated by PNGAN is highly similar to real noise in terms of intensity and distribution. Quantitative experiments demonstrate that a series of denoisers trained with the generated noisy images achieve state-of-the-art (SOTA) results on four real denoising benchmarks. Part of codes, pre-trained models, and results are available at https://github.com/caiyuanhao1998/PNGAN for comparisons.

Analysis of "Learning to Generate Realistic Noisy Images via Pixel-level Noise-aware Adversarial Training"

The paper "Learning to Generate Realistic Noisy Images via Pixel-level Noise-aware Adversarial Training" presents an advanced framework for generating realistic noisy images to improve the training of real denoisers, which traditionally require extensive noisy-clean image pairs. Capturing such pairs in real-world scenarios is costly, prompting this exploration into synthetic data generation.

Central to this work is the introduction of a Pixel-level Noise-aware Generative Adversarial Network (PNGAN), which is designed to model the complexities of real camera noise more accurately than existing synthesis techniques. Conventional approaches often rely on applying Additive White Gaussian Noise (AWGN) or modeling noise based on Poisson-Gaussian distributions coupled with an ISP pipeline for RGB images. These methods, however, fall short in capturing the nuanced, signal-dependent, and device-specific characteristics of real-world noise, leading to challenges in aligning synthesized with actual noisy data.

Core Methodologies

Pixel-level Noise Model

The authors propose a novel pixel-level noise model where each noisy pixel is treated as a random variable. This model is coupled with a real denoiser network to separate image domain alignment from noise domain alignment, enhancing the generator's ability to produce noise that closely resembles real camera noise.

Structure of PNGAN

PNGAN employs a pre-trained denoiser to direct the generator towards producing noise-free representations, facilitating image domain alignment. Concurrently, a pixel-level discriminator ensures noise domain alignment through adversarial training. This dual alignment approach aims to minimize the distributional discrepancies between synthetic and real noisy images.

Simple Multi-scale Network (SMNet)

A specialized network architecture, SMNet, serves as the backbone of the generator in PNGAN. Designed for efficient noise fitting, this architecture leverages multi-scale feature aggregation and shift-invariant downsampling to model complex noise patterns effectively. SMNet balances computational efficiency with architectural capacity, operating with only 0.8M parameters while capturing intricate spatial representations.

Empirical Analysis

The paper presents rigorous validation across multiple benchmarks, demonstrating that denoisers trained on PNGAN-generated data achieve state-of-the-art results compared to those trained on either real noisy datasets or traditional synthetic images. Specifically, PNGAN-generated data significantly narrows the domain discrepancy, as evidenced by Maximum Mean Discrepancy (MMD) evaluations. These evaluations point to a substantial reduction in alignment errors across widely used real-world denoising datasets like SIDD, DND, PolyU, and Nam.

Implications and Future Prospects

PNGAN's architecture and training framework suggest several broader implications beyond immediate denoising tasks. The methodology could be extrapolated to other restoration tasks where data acquisition is non-trivial, such as super-resolution or inpainting. Furthermore, the pixel-level adversarial training paradigm introduced could inspire future works to refine noise modeling in more diversified and context-dependent imaging conditions, addressing variations at a granular level.

Conclusion and Further Research

The contributions of this paper are twofold: providing a framework to synthesize realistic noise and extending the efficacy of deep learning models in denoising through synthetic augmentation. While the document signals advancements in real image synthesis for denoising applications, future research may focus on extending these concepts to broader datasets and more heterogeneous environmental conditions, potentially integrating real-time adaptation of denoisers as part of an active learning pipeline. This could lead to models that dynamically improve from continuous real-world usage, bridging the gap between synthetic and real data with minimal human oversight. As AI models evolve, the ability to leverage such frameworks could curtail the resources traditionally earmarked for high-fidelity data acquisition, rendering robust models more accessible across varying computational landscapes.