Conditional Denoising UNet Architecture

• Conditional Denoising UNet is a neural architecture that extends the classic UNet by integrating noise-characterized conditioning for handling diverse and variable noise levels.
• It employs FiLM and RS-CFM modules for multi-layer feature modulation to achieve superior performance with high PSNR and SSIM on standard benchmarks.
• Its adaptable conditioning mechanism enables spatially-varying and continuous noise control in both image and speech denoising, streamlining model deployment.

A Conditional Denoising UNet is a deep neural architecture for noise reduction that extends the classical UNet by integrating explicit conditioning on parameters that characterize the noise process. This conditioning enables a single model to handle diverse and variable noise types and levels, and to interpolate or adapt its denoising behavior across a continuum of noise regimes. Such architectures are widely adopted in image and speech denoising, including non-blind and flexible (parametric or spatially-varying) scenarios, and are typically realized by modulating feature representations using side information (e.g., noise level, distribution, or controlling factors) at multiple layers.

1. Architectures and Conditioning Mechanisms

UNet serves as the structural backbone for conditional denoising. The baseline UNet is a fully convolutional network embodying an encoder-decoder path with skip connections, facilitating hierarchical feature extraction and precise spatial localization. Conditional Denoising UNets augment this setup via feature modulation driven by noise-characterizing parameters.

Feature-wise Linear Modulation (FiLM)

Distribution conditional denoising networks introduce FiLM layers after each convolutional block. The activations $\mathbf{h}_\ell(x)$ at each layer $\ell$ are modulated by noise parameters $\eta$ through an affine transformation:

$$a_\ell(x,\eta) = \gamma_\ell(\eta)\odot h_\ell(x) + \beta_\ell(\eta)$$

where $\gamma_\ell(\eta)$ and $\beta_\ell(\eta)$ are learned functions (typically MLPs) of the noise parameter(s), applying per-channel scaling and shifting (Kelly, 2020).

Multi-Layer Conditional Feature Modulation (CFM)

CFMNet advances the approach by embedding the noise level map throughout the network’s depth. A dual-stream UNet processes both the noisy image and the noise-level map, and multi-layer RS-CFM (Residual-Shifting Conditional Feature Modulation) modules are inserted at each encoder and decoder stage. Each RS-CFM module computes spatially varying shifts as functions of both the image features and noise-level features:

• At each layer, two streams produce features $f_\ell$ (image) and $g_\ell$ (noise map).
• The shift $s_\ell$ is computed by small conv subnets from the concatenated $[f_\ell; g_\ell]$.
• Modulation occurs as $\ell$0, where $\ell$1 is a transformed image feature. This mechanism enables better adaptation to local image structure and noise variability (Du et al., 2020).

2. Conditioning Inputs and Noise Modeling

Conditional Denoising UNets are designed to explicitly incorporate parameterizations of the noise distribution.

• Parameter Encoding: For additive Gaussian noise, $\ell$2 is the standard deviation. For Poisson-Gaussian, $\ell$3, with $\ell$4 modulating signal-dependent noise. Both scalar and spatially-varying maps are supported.
• Continuous State Variables for Speech: In continuous process modeling for speech, a state variable traverses from noisy to clean; noise control at inference is achieved by embedding a controlling factor $\ell$5 in $\ell$6, modifying the network’s operation to allow controllable denoising strength (Guo et al., 2023).

This conditioning enables the model to generalize across a continuum of noise scenarios, including unseen levels or complex, signal-dependent regimes.

3. Training Protocols and Objectives

Conditional Denoising UNets are trained to map noisy observations to the clean target, conditioned on the known (or estimated) noise parameters.

Objective Functions

• MSE Loss: Training typically uses mean squared error (MSE) between the denoised output and the ground truth, conditioned on both the input and noise parameters:

$\ell$7

where $\ell$8 applies parameterized noise (Kelly, 2020, Du et al., 2020).

• Residual Prediction: CFMNet applies MSE to the residual between the clean and noisy plus the network’s output, with loss:

$\ell$9

Training Schedules

• Distribution of Noise Levels: During training, each sample is corrupted with noise drawn from a prescribed (often uniform) distribution over permissible noise parameters to ensure coverage and interpolation capability.
• Two-Phase Training for FiLM-U-Net: Phase 1 freezes FiLM layers and trains the UNet core on high, fixed noise; Phase 2 freezes core weights and trains only FiLM parameters under varying noise (Kelly, 2020).

4. Inference, Generalization, and Adaptivity

Conditional Denoising UNets support user- or system-controlled adaptation at test time:

• Noise Parameter Control: Users specify any noise parameter $\eta$0 (including values unobserved during training), and the model denoises accordingly, interpolating across noise intensities or types.
• Spatially Varying Noise: By providing spatial maps (rather than scalars) of noise level, the network performs non-uniform, locally adaptive denoising without retraining (Kelly, 2020, Du et al., 2020).
• Controllable Speech Denoising: Embedded controlling factors modulate the degree of processed noise in the output, supporting application-dependent trade-offs in speech enhancement (Guo et al., 2023).

This adaptivity eliminates the need for deploying multiple specialized models and permits seamless operation in variable or unpredictable noise environments.

5. Quantitative Performance and Comparative Evaluation

Conditional Denoising UNets have achieved or surpassed state-of-the-art performance on standard image denoising benchmarks across a range of noise regimes.

Table: Example PSNR (dB) on CBSD68, Gaussian / Poisson Noise (Kelly, 2020)

$\eta$1	BM3D	FFDNet	FiLM-UNet
0.05	31.2/31.4	32.7/32.8	31.6/34.0
0.10	27.8/27.9	28.6/28.6	29.0/31.3
0.20	26.3/25.3	26.4/24.9	27.7/28.5
0.30	23.5/24.0	21.9/21.9	25.2/27.0

Table: SSIM on CBSD68 (Kelly, 2020)

$\eta$2	BM3D	FFDNet	FiLM-UNet
0.05	0.88/0.87	0.92/0.92	0.92/0.96
0.10	0.78/0.77	0.82/0.82	0.86/0.91
0.20	0.72/0.66	0.75/0.72	0.82/0.84
0.30	0.60/0.61	0.63/0.63	0.71/0.79

CFMNet yields superior PSNR/SSIM on both uniform and spatially varying Gaussian noise benchmarks, outperforming BM3D, FFDNet, and related approaches; for example, on Set12 (AWGN, $\eta$3), BM3D achieves $\eta$4 dB vs CFMNet's $\eta$5 dB (Du et al., 2020). Visual results indicate improved detail preservation, especially in textured or structurally complex regions.

6. Applications and Limitations

Conditional Denoising UNets are applied extensively in image denoising (with both additive white Gaussian noise and Poisson-Gaussian noise models) and speech enhancement. The architecture accommodates a variety of practical noise settings, including:

• Non-blind denoising: where noise parameters are known or estimated, e.g., from acquisition metadata or pre-processing.
• Continuous and controllable denoising: such as in speech enhancement, enabling application-specific modulation of residual noise (Guo et al., 2023).
• Spatially varying noise: e.g., in non-uniform imaging environments.
• Multi-task denoising: single models flexibly adapt across noise types and intensities.

Limitations include increased computational complexity compared to simpler modulations (e.g., FFDNet’s input-level shift), and focus on additive noise models; generalization to truly blind or non-Gaussian real noise scenarios may require upstream estimation or adversarial training (Du et al., 2020). Compared to single-layer or input-only modulation, multi-layer conditional modulation yields substantial gains in detail preservation for challenging tasks but at higher inference cost.

7. Summary and Future Extensions

Conditional Denoising UNets extend the classical UNet by multi-layer, parameter-aware feature modulation, supporting flexible, controllable, and generalizable denoising across diverse image and speech domains. Innovations in feature modulation (FiLM, RS-CFM) and conditioning strategies have led to performance improvements and broader applicability, including spatial adaptation and multi-task capabilities.

Suggested extensions include adopting full affine modulation (i.e., learning both scale and shift, instead of shift only), integration with non-local or attention mechanisms for improved texture recovery, and adaptation to blind denoising or cross-domain restoration tasks, as well as applications to deblurring, super-resolution, and burst denoising by swapping the backbone architecture (Du et al., 2020).