Learned Denoising Networks (LDNets)

• LDNets are neural architectures that replace manual denoisers in iterative inference algorithms with learnable modules to achieve state-of-the-art performance.
• They unroll classical algorithms like AMP and ISTA, combining theoretical guarantees with empirical robustness in compressed sensing and image recovery.
• LDNets offer interpretable components and flexible training protocols that bridge classical estimation methods and modern deep learning for optimal denoising.

Learned Denoising Networks (LDNets) are a broad class of neural architectures that deliver state-of-the-art and, in key cases, theoretically guaranteed denoising performance by “unrolling” iterative inference algorithms—most notably Approximate Message Passing (AMP), Iterative Soft-Thresholding Algorithm (ISTA), and their generalizations—where classical algorithm steps are replaced by learnable neural network modules. LDNets facilitate provable Bayes-optimal inference, highly structured interpretability, and adaptability to prior uncertainty, with empirical advances across compressed sensing, rank-one estimation, and image denoising domains (Karan et al., 2024, Janjušević et al., 2021, Heckel et al., 2018, Janjušević et al., 2021).

1. Core Principles and Definitions

An LDNet is defined by the replacement of hand-crafted denoising functions within an iterative inference algorithm by parameterized neural mappings, which are then learned from data generated by a (possibly unknown) signal prior. This paradigm was initially motivated by the success of algorithm unrolling for sparse coding, but has found broader scope in generic linear and non-linear inverse problems. The network mirrors the iteration dynamics of an underlying estimator—such as AMP in compressed sensing—while its learnable denoisers enable adaptation to the true, possibly unknown, data distribution.

The LDNet architecture typically involves two classes:

• Unrolled inference networks: Each “layer” implements one iteration of the base algorithm, e.g., AMP or ISTA. Operators (linear transforms, thresholding, denoisers) are replaced by neural networks or parameterized convolutions.
• Latent-code denoising networks: A generative network (e.g., deep ReLU MLP) represents the data manifold, and denoising is achieved by projection/optimization onto this manifold or via an encoder-decoder.

2. Architecture: AMP Unrolling and Neural Denoisers

The canonical LDNet for linear inverse problems in compressed sensing is constructed by unrolling $L$ iterations of the AMP algorithm (Karan et al., 2024):

$$\begin{aligned} x^{(0)} &= 0 \ v^{(0)} &= y \ z^{(t)} &= A^\top v^{(t)} + x^{(t)} \ x^{(t+1)} &= f_t(z^{(t)}; \tau_t) \ v^{(t+1)} &= y - A x^{(t+1)} + \frac{1}{\delta} v^{(t)} \left\langle \partial_1 f_t(z^{(t)};\tau_t) \right\rangle \end{aligned}$$

where each $f_t$ is a learned neural denoiser (typically a small MLP, sometimes parameterized also by the estimated effective noise level $\tau_t$). The Onsager term, $$\frac{1}{\delta} v^{(t)} \langle \partial_1 f_t \rangle$$, is critical for ensuring that the iterates maintain the independent signal-plus-Gaussian-noise property.

Key architectural components:

• Learned scalar/vector denoisers: $f_t$ can be an MLP ($\mathbb{R} \to \mathbb{R}$) or a CNN ($\mathbb{R}^d \to \mathbb{R}^d$) for non-product priors.
• Auxiliary learned matrices: For non-Gaussian measurement operators, the transpose $A^\top$ is replaced by a trainable $B$, providing finite-sample flexibility.
• State evolution tracking: $\tau_t$ is estimated empirically at each layer, ensuring adaptive denoiser behavior.

Feedforward and autoencoder approaches, as in latent-code denoising networks (Heckel et al., 2018), leverage deep generator networks to either project noisy data to the image manifold or pass through an encoder-decoder, both achieving strong denoising in high-dimensional settings.

3. Training Protocols and Optimization

LDNets admit both end-to-end and layerwise training. Empirical findings across unrolling literature indicate that layerwise training—freezing all but the current layer at each phase—avoids poor local minima and enhances convergence toward Bayes-optimal denoisers (Karan et al., 2024). Algorithmic specifics include:

• Layerwise procedure: For each $t \in [0, L-1]$, parameters of $f_0, \dots, f_{t-1}$ are fixed. $f_t$ is initialized (optionally from $f_{t-1}$) and optimized via SGD or Adam on partial network loss.

$$\min_{f_t} \frac{1}{N} \sum_{i=1}^{N} \frac{1}{d} \| x^{(i)}_{t+1} - x^{(i)} \|^2$$

• Data generation: At each mini-batch, sample $(A, x, y)$ afresh (for random measurement models).

Other settings include end-to-end training with careful initialization. The final loss is typically the $\ell_2$ error between reconstructed and ground-truth signal; regularization and normalization choices mirror the requirements of the base iterative algorithm.

For convolutional dictionary learning networks (CDLNets), parameter learning involves untied (per-layer) convolutions and adaptive channelwise thresholds, trained with projected gradient descent to enforce positivity and norm constraints (Janjušević et al., 2021, Janjušević et al., 2021).

4. Theoretical Guarantees and Optimality

LDNets are distinguished by rigorous performance guarantees under broad statistical regimes. For unrolled AMP-based LDNets, the main proof (Karan et al., 2024) establishes:

• Exact Bayes-optimality: In the high-dimensional limit ($d \to \infty$), with Gaussian measurements and product priors, layerwise-trained LDNets provably achieve the same mean squared error as Bayes-AMP, matching the optimal minimum MSE.
• Parameter requirements: Neural denoiser widths and sample sizes scale polynomially in the approximation complexity of the Bayes denoising function and are independent of the ambient dimension.
• Proof techniques: Key ingredients include NTK-based gradient descent analysis for 1-D function fitting, rigorous state evolution reduction, and stability lemmas ensuring the robust transfer of convergence to the learned network.

A notable extension is the empirical superiority of LDNets when $A$ is non-Gaussian, when the prior is non-product, when both $B$ and denoisers are learned jointly, or in finite dimensional, low-sample regimes.

For latent-code denoising networks, projection onto the generative manifold affords a provable $O(k/n)$ noise-reduction rate, with both generator-only and autoencoder variants (Heckel et al., 2018).

5. Interpretability, Empirical Performance, and Practical Implementation

An important property of LDNets—especially those constructed by algorithm unrolling (e.g., CDLNet)—is that their components (filters, thresholds, denoisers) remain interpretable, since they can be mapped directly onto steps in traditional optimization or inference algorithms (Janjušević et al., 2021, Janjušević et al., 2021). Empirical observations include:

• Filter interpretability: Small CDLNets yield Gabor-like edge and a few texture atoms, while larger models learn bases covering a broad set of spatial primitives including edges, blobs, corners.
• State evolution of sparse codes: Deep layers induce stronger sparsity, reflecting increasing adherence to the learned $\ell_1$ prior.
• Noise-adaptive denoising: Thresholds parameterized as a function of estimated $\sigma$ generalize robustly to out-of-training-distribution noise levels.
• Empirical PSNR gains: In supervised and unsupervised settings, CDLNet matches or surpasses parameter-matched deep convolutional baselines (e.g., DnCNN, FFDNet), especially in blind denoising and joint demosaicing (Janjušević et al., 2021).

Autoencoder-based LDNets on image data, e.g., MNIST, demonstrate denoising performance scaling linearly with the code length over ambient dimension, confirming theory (Heckel et al., 2018).

6. Extensions, Limitations, and Research Directions

Current LDNet frameworks extend beyond classic scenarios:

• Non-product and structured priors: Vector denoisers learned as deep MLPs or CNNs enable denoising and inference under arbitrary prior geometries (Karan et al., 2024).
• Non-Gaussian and ill-conditioned measurement ensembles: Learning auxiliary operators such as $B$ within the unrolling allows robust adaptation; empirical evidence shows that LDNet outperforms optimal Bayes-AMP in such regimes by up to 7–37% NMSE in finite dimensions.
• General denoising vistas: Algorithmic unrolling has inspired LDNets under general iterative solvers, including ISTA/FISTA (e.g., CDLNet), with successful application to challenging tasks such as blind denoising, color demosaicing, and unsupervised denoising.

An open theoretical problem remains the absence of a full guarantee for high-dimensional vector denoisers with non-product priors, though empirical results are strong (Karan et al., 2024). Pragmatic guidance includes layer depth, denoiser width scaling with function complexity, and best practices for optimizer and initialization.

7. Connections to Broader Methodologies

Learned Denoising Networks provide a rigorous template for “learning to infer” in inverse problems by coupling the inductive bias of classical algorithms with the adaptability of deep learning. They offer a data-driven alternative to traditional analytical inference procedures, enabling both interpretability and near-optimal statistical efficiency without explicit prior knowledge. Their mechanism, bridging algorithm unrolling and neural function approximation, has led to widespread adoption in advanced computational imaging, signal processing, and scientific machine learning pipelines (Karan et al., 2024, Janjušević et al., 2021, Janjušević et al., 2021, Heckel et al., 2018).