ReLU Denoising Autoencoder (DAE)

• ReLU Denoising Autoencoder is a neural network architecture that uses ReLU activations in a bottleneck encoder-decoder framework to effectively reconstruct high-dimensional signals.
• The design leverages feedforward or convolutional mappings and self-normalizing features to achieve rate-optimal noise reduction and stability across varying noise levels.
• Empirical results on datasets like MNIST and CelebA validate its low mean-squared error and robust performance in denoising, underpinning both practical and theoretical advancements.

A ReLU Denoising Autoencoder (DAE) is a neural network architecture designed for reconstruction and denoising of high-dimensional signals, leveraging feedforward or convolutional mappings with ReLU activation functions. The ReLU DAE performs dimensionality reduction via a bottleneck (encoder) and subsequent signal restoration via a decoder, achieving provable denoising performance rates and broad stability under varying noise regimes. Architectures such as the self-normalizing ReLU DAE (“NeLU”) extend classical sparse encoding principles to provide invariance against unknown test-time noise scales.

1. Architectural Principles and Formulation

The canonical ReLU DAE maps an input $y \in \mathbb{R}^n$ to a reconstruction $\hat{x} \in \mathbb{R}^n$ via a composition $F = D \circ E$, where $E: \mathbb{R}^n \to \mathbb{R}^k$ (with $k \ll n$) is the encoder and $D: \mathbb{R}^k \to \mathbb{R}^n$ is the decoder. Each module consists of linear transformations followed by entrywise ReLU activations, yielding a piecewise-linear mapping. For deep DAEs, the encoder and decoder are typically parameterized as multi-layer feedforward or convolutional neural networks (Heckel et al., 2018, Dhaliwal et al., 2021).

A representative model structure is:

• Encoder: $E(y) = \mathrm{ReLU}(W' y)$ or as a sequence of convolutional layers with ReLU functions.
• Decoder: $$G(x) = \mathrm{ReLU}(W_d\,\mathrm{ReLU}(W_{d-1}\ldots\mathrm{ReLU}(W_1 x)\ldots))$$.

Self-normalizing ReLU DAEs (“NeLU”) introduce an unrolled proximal-gradient solver enforcing noise invariance, formalized as the solution to a square-root lasso objective with a ReLU or soft-threshold nonlinearity (Goldenstein et al., 23 Jun 2024).

2. Denoising Mechanism and Rate-Optimality

The ReLU DAE is trained to minimize mean-squared error between reconstructed and clean signals, typically using additive Gaussian noise during the training phase. When observing $y = x + \eta$, with $\eta \sim \mathcal{N}(0, \sigma^2/n\,I_n)$, the residual energy of the noise in the reconstruction admits rigorous characterization.

Proposition (Rate-optimal denoising): If the active-masks (“ReLU patterns”) induce low-rank matrices $U$ where $\|U\|^2 \leq 2$ and the bottleneck dimension $k$ satisfies $k \cdot 32 \cdot \log(2 n_1 \cdots n_d) \leq n$, then with high probability

$\mathbb{E}[ \|H(\eta)\|^2 ] \leq C (k/n) \sigma^2,$

where $C = 5 \log(2n_1 \cdots n_d)$ is an architecture-dependent constant (Heckel et al., 2018).

This result shows that DAEs remove a fraction $O(k/n)$ of the noise energy, approaching optimality relative to subspace projection in high dimensions.

3. Theoretical Guarantees and Provable Recovery

Rigorous recovery guarantees are available for ReLU DAEs, including in the context of linear inverse problems. For an observation $y = Ax$ with $x \in S$ and $A$ satisfying a restricted isometry property (RIP) on $S$, projected gradient descent onto the range of a ReLU DAE $F$ yields geometric convergence:

$$\|x_T - x\| \leq (2\gamma)^T \|x_0 - x\| + \alpha \frac{1-(2\gamma)^T}{1-2\gamma}$$

for projection constant $\alpha$ and step control $$\gamma = \sqrt{ \eta^2 M(1+\delta) + 2\eta(\delta-1) + 1 }$$ (Dhaliwal et al., 2021). Under multi-scale Gaussian noise during training, the projection operator $F$ achieves a small constant $\alpha$ across evaluation conditions.

Self-normalizing ReLU DAEs exhibit invariance to noise level due to the pivotal regularization parameter $\lambda$, which can be set as $\lambda = a\sqrt{2\ln d/n}$ independently of the true $\sigma$. The analysis proves support recovery and estimation error bounds are unaffected by noise variance (Goldenstein et al., 23 Jun 2024).

4. Optimization Algorithms and Training Regimes

Standard ReLU DAEs employ feedforward architectures with strided convolutional layers and ReLU activations. DAEs are trained end-to-end using MSE loss between noisy inputs $x_i + e_i$ and clean targets $x_i$, with noise $e_i \sim \mathcal{N}(0, \sigma_i^2)$ spanning multiple scales within the training set (Dhaliwal et al., 2021).

The NeLU DAE architecture unrolls $N$ steps of an accelerated proximal-gradient algorithm for the pivotal lasso objective:

$$z^{(k+1)} = \max \left\{ z^{(k)} + \alpha v^{(k+1)} - \beta\lambda,\, 0 \right\}$$

with row-normalization of $W$, step-size tuning, and momentum $\alpha \approx 0.8$. The decoder applies the (pseudo)inverse $W^+$, often implemented as $W^T$ in convolutional networks (Goldenstein et al., 23 Jun 2024).

Empirical best practices include:

• Normalizing $W$ rows after each gradient update.
• Using $N \in [5,20]$ unrolled steps.
• AdamW optimizer with weight decay $10^{-4}$.
• Fixed $\lambda$ across train/test, enabling robust generalization.

5. Empirical Performance and Benchmarks

Numerical experiments validate theoretical denoising rates for various ReLU DAE topologies:

• Synthetic experiments: A two-layer generator with $n=1500$, $n_1=500$, varying $k$, and iid Gaussian weights $W$; reconstruction MSE scales as $O(k/n)$ with noise variance $\sigma^2$ (Heckel et al., 2018).
• MNIST and CelebA datasets: Deep convolutional DAEs achieve 10x lower MSE and >100x speedup in compressive sensing versus GAN-based methods, with no hyperparameter tuning required (Dhaliwal et al., 2021).
• Noise-level robustness: Self-normalizing NeLU DAEs demonstrate stable performance across a broad range of test-time noise levels, consistently outperforming classical ReLU architectures, with empirical PSNR improvements that widen with deviation from training $\sigma$ (Goldenstein et al., 23 Jun 2024).

6. Extensions: Generative Priors and Alternative Denoising Schemes

A related denoising strategy involves optimizing over the range of a generative model—finding $x \in \mathbb{R}^k$ such that $G(x)$ is closest to the noisy observation $y$. Under expansivity and Gaussian initialization assumptions, “sign-flip” gradient descent achieves an $O(k/n)$ noise reduction rate. In the noiseless case, exact recovery is possible (Heckel et al., 2018).

DAEs have also integrated VAE-style bottlenecks and multi-scale noise training (partitioning data by $\sigma_i$) to improve the range of effective denoising. Projected gradient descent algorithms using DAEs as priors substantially accelerate recovery in linear inverse problems (Dhaliwal et al., 2021).

7. Practical Implementation Considerations

Implementation guidelines include:

• Enforce row-normalization on $W$ in sparse auto-encoders and NeLU DAEs.
• Use $N=5$–$20$ proximal-gradient iterations (unrolling) for NeLU DAEs.
• Set the pivotal regularization parameter $\lambda = a\sqrt{2\ln d/n} / 2$ ($a = 3$–$5$).
• Adopt learning rate decay schedules and batch sizes ($64$–$256$).
• Evaluate reconstruction error via MSE or PSNR, ensuring the robustness of $\lambda$ without re-tuning for unknown test-time noise levels (Goldenstein et al., 23 Jun 2024).

A plausible implication is that DAEs employing self-normalizing mechanisms (“NeLU”) substantially alleviate the sensitivity to mismatch between training and testing noise levels, representing an advance in robust unsupervised and supervised denoising architectures.