Noisy Linear Layers in Neural Networks

• Noisy linear layers are defined as linear transformations incorporating explicit stochastic noise to boost robustness against adversarial inputs and hardware-induced perturbations.
• They are analyzed using mathematical formalisms that reveal dynamics like weight norm scaling in additive regimes and self-binarization in multiplicative noise settings.
• Empirical studies demonstrate that appropriate noise injection strategies enhance adversarial defense, semi-supervised learning, and performance in hardware-constrained environments.

Noisy linear layers are linear transformations in neural networks subject to explicit stochastic perturbations, generally modeled as additive or multiplicative noise on their activations or weights. These constructs are essential for analyzing and improving robustness to both adversarial inputs and inherent computational noise, especially in hardware-constrained or semi-supervised learning settings.

1. Mathematical Formalisms of Noisy Linear Layers

A canonical noisy linear layer computes its pre-activation as

$z = W x + b,$

where $W\in\mathbb{R}^{m\times n}$ is the weight matrix, $x\in\mathbb{R}^n$ is the input, and $b\in\mathbb{R}^m$ the bias. "Noise injection" consists of either:

• Additive noise on activations:

$$\tilde{z} = W x + b + \epsilon_\text{add}, \quad \epsilon_\text{add}\sim\mathcal{N}(0, \sigma^2 I_m)$$

• Multiplicative noise on activations:

$$\tilde{z} = \epsilon_\text{mul} \odot (W x + b), \quad \epsilon_\text{mul}\sim\mathcal{N}(1, \sigma^2 I_m)$$

• Additive (or multiplicative) noise on weights:

$$\tilde{z} = (W + \Delta W)x + b, \;\; \Delta W\sim\mathcal{N}(0, \sigma^2 I_{m\times n})$$

Noise injection is generally performed at the layer's output (post-linear, pre-activation for the next layer) and can be employed either globally (across all layers) or at a specific layer ("walking noise") (Borras et al., 2022).

2. Layerwise Robustness Under Additive and Multiplicative Noise

The "Walking Noise" methodology measures robustness to noise per layer by sweeping the noise variance $\sigma^2$ from 0 until accuracy degrades to random-guess levels. In additive noise regimes, network weights spontaneously grow in magnitude to maintain a high signal-to-noise ratio, exploiting the fact that increasing $\|W\|_F$ raises SNR:

$\mathrm{SNR}_l \approx \frac{\|W^l\|_F}{\sigma}$

In contrast, multiplicative noise induces self-binarization in layer activations: output distributions become bimodal, rendering the network robust even for extreme $\sigma$, as only the binary presence/absence of activations carries information. BatchNorm stabilizes both effects, enabling large weights (additive) and consistent binarization (multiplicative) (Borras et al., 2022).

Layerwise findings are:

Layer Type	Max Robustness (μ)	Typical Failure Mode
Hidden layers	Highest	Collapse at extreme σ
Input/output	Lowest	Early/late signal corruption

3. Information-Theoretic Properties in Gaussian Noisy Linear Layers

The mutual information and minimum mean-squared error (MMSE) of noisy linear layers under additive Gaussian noise can be characterized by additive Gaussian-noise transforms (Reeves, 2017). For $y = A x + w$, $w \sim \mathcal{N}(0, \sigma^2 I)$, and prior $P_X$, the per-layer information quantities admit closed-form state-evolution formulas:

• Per-component mutual information:

$$\frac{1}{N} I(x; y) = \min_{0 \leq u \leq M_X(0)} \left\{ I_X^*(u) + J_{A^T A}(u) \right\}$$

where $I_X^*(u)$ is the convex dual of I-MMSE and $J_{A^T A}(u)$ encodes the spectrum of $A$.

• For $L$ stacked noisy linear layers, the total "Legendre potential" is the sum of each layer's transform:

$$I^*_{x^{(0)} \to x^{(L)}}(u) = I_X^*(u) + \sum_{\ell=1}^L J^*_{A_\ell^T A_\ell}(u)$$

Additivity extends to phase transitions, with noise-induced changes in the number of optimal MMSE solutions marking statistical boundaries in representation capacity under noise (Reeves, 2017).

4. Empirical and Theoretical Insights From Hardware and Numerical Studies

In analog-hardware settings, noise is classified as either local (uncorrelated) or global (correlated). Analytical and simulation results for deep and recurrent feedforward networks with noisy linear nodes indicate:

• For a single-node:

$$\operatorname{SNR}(y) = \frac{E[y']}{\sqrt{\sigma^2_{\rm add} + \sigma^2_{\rm mult} (E[y'])^2}}$$

Additive noise SNR grows linearly with signal, multiplicative noise imposes a constant SNR floor.

• Multi-layer deep networks average out uncorrelated noise, but correlated noise accumulates with depth and fundamentally limits network-level SNR.
• Mitigation strategies: minimize correlated noise sources, duplicate input neurons for first-layer decorrelation, enforce high last-layer fan-in, and design feedback parameters in RNNs to limit correlated amplification.

Under realistic noise parameters, these measures can improve SNR by up to an order of magnitude, confirming the practical resilience of noisy linear architectures (Semenova et al., 2019).

5. Implications for Robustness, Adversarial Defense, and Semi-supervised Learning

Noisy linear layers play several roles depending on context:

• Adversarial Robustness: Inference-time Gaussian noise in inputs or logits, combined with model ensembles and majority-voting, produces a "smoothed classifier" to which certified $\ell_2$-robustness guarantees (randomized smoothing) apply. The certified radius is

$R = \sigma \cdot \Phi^{-1}(\underline{p}_A)$

where $\underline{p}_A$ is a lower bound on the correct class's probability under random noise. This approach was shown to require larger adversarial perturbations to successfully attack, especially on CIFAR-10 (Liang et al., 2020).

• Semi-supervised Training: In the presence of noisy pseudo-labels, linear classifier layers are particularly susceptible to destructive gradients. The Grad-ReLU strategy, which zeros out noisy-unsupervised gradients for the linear layer, isolates the feature extractor's learning from label noise while preserving classifier accuracy. Empirically, this raises classification accuracy by over 10% on strong benchmarks compared to baseline noisy label approaches (Liang et al., 2024).

6. Practical Considerations and Implementation in Deep Learning Systems

When implementing noisy linear layers, several operational choices are critical:

• Training vs. Inference Injection: Noise can be injected only at inference (e.g., for robustness certification (Liang et al., 2020)) or during training (to induce feature-level noise immunity (Borras et al., 2022)). Training-time injection creates opportunities for the network to adapt its parameters (e.g., by weight magnification or activation binarization).
• Gradient Handling: Backward propagation through noise (straight-through on clean activations (Borras et al., 2022)) simplifies analysis and avoids creating invariances that can be exploited by adversaries (Liang et al., 2020).
• Hardware Constraints: On analog or photonic accelerators, mixtures of additive and multiplicative noise modes are present; defense mechanisms (weight growth, binarization) must be tuned to the empirical noise model (Borras et al., 2022, Semenova et al., 2019).

7. Summary Table of Noise Types and Effects

Noise Type	Typical Effect	Key Defense Mechanism	Contexts Addressed
Additive	SNR drops with $\sigma$	Weight norm increase	Hardware, Adversarial, SSL
Multiplicative	SNR saturates (binarization)	Self-binarization of activ.	Hardware, Resilience
Pseudo-label	Harmful linear gradients	Gradient masking (Grad-ReLU)	Semi-supervised Learning
Input/Logit	Robustness via smoothing	Ensemble, majority vote	Adversarial

This typology captures the distinctive structural, statistical, and algorithmic consequences of noise in linear layers across both theoretical and applied domains.