White-box Weight Noising in Neural Networks

• White-box weight noising is a technique where stochastic noise is explicitly injected into neural network weights using known distributions to enhance transparency and interpretability.
• It employs methods such as independent Gaussian and colored noise injections to improve adversarial robustness and facilitate principled variational inference.
• The approach provides analytical tractability through closed-form variance propagation and efficient gradient updates, balancing robustness gains with modest clean accuracy trade-offs.

White-box weight noising refers to a class of techniques in which stochastic noise is explicitly injected into the weights of neural networks during training and/or inference, with all noise-generation mechanisms and parameters fully visible (“white-box”) to both the practitioner and—importantly—the adversary in threat settings. Distinct from black-box randomization or heuristic robustness tricks, white-box weight noising encompasses approaches where the distributional form, parameterization, and training protocol for noise are known and tractable, often leveraged for adversarial robustness, Bayesian inference, or regularization. Recent work distinguishes between “white” (i.e. independent Gaussian) and “colored” (i.e. correlated/LR) noise, and between fixed/noise-level–tuned and noise-level–learned methodologies. Notable frameworks include Fast Adaptive Weight Noise (FAWN), Pathwise Noise Optimization, and Colored Noise Injection (CNI) for adversarial defense.

1. Mathematical Formalism and Noise Models

White-box weight noising models neural network parameters not as fixed values but as random variables drawn from known distributions. The most principled formalism is to assign to each weight or bias $\theta_i$ a (learned) distribution $q(\theta_i)$, commonly factorized as

• Independent Gaussian: $q(\theta_i) = \mathcal{N}(\mu_i, \sigma^2_i)$, where each $\sigma_i$ can be fixed or optimized during training (Bayer et al., 2015).
• Correlated/Colored Gaussian: For the vector of weights $w \in \mathbb{R}^N$ in a layer, additive noise $\epsilon \sim \mathcal{N}(0, \Sigma)$ is injected, with covariance structure

$\Sigma = \Lambda + VV^\top$

where $\Lambda$ is diagonal (white noise) and $V \in \mathbb{R}^{N \times M}$ encodes a low-rank correlation structure (“coloring”) (Zheltonozhskii et al., 2020).

• Bernoulli/Binary Noise (Dropout-like): $b_i \sim \mathrm{Bern}(p_i)$, so that $q(\theta_i)$0, providing both mean and variance characterization (Bayer et al., 2015).

By controlling $q(\theta_i)$1, the practitioner can marginalize out uncertainty, regularize the model, or attempt to smooth the loss landscape to resist adversarial attacks. All first- and second-moment propagation calculations proceed analytically.

2. Optimization and Training Protocols

Optimization in white-box weight noising proceeds via explicit gradients with respect to both standard neural parameters (means $q(\theta_i)$2) and noise parameters (variances $q(\theta_i)$3, correlations $q(\theta_i)$4).

• Pathwise (Reparameterization) Gradients: For Gaussian noise injected per neuron pre-activation (e.g., $q(\theta_i)$5), gradients with respect to $q(\theta_i)$6 follow directly via

$q(\theta_i)$7

where $q(\theta_i)$8 is computed via standard backprop. Thus, noise parameters can be updated “for free” by accumulating $q(\theta_i)$9 during backprop, with negligible overhead versus conventional gradient calculations (Xiao et al., 2021).

• Variance Propagation / Moment Matching: In the FAWN framework, the means and variances of all intermediate activations are computed and propagated analytically through each layer, avoiding sampling and obviating high-variance MC estimators. This enables closed-form marginal likelihoods for output predictions and KL-regularized VI objectives (Bayer et al., 2015).
• Adversarial Objective (CNI): With colored noise, gradients are aggregated over both clean and adversarial mini-batches, and all noise-distribution parameters ($q(\theta_i) = \mathcal{N}(\mu_i, \sigma^2_i)$0, $q(\theta_i) = \mathcal{N}(\mu_i, \sigma^2_i)$1) are updated jointly with weights $q(\theta_i) = \mathcal{N}(\mu_i, \sigma^2_i)$2. An explicit $q(\theta_i) = \mathcal{N}(\mu_i, \sigma^2_i)$3-regularization on $q(\theta_i) = \mathcal{N}(\mu_i, \sigma^2_i)$4 constrains low-rank noise to prevent degenerate solutions (Zheltonozhskii et al., 2020).

Pseudocode for representative methods is given below (Pathwise Noise Optimization (Xiao et al., 2021)):

$w \in \mathbb{R}^N$7

3. Analytical Tractability and Variational Inference

A distinctive feature of white-box weight noising is its analytical tractability, allowing marginalization over the noise at each layer—“moment propagation”—without resorting to Monte Carlo sampling. In the case of FAWN, this enables closed-form approximations of both marginal likelihoods and predictive distributions:

• For a single layer with $q(\theta_i) = \mathcal{N}(\mu_i, \sigma^2_i)$5,

$q(\theta_i) = \mathcal{N}(\mu_i, \sigma^2_i)$6

$q(\theta_i) = \mathcal{N}(\mu_i, \sigma^2_i)$7

enabling layerwise propagation of mean and variance (Bayer et al., 2015).

The optimization objective can realize variational-Bayes, with a KL divergence term against the prior $q(\theta_i) = \mathcal{N}(\mu_i, \sigma^2_i)$8:

$q(\theta_i) = \mathcal{N}(\mu_i, \sigma^2_i)$9

where all terms are analytically computable via variance propagation.

4. White-Box Robustness and Empirical Results

In adversarial robustness settings, white-box weight noising explicitly assumes that the attacker knows all sources of randomness, noise levels, and their parameters. Adversarial attacks, e.g., FGSM, PGD, and L-BFGS, are applied using “Expectation over Transformation” to account for stochasticity (Xiao et al., 2021).

Empirical results highlight substantial gains in white-box and black-box robustness:

• Pathwise Noise Optimization: On MNIST, CIFAR-10, and Tiny-ImageNet, trainable per-neuron noise yields
  • FGSM (MNIST-MLP): baseline 0.149 $\sigma_i$0 0.295
  • PGD (CIFAR-10): baseline 0.114 $\sigma_i$1 0.203
  • PGD (Tiny-ImageNet): baseline 0.019 $\sigma_i$2 0.055
• Colored Noise Injection: For WideResNet-28-4 on CIFAR-10, injecting low-rank correlated noise achieves
  • PGD accuracy: PNI (rank 0) 53.3% $\sigma_i$3 CNI-W (rank 5) 55.8%
  • Classical Madry Adv. Training: 38.6%
  • TRADES: 56.5%

A summary table for CNI results on CIFAR-10 (WideResNet-28-4, PGD $\sigma_i$4) (Zheltonozhskii et al., 2020):

Method	Clean (%)	PGD (%)
Adv. training [Madry]	86.1	38.6
MMA [Ding et al.]	86.2	54.9
PNI [Rakin et al.]	84.6	53.3
CNI-W (ours)	84.4	55.8
TRADES [Zhang et al.]	84.9	56.5
MART [Zhang et al.]	83.6	57.3

These results demonstrate that learnable/noise-optimized defenses substantially increase robustness over non-noised or fixed-noise baselines in the fully disclosed (white-box) threat model.

5. Computational Complexity and Implementation Considerations

The computational cost of white-box weight noising is marginally higher than standard deterministic training:

• Pathwise Gradient Methods: Per-sample, one additional multiplication $\sigma_i$5 per neuron (negligible against standard gradient calculation) (Xiao et al., 2021).
• Variance Propagation (FAWN): Overall cost is $\sigma_i$6 for the forward pass, with a 2–3× constant factor overhead compared to a deterministic network. The backward pass presents similar scaling, due to the propagation of means and variances per layer (Bayer et al., 2015).
• Colored Noise (CNI): Increases the parameter count by $\sigma_i$7 per layer, for $\sigma_i$8 weights and rank $\sigma_i$9, and introduces sampling overhead for $w \in \mathbb{R}^N$0 in the noise computation (Zheltonozhskii et al., 2020).

6. Practical Trade-Offs, Limitations, and Future Directions

Key trade-offs are documented:

• Adversarial Robustness vs. Clean Accuracy: While white-box weight noising increases adversarial accuracy (by 2–3% absolute PGD gains for CNI), clean-set accuracy can decrease modestly (e.g., from 84.6% to 84.4% for WideResNet-28-4 on CIFAR-10 under CNI) (Zheltonozhskii et al., 2020).
• Hyperparameter Tuning: Colored noise requires selection of rank $w \in \mathbb{R}^N$1 and weight decay for $w \in \mathbb{R}^N$2; over-parameterization ($w \in \mathbb{R}^N$3) can degrade performance (Zheltonozhskii et al., 2020).
• Modeling Choices: Only Gaussian noise has been extensively studied; extensions to non-Gaussian forms (using normalizing flows) or adaptation of rank $w \in \mathbb{R}^N$4 per layer are open research topics (Zheltonozhskii et al., 2020).

This suggests further fusion of white-box weight noising with certified smoothing, batch-norm noise injection, or non-Gaussian parameterizations as promising avenues for improved adversarial robustness and uncertainty calibration.

7. Connections to Related Bayesian and Regularization Methods

White-box weight noising has substantive linkage to Bayesian neural networks, variational inference, and information-theoretic regularization:

• Bayesian Interpretation: Treating $w \in \mathbb{R}^N$5 as a factorized variational posterior enables minimization of the negative variational bound with analytic KL regularization (Bayer et al., 2015).
• Minimum Description Length: Empirical Bayes priors (MDL-inspired) can be incorporated, optimizing the regularized predictive distribution directly, as in FAWN-ROPD (Bayer et al., 2015).
• Relationship to Dropout and PNI: Standard dropout is a special case of Bernoulli-distributed weight noise; vanilla Parameter Noise Injection (PNI) is a CNI variant with $w \in \mathbb{R}^N$6 (pure white-noise, diagonal covariance) (Zheltonozhskii et al., 2020).

The empirical evidence consolidates white-box weight noising as a theoretically justified, computationally tractable mechanism for achieving robust, regularized, and fully interpretable stochasticity in deep learning architectures.