Non-Correlating Multiplicative Noise (NCMN)

• Non-Correlating Multiplicative Noise (NCMN) is a stochastic process characterized by i.i.d. multiplicative perturbations that lead to heavy-tailed stationary laws and feature decorrelation.
• It is applied across machine learning, stochastic optimization, and physical systems, where it enhances exploration, synchronization, and robustness through noise-induced dynamics.
• Empirical studies demonstrate that NCMN improves neural network generalization and aids in escaping narrow minima, with performance sensitive to hyperparameter scaling and network architecture.

Non-Correlating Multiplicative Noise (NCMN) refers to a type of stochastic perturbation where the noise acts multiplicatively on the system's state (rather than additively), with the crucial property that the multiplicative factors are independently and identically distributed (i.i.d.) across iterations or time, exhibiting no temporal correlation. NCMN arises in diverse contexts including stochastic optimization, machine learning regularization, and nonlinear physical systems under stochastic drives. The defining feature is the i.i.d. sampling of random scaling operators or fields, leading to distinctive statistical and dynamical phenomena such as heavy-tailed stationary laws, decorrelation of internal representations, and resonance-enhanced coherence.

1. Formal Definition and Characterization

In a general iterative or dynamical framework, NCMN can be captured by updates of the form:

$W_{k+1} = A_k W_k + B_k,$

where $A_k$ is a random linear operator (matrix) and $B_k$ an additive noise or drift term. The "non-correlating" aspect requires $\{A_k, B_k\}$ to be drawn i.i.d. at each step, with no memory or statistical dependence on previous steps. This form models, for instance, mini-batch SGD when linearized locally, with $A_k$ reflecting random local contraction/expansion rates and $B_k$ the stochastic gradient noise, both dependent on the current mini-batch (Hodgkinson et al., 2020).

A closely related scenario is encountered in physical systems, such as charge-density wave (CDW) dynamics, where the external drive is modulated as $V_\mathrm{noise}(t) = V_0(t)[1+\eta(t)]$ with $\eta(t)$ a zero-mean Gaussian white noise with vanishing correlation time:

$\langle \eta(t)\eta(t') \rangle = 2D\delta(t-t'),$

and updates are described by stochastic differential equations of the type:

$\frac{d\phi}{dt} = f(\phi) + g(\phi)\eta(t),$

again with noise $\eta$ sampled independently at every infinitesimal time increment (Dumas et al., 2012).

In deep neural networks, NCMN also refers to protocols for injecting multiplicative noise into activations or weights, modified to avoid inducing correlations between features, as standard multiplicative noise does (Zhang et al., 2018).

2. Discrete-Time Markov Recurrence, Stationary Law, and Heavy Tails

The iterated random affine system inherent in NCMN leads, under broad contractivity and ergodicity conditions, to a unique stationary law described by the fixed-point equation:

$W_\infty \stackrel{d}{=} A W_\infty + B,$

where $(A,B)$ are independent of $W_\infty$. The Kesten–Goldie theorem determines that, when $A$ is non-degenerate, the stationary distribution of $W_\infty$ exhibits heavy tails:

$\mathbb{P}(\|W_\infty\| > t) \sim C t^{-\alpha}$

for $t \to \infty$, with the tail index $\alpha$ determined by $E[\sigma_{\min}(A)^\alpha]=1$ and further integrability conditions (Hodgkinson et al., 2020). Notably, this heavy-tailed behavior can arise even if $B$ is strictly light-tailed, indicating that multiplicative fluctuations alone govern the high-probability outliers in the stationary distribution.

3. Parameter Dependence and Exploration–Exploitation Tradeoff

The statistical properties of NCMN in stochastic optimization depend on algorithmic hyperparameters and data statistics. In the canonical linear regression setting with

$A = I - \eta(XX^\top + \lambda I)/n,$

where $X$ is a batch of data vectors and $n$ is the batch size, the tail exponent $\alpha$ solving

$$E[|\mathrm{1} - \eta \lambda_{\min}(XX^\top)/n|^\alpha] = 1$$

is a function of step size $\eta$, batch size $n$, and data dispersion. Larger $\eta$ and smaller $n$ increase spectral fluctuations in $A$, decrease $\alpha$, and thereby yield heavier tails. Similarly, greater data variability enhances exploration via heavier-tailed updates (Hodgkinson et al., 2020). This mechanism has been linked to improved ability to escape narrow local minima in non-convex loss landscapes ("basin hopping"), especially early in training, whereas subsequent step size annealing reverts to more localized, light-tailed stationary behavior.

4. NCMN in Deep Neural Networks and Feature Decorrelation

Multiplicative noise has long been used as a regularization technique (e.g., dropout, dropconnect) in deep networks. Traditional multiplicative noise, however, has been shown to increase pairwise correlations between features, as the network is implicitly pushed to maximize the signal-to-noise ratio by aligning activations (Zhang et al., 2018). NCMN methods for DNNs are designed to remove this correlation effect. This is achieved by (i) decomposing activations into signal and noise components, (ii) applying batch normalization solely to the signal, and (iii) injecting the noise component as a "constant" during backprop, blocking gradients through the noise term:

$$\hat{z}_j' = \mathrm{BN}(z_j^s) + \mathrm{AsConst}(\mathrm{BN}(z_j) - \mathrm{BN}(z_j^s)),$$

where the operator $\mathrm{AsConst}(\cdot)$ ensures the noise does not bias feature learning (Zhang et al., 2018). This approach consistently reduces feature correlations after batch normalization and improves generalization, outperforming both standard multiplicative noise and two-branch "shake-shake" regularization techniques.

5. Physical Systems: NCMN and Noise-Enhanced Coherence

In materials physics, non-correlating multiplicative noise manifests in systems such as CDWs under voltage pulses modulated by fast, uncorrelated noise. Experimental and theoretical analyses show that such NCMN can reduce average signal amplitudes, shift the onset of dynamic transitions, and create broad maxima in observables such as the normalized signal drop $\Delta V / V$ as a function of drive amplitude (Dumas et al., 2012). These effects correspond to noise-enhanced synchronization and coherence (stochastic resonance) when the noise intensity matches the scale of deterministic forcing. The mathematical description is provided by Langevin and Fokker–Planck equations with multiplicative white noise terms.

6. Empirical Evidence and Extensions to Advanced Optimizers

Empirical investigations of NCMN in optimization scenarios include analyses of SGD on high-dimensional tasks, where principal component projections of parameter trajectories and step norm statistics confirm the presence of heavy-tailed dynamics predicted by theory ($\alpha \approx 1.5$) (Hodgkinson et al., 2020). Basin hopping and escape from narrow minima are observed to be more prevalent relative to purely additive-noise schemes. Similar random linear recurrence principles apply to optimization algorithms incorporating momentum, Adam, or stochastic Newton steps, with the caveat that heavy tails persist only if the random Lipschitz components can stochastically expand; in Adam, the normalization mechanism often dampens the effect, yielding lighter tails.

7. Practical Implementation and Recommendations

NCMN in neural networks is simple to implement in BN-equipped architectures. The method termed NCMN$_0$, where noise is injected post-normalization, demands minimal code modifications and matches the capacity of standard networks. On architectures such as ResNets, versions (e.g., NCMN$_2$) aligned with residual blocks yield strong performance improvements, comparable to, but more flexible than, shake-shake (Zhang et al., 2018). Hyperparameter sweeps for noise variance suggest linear scaling with network width is effective. While batch normalization is integral to current NCMN implementations, future work could consider layer or group normalization to extend the decorrelating mechanism beyond BN.

---

Key References:

• "Multiplicative noise and heavy tails in stochastic optimization" (Hodgkinson et al., 2020)
• "Removing the Feature Correlation Effect of Multiplicative Noise" (Zhang et al., 2018)
• "Multiplicative noise and charge density wave dynamics in Rb0.30MoO3" (Dumas et al., 2012)