Local Correlation Regularization

• Local correlation regularization is a method that enforces structured dependencies among neighboring weights, activations, or image patches to reduce redundancy.
• It uses techniques such as correlated noise injection, local decorrelation penalties, and topological persistence to improve model robustness and generalization.
• Empirical results across image classification and inverse problems demonstrate that these strategies improve accuracy and mitigate overfitting in deep learning models.

Local correlation regularization refers to a class of regularization strategies in neural networks and inverse problems where either model weights, activations, or image patches are encouraged to exhibit (or avoid) certain correlation structures at a local scale. The aim is often to reduce redundancy, enhance robustness, or enforce prior statistical structure, especially motivated by properties observed in biological neural systems or in natural signals.

1. Correlation-Based Regularization in Neural Networks

Local correlation regularization in deep learning primarily targets the dependencies among neurons, filters, or activations, frequently through explicit mathematical constraints or noise-injection mechanisms. Unlike global decorrelation, local correlation approaches often act only on subsets, such as spatially close filters, positively correlated pairs, or small groups in a feature space, reflecting the hypothesis that structured (but not total) redundancy is beneficial.

One biologically inspired method directly implements correlated stochasticity: in "Convolutional Neural Networks Regularized by Correlated Noise" (Dutta et al., 2018), correlated Gaussian noise is injected into CNN activations during training. The pairwise correlation $r(i, j)$ between units $i$ and $j$ at a given layer is defined as

$$r(i, j) = [a - b d_{ij}]_+ \exp\Bigl(\frac{k_{ij} - 1}{\tau} + c\Bigr)$$

where $d_{ij}$ is the Euclidean distance between the spatial locations of the units, $k_{ij}$ is the cosine similarity between their convolutional kernels, and $a, b, c, \tau$ are hyperparameters controlling spatial and tuning dependence. All $r(i, j)$ are assembled into a correlation matrix $\Sigma$, which determines the covariance of the added Gaussian noise.

Alternatively, decorrelation strategies, as in OrthoReg (Rodríguez et al., 2016), penalize only positively correlated filter pairs in weight space, leaving negatively correlated (inhibitory or redundant) feature detectors unaffected, and focus their loss around small angular neighborhoods of filter similarity.

Persistence-based approaches (Ballester et al., 2023) leverage topological constructs: by computing the minimum spanning tree (MST) over neuron activations (where edge weights are correlation dissimilarities), the regularizer penalizes only the highest-magnitude neuron correlations in a batch, preserving necessary redundancies.

2. Mathematical Formulations and Mechanisms

Different realizations of local correlation regularization reflect distinct mathematical philosophies:

Correlated Noise Regularization

The noise vector $\mathbf{z} \sim \mathcal{N}(\mathbf{0}, \Sigma)$ is sampled using a differentiable reparameterization, typically via Cholesky decomposition: $$\mathbf{z} = L \mathbf{X},\quad \mathbf{X} \sim \mathcal{N}(0, I),\quad \Sigma = LL^\top$$ This stochastic variable is added to the deterministic pre-activation: $\mathbf{h} = \mathbf{a} + \mathbf{z}$ Gradients flow through $\Sigma$ into the original weights thanks to differentiability of both the Cholesky operation and the noise sampling (Dutta et al., 2018).

Local Decorrelating Penalties (Weight Space)

OrthoReg defines a weight-based local regularizer acting on the (normalized) cosine similarities among filters: $$R_{\text{ortho}}(\Theta) = \sum_{i \neq j} \log\left(1 + \exp\left[\lambda (\theta_i^\top \theta_j - 1)\right]\right)$$ where $\Theta \in \mathbb{R}^{n \times d}$ collects $n$ normalized filter vectors and $\lambda$ controls locality of the penalty. Only pairs with $\theta_i^\top \theta_j > 0$ are penalized significantly (Rodríguez et al., 2016).

Topological Persistence-Based Regularization

The MST-based regularization proceeds by building a graph over neuron activations with dissimilarities $d(u, v) = 1 - |\mathrm{corr}(u, v)|$. Regularizers include: $$\mathcal{T}_1 = -\sum_{i=1}^{m} y_i,\qquad \mathcal{T}_2 = -\alpha \bar{y} + \beta \sigma_y$$ where $\{y_i\}$ are the $m = |V'| - 1$ edge weights of the MST, with $\bar{y}$ their mean and $\sigma_y$ their standard deviation (Ballester et al., 2023).

3. Integration and Interaction with Other Regularizers

Local correlation regularization mechanisms are generally incorporated as either a direct modification of the forward pass (stochastic noise injection) or an explicit term in the loss function (decorrelation penalties). In the correlated noise framework, no explicit penalty is added; rather, the noise covariance structure enforces implicit regularization. For loss-based decorrelation or persistence regularizers, a weighted sum augments the task loss: $$\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{task}} + \gamma R_{\text{ortho}} \quad \text{or} \quad \mathcal{L}_{\text{total}} = \mathcal{L}_{\text{task}} + \omega \mathcal{T}_{k}$$ where $\gamma, \omega$ are tunable hyperparameters.

Practically, such local correlation regularizers are compatible with standard methods like dropout or batch normalization. Experiments in (Dutta et al., 2018) and (Rodríguez et al., 2016) demonstrate that the combination with dropout is additive, with correlated noise yielding robust improvements on occluded image variants—even when dropout, batch normalization, or advanced initialization schemes (e.g., LSUV) are used.

4. Empirical Studies and Application Contexts

Empirical evaluation of local correlation regularization typically focuses on generalization performance, robustness to corruption, and reduction of overfitting. Key experimental results include:

• (Dutta et al., 2018): On CIFAR-10, correlated Gaussian noise injected in the first convolutional layer increased baseline clean set accuracy from 75.5% to 80.8%. On occluded test sets, improvements of 5–12 points over non-regularized baselines were observed. The benefit extended to the combined use of dropout and correlated noise.
• (Rodríguez et al., 2016): OrthoReg decreased test error rates across MNIST, CIFAR-10/100, and SVHN, notably reducing overfitting gaps. For example, on Wide ResNet v2, test error dropped from 3.89% to 3.69% (CIFAR-10).
• (Ballester et al., 2023): On CIFAR-10 with VGG-like architectures, MST-based regularization outperformed both the absence of correlation regularization and naive all-pair decorrelation, with test accuracy improving from ~68.5% (no reg) or ~68.0% (all-pairs) to ~69.1% ($\mathcal{T}_1$) and ~69.2% ($\mathcal{T}_2$).

In inverse problems, such as image denoising or deblurring, local regularization is constructed over small image patches by learning a critic network to distinguish clean from degraded patches, as presented in (Prost et al., 2021). The regularizer aggregates local scores, enforcing global structure via local constraints learned from data.

5. Theoretical and Biological Motivation

The empirical and theoretical grounding of local correlation regularization aligns with both information-theoretic and neuroscience perspectives. In biological systems, correlated variability among nearby neurons or similarly tuned populations is a persistent observation, and is hypothesized to contribute to robustness against partial corruption and to enhance the representation of invariant features (Dutta et al., 2018).

From an optimization standpoint, local regularization of correlations (as opposed to blanket global decorrelation) preserves necessary redundancies, avoids destructive interference between divergent directions in high-dimensional parameter or feature spaces, and guarantees the preservation of global connectivity or information flow, e.g., via MST-based constructions (Ballester et al., 2023). These strategies often outperform global decorrelation, which can inadvertently penalize beneficial complementarity among features.

6. Computational Considerations and Limitations

Many correlation regularization schemes involve nontrivial computational overhead:

• For correlated noise injection, constructing and factorizing a dense $d \times d$ correlation matrix $\Sigma$ using Cholesky decomposition has $O(d^3)$ cost per batch, hampering scalability beyond early convolutional layers. Practical implementations often restrict noise to the first layer or use low-rank approximations (Dutta et al., 2018).
• Persistence-based regularization requires batch-wise computation of Pearson correlations and MST extraction. In large models, importance sampling of neurons or filters is necessary, and strategies such as GPU implementations of Kruskal’s algorithm are suggested for efficiency (Ballester et al., 2023).
• Weight-space local regularizers like OrthoReg (Rodríguez et al., 2016), acting on pooled filter matrices, are orders of magnitude less costly than decorrelating activations and integrate readily within batched SGD or Adam updates.

A further open question concerns the generalization of regularization hyperparameters, such as $(a, b, c, \tau)$ in correlated-noise models and the selection of penalty weights. Scaling to very deep architectures, learning tasks beyond vision, and the extension to layer-wise or structured (e.g., Toeplitz) correlation models remain active research areas.

7. Advances in Local Regularization for Inverse Problems

In variational image restoration, local correlation regularization is learned via adversarial approaches:

• (Prost et al., 2021) employs a patch-based critic trained by Wasserstein GAN (with gradient penalty) to assign low regularization scores to clean patches and higher scores to degraded ones. The global regularizer then sums the critic’s values over all overlapping patches. This model generalizes across noise levels and achieves state-of-the-art performance on denoising and deblurring tasks, as measured by PSNR and perceptual LPIPS.

The learned regularizer adapts to complex patchwise statistics and enforces that every local region of the restored image exhibits the high-order dependencies characteristic of clean data, explicitly leveraging local correlations learned from data, rather than relying on hand-engineered priors.

---

References:

• "Convolutional Neural Networks Regularized by Correlated Noise" (Dutta et al., 2018)
• "Regularizing CNNs with Locally Constrained Decorrelations" (Rodríguez et al., 2016)
• "Decorrelating neurons using persistence" (Ballester et al., 2023)
• "Learning local regularization for variational image restoration" (Prost et al., 2021)