Layer-Wise Covariance Initialization

• Layer-Wise Covariance Initialization is a family of methods that explicitly models weight correlations to enhance signal propagation and trainability in deep networks.
• It employs data-driven Sylvester solvers, closed-form covariance in convolutional filters, and PCA-based strategies to deliver accuracy gains and faster convergence.
• These approaches transfer initialization statistics across models and datasets, providing robust, architecture-agnostic improvements and stabilized training in deep and wide architectures.

Layer-wise covariance initialization encompasses a family of methods that explicitly model, transfer, or control the covariance structure of neural network weights at initialization, rather than relying on conventional schemes that treat each parameter independently. This approach has been motivated by empirical observations of highly structured weight covariances in trained deep models—especially in convolutional architectures—and by theoretical advances clarifying the impact of correlated weights on signal propagation and trainability. Layer-wise covariance initialization includes closed-form, data-driven, and SDE-guided strategies, each targeting different facets of neural network expressivity, stabilization, and data alignment. Key developments have recently enabled robust transfer of initialization statistics across model scales and the principled stabilization of very deep or wide architectures through explicit control of the covariance propagation.

1. Data-Driven and Latent Code-Based Layer-Wise Covariance Initialization

A representative data-driven approach to layer-wise covariance initialization is embodied by the Sylvester-based solver of "Data-driven Weight Initialization with Sylvester Solvers" (Das et al., 2021). This technique frames the layer-wise initialization as a minimization of the joint sum of encoding and decoding losses, encouraging weights to both encode samples into a predefined latent representation and decode back to the inputs. Formally, for input activations $X\in\mathbb{R}^{d_i\times n}$ and latent codes $S\in\mathbb{R}^{d_o\times n}$, the optimization objective

$$L(W; X, S) = \| WX - S \|_F^2 + \lambda \| X - W^T S \|_F^2$$

is solved for $W$, yielding a Sylvester equation:

$A W + W B = C$

where $A = SS^T$, $B = \lambda XX^T$, $C = (1+\lambda) SX^T$. The solution via spectral (eigen) decomposition or direct Sylvester solvers is efficient and fully gradient-free. The latent code $S$ can be chosen layer-wise, e.g., as top PCA components for hidden layers or one-hot encodings for classification. The process is iterated layer by layer, with subsequent layers initialized on the activations produced by the previously initialized layers.

Empirically, this method delivers substantially higher zero-shot accuracy (e.g., 25% vs 10% with He-uniform on CIFAR-10), persistent gains after full training (1–2% improvement), and up to 5–10% greater accuracy in few-shot regimes. Initialization runtime is dominated by PCA (when used), with overall complexity scaling as $O(d_i^3 + d_i^2 n)$ per layer, significantly more expensive than random schemes but negligible relative to full training (Das et al., 2021).

2. Multivariate Covariance Structure in Convolutional Filter Initialization

Empirical analysis of convolutional weights in modern architectures such as ConvMixer and ConvNeXt has revealed that learned filters exhibit highly nontrivial covariance structures, characterized by block-wise stationarity, local smoothness, and model-independent patterns across depths, kernel sizes, and dataset scales (Trockman et al., 2022). Traditional univariate methods, such as He or Xavier initialization, fail to capture these spatial covariances or the associated translation invariance.

To address this, a closed-form initialization scheme generates each layer's set of depthwise convolutional filters as samples from a structured multivariate normal $\mathcal{N}(0, \Sigma)$, where the covariance $\Sigma\in\mathbb{R}^{k^2\times k^2}$ is constructed using analytically defined kernels. The process is as follows:

1. Compute a 2D Gaussian $Z_\sigma$, parameterized by a "spread" $\sigma$.
2. Use Kronecker products and blocks to generate components $C$, $S$, and $M$.
3. Assemble a prototype covariance $\widehat\Sigma = M \odot (C-\frac{1}{2}S)$, symmetrize to obtain $\Sigma$, and ensure positive semi-definiteness.
4. Vary $\sigma$ per layer as a quadratic function of normalized layer depth.

This initialization is learning-free and incorporates empirically observed covariances, leading to faster convergence and higher accuracy, especially in large-kernel models. Covariances transferred from small models are effective for larger or deeper variants, and from different datasets, indicating strong model-independence in the spatial filter statistics. Table 1 summarizes empirical results for key models and datasets.

Initialization	CIFAR-10 Top-1 (%)	ImageNet Top-1 (%)	Notes
Uniform (He/Xavier)	93.1	65.0	Baseline 200 epochs
Empirical-cov transfer	92.9	—	Covariances from small models
Closed-form covariance	93.4	68.8	(+0.3–3.8%)

Closed-form covariance initialization typically yields $0.1$–$1.1$% improvements on CIFAR-10 and up to $3.8$% on ImageNet, with gain persistence even when filters are frozen throughout training (Trockman et al., 2022).

3. Layer-Wise PCA-Based Covariance Initialization

Layer-wise PCA (Principal Component Analysis) initialization explicitly diagonalizes the empirical covariance of input data or activations at each layer, mapping the leading principal components to weight rows. For $X \in \mathbb{R}^{d \times n}$, compute mean $\mu$ and covariance $\Sigma$, obtain the top $k$ eigenspectrum $(E_k,\,\Lambda_k)$, and set encoder weights $W_{\text{enc}}=E_k^T$ and bias $b_{\text{enc}}=-E_k^T\mu$. Optionally, build a decoder as $W_{\text{dec}}=E_k$ and $b_{\text{dec}}=\mu$. By stacking these units and applying PCA to activations from prior layers, one can initialize deep stacks of PCA-structured autoencoder layers (Seuret et al., 2017).

This technique preserves covariance structure and aligns hidden units with directions of maximal variance from the outset, conferring stable, data-aligned sensitivity across the network. Experiments in document image analysis demonstrated:

• Fast convergence: >90% of final accuracy reached in 10–20% of training steps required by Xavier initialization.
• High stability: Learned filters reproduced stably with as few as 500 patches.
• Consistently higher initial accuracy and faster adaptation, with comparable final test accuracy and variance across runs (Seuret et al., 2017).

4. Covariance Evolution and Initialization in Deep and Wide Neural Networks

Theoretical analysis of infinitely deep and wide feedforward networks demonstrates that the random covariance matrices of layer activations converge to the solution of a nonlinear stochastic differential equation ("Neural Covariance SDE") characterized by both drift and diffusion---with coefficients dependent on the choice and scaling of the per-layer activation function (Li et al., 2022). In this framework, the unique non-degenerate limit arises only if the activation function is critically "shaped" (e.g., ReLU scaling as $1 + c_{\pm}/\sqrt{n}$ or smooth functions as $a \sqrt{n}$).

The SDE description reveals an exact if-and-only-if criterion for signal propagation stability:

$$b = [\frac{3}{4}\phi''(0)^2 + \phi'''(0)] / a^2 \leq 0$$

In practice, initialization adheres to this analytical regime by setting each weight $W_{\ell}$ as i.i.d. $\mathcal{N}(0, c/n)$, where $c^{-1} = \mathbb{E}[\phi_s(g)^2]$. This enables direct prediction of the joint covariance evolution and guides the selection of scaling constants to prevent norm explosion or vanishing, a critical consideration for deep architectures and for avoiding reliance on batch normalization or explicit residual connections.

Empirical results show close agreement between SDE-predicted and observed layerwise covariance histograms in finite networks, and Kolmogorov-Smirnov distances decrease as $O(n^{-1/2})$ (Li et al., 2022).

5. Practical Algorithms and Pseudocode

Representative pseudocode and algorithmic recipes have been published for each major method:

• Sylvester initialization: Solve the Sylvester equation for each layer, propagate samples through initialized weights to subsequent layers, and reshape as necessary for convolutional kernels (Das et al., 2021).
• Closed-form convolutional covariance: Loop through layers, build depth-adjusted $\Sigma$ using block-wise Kronecker/Hadamard combinations, sample multivariate normal filters, and assign to each output channel (Trockman et al., 2022).
• PCA-layer initialization: Sequentially sample patches, compute mean/variance, extract top principal components, set weights and biases accordingly, and iterate stacking if desired (Seuret et al., 2017).

All these routines concretely implement covariance-aware initialization and are supported by empirical benchmarks documenting their speed and convergence profiles.

6. Significance and Transferability of Covariance Statistics

Layer-wise covariance initialization harnesses both theoretical and empirical insights: by controlling the inter-weight correlation structure at initialization, it enables three key advantages:

• Enhanced convergence speed: Covariance-matched or PCA-initialized networks require substantially fewer epochs to reach high accuracy versus conventional random initializations.
• Model-independence and transferability: Learned and closed-form covariances can be transferred between models of different depths, widths, kernel sizes, and even across datasets, providing robust, architecture-agnostic initialization priors (Trockman et al., 2022).
• Stabilized training of deep and wide networks: SDE-guided scaling enables explicit control over activation norm propagation, permitting very deep architectures to be trained without additional normalization layers (Li et al., 2022).

This suggests that layer-wise covariance initialization not only captures essential invariances and priors from data or trained models, but can also be deployed in learning-free and analytically tractable forms, substantially extending both practical performance and theoretical understanding of deep model initialization.