Layered Asymmetric Initialization in Deep Models

• Layered asymmetric initialization is a method that assigns distinct priors, scales, and subspaces to different layers or factors to address issues like neuron death and slow convergence.
• It is applied in deep ReLU networks, matrix factorization, LoRA finetuning, and Lipschitz-constrained networks to ensure stable activation and efficient optimization.
• Empirical and theoretical studies show that this tailored initialization boosts training stability and performance by embedding structure-aware biases into model parameters.

“Layered asymmetric initialization” (Editor's term) denotes a family of initialization schemes in which different layers, factors, projections, or coordinates are not initialized from a common symmetric law, but are instead assigned distinct priors, scales, signs, or subspaces. In the literature, this pattern appears in several technically distinct settings: deep ReLU networks, where asymmetry is introduced to prevent dead activations; matrix factorization and matrix sensing, where imbalance between factors changes convergence rates; low-rank adaptation of large models, where adapter factors or attention projections are initialized according to the structure of pretrained weights; and Lipschitz-constrained networks, where initialization must be designed in the transformed parameterization rather than in the raw weight space (Lu et al., 2019, Xiong et al., 2023, Li et al., 2024, Ji et al., 9 Oct 2025, Juston et al., 13 Jan 2026).

1. Conceptual scope and defining features

The unifying idea is not merely that initialization is non-random, but that it is intentionally non-symmetric in a layer-aware or role-aware way. In one line of work, the asymmetry is per neuron and per layer: exactly one randomly chosen weight or bias per neuron is drawn from a strictly positive distribution, while the remaining incident parameters remain symmetric (Lu et al., 2019). In another, the asymmetry is between low-rank factors: one factor is initialized by a Nyström sketch and the other is initialized at zero, or one factor is made deliberately larger than the other so that the Gram difference is nontrivial (Li et al., 2024, Xiong et al., 2023). In transformer finetuning, the asymmetry is between functions inside the same self-attention block: query and value projections are assigned different low-rank priors because their functional roles differ (Ji et al., 9 Oct 2025).

The adjective “layered” also has more than one technical meaning. In deep ReLU theory, it refers to per-layer distributions $P_\ell$ and to recursive control of activation survival across depth (Lu et al., 2019). In LoRA-style finetuning, it refers to per-adapter, per-layer initialization such as $X_0^{(\ell)} = W_0^{(\ell)} \Omega^{(\ell)}$, $Y_0^{(\ell)} = 0$, or to layer-specific distinctions between $W^Q$ and $W^V$ (Li et al., 2024, Ji et al., 9 Oct 2025). In Lipschitz-constrained architectures, it refers to choosing $(\sigma_\ell^2,\alpha_\ell,\gamma_\ell)$ under a layer-wise normalization that changes both forward variance and gradient sensitivity (Juston et al., 13 Jan 2026).

What these variants share is a rejection of the assumption that symmetric or balanced initialization is a neutral default. The cited works instead treat symmetry as a source of pathologies: dying ReLU, polynomially slow decay of redundant directions, suboptimal low-rank adaptation, or rapid information loss under Lipschitz normalization (Lu et al., 2019, Xiong et al., 2023, Ji et al., 9 Oct 2025, Juston et al., 13 Jan 2026).

2. Randomized asymmetric initialization in deep ReLU networks

A foundational theoretical treatment appears in “Dying ReLU and Initialization: Theory and Numerical Examples” (Lu et al., 2019). For a fully-connected feedforward ReLU network

$$z^\ell = W^\ell a^{\ell-1} + b^\ell,\qquad a^\ell = \sigma(z^\ell),\quad \sigma(x)=\max\{0,x\},$$

the paper defines a neuron as dead if $a^\ell_j(x)\equiv 0$ on the domain of interest. Under common symmetric initialization schemes, the paper proves that a deep ReLU network will eventually die in probability as depth goes to infinite, and in the simplified 1D setting it establishes

$$\mathbb{P}(\mathcal{D}_L)\longrightarrow 1 \qquad\text{as }L\to\infty$$

for fixed width $N$ (Lu et al., 2019).

The proposed remedy is Randomized Asymmetric Initialization (RAI). For each layer $X_0^{(\ell)} = W_0^{(\ell)} \Omega^{(\ell)}$0 and neuron $X_0^{(\ell)} = W_0^{(\ell)} \Omega^{(\ell)}$1, one special index is selected uniformly from $X_0^{(\ell)} = W_0^{(\ell)} \Omega^{(\ell)}$2,

$X_0^{(\ell)} = W_0^{(\ell)} \Omega^{(\ell)}$3

where index $X_0^{(\ell)} = W_0^{(\ell)} \Omega^{(\ell)}$4 corresponds to the bias. All non-special weights are initialized from a zero-mean symmetric Gaussian, while the special weight or bias is drawn from a distribution $X_0^{(\ell)} = W_0^{(\ell)} \Omega^{(\ell)}$5 supported on $X_0^{(\ell)} = W_0^{(\ell)} \Omega^{(\ell)}$6 (Lu et al., 2019). The construction is asymmetric because each neuron is guaranteed one strictly positive incident parameter, and layered because $X_0^{(\ell)} = W_0^{(\ell)} \Omega^{(\ell)}$7, its mean $X_0^{(\ell)} = W_0^{(\ell)} \Omega^{(\ell)}$8, and the survival analysis are all layer dependent.

The theoretical significance is that RAI blocks the “eventually die in probability” behavior. The paper states that with suitable choices of $X_0^{(\ell)} = W_0^{(\ell)} \Omega^{(\ell)}$9 and $Y_0^{(\ell)} = 0$0, the per-layer survival probability does not degrade with depth, and the probability that a deep ReLU network is dead can be made extremely small, in contrast to symmetric initialization (Lu et al., 2019). Numerical experiments further report that, under RAI, the fraction of dead neurons at initialization is dramatically reduced; the probability that a whole layer or the whole net is dead is extremely small even for large depths; and training curves show faster convergence and better final training loss and test accuracy, especially for deeper networks (Lu et al., 2019).

Within the broader history of initialization, RAI is important because it gives a clean example of asymmetry as a designed statistical prior rather than an optimizer side effect. The asymmetry is neither heuristic noise nor an architectural modification such as Leaky ReLU. It is a per-layer initialization rule whose purpose is to maintain a stable activation process across depth (Lu et al., 2019).

3. Symmetry breaking, imbalance, and convergence in matrix factorization

A second major strand concerns low-rank matrix factorization, where the role of asymmetry is not primarily activation survival but convergence geometry. “How Over-Parameterization Slows Down Gradient Descent in Matrix Sensing: The Curses of Symmetry and Initialization” studies symmetric parameterization $Y_0^{(\ell)} = 0$1 and asymmetric parameterization $Y_0^{(\ell)} = 0$2 (Xiong et al., 2023). In the over-parameterized symmetric case $Y_0^{(\ell)} = 0$3, the paper gives a novel $Y_0^{(\ell)} = 0$4 lower bound for randomly initialized gradient descent, contrasting sharply with the exact-parameterization scenario $Y_0^{(\ell)} = 0$5, where the convergence rate is $Y_0^{(\ell)} = 0$6 (Xiong et al., 2023). In the asymmetric setting, it gives a global exact convergence result for the over-parameterization case with an $Y_0^{(\ell)} = 0$7 rate, where $Y_0^{(\ell)} = 0$8 is the initialization scale (Xiong et al., 2023).

The mechanism is deliberate imbalance. In the over-parameterized asymmetric analysis, the initialization is

$Y_0^{(\ell)} = 0$9

with Gaussian $W^Q$0, so that the Gram difference

$W^Q$1

is positive definite with high probability at scale $W^Q$2 (Xiong et al., 2023). The paper’s central claim is that imbalance is not a nuisance but the driver of fast convergence in redundant directions. It further proposes a novel method that only modifies one step of GD and obtains a convergence rate independent of $W^Q$3, thereby recovering the rate in the exact-parameterization case (Xiong et al., 2023).

“On the Crucial Role of Initialization for Matrix Factorization” develops this theme through Nyström initialization and ScaledGD (Li et al., 2024). For symmetric factorization, the Nyström initialization is

$W^Q$4

and for asymmetric factorization it is

$W^Q$5

with Gaussian $W^Q$6 (Li et al., 2024). The paper proves that ScaledGD with Nystrom initialization achieves quadratic convergence in cases where only linear rates were previously known, and in the asymmetric exact- and over-parametrized settings it proves one-step convergence: with $W^Q$7, the modified ScaledGD guarantees $W^Q$8 with high probability (Li et al., 2024).

These results matter for the idea of layered asymmetry because they convert asymmetry from a local tweak into a geometric principle. Symmetric over-parameterization leaves large flat or weakly contracting directions; asymmetric factorization with structured initialization aligns the factors with the signal subspaces from the outset (Xiong et al., 2023, Li et al., 2024). This suggests that, in layered models built from repeated low-rank factors, asymmetry is not only a way to avoid bad minima or saddle behavior, but a way to shape the effective optimization regime itself.

4. Layer-wise asymmetric initialization in low-rank adaptation

The most direct modern use of layered asymmetric initialization appears in parameter-efficient finetuning of large models. “AILoRA: Function-Aware Asymmetric Initialization for Low-Rank Adaptation of LLMs” states that LoRA is typically applied to the $W^Q$9 and $W^V$0 projection matrices of self-attention modules, and that LoRA still encounters challenges such as suboptimal performance and slow convergence (Ji et al., 9 Oct 2025). The paper introduces AILoRA, a parameter-efficient method that incorporates function-aware asymmetric low-rank priors. Its empirical analysis reports that $W^V$1 and $W^V$2 exhibit distinct parameter characteristics stemming from their functional differences: $W^V$3 captures task-specific semantic space knowledge essential for attention distributions computation and is highly sensitive to downstream task variations, whereas $W^V$4 encodes token-level feature representations that tend to remain stable across tasks and layers (Ji et al., 9 Oct 2025).

The initialization rule is correspondingly asymmetric. AILoRA performs a function-aware initialization by injecting the principal components of $W^V$5 to retain task-adaptive capacity, and the minor components of $W^V$6 to preserve generalizable feature representations (Ji et al., 9 Oct 2025). The stated consequence is that this asymmetric initialization strategy enables LoRA modules to better capture the specialized roles of attention parameters, thereby enhancing both finetuning performance and convergence efficiency (Ji et al., 9 Oct 2025). In this usage, “layered” is naturally understood as per-transformer-layer deployment, since LoRA is attached to repeated attention projections across depth.

A related but distinct line is “NoRA,” introduced in the matrix-factorization paper (Li et al., 2024). There the per-layer LoRA initialization is

$W^V$7

applied independently to each adapter layer. The paper reports superior performance across various downstream tasks and model scales, from 1B to 7B parameters, in large language and diffusion models (Li et al., 2024). On OPT-1.3B few-shot evaluation, the average test accuracy is reported as 73.2 for LoRA, 75.0 for NoRA, and 75.1 for NoRA+; on Stable Diffusion v1.4 subject-driven finetuning, the average training loss is 0.092 for LoRA and 0.084 for both NoRA and NoRA+; and on Gemma-7B GSM8K, the test accuracy is 76.72 for LoRA, 77.94 for PiSSA, 78.62 for NoRA, 77.03 for LoRA-P, and 78.47 for NoRA+ (Li et al., 2024).

Taken together, these works show two distinct forms of asymmetry in adapter initialization. One is factor-asymmetric: one low-rank factor is initialized by a structured sketch while the other is zero (Li et al., 2024). The other is function-asymmetric: different projections inside self-attention receive different spectral priors because their roles differ (Ji et al., 9 Oct 2025). This suggests that layered asymmetric initialization in PEFT is not a single algorithmic trick but a broader design pattern for embedding pretrained structure into low-rank subspaces before finetuning starts.

5. Lipschitz-constrained layers, variance preservation, and gradient sensitivity

A different interpretation of layered asymmetric initialization arises in Lipschitz-constrained networks. “LDLT L-Lipschitz Network Weight Parameterization Initialization” analyzes layers of the form

$W^V$8

or equivalently $W^V$9 with $(\sigma_\ell^2,\alpha_\ell,\gamma_\ell)$0 (Juston et al., 13 Jan 2026). Here the effective weight matrix is a nonlinear transformation of an unconstrained Gaussian matrix $(\sigma_\ell^2,\alpha_\ell,\gamma_\ell)$1, and initialization must therefore be analyzed in the parameterized operator space rather than in the raw weight space. The paper derives the exact marginal output variance when $(\sigma_\ell^2,\alpha_\ell,\gamma_\ell)$2 has IID Gaussian entries and shows that the relevant random-matrix object is the Wishart matrix $(\sigma_\ell^2,\alpha_\ell,\gamma_\ell)$3 (Juston et al., 13 Jan 2026).

Its headline quantitative comparison is explicit. Using current He or Kaiming initialization with scaling $(\sigma_\ell^2,\alpha_\ell,\gamma_\ell)$4, the output variance is 0.41, whereas the new parameterization with $(\sigma_\ell^2,\alpha_\ell,\gamma_\ell)$5 for $(\sigma_\ell^2,\alpha_\ell,\gamma_\ell)$6 results in an output variance of 0.9 (Juston et al., 13 Jan 2026). The paper concludes that the findings clarify why deep $(\sigma_\ell^2,\alpha_\ell,\gamma_\ell)$7-Lipschitz networks suffer rapid information loss at initialization and offer practical prescriptions for choosing initialization hyperparameters to mitigate this effect (Juston et al., 13 Jan 2026).

The same paper also provides an important caution. Using the Higgs boson classification dataset, a hyperparameter sweep over optimizers, initialization scale, and depth was conducted to validate the results on real-world data, showing that although the derivation ensures variance preservation, empirical results indicate He initialization still performs better (Juston et al., 13 Jan 2026). In its detailed analysis, larger initialization scales consistently hurt performance, reduce gradient norms, and move the normalized weights toward the Lipschitz boundary where the sensitivity of the parameterization to changes in $(\sigma_\ell^2,\alpha_\ell,\gamma_\ell)$8 becomes very small (Juston et al., 13 Jan 2026).

This is a central qualification for the wider topic. In this setting, asymmetric or scale-aware initialization is necessary because the parameterization is nonlinear and depth dependent, but higher forward variance is not by itself a sufficient objective. The paper therefore reframes initialization as a trade-off among variance transmission, spectral saturation, and trainability under the specific layer normalization (Juston et al., 13 Jan 2026).

6. Common principles, limitations, and recurring misconceptions

Across these literatures, a common principle is that asymmetry is usually designed to target a specific failure mode. In RAI, the target is neuron death under symmetric distributions (Lu et al., 2019). In matrix sensing and factorization, the target is slow convergence caused by over-parameterized symmetry and redundant directions (Xiong et al., 2023, Li et al., 2024). In LoRA, the target is the mismatch between generic low-rank random initialization and the structured functional roles of transformer projections or pretrained weights (Ji et al., 9 Oct 2025, Li et al., 2024). In LDLT-based Lipschitz networks, the target is rapid information loss under spectral normalization (Juston et al., 13 Jan 2026).

A recurrent misconception is that asymmetry simply means “larger scale” or “unbalanced norms.” The evidence is more specific. RAI uses exactly one positive weight or bias per neuron, not a global positive bias (Lu et al., 2019). The matrix-factorization papers rely on structured imbalance or subspace alignment, not arbitrary asymmetry (Xiong et al., 2023, Li et al., 2024). AILoRA distinguishes between principal components of $(\sigma_\ell^2,\alpha_\ell,\gamma_\ell)$9 and minor components of $$z^\ell = W^\ell a^{\ell-1} + b^\ell,\qquad a^\ell = \sigma(z^\ell),\quad \sigma(x)=\max\{0,x\},$$0, rather than advocating uniform spectral bias (Ji et al., 9 Oct 2025). The LDLT study shows that even when a larger scale improves output variance from 0.41 to 0.9, empirical performance may still degrade (Juston et al., 13 Jan 2026).

Another misconception is that symmetric parameterization and symmetric initialization are interchangeable issues. The matrix-sensing results show otherwise: even when the target matrix is symmetric, an asymmetric parameterization $$z^\ell = W^\ell a^{\ell-1} + b^\ell,\qquad a^\ell = \sigma(z^\ell),\quad \sigma(x)=\max\{0,x\},$$1 can change the convergence class from a polynomial $$z^\ell = W^\ell a^{\ell-1} + b^\ell,\qquad a^\ell = \sigma(z^\ell),\quad \sigma(x)=\max\{0,x\},$$2 lower bound to linear convergence, and a one-step modification can remove the dependence on the initialization scale $$z^\ell = W^\ell a^{\ell-1} + b^\ell,\qquad a^\ell = \sigma(z^\ell),\quad \sigma(x)=\max\{0,x\},$$3 (Xiong et al., 2023). By contrast, Nyström initialization shows that a suitable initialization can also transform ScaledGD without changing the basic optimization method (Li et al., 2024). The distinction between parameterization asymmetry and initialization asymmetry is therefore analytically significant.

The limitations are equally clear. The RAI theory is developed for fully-connected ReLU nets with 1D inputs and simple input distributions, and extension to high-dimensional, convolutional, or residual architectures is not proven (Lu et al., 2019). The matrix-sensing analysis assumes noise-free observations, RIP, small initialization, and small step size, and does not cover deep multi-layer networks beyond the two-factor setting (Xiong et al., 2023). The Lipschitz study concerns a specific LDLT-based parameterization and shows that variance-preserving prescriptions do not directly predict downstream performance (Juston et al., 13 Jan 2026). These constraints suggest that layered asymmetric initialization is best understood as a design principle with multiple mathematically distinct realizations, not as a universal theorem schema.

In that sense, the modern significance of layered asymmetric initialization lies in its redefinition of what initialization is supposed to do. The older objective of preserving variance under a symmetric prior remains relevant, but the cited works broaden the agenda to include activation survival, subspace alignment, factor imbalance, function-aware specialization, and compatibility with constrained parameterizations. This suggests a general research program: initialization should encode the geometry, function, and depth-specific role of the parameters it seeds, rather than treating all layers and factors as statistically exchangeable (Lu et al., 2019, Xiong et al., 2023, Li et al., 2024, Ji et al., 9 Oct 2025, Juston et al., 13 Jan 2026).