Rank-Expanding Initialization

• Rank-expanding initialization is a technique that increases the effective rank of feature representations to overcome bottlenecks in convergence and stability.
• It employs structured methods such as covariance-guided orthogonalization and identity-like constructs to ensure diverse, nearly orthogonal initializations.
• These strategies are applied across architectures like PINNs, transformers, and implicit networks to accelerate learning and improve overall performance.

Rank-expanding initialization refers to a class of initialization and preconditioning strategies in machine learning and related computational disciplines that increase or preserve the effective rank, diversity, or linear independence of feature representations at initialization. These methods are designed to address structural bottlenecks associated with traditional random or identity-inspired initialization schemes, promote rapid convergence, enhance stability, and mitigate phenomena like spectral bias, pathologically ill-conditioned least-squares problems, and initialization sensitivity. Rank-expanding initialization encompasses techniques for neural networks, matrix factorization, low-rank adaptation, and implicit representations, with formal definitions, theoretical guarantees, and empirical justifications across a range of architectures and applications.

1. Mathematical Foundations of Rank and Initialization Bottlenecks

The central mathematical motivation for rank-expanding initialization is the observation that the initial “expressivity” of a model—manifested via the rank of feature matrices, Gram matrices, or neural basis outputs—directly determines both its representational capacity and the effectiveness of first-order optimization. For feedforward and coordinate-based networks, when input dimensionality $P\ll D$ (with $D$ the hidden width), generic random initializations (e.g., Xavier, Kaiming) yield feature matrices or Jacobians whose rank is at most $P$, causing “inlet rank collapse” and impeding propagation of independent gradients (Zheng et al., 2 Feb 2026). In residual and low-rank networks, zero-initialization or identity-inspired approaches can limit the attainable rank in non-square layers, forming a structural bottleneck (Pan et al., 6 Mar 2025).

For low-rank adaptation and matrix sensing, the initial rank of parameter factorizations (e.g., product $BA$ in LoRA, or factor $U$ in $X=UU^\top$) can strictly restrict the subspace in which learning dynamics unfold; high-rank or vanishing initializations may either throttle convergence or violate desirable implicit regularization properties (Xue, 4 Oct 2025, Eftekhari et al., 2020). In nonnegative matrix factorization (NMF), intelligent initialization can attain tight error bounds and accelerate convergence versus generic schemes (Liu et al., 2016).

2. Core Methodologies for Rank-Expanding Initialization

2.1 Structured First-Layer and Covariance-Orthogonalizing Schemes

Several approaches design the first layer so that hidden unit functions are maximally diverse, nearly orthonormal, or linearly independent:

• Covariance-guided Orthogonalization: RINN (Peng et al., 21 Jun 2025) constructs a neural basis matrix $\Phi\in\mathbb{R}^{K\times N_\ell}$ on collocation points and introduces a regularization loss $$L_\mathrm{reg}(\theta) = \epsilon L_\mathrm{diag}(\theta) + L_\mathrm{ortho}(\theta)$$, where $L_\mathrm{ortho}$ penalizes off-diagonal covariance and $L_\mathrm{diag}$ normalizes variance. This yields a nearly identity covariance structure, enforcing pairwise orthogonality and maximal effective rank.
• Structured First-Layer Initialization (SFLI): SFLI (Tang et al., 16 Jul 2025) explicitly constructs first-layer neuron functions so that the Gram matrix $M_{ij}=\int \phi_i(x)\phi_j(x)\,dx$ is numerically full rank up to $\varepsilon$, across various activation functions. Parameterizations for weights and biases are made so features tile the input space with minimal overlap.

2.2 Rank-Expanding Initialization in Coordinate-Based MLPs

In INR architectures, “rank-expanding initialization” (REI) (Zheng et al., 2 Feb 2026) analytically constructs first-layer weights and biases such that, for $N$ inputs and $D\geq N$ hidden units:

• For $P=1$ (scalar input): $w_j=1$, $b_j=-x_j+\varepsilon$ yields a lower-triangular activation matrix with full rank $N$.
• For $P=2$: Weights and biases correspond to grid points in $[-1,1]^2$; ReLU boundaries are arranged so the $N\times D$ matrix of first-layer outputs achieves maximum numerical rank. These constructions generalize to higher dimensions by sampling weights on the unit sphere.

2.3 Identity-Like Constructive Initialization

IDInit (Pan et al., 6 Mar 2025) employs padded identity matrices for both square and non-square layers, breaking the zeros-only bottleneck and ensuring each layer’s rank is as large as its dimension. This approach is applied also to convolutional and higher-order weights, guaranteeing the preservation of high-rank structure through residual branches and robust gradient propagation even in very deep settings.

2.4 Low-Rank Adapter and Matrix Sensing Initializations

• Task-Aligned Low-Rank Initialization: LoRA-SB (Ponkshe et al., 2024) and IniLoRA (Xue, 4 Oct 2025) for parameter-efficient fine-tuning use SVD-based or loss-gradient-based initialization so that the low-rank subspace is optimally aligned with the dominant update direction or the pretrained weights themselves. This ensures the learnable parameters span maximum-variance directions immediately and preserves rank richness throughout adaptation.
• Matrix Sensing: The gradient flow in low-rank matrix sensing converges to a rank at most that of the initialization and cannot exceed it. Adaptive restarting and low-rank factor initialization are critical for both zero-error convergence and generalization, as the flow is “rank-invariant” after initialization (Eftekhari et al., 2020).

2.5 Rank-One Expansion in NMF

The cr1-nmf initialization (Liu et al., 2016) partitions the data matrix into geometrically-separated clusters and performs independent rank-one NMF on each, assembling a rank-$K$ factorization with guaranteed small approximate error and rapid subsequent convergence.

3. Theoretical Guarantees and Convergence Properties

Rank-expanding initialization methods exhibit several provable benefits:

• Covariance preconditioning in RINN yields well-conditioned design matrices and stabilized least-squares solutions (Peng et al., 21 Jun 2025).
• SFLI achieves full $\varepsilon$-rank at initialization, directly lowering minimax loss bounds due to enhanced representational capacity and removing loss plateaus caused by rank bottlenecks (Tang et al., 16 Jul 2025).
• REI analytically guarantees NTK rank scaling with hidden width, achieving $\operatorname{rank}(\mathrm{NTK}) = \min(N, D)$ and unblocking the “inlet rank collapse” bottleneck described for standard coordinate MLPs (Zheng et al., 2 Feb 2026).
• IDInit ensures that even non-square layers propagate full possible rank through SGD, breaking the rank-deficient constraint of pure zero-padded or identity inits (Pan et al., 6 Mar 2025).
• cr1-nmf deterministic and probabilistic error bounds guarantee that the initialization error is $\leq \max_k \sin\alpha_k$ under conic separation (Liu et al., 2016).
• Low-rank adaptation methods initialized via principal subspace or task-aligned directions retain optimal update directions throughout training and, under certain assumptions, simulate full fine-tuning (Ponkshe et al., 2024, Xue, 4 Oct 2025).
• For matrix sensing, rank-invariance is rigorously established: the gradient flow initialized at rank $p$ never increases rank and converges within a neighborhood whose radius can be much larger than those of local-refinement-only results (Eftekhari et al., 2020).

4. Practical Algorithms and Empirical Evidence

Rank-expanding initialization is operationalized through explicit pseudocode and is typically implemented as a drop-in replacement for the standard random initialization of the relevant layers. Representative workflow recipes include:

• Stagewise orthogonalization by first-order optimization (RINN) (Peng et al., 21 Jun 2025).
• Sampling-based selection of weight/bias parameters to span input space (SFLI, REI) (Tang et al., 16 Jul 2025, Zheng et al., 2 Feb 2026).
• Padded identity and variance-calibrated constructions (IDInit) (Pan et al., 6 Mar 2025).
• SVD- or loss-gradient-based subspace construction for low-rank modules, followed by residual correction (IniLoRA, LoRA-SB) (Ponkshe et al., 2024, Xue, 4 Oct 2025).
• Hierarchical clustering and independent rank-one expansions per cluster (cr1-nmf) (Liu et al., 2016).

Empirical results consistently show:

• Faster convergence and earlier loss reductions (often by 20–100× versus baseline) (Peng et al., 21 Jun 2025, Ponkshe et al., 2024, Tang et al., 16 Jul 2025, Zheng et al., 2 Feb 2026, Pan et al., 6 Mar 2025).
• Higher and more stable final accuracy across standard language modeling (GLUE, GSM8K), vision (Cifar-10, ImageNet), function approximation, and PDE benchmarks (Xue, 4 Oct 2025, Pan et al., 6 Mar 2025).
• Robustness to changes in initialization variance and parameter scaling (Xue, 4 Oct 2025).
• Dramatic boosts to $\varepsilon$-rank and spectral flatness in the first hidden layer, eliminating the need for later “rank jumps” to achieve nontrivial approximation (Tang et al., 16 Jul 2025).

5. Applications Across Architectures and Domains

Rank-expanding initialization methods apply to a variety of architectures:

• Physics-informed networks: RINN and SFLI are directly applicable to PINNs, residual adaptive networks, and scientific computing models (Peng et al., 21 Jun 2025, Tang et al., 16 Jul 2025).
• Transformer LLMs: Low-rank adaptation techniques (LoRA, LoRA-SB, IniLoRA) are the state-of-the-art in parameter-efficient fine-tuning of LLMs (Ponkshe et al., 2024, Xue, 4 Oct 2025).
• Residual deep networks: IDInit’s padded identities significantly accelerate and stabilize training of deep ResNets and ViTs (Pan et al., 6 Mar 2025).
• Implicit neural representations (INRs): Rank-expanding methods both clarify and solve the challenge of representing high-frequency details in continuous signal models (Zheng et al., 2 Feb 2026).
• Nonnegative matrix factorization: Rank-one initialization is essential for scalable, tight-error NMF algorithms, improving clustering and representation learning (Liu et al., 2016).
• Matrix sensing: Initial rank selection governs both trajectory and generalization in convex and non-convex matrix recovery (Eftekhari et al., 2020).

6. Limitations and Contextual Considerations

While rank-expanding initialization resolves key bottlenecks, certain settings present nuances:

• In matrix sensing, large initial rank or excessive norm can increase the likelihood of converging to poorly-generalizing or high-rank interpolators; the most effective regimes couple moderate or minimal rank with small norm (Eftekhari et al., 2020).
• For low-rank adapters, as rank approaches full, computational savings diminish; careful tuning of rank parameter yields optimal trade-offs between parameter count and performance (Xue, 4 Oct 2025).
• Over-decorrelation during covariance preconditioning can degrade the fit to PDE constraints unless stopped early based on loss minima (Peng et al., 21 Jun 2025).
• Empirical validation of $\varepsilon$-rank, condition number, and spectral flatness is advised to diagnose any residual bottleneck (Tang et al., 16 Jul 2025).

7. Comparative Perspective and Integration

Rank-expanding initialization joins a broader ecosystem of “diversity-enhancing” methods, including positional encodings, SIREN, BatchNorm, and other spectral-reshaping techniques. Notably, the structural diagnosis in (Zheng et al., 2 Feb 2026) combines and unifies these mechanisms, demonstrating that optimized initialization alone is often sufficient to enable full-rank NTKs and maximal downstream expressivity, without extra computational or architectural overhead. These strategies are easily integrable, often requiring only a single code-line or minimal precomputation, and generalize across tasks, network types, and input dimensionality.

In summary, rank-expanding initialization forms a mathematically and empirically grounded paradigm for unlocking the capacity, efficiency, and stability of modern ML architectures through deliberate structural optimization of initial representations.