---
title: Deep Linear Network Theory
url: https://www.emergentmind.com/topics/deep-linear-network-dln
type: topic
---

# Deep Linear Network Theory

Searching arXiv for recent and foundational DLN papers to ground the article.
A deep linear network (DLN) is a multilayer factorization of a linear map, typically written as a tuple of matrices \(\mathbf W=(W_N,\dots,W_1)\) whose observable end-to-end map is the product \(W=W_N\cdots W_1\). Although the represented function class is still linear, depth turns a \(d^2\)-dimensional predictor into an overparameterized system with \(Nd^2\) parameters, and that change in parameterization induces nontrivial optimization dynamics, invariant manifolds, symmetry orbits, and architecture-dependent implicit regularization [2210.12497]. For this reason, DLNs have become a mathematically tractable model for studying gradient flow, balancedness, initialization-sensitive training regimes, low-rank bias, entropy-based selection effects, and geometric formulations of deep learning more broadly [2411.09004].

## 1. Formal model and overparameterized factorization

In the square-matrix setting emphasized in several geometric papers, a depth-\(N\) DLN is specified by matrices \(W_i\in\mathbb M_d\) and the product map
\[
\phi(\mathbf W)=W_NW_{N-1}\cdots W_1.
\]
A rectangular formulation is also standard: for depth \(L\) and widths \(n_0,\dots,n_L\), one writes \(A_\theta=W_L\cdots W_1\), with hidden widths often specialized to a common value \(w\) in \((L,w)\)-DLNs [2210.12497; 2106.15933]. In supervised regression form, the predictor may be normalized as
\[
N_W(x)=\frac{1}{\sqrt{m^{L-1}d_y}}\,W^{(L)}\cdots W^{(1)}x,
\]
and trained by squared loss on samples \(\{(x_i,y_i)\}_{i=1}^n\) [2010.01618].

The decisive structural fact is that depth changes parameterization rather than expressivity. In the square case, the parameter space has dimension \(Nd^2\) while the end-to-end matrix remains \(d\times d\), so many parameter tuples represent the same observable [2210.12497]. This redundancy is organized by reparameterization symmetries. For full-rank \(X\), the fiber
\[
\mathcal F_X=\{\mathbf W:W_N\cdots W_1=X\}
\]
is an orbit of \(\mathrm{GL}_d^{N-1}\), and its balanced representatives form a compact \(U_d^{N-1}\)-orbit in the complex formulation [2511.01137]. The later entropy theory refines this point further: on the balanced manifold, the fibers over invertible end-to-end matrices are orthogonal-group orbits, so overparameterization becomes an explicit foliation by compact leaves [2509.09088].

Losses studied on DLNs range from generic lifted objectives \(L=E\circ\phi\) to structured problems such as matrix completion. A representative completion objective is
\[
E_\mathcal B(W)=\frac12\|\mathcal B\circ(\Phi-W)\|_2^2,
\]
with minimizer manifold \(\mathcal N_\mathcal B=\{W:\mathcal B\circ(\Phi-W)=0\}\) [2210.12497]. Classification-oriented DLN analyses also use mean squared error, either on fixed data matrices or in unconstrained feature models where the feature matrix itself is optimized [2311.02960; 2404.06106].

## 2. Balancedness and the induced Riemannian geometry

A central DLN invariant is the family of adjacent Gram differences
\[
G_j=W_{j+1}^TW_{j+1}-W_jW_j^T,\qquad 1\le j\le N-1.
\]
If the network is initialized with \(G_j=0\), then it remains balanced under gradient flow, and on the balanced manifold all layers share the same singular values [2210.12497]. Later work recast this more conceptually: the \(G_j\) are the components of the moment map for the unitary reparameterization symmetry, so balancedness is the zero level set of that moment map rather than merely a convenient algebraic constraint [2511.01137].

On the balanced manifold, DLN training reduces to a Riemannian gradient flow on the end-to-end matrix. In one formulation, the architecture induces an operator
\[
\mathcal A_{N,W}(Z)=\frac1N\sum_{j=1}^N (WW^T)^{\frac{N-j}{N}}\,Z\,(W^TW)^{\frac{j-1}{N}},
\]
and the metric
\[
g^N(Z_1,Z_2)=\operatorname{tr}\!\left(\mathcal A_{N,W}^{-1}(Z_1)^T Z_2\right)
\]
makes the end-to-end dynamics
\[
\dot W=-\operatorname{grad}_{g^N}E(W)
\]
exact [2210.12497]. The geometry is therefore architecture-induced: the loss \(E\) is task-dependent, but the metric \(g^N\) is determined by the factorization depth.

This metric is explicit in singular-value coordinates. If \(W=U\Sigma V^T\) with singular values \(\sigma_1\ge\cdots\ge\sigma_d\), then the basis elements \(T_{il}=UE_{il}V^T\) diagonalize \(\mathcal A_{N,W}\), with eigenvalues
\[
\lambda^N_{il}=\frac1N\sum_{j=1}^N (\sigma_i^2)^{\frac{N-j}{N}}(\sigma_l^2)^{\frac{j-1}{N}}.
\]
Thus the metric weights tangent directions according to the current spectrum of the end-to-end map [2210.12497]. This SVD-level diagonalization is the basic analytic mechanism behind later formulas for volume, entropy, Brownian motion, and geodesics [2509.09088; 2411.09004].

Depth also admits an explicit infinite-depth limit. As \(N\to\infty\),
\[
\mathcal A_{\infty,W}(Z)=\int_0^1 (WW^T)^{1-\tau}Z(W^TW)^\tau\,d\tau,
\]
yielding a limiting metric \(g^\infty\) on invertible matrices [2210.12497]. The associated geometric theory is not merely formal: the balanced-manifold papers prove that the projection \(\phi:(\mathcal M,\iota)\to(\mathfrak M_d,g^N)\) is a Riemannian submersion, so the DLN end-to-end metric is the quotient metric induced from Euclidean geometry upstairs on balanced parameters [2509.09088].

## 3. Dynamical regimes, initialization scale, and accelerated optimization

DLN training is highly sensitive to initialization scale. For rectangular \((L,w)\)-DLNs with i.i.d. Gaussian initialization of variance \(\sigma^2=w^{-\gamma}\), there is a phase transition at \(\gamma=1\). When \(1-\frac1L\le \gamma<1\), initialization is asymptotically close to global minima and far from saddles, corresponding to an NTK-like lazy regime. When \(\gamma>1\), initialization is asymptotically close to saddles and far from minima, producing a qualitatively different small-initialization regime with long plateaus and a conjectured saddle-to-saddle dynamics through increasing effective rank [2106.15933]. A common misconception is therefore that DLNs are intrinsically “lazy”; the small-initialization regime shows the opposite.

The small-initialization picture is especially sharp in the \(\alpha\searrow0\) limit. The origin is a saddle, escape times diverge, and the first rigorously controlled nontrivial segment of training follows a width-1 path aligned with the top singular mode of \(\nabla C(0)\). Under a simple leading-singular-value assumption, the first escape path has rank-1 form up to rotation and width inclusion, providing a rigorous “rank-1 emergence” theorem [2106.15933]. The broader conjecture is a sequence
\[
0 \leadsto \text{rank-1 saddle} \leadsto \text{rank-2 saddle} \leadsto \cdots,
\]
which suggests greedy low-rank search in the vanishing-initialization regime.

Separate optimization theory shows that acceleration can survive the nonconvex factorization. For deep linear regression with squared loss, orthogonal initialization, and sufficient width, layerwise Polyak momentum yields a non-asymptotic residual rate
\[
\left(1-\Theta\!\left(\frac1{\sqrt\kappa}\right)\right)^t
\]
instead of the vanilla gradient-descent rate
\[
\left(1-\Theta\!\left(\frac1\kappa\right)\right)^t,
\]
where \(\kappa\) is the condition number of the data matrix [2010.01618]. The proof isolates an accelerated linear recursion for the end-to-end residual and controls the factorization-induced perturbations \(\phi_t,\psi_t,\iota_t\) through overparameterization and orthogonal near-isometry. The result is discrete-time and non-asymptotic, but it comes with a stronger width requirement than the corresponding vanilla-GD theory [2010.01618].

## 4. Representation geometry and low-dimensional structure

DLNs are also used to analyze how representations evolve across layers. In multiclass classification with nearly orthogonal data and a trained solution that is minimum-norm, balanced, and approximately low-rank, one can define the layerwise compression and discrimination metrics
\[
C_l=\frac{\operatorname{Tr}(\Sigma_W^l)}{\operatorname{Tr}(\Sigma_B^l)},
\qquad
D_l=1-\max_{k\neq k'}\frac{\langle \mu_k^l,\mu_{k'}^l\rangle}{\|\mu_k^l\|\,\|\mu_{k'}^l\|},
\]
where \(\Sigma_W^l\) and \(\Sigma_B^l\) are within-class and between-class scatter at layer \(l\) [2311.02960]. Under the stated assumptions, the theory shows that \(C_l\) decays geometrically across depth while \(D_l\) improves linearly in \(l\). In this sense, each layer progressively compresses within-class variability and increases between-class discrimination [2311.02960].

A related but distinct line of work studies a deep linear unconstrained feature model, where the explicit deep linear layers are trained jointly with a free feature matrix. In that setting, global optima exhibit exact Deep Neural Collapse: classwise feature collapse \(H_1^*=\bar H^*\otimes 1_n^T\), orthogonal/equal-norm class geometry at every analyzed layer, and multilayer self-duality aligning downstream weight products with class means [2404.06106]. These collapse relations imply several low-dimensional consequences. The layerwise Hessian has rank \(K^2\), with nonzero eigenspace spanned by tensor products \(\mu_c^{(l+1)}\otimes \mu_{c'}^{(l)}\), while the update-relevant gradient occupies only the \(K\)-dimensional matched-class slice \(\mu_c^{(l+1)}\otimes \mu_c^{(l)}\) of that outlier space [2404.06106]. This provides a unified explanation of low-rank weights, Hessian outliers, and gradient concentration, though it applies to the unconstrained-feature model rather than directly to fixed-data DLNs.

These results suggest that DLNs capture two complementary aspects of deep representation theory. First, fixed-data DLNs can produce quantitative layerwise compression and discrimination laws. Second, in unconstrained-feature form they expose exact class-collapse geometries from which Hessian and gradient low-dimensionality follow algebraically [2311.02960; 2404.06106].

## 5. Matrix completion and implicit regularization beyond rank alone

Matrix completion is the setting in which the geometry of DLN implicit regularization has been analyzed most explicitly. For the masked quadratic loss
\[
E_\mathcal B(W)=\frac12\|\mathcal B\circ(\Phi-W)\|_2^2,
\]
the minimizer set \(\mathcal N_\mathcal B\) can contain completions of many different ranks [2210.12497]. On the DLN metric, the associated volume form becomes singular near rank-deficient matrices. In the infinite-depth case,
\[
\sqrt{\det g^\infty}\,dW=
\frac{\operatorname{van}(\log \Sigma^2)}{\sqrt{\det(\Sigma^2)}}\,d\Sigma\,dU\,dV,
\]
so the density diverges as any singular value \(\sigma_i\to0\) [2210.12497]. This motivates an entropic interpretation of implicit regularization: trajectories started near the origin are biased toward minimizers surrounded by large Riemannian state-space volume.

The matrix-completion numerics support that interpretation. In a \(20\times20\) diagonal completion experiment, even \(N=3\) already shows a strong bias toward low effective rank, while \(N=10\) and \(N=\infty\) produce nearly identical histograms concentrated near rank one [2210.12497]. In a \(2\times2\) example, finite depths \(N=5,10,20\) and the infinite-depth limit all concentrate near the rank-one completion hyperbola, with \(N=20\) already close to \(N=\infty\) [2210.12497]. This suggests that the infinite-depth geometry is a good asymptotic model for sufficiently deep finite DLNs.

A central correction in this literature is that low rank is not by itself the selecting principle. The same paper gives a \(3\times3\) example with an entire family of rank-two minimizers, yet trajectories concentrate near one particular member; Monte Carlo estimates of local Riemannian volume show that the empirically selected point lies in a region whose local volume is orders of magnitude larger than around other equal-rank solutions [2210.12497]. It also presents a “pathological” rank-one state space where rank carries no information at all, yet the geometry still favors smaller singular value. A plausible implication is that DLN implicit bias is more accurately described as preference for high state-space volume than as preference for low rank simpliciter [2210.12497].

## 6. Explicit regularization, exact solutions, and the special role of the origin

When one adds weight decay and stochastic hidden neurons, the benign unregularized DLN landscape changes qualitatively. For a scalar-output deep linear network with multiplicative hidden noise and layerwise \(L_2\) penalties, the global minimizers admit exact closed-form structure: all deeper layers are rank-one sign-symmetric outer products, the first layer aligns with \(E[xy]\) through a shrunk inverse involving \(A_0=E[xx^T]\), and the entire optimization reduces to a one-dimensional scalar equation after balancing and symmetry reduction [2202.04777]. This exact-solvability result is one of the clearest demonstrations that DLNs with regularization can be analyzed beyond generic “all local minima are global” folklore.

The origin becomes especially significant once depth exceeds one hidden layer. For depth \(D=1\), the zero solution is global iff \(\|E[xy]\|^2\le \gamma_u\gamma_w\), and otherwise it is not a local minimum. For \(D\ge2\), however, the Hessian at the origin is strictly positive definite because the data-fit term has no quadratic contribution there, so zero is always a local minimum; when nontrivial global minima also exist, zero becomes a bad local minimum [2202.04777]. This is a direct refutation of the idea that deep linear landscapes remain benign after adding ordinary regularization.

A different regularization theory reaches balancedness from symmetry. Using geometric invariant theory and the Kempf–Ness theorem, one can show that for any full-rank end-to-end matrix \(X\),
\[
\arg\min_{\mathbf W\in\mathcal F_X}\|\mathbf W\|_2=\mathcal F_X\cap\mathcal M,
\]
so the \(L^2\)-minimal factorizations on a fiber are exactly the balanced ones [2511.01137]. This is stronger than a heuristic preference: balancedness is the unique minimum-norm representative up to unitary gauge symmetry. The corresponding regularizing flow on a fixed fiber is exactly solvable in moment-map coordinates,
\[
G_k(t)=e^{-4t}G_k(0),
\]
which gives exponential decay of imbalance independent of the learning objective [2511.01137]. The same framework also motivates model-reduction and Bayesian interpretations of balanced factorizations.

## 7. Entropy, free energy, and geodesic structure

The thermodynamic extension of DLN theory defines a Boltzmann entropy as the logarithmic volume of the balanced orbit over a fixed end-to-end matrix:
\[
S(X)=\log \operatorname{vol}(\mathcal O_X).
\]
For full-rank \(X=Q_N\Sigma Q_0^T\), this entropy is
\[
S(X)=(N-1)\log c_d+\frac12\log\frac{\operatorname{van}(\Sigma^2)}{\operatorname{van}(\Sigma^{2/N})},
\]
so it depends only on the singular values of \(X\) [2509.09088]. The same work proves that the DLN metric on end-to-end matrices is obtained by Riemannian submersion from the balanced manifold, linking symmetry, entropy, and optimization in a single quotient-geometric picture [2509.09088].

For spectral energies \(E(\sigma)=g(\sum_i f(\sigma_i))\), adding the DLN entropy yields a free energy
\[
F_\beta(X)=E(X)-\beta^{-1}S_N(X).
\]
In this setting the singular-value dynamics reduce exactly to a weighted gradient flow, and the only equilibria are isotropic minimizers with \(\sigma_1=\cdots=\sigma_d=\sigma_\star\) [2512.06137]. A notable contrast with random matrix theory is that the DLN entropy does not produce singular-value repulsion: the entropy is analytic across collisions, and equal singular values are not forbidden [2512.06137]. The entropy formula has also been extended uniformly from real DLNs to complex and quaternionic DLNs, with a \(\beta\in\{1,2,4\}\) dependence analogous to Dyson’s index but with additional diagonal contributions for \(\beta>1\) [2606.16579].

The same geometry supports a geodesic theory. On the full-rank manifold \((\mathfrak M_d,g^N)\), the geodesic equations admit a Hamiltonian formulation, and the product map from the balanced manifold is again the organizing device [2510.07324]. In a special case where the singular vectors of the endpoints are related by a common orthogonal transformation on both left and right, explicit DLN geodesics can be obtained by projecting horizontal straight lines upstairs. In the commuting diagonal case, these geodesics reduce to
\[
X(t)=\bigl((1-t)A^{1/N}+tB^{1/N}\bigr)^N,
\]
and as \(N\to\infty\),
\[
X(t)\to A^{1-t}B^t
\]
[2510.07324]. This suggests that infinite-depth DLN geometry interpolates multiplicatively along commuting positive directions.

Taken together, these developments recast the DLN as more than a factorized linear regression model. It is simultaneously a model of invariant-manifold dynamics, a quotient-Riemannian optimization problem, a setting for entropy and free energy, and a controlled laboratory for studying how depth alters training without altering end-to-end expressivity [2411.09004].

Source: https://www.emergentmind.com/topics/deep-linear-network-dln