Deep Linear Network Theory

• Deep linear networks are multilayer factorizations of linear maps that introduce overparameterization while preserving linear expressivity.
• They reveal architecture-dependent dynamics through balanced training, reparameterization symmetries, and gradient-driven implicit regularization.
• Applications span matrix completion, regression, and geometric analysis of representation compression and accelerated optimization.

Searching arXiv for recent and foundational DLN papers to ground the article. A deep linear network (DLN) is a multilayer factorization of a linear map, typically written as a tuple of matrices $\mathbf W=(W_N,\dots,W_1)$ whose observable end-to-end map is the product $W=W_N\cdots W_1$. Although the represented function class is still linear, depth turns a $d^2$-dimensional predictor into an overparameterized system with $Nd^2$ parameters, and that change in parameterization induces nontrivial optimization dynamics, invariant manifolds, symmetry orbits, and architecture-dependent implicit regularization (Cohen et al., 2022). For this reason, DLNs have become a mathematically tractable model for studying gradient flow, balancedness, initialization-sensitive training regimes, low-rank bias, entropy-based selection effects, and geometric formulations of deep learning more broadly (Menon, 2024).

1. Formal model and overparameterized factorization

In the square-matrix setting emphasized in several geometric papers, a depth-$N$ DLN is specified by matrices $W_i\in\mathbb M_d$ and the product map

$\phi(\mathbf W)=W_NW_{N-1}\cdots W_1.$

A rectangular formulation is also standard: for depth $L$ and widths $n_0,\dots,n_L$, one writes $A_\theta=W_L\cdots W_1$, with hidden widths often specialized to a common value $W=W_N\cdots W_1$0 in $W=W_N\cdots W_1$1-DLNs (Cohen et al., 2022, Jacot et al., 2021). In supervised regression form, the predictor may be normalized as

$W=W_N\cdots W_1$2

and trained by squared loss on samples $W=W_N\cdots W_1$3 (Wang et al., 2020).

The decisive structural fact is that depth changes parameterization rather than expressivity. In the square case, the parameter space has dimension $W=W_N\cdots W_1$4 while the end-to-end matrix remains $W=W_N\cdots W_1$5, so many parameter tuples represent the same observable (Cohen et al., 2022). This redundancy is organized by reparameterization symmetries. For full-rank $W=W_N\cdots W_1$6, the fiber

$W=W_N\cdots W_1$7

is an orbit of $W=W_N\cdots W_1$8, and its balanced representatives form a compact $W=W_N\cdots W_1$9-orbit in the complex formulation (Lindsey et al., 3 Nov 2025). The later entropy theory refines this point further: on the balanced manifold, the fibers over invertible end-to-end matrices are orthogonal-group orbits, so overparameterization becomes an explicit foliation by compact leaves (Menon et al., 11 Sep 2025).

Losses studied on DLNs range from generic lifted objectives $d^2$0 to structured problems such as matrix completion. A representative completion objective is

$d^2$1

with minimizer manifold $d^2$2 (Cohen et al., 2022). Classification-oriented DLN analyses also use mean squared error, either on fixed data matrices or in unconstrained feature models where the feature matrix itself is optimized (Wang et al., 2023, Garrod et al., 2024).

2. Balancedness and the induced Riemannian geometry

A central DLN invariant is the family of adjacent Gram differences

$d^2$3

If the network is initialized with $d^2$4, then it remains balanced under gradient flow, and on the balanced manifold all layers share the same singular values (Cohen et al., 2022). Later work recast this more conceptually: the $d^2$5 are the components of the moment map for the unitary reparameterization symmetry, so balancedness is the zero level set of that moment map rather than merely a convenient algebraic constraint (Lindsey et al., 3 Nov 2025).

On the balanced manifold, DLN training reduces to a Riemannian gradient flow on the end-to-end matrix. In one formulation, the architecture induces an operator

$d^2$6

and the metric

$d^2$7

makes the end-to-end dynamics

$d^2$8

exact (Cohen et al., 2022). The geometry is therefore architecture-induced: the loss $d^2$9 is task-dependent, but the metric $Nd^2$0 is determined by the factorization depth.

This metric is explicit in singular-value coordinates. If $Nd^2$1 with singular values $Nd^2$2, then the basis elements $Nd^2$3 diagonalize $Nd^2$4, with eigenvalues

$Nd^2$5

Thus the metric weights tangent directions according to the current spectrum of the end-to-end map (Cohen et al., 2022). This SVD-level diagonalization is the basic analytic mechanism behind later formulas for volume, entropy, Brownian motion, and geodesics (Menon et al., 11 Sep 2025, Menon, 2024).

Depth also admits an explicit infinite-depth limit. As $Nd^2$6,

$Nd^2$7

yielding a limiting metric $Nd^2$8 on invertible matrices (Cohen et al., 2022). The associated geometric theory is not merely formal: the balanced-manifold papers prove that the projection $Nd^2$9 is a Riemannian submersion, so the DLN end-to-end metric is the quotient metric induced from Euclidean geometry upstairs on balanced parameters (Menon et al., 11 Sep 2025).

3. Dynamical regimes, initialization scale, and accelerated optimization

DLN training is highly sensitive to initialization scale. For rectangular $N$0-DLNs with i.i.d. Gaussian initialization of variance $N$1, there is a phase transition at $N$2. When $N$3, initialization is asymptotically close to global minima and far from saddles, corresponding to an NTK-like lazy regime. When $N$4, initialization is asymptotically close to saddles and far from minima, producing a qualitatively different small-initialization regime with long plateaus and a conjectured saddle-to-saddle dynamics through increasing effective rank (Jacot et al., 2021). A common misconception is therefore that DLNs are intrinsically “lazy”; the small-initialization regime shows the opposite.

The small-initialization picture is especially sharp in the $N$5 limit. The origin is a saddle, escape times diverge, and the first rigorously controlled nontrivial segment of training follows a width-1 path aligned with the top singular mode of $N$6. Under a simple leading-singular-value assumption, the first escape path has rank-1 form up to rotation and width inclusion, providing a rigorous “rank-1 emergence” theorem (Jacot et al., 2021). The broader conjecture is a sequence

$N$7

which suggests greedy low-rank search in the vanishing-initialization regime.

Separate optimization theory shows that acceleration can survive the nonconvex factorization. For deep linear regression with squared loss, orthogonal initialization, and sufficient width, layerwise Polyak momentum yields a non-asymptotic residual rate

$N$8

instead of the vanilla gradient-descent rate

$N$9

where $W_i\in\mathbb M_d$0 is the condition number of the data matrix (Wang et al., 2020). The proof isolates an accelerated linear recursion for the end-to-end residual and controls the factorization-induced perturbations $W_i\in\mathbb M_d$1 through overparameterization and orthogonal near-isometry. The result is discrete-time and non-asymptotic, but it comes with a stronger width requirement than the corresponding vanilla-GD theory (Wang et al., 2020).

4. Representation geometry and low-dimensional structure

DLNs are also used to analyze how representations evolve across layers. In multiclass classification with nearly orthogonal data and a trained solution that is minimum-norm, balanced, and approximately low-rank, one can define the layerwise compression and discrimination metrics

$W_i\in\mathbb M_d$2

where $W_i\in\mathbb M_d$3 and $W_i\in\mathbb M_d$4 are within-class and between-class scatter at layer $W_i\in\mathbb M_d$5 (Wang et al., 2023). Under the stated assumptions, the theory shows that $W_i\in\mathbb M_d$6 decays geometrically across depth while $W_i\in\mathbb M_d$7 improves linearly in $W_i\in\mathbb M_d$8. In this sense, each layer progressively compresses within-class variability and increases between-class discrimination (Wang et al., 2023).

A related but distinct line of work studies a deep linear unconstrained feature model, where the explicit deep linear layers are trained jointly with a free feature matrix. In that setting, global optima exhibit exact Deep Neural Collapse: classwise feature collapse $W_i\in\mathbb M_d$9, orthogonal/equal-norm class geometry at every analyzed layer, and multilayer self-duality aligning downstream weight products with class means (Garrod et al., 2024). These collapse relations imply several low-dimensional consequences. The layerwise Hessian has rank $\phi(\mathbf W)=W_NW_{N-1}\cdots W_1.$0, with nonzero eigenspace spanned by tensor products $\phi(\mathbf W)=W_NW_{N-1}\cdots W_1.$1, while the update-relevant gradient occupies only the $\phi(\mathbf W)=W_NW_{N-1}\cdots W_1.$2-dimensional matched-class slice $\phi(\mathbf W)=W_NW_{N-1}\cdots W_1.$3 of that outlier space (Garrod et al., 2024). This provides a unified explanation of low-rank weights, Hessian outliers, and gradient concentration, though it applies to the unconstrained-feature model rather than directly to fixed-data DLNs.

These results suggest that DLNs capture two complementary aspects of deep representation theory. First, fixed-data DLNs can produce quantitative layerwise compression and discrimination laws. Second, in unconstrained-feature form they expose exact class-collapse geometries from which Hessian and gradient low-dimensionality follow algebraically (Wang et al., 2023, Garrod et al., 2024).

5. Matrix completion and implicit regularization beyond rank alone

Matrix completion is the setting in which the geometry of DLN implicit regularization has been analyzed most explicitly. For the masked quadratic loss

$\phi(\mathbf W)=W_NW_{N-1}\cdots W_1.$4

the minimizer set $\phi(\mathbf W)=W_NW_{N-1}\cdots W_1.$5 can contain completions of many different ranks (Cohen et al., 2022). On the DLN metric, the associated volume form becomes singular near rank-deficient matrices. In the infinite-depth case,

$\phi(\mathbf W)=W_NW_{N-1}\cdots W_1.$6

so the density diverges as any singular value $\phi(\mathbf W)=W_NW_{N-1}\cdots W_1.$7 (Cohen et al., 2022). This motivates an entropic interpretation of implicit regularization: trajectories started near the origin are biased toward minimizers surrounded by large Riemannian state-space volume.

The matrix-completion numerics support that interpretation. In a $\phi(\mathbf W)=W_NW_{N-1}\cdots W_1.$8 diagonal completion experiment, even $\phi(\mathbf W)=W_NW_{N-1}\cdots W_1.$9 already shows a strong bias toward low effective rank, while $L$0 and $L$1 produce nearly identical histograms concentrated near rank one (Cohen et al., 2022). In a $L$2 example, finite depths $L$3 and the infinite-depth limit all concentrate near the rank-one completion hyperbola, with $L$4 already close to $L$5 (Cohen et al., 2022). This suggests that the infinite-depth geometry is a good asymptotic model for sufficiently deep finite DLNs.

A central correction in this literature is that low rank is not by itself the selecting principle. The same paper gives a $L$6 example with an entire family of rank-two minimizers, yet trajectories concentrate near one particular member; Monte Carlo estimates of local Riemannian volume show that the empirically selected point lies in a region whose local volume is orders of magnitude larger than around other equal-rank solutions (Cohen et al., 2022). It also presents a “pathological” rank-one state space where rank carries no information at all, yet the geometry still favors smaller singular value. A plausible implication is that DLN implicit bias is more accurately described as preference for high state-space volume than as preference for low rank simpliciter (Cohen et al., 2022).

6. Explicit regularization, exact solutions, and the special role of the origin

When one adds weight decay and stochastic hidden neurons, the benign unregularized DLN landscape changes qualitatively. For a scalar-output deep linear network with multiplicative hidden noise and layerwise $L$7 penalties, the global minimizers admit exact closed-form structure: all deeper layers are rank-one sign-symmetric outer products, the first layer aligns with $L$8 through a shrunk inverse involving $L$9, and the entire optimization reduces to a one-dimensional scalar equation after balancing and symmetry reduction (Ziyin et al., 2022). This exact-solvability result is one of the clearest demonstrations that DLNs with regularization can be analyzed beyond generic “all local minima are global” folklore.

The origin becomes especially significant once depth exceeds one hidden layer. For depth $n_0,\dots,n_L$0, the zero solution is global iff $n_0,\dots,n_L$1, and otherwise it is not a local minimum. For $n_0,\dots,n_L$2, however, the Hessian at the origin is strictly positive definite because the data-fit term has no quadratic contribution there, so zero is always a local minimum; when nontrivial global minima also exist, zero becomes a bad local minimum (Ziyin et al., 2022). This is a direct refutation of the idea that deep linear landscapes remain benign after adding ordinary regularization.

A different regularization theory reaches balancedness from symmetry. Using geometric invariant theory and the Kempf–Ness theorem, one can show that for any full-rank end-to-end matrix $n_0,\dots,n_L$3,

$n_0,\dots,n_L$4

so the $n_0,\dots,n_L$5-minimal factorizations on a fiber are exactly the balanced ones (Lindsey et al., 3 Nov 2025). This is stronger than a heuristic preference: balancedness is the unique minimum-norm representative up to unitary gauge symmetry. The corresponding regularizing flow on a fixed fiber is exactly solvable in moment-map coordinates,

$n_0,\dots,n_L$6

which gives exponential decay of imbalance independent of the learning objective (Lindsey et al., 3 Nov 2025). The same framework also motivates model-reduction and Bayesian interpretations of balanced factorizations.

7. Entropy, free energy, and geodesic structure

The thermodynamic extension of DLN theory defines a Boltzmann entropy as the logarithmic volume of the balanced orbit over a fixed end-to-end matrix: $n_0,\dots,n_L$7 For full-rank $n_0,\dots,n_L$8, this entropy is

$n_0,\dots,n_L$9

so it depends only on the singular values of $A_\theta=W_L\cdots W_1$0 (Menon et al., 11 Sep 2025). The same work proves that the DLN metric on end-to-end matrices is obtained by Riemannian submersion from the balanced manifold, linking symmetry, entropy, and optimization in a single quotient-geometric picture (Menon et al., 11 Sep 2025).

For spectral energies $A_\theta=W_L\cdots W_1$1, adding the DLN entropy yields a free energy

$A_\theta=W_L\cdots W_1$2

In this setting the singular-value dynamics reduce exactly to a weighted gradient flow, and the only equilibria are isotropic minimizers with $A_\theta=W_L\cdots W_1$3 (Chen et al., 5 Dec 2025). A notable contrast with random matrix theory is that the DLN entropy does not produce singular-value repulsion: the entropy is analytic across collisions, and equal singular values are not forbidden (Chen et al., 5 Dec 2025). The entropy formula has also been extended uniformly from real DLNs to complex and quaternionic DLNs, with a $A_\theta=W_L\cdots W_1$4 dependence analogous to Dyson’s index but with additional diagonal contributions for $A_\theta=W_L\cdots W_1$5 (Contreras et al., 15 Jun 2026).

The same geometry supports a geodesic theory. On the full-rank manifold $A_\theta=W_L\cdots W_1$6, the geodesic equations admit a Hamiltonian formulation, and the product map from the balanced manifold is again the organizing device (Chen, 18 Sep 2025). In a special case where the singular vectors of the endpoints are related by a common orthogonal transformation on both left and right, explicit DLN geodesics can be obtained by projecting horizontal straight lines upstairs. In the commuting diagonal case, these geodesics reduce to

$A_\theta=W_L\cdots W_1$7

and as $A_\theta=W_L\cdots W_1$8,

$A_\theta=W_L\cdots W_1$9

(Chen, 18 Sep 2025). This suggests that infinite-depth DLN geometry interpolates multiplicatively along commuting positive directions.

Taken together, these developments recast the DLN as more than a factorized linear regression model. It is simultaneously a model of invariant-manifold dynamics, a quotient-Riemannian optimization problem, a setting for entropy and free energy, and a controlled laboratory for studying how depth alters training without altering end-to-end expressivity (Menon, 2024).