Diagonal Linear Networks: Dynamics

• Diagonal Linear Networks are overparameterized linear models that reparameterize a linear predictor via diagonal-layer factors, preserving linearity while introducing nonconvexity.
• They employ factorizations (e.g., u ⊙ v, deep products) to analyze gradient-flow dynamics, resulting in sparse ℓ1-type regularization and stagewise support activation.
• Researchers use DLNs to investigate trade-offs between speed and generalization, incremental learning paths, and applications such as reweighted least-squares and entropic regularization.

Diagonal linear networks (DLNs) are overparameterized linear models in which the effective predictor is represented through diagonal-layer parameters rather than as a single weight vector. In the canonical two-layer regression model, one writes $\beta=u\odot v$ and optimizes $F(u,v)=L(u\odot v)$, so the hypothesis class remains linear while the parameterization becomes nonconvex. This minimal reparameterization is sufficient to produce mirror-flow structure, initialization-dependent implicit regularization, sparse minimum-$\ell_1$ solutions under tiny initialization, stagewise support activation, and high-dimensional speed–generalization trade-offs (Pesme et al., 2023, Labarrière et al., 2024, Nishiyama et al., 2 Oct 2025).

1. Formal models and parameterizations

The basic supervised setting is least-squares regression with data $(x_i,y_i)\in\mathbb R^d\times\mathbb R$, $i=1,\dots,n$, and loss

$$L(\beta)=\frac1{2n}\sum_{i=1}^n(\langle \beta,x_i\rangle-y_i)^2.$$

A two-layer DLN introduces $u,v\in\mathbb R^d$ and sets $\beta=u\odot v$, giving predictor

$f(u,v;x)=\langle u\odot v,x\rangle,$

and induced loss

$F(u,v)=L(u\odot v).$

Equivalent presentations appear throughout the literature, including $F(u,v)=L(u\odot v)$0, $F(u,v)=L(u\odot v)$1, and $F(u,v)=L(u\odot v)$2 parameterizations, all of which preserve linearity in the input while making optimization nonconvex in the parameters (Pesme et al., 2023, Papazov et al., 2024, Pesme et al., 2021, Nishiyama et al., 2 Oct 2025).

Setting	Effective predictor	Representative source
Two-layer Hadamard DLN	$F(u,v)=L(u\odot v)$3	(Pesme et al., 2023)
Difference-of-squares DLN	$F(u,v)=L(u\odot v)$4	(Pesme et al., 2021)
Two-layer symmetric form	$F(u,v)=L(u\odot v)$5	(Nishiyama et al., 2 Oct 2025)
Deep DLN	$F(u,v)=L(u\odot v)$6	(Labarrière et al., 2024)
Depth-$F(u,v)=L(u\odot v)$7 positive/negative DLN	$F(u,v)=L(u\odot v)$8	(Wind et al., 2023)

The deep case replaces the two factors by $F(u,v)=L(u\odot v)$9 diagonal layers. With $\ell_1$0, the effective weight is

$\ell_1$1

This depth does not enlarge the underlying linear function class, but it changes the induced geometry and the selected solution of the training problem (Labarrière et al., 2024, Wind et al., 2023).

A central point, often misunderstood, is that DLNs are not interesting because they define a richer predictor class than linear regression. Their significance lies in the reparameterization: diagonal multiplicative structure creates a tractable nonconvex model whose optimization dynamics can be analyzed exactly or asymptotically, and whose implicit bias is often sparse or sparsity-leaning (Berthier, 2022, Pesme et al., 2023).

2. Gradient flow, preconditioning, and mirror-flow structure

For the two-layer Hadamard model, the gradient-flow ODE is explicit. Writing $\ell_1$2,

$\ell_1$3

and hence

$\ell_1$4

This already shows that the effective predictor does not follow ordinary gradient flow in $\ell_1$5-space; its dynamics are multiplicatively preconditioned by the current factorization (Pesme et al., 2023).

Under vanishing initialization, a mirror-flow description becomes exact. For

$\ell_1$6

the trajectory satisfies a mirror-ODE based on the hyperbolic-entropy potential

$\ell_1$7

namely

$\ell_1$8

After the renormalization $\ell_1$9, the rescaled potential $(x_i,y_i)\in\mathbb R^d\times\mathbb R$0 satisfies $(x_i,y_i)\in\mathbb R^d\times\mathbb R$1, which is the source of the $(x_i,y_i)\in\mathbb R^d\times\mathbb R$2-type bias in the tiny-initialization regime (Pesme et al., 2023).

For deep DLNs, the effective parameter $(x_i,y_i)\in\mathbb R^d\times\mathbb R$3 obeys a preconditioned flow

$(x_i,y_i)\in\mathbb R^d\times\mathbb R$4

where $(x_i,y_i)\in\mathbb R^d\times\mathbb R$5 with

$(x_i,y_i)\in\mathbb R^d\times\mathbb R$6

Under a mild initialization assumption ensuring invertibility of $(x_i,y_i)\in\mathbb R^d\times\mathbb R$7, this becomes a mirror flow

$(x_i,y_i)\in\mathbb R^d\times\mathbb R$8

for a convex entropy $(x_i,y_i)\in\mathbb R^d\times\mathbb R$9 satisfying $i=1,\dots,n$0 (Labarrière et al., 2024).

This mirror-flow viewpoint is structurally important. It explains why DLN training selects constrained minimizers of a separable potential rather than simply following Euclidean steepest descent in predictor space. It also clarifies why initialization scale, layer imbalance, and stochastic perturbations change the selected solution even when the training constraints are identical (Labarrière et al., 2024, Pesme et al., 2021).

3. Implicit regularization, initialization scale, and depth dependence

In overparameterized regression, a defining result is that small initialization drives DLNs toward sparse interpolants. In the two-layer setting analyzed via vanishing initialization and renormalized time, the final saddle is the unique minimum-$i=1,\dots,n$1-norm interpolant of the data (Pesme et al., 2023). Closely related work on basis pursuit shows more sharply that, as $i=1,\dots,n$2, the gradient flow of a depth-$i=1,\dots,n$3 DLN approximates minimizers of

$i=1,\dots,n$4

not merely the basis-pursuit objective value (Wind et al., 2023).

The finite-$i=1,\dots,n$5 limit is not arbitrary. For every fixed $i=1,\dots,n$6 and $i=1,\dots,n$7, there exist constants $i=1,\dots,n$8 such that

$i=1,\dots,n$9

and for $$L(\beta)=\frac1{2n}\sum_{i=1}^n(\langle \beta,x_i\rangle-y_i)^2.$$0 one has $$L(\beta)=\frac1{2n}\sum_{i=1}^n(\langle \beta,x_i\rangle-y_i)^2.$$1, hence

$$L(\beta)=\frac1{2n}\sum_{i=1}^n(\langle \beta,x_i\rangle-y_i)^2.$$2

The selected $$L(\beta)=\frac1{2n}\sum_{i=1}^n(\langle \beta,x_i\rangle-y_i)^2.$$3 minimizer depends on depth: for $$L(\beta)=\frac1{2n}\sum_{i=1}^n(\langle \beta,x_i\rangle-y_i)^2.$$4, $$L(\beta)=\frac1{2n}\sum_{i=1}^n(\langle \beta,x_i\rangle-y_i)^2.$$5 with $$L(\beta)=\frac1{2n}\sum_{i=1}^n(\langle \beta,x_i\rangle-y_i)^2.$$6, whereas for $$L(\beta)=\frac1{2n}\sum_{i=1}^n(\langle \beta,x_i\rangle-y_i)^2.$$7, $$L(\beta)=\frac1{2n}\sum_{i=1}^n(\langle \beta,x_i\rangle-y_i)^2.$$8 (Wind et al., 2023).

This depth dependence corrects a common oversimplification. Tiny-initialization DLNs do not merely “solve $$L(\beta)=\frac1{2n}\sum_{i=1}^n(\langle \beta,x_i\rangle-y_i)^2.$$9 minimization” in a universal sense; when the basis-pursuit minimizer is nonunique, the particular minimizer depends on the network depth through the selection rule $u,v\in\mathbb R^d$0 (Wind et al., 2023).

Initialization scale controls a second, broader interpolation between dense and sparse bias. In the proportional high-dimensional limit with $u,v\in\mathbb R^d$1 and $u,v\in\mathbb R^d$2, the long-time limit minimizes a data-dependent norm

$u,v\in\mathbb R^d$3

As $u,v\in\mathbb R^d$4, this norm tends to $u,v\in\mathbb R^d$5; as $u,v\in\mathbb R^d$6, it tends to $u,v\in\mathbb R^d$7 (Nishiyama et al., 2 Oct 2025). The deep-DLN optimization perspective states the same dichotomy as a kernel/lazy versus rich regime: large initialization produces an $u,v\in\mathbb R^d$8 behavior associated with minimum-$u,v\in\mathbb R^d$9 bias, whereas small initialization makes the mirror map $\beta=u\odot v$0 $\beta=u\odot v$1-like but slows convergence because the rate constant depends on initialization gaps (Labarrière et al., 2024).

A plausible implication is that DLNs are best viewed not as single-bias models, but as a family of initialization- and depth-indexed implicit regularizers whose convex surrogates become increasingly sparse as scale shrinks or homogeneity increases.

4. Trajectory geometry: incremental learning, saddles, and the Lasso path

The full training trajectory of DLNs is now understood in considerably more detail than the endpoint alone. An early rigorous result, under the underparameterized regime and anti-correlated features $\beta=u\odot v$2 for $\beta=u\odot v$3, showed that coordinates are successively activated as initialization tends to zero. In that setting, there is a piecewise-constant activation set $\beta=u\odot v$4 such that

$\beta=u\odot v$5

where $\beta=u\odot v$6 is the unique fixed point supported on $\beta=u\odot v$7, and the time-average recovers

$\beta=u\odot v$8

This identifies incremental learning with a decaying weighted $\beta=u\odot v$9 penalty $f(u,v;x)=\langle u\odot v,x\rangle,$0 (Berthier, 2022).

A later and more general analysis removed the need for monotone support growth. For any index set $f(u,v;x)=\langle u\odot v,x\rangle,$1, a saddle of the training loss maps to a vector $f(u,v;x)=\langle u\odot v,x\rangle,$2 satisfying

$f(u,v;x)=\langle u\odot v,x\rangle,$3

Thus every visited saddle is the exact minimizer of the least-squares loss restricted to its support. After the $f(u,v;x)=\langle u\odot v,x\rangle,$4 time renormalization, the vanishing-initialization flow converges to a limiting piecewise-constant process that sticks at saddles $f(u,v;x)=\langle u\odot v,x\rangle,$5 and jumps instantaneously between them along heteroclinic orbits. The visited saddles and jump times are characterized by a recursive algorithm reminiscent of the LARS algorithm used for the Lasso path, and the recursion terminates after at most $f(u,v;x)=\langle u\odot v,x\rangle,$6 steps at the minimal-$f(u,v;x)=\langle u\odot v,x\rangle,$7 solution (Pesme et al., 2023).

One notable consequence is that active-set size need not be monotone. The general theory explicitly allows both activations and deactivations, so coordinates can leave the active set later in training. This is the opposite of the monotone support-growth picture that one might infer from simpler sparse-path analogies (Pesme et al., 2023).

A further refinement identifies an explicit relationship to the Lasso regularization path. For the averaged trajectory $f(u,v;x)=\langle u\odot v,x\rangle,$8 under the rescaling

$f(u,v;x)=\langle u\odot v,x\rangle,$9

one has

$F(u,v)=L(u\odot v).$0

so training time plays the role of an inverse regularization parameter. If the parametric Lasso path satisfies the monotonicity condition that $F(u,v)=L(u\odot v).$1 is coordinate-wise monotone, then the match is exact at the level of the Lasso objective value; without monotonicity, the gap is controlled by the downward variation $F(u,v)=L(u\odot v).$2 and remains small whenever $F(u,v)=L(u\odot v).$3 is small (Berthier, 23 Sep 2025).

Taken together, these results make DLNs one of the rare nonconvex neural models for which the entire optimization trajectory can be interpreted as a stagewise sparse path rather than merely as convergence to a sparse endpoint.

5. Stochasticity, momentum, stepsize, and sharpness-aware perturbations

Stochasticity changes the implicit bias in a quantitatively analyzable way. In the two-layer difference-of-squares model, stochastic gradient flow satisfies a stochastic mirror-descent equation with a time-varying hyperbolic-entropy potential. If the trajectory converges, the limit solves

$F(u,v)=L(u\odot v).$4

with

$F(u,v)=L(u\odot v).$5

Since $F(u,v)=L(u\odot v).$6 coordinate-wise, the stochastic flow is strictly more $F(u,v)=L(u\odot v).$7-biased than batch gradient flow. The same analysis shows that slower loss convergence yields stronger bias because the biasing effect is controlled by $F(u,v)=L(u\odot v).$8 (Pesme et al., 2021).

Constant stepsizes also matter. For GD and SGD with macroscopic stepsizes in the overparameterized regression setting, the limit is still characterized as the unique minimizer of a hypentropy potential $F(u,v)=L(u\odot v).$9 over the interpolation set, but the effective scale $F(u,v)=L(u\odot v)$00 depends on the entire trajectory and on the stepsize. The resulting qualitative picture is asymmetric: large stepsizes consistently benefit SGD for sparse regression problems, while they can hinder the recovery of sparse solutions for GD, and these effects are magnified in the edge-of-stability regime just below the divergence threshold (Even et al., 2023).

Momentum introduces a second-order continuous-time dynamics

$F(u,v)=L(u\odot v)$01

In the two-layer diagonal network, $F(u,v)=L(u\odot v)$02 is the intrinsic parameter that uniquely defines the optimization path up to time-reparameterization. The recovered solution is characterized through an implicit regularization problem involving the Bregman divergence of a hyperbolic entropy $F(u,v)=L(u\odot v)$03, and small values of $F(u,v)=L(u\odot v)$04 help to recover sparse solutions. Comparable, though weaker, results are obtained for stochastic momentum gradient descent (Papazov et al., 2024).

Noise added directly to the factors yields another distinct regularization mechanism. In stochastic sharpness-aware minimization for depth-$F(u,v)=L(u\odot v)$05 DLNs, isotropic Gaussian perturbations transform the expected objective into $F(u,v)=L(u\odot v)$06, where

$F(u,v)=L(u\odot v)$07

Every critical point of $F(u,v)=L(u\odot v)$08 is balanced in the sense that $F(u,v)=L(u\odot v)$09, and gradient flow or descent on $F(u,v)=L(u\odot v)$10 enforces balancing at an exponential or geometric rate. Among factorizations of a fixed end-to-end matrix, the balanced one uniquely minimizes both the induced average-sharpness penalty and the trace of the Hessian. The resulting stationary condition yields coordinatewise shrinkage-thresholding of the teacher parameter, with the noise level regulating both the shrinkage factor and the threshold (Clara et al., 14 Mar 2025).

These variants show that DLNs are not tied to a single optimization story. Stochasticity, inertial terms, and noise-based sharpness penalties all alter the effective convex potential seen in predictor space, yet they do so in ways that remain analytically explicit.

6. High-dimensional theories, algorithmic links, and broader scope

In the proportional high-dimensional limit $F(u,v)=L(u\odot v)$11 with $F(u,v)=L(u\odot v)$12, DLNs admit a unifying Dynamical Mean-Field Theory reduction. For the symmetric parameterization

$F(u,v)=L(u\odot v)$13

the gradient-flow dynamics concentrate onto a scalar stochastic integro-differential effective process coupled to kernels $F(u,v)=L(u\odot v)$14. This reduction yields fixed-point laws, convergence rates, phase-specific timescales, and an explicit speed–generalization trade-off. Among the reported consequences are: for $F(u,v)=L(u\odot v)$15, the limit solves $F(u,v)=L(u\odot v)$16-regularized regression; for $F(u,v)=L(u\odot v)$17 and $F(u,v)=L(u\odot v)$18, it is the unique ridgeless least-squares solution; and for $F(u,v)=L(u\odot v)$19 and $F(u,v)=L(u\odot v)$20, it is an interpolator minimizing the data-dependent norm $F(u,v)=L(u\odot v)$21. The same framework reproduces lazy versus rich training, search versus descent in the small-$F(u,v)=L(u\odot v)$22 regime, coordinate-wise activation times, and the monotonic dependence of the convergence-rate parameter $F(u,v)=L(u\odot v)$23 on the initialization scale (Nishiyama et al., 2 Oct 2025).

A complementary high-dimensional program analyzes fixed-stepsize SGD directly. In that setting, the iterates are approximated by an SDE whose drift and gradient-noise terms are explicitly decoupled, and a deterministic nonlinear PDE propagates observable statistics such as risk and curvature. In the isotropic squared parametrization with constant stepsize below a threshold, the resulting SDE is globally well posed and the risk decays exponentially fast to zero with high probability (Malaxechebarría et al., 16 May 2026).

DLNs also serve as algorithmic solvers for classical optimization problems. For linear programming, the nonnegative parameterization $F(u,v)=L(u\odot v)$24 combined with the slack-penalized objective $F(u,v)=L(u\odot v)$25 leads, under gradient flow, to the unique minimizer of the entropically regularized linear program

$F(u,v)=L(u\odot v)$26

with the regularization strength controlled by the initialization. In discrete time, under strict feasibility and full-row-rank assumptions, the same framework gives global linear convergence and an “almost” entropic LP in which the extra term vanishes as the stepsize tends to zero. Basis pursuit and optimal transport are explicit examples (Wang et al., 2023).

Another algorithmic line connects DLNs to reweighted least-squares. In a batched Gaussian design model, a unified family encompassing IRLS, the lin-RFM algorithm, and alternating minimization over a linear diagonal network admits a precise state-evolution recursion for the joint empirical law of the iterates. This theory supports optimized reweighting policies, shows that only a handful of iterations can suffice asymptotically, and extends naturally to group-sparse recovery, where group-aware reweighting improves test error relative to coordinate-wise reweighting (Kaushik et al., 2024).

Recent work also calibrates the DLN initialization scale $F(u,v)=L(u\odot v)$27 to an effective exponent $F(u,v)=L(u\odot v)$28 by matching the separable DLN potential to explicit minimum-$F(u,v)=L(u\odot v)$29 interpolation. In that formulation, the family of norms $F(u,v)=L(u\odot v)$30 exhibits a data-dependent elbow $F(u,v)=L(u\odot v)$31 and a universal threshold $F(u,v)=L(u\odot v)$32 separating norms that plateau from norms that continue to grow with sample size, and empirical DLN experiments are reported to inherit the same elbow/threshold laws once $F(u,v)=L(u\odot v)$33 is replaced by $F(u,v)=L(u\odot v)$34 (Zhang et al., 25 Sep 2025).

The cumulative picture is that DLNs constitute a compact analytical laboratory for nonconvex learning dynamics. They connect mirror descent, sparse interpolation, heteroclinic optimization paths, stochastic regularization, mean-field limits, and classical optimization algorithms within a single diagonal framework.