---
title: 'Diagonal Linear Networks: Dynamics'
url: https://www.emergentmind.com/topics/diagonal-linear-networks
type: topic
---

# Diagonal Linear Networks: Dynamics

Diagonal linear networks (DLNs) are overparameterized linear models in which the effective predictor is represented through diagonal-layer parameters rather than as a single weight vector. In the canonical two-layer regression model, one writes $\beta=u\odot v$ and optimizes $F(u,v)=L(u\odot v)$, so the hypothesis class remains linear while the parameterization becomes nonconvex. This minimal reparameterization is sufficient to produce mirror-flow structure, initialization-dependent implicit regularization, sparse minimum-$\ell_1$ solutions under tiny initialization, stagewise support activation, and high-dimensional speed–generalization trade-offs [2304.00488] [2412.16765] [2510.01930].

## 1. Formal models and parameterizations

The basic supervised setting is least-squares regression with data $(x_i,y_i)\in\mathbb R^d\times\mathbb R$, $i=1,\dots,n$, and loss
$$
L(\beta)=\frac1{2n}\sum_{i=1}^n(\langle \beta,x_i\rangle-y_i)^2.
$$
A two-layer DLN introduces $u,v\in\mathbb R^d$ and sets $\beta=u\odot v$, giving predictor
$$
f(u,v;x)=\langle u\odot v,x\rangle,
$$
and induced loss
$$
F(u,v)=L(u\odot v).
$$
Equivalent presentations appear throughout the literature, including $\theta=u\odot v$, $w_+^2-w_-^2$, and $\frac12(u\odot u-v\odot v)$ parameterizations, all of which preserve linearity in the input while making optimization nonconvex in the parameters [2304.00488] [2403.05293] [2106.09524] [2510.01930].

| Setting | Effective predictor | Representative source |
|---|---|---|
| Two-layer Hadamard DLN | $\beta=u\odot v$ | [2304.00488] |
| Difference-of-squares DLN | $\beta_w=w_+^2-w_-^2$ | [2106.09524] |
| Two-layer symmetric form | $w=\frac12(u\odot u-v\odot v)$ | [2510.01930] |
| Deep DLN | $\theta=u^1\odot\cdots\odot u^L$ | [2412.16765] |
| Depth-$p$ positive/negative DLN | $\psi_\theta=\theta_+^p-\theta_-^p$ | [2307.07410] |

The deep case replaces the two factors by $L$ diagonal layers. With $u^1,\dots,u^L\in\mathbb R^d$, the effective weight is
$$
\theta=u^1\odot u^2\odot\cdots\odot u^L.
$$
This depth does not enlarge the underlying linear function class, but it changes the induced geometry and the selected solution of the training problem [2412.16765] [2307.07410].

A central point, often misunderstood, is that DLNs are not interesting because they define a richer predictor class than linear regression. Their significance lies in the reparameterization: diagonal multiplicative structure creates a tractable nonconvex model whose optimization dynamics can be analyzed exactly or asymptotically, and whose implicit bias is often sparse or sparsity-leaning [2208.14673] [2304.00488].

## 2. Gradient flow, preconditioning, and mirror-flow structure

For the two-layer Hadamard model, the gradient-flow ODE is explicit. Writing $\beta=u\odot v$,
$$
\nabla_uF(u,v)=\nabla L(\beta)\odot v,\qquad
\nabla_vF(u,v)=\nabla L(\beta)\odot u,
$$
and hence
$$
\dot u=-\nabla L(\beta)\odot v,\qquad
\dot v=-\nabla L(\beta)\odot u.
$$
This already shows that the effective predictor does not follow ordinary gradient flow in $\beta$-space; its dynamics are multiplicatively preconditioned by the current factorization [2304.00488].

Under vanishing initialization, a mirror-flow description becomes exact. For
$$
u(0)=\sqrt2\,\alpha\cdot\mathbf1,\qquad v(0)=0,\qquad \beta(0)=0,
$$
the trajectory satisfies a mirror-ODE based on the hyperbolic-entropy potential
$$
\tilde\phi_\alpha(\beta)
=
\frac12\sum_{j=1}^d\Bigl[\beta_j\operatorname{Arcsinh}\frac{\beta_j}{\alpha^2}
-\sqrt{\beta_j^2+\alpha^4}
+\alpha^2\Bigr],
$$
namely
$$
d[\nabla \tilde\phi_\alpha(\beta_t)]=-\nabla L(\beta_t)\,dt.
$$
After the renormalization $t'=\ln(1/\alpha)\,t$, the rescaled potential $\Phi_\alpha=\tilde\phi_\alpha/\ln(1/\alpha)$ satisfies $\Phi_\alpha\sim \|\beta\|_1$, which is the source of the $\ell_1$-type bias in the tiny-initialization regime [2304.00488].

For deep DLNs, the effective parameter $\theta=u^1\odot\cdots\odot u^L$ obeys a preconditioned flow
$$
\dot\theta + M(t)\nabla\mathcal L(\theta)=0,
$$
where $M(t)=\operatorname{diag}(m_i(t))$ with
$$
m_i(t)=\sum_{j=1}^L\prod_{k\neq j}(u_i^k(t))^2.
$$
Under a mild initialization assumption ensuring invertibility of $M(t)$, this becomes a mirror flow
$$
\frac{d}{dt}[\nabla \mathcal Q(\theta(t))]+\nabla\mathcal L(\theta(t))=0,
$$
for a convex entropy $\mathcal Q$ satisfying $\nabla^2\mathcal Q(\theta(t))=M(t)^{-1}$ [2412.16765].

This mirror-flow viewpoint is structurally important. It explains why DLN training selects constrained minimizers of a separable potential rather than simply following Euclidean steepest descent in predictor space. It also clarifies why initialization scale, layer imbalance, and stochastic perturbations change the selected solution even when the training constraints are identical [2412.16765] [2106.09524].

## 3. Implicit regularization, initialization scale, and depth dependence

In overparameterized regression, a defining result is that small initialization drives DLNs toward sparse interpolants. In the two-layer setting analyzed via vanishing initialization and renormalized time, the final saddle is the unique minimum-$\ell_1$-norm interpolant of the data [2304.00488]. Closely related work on basis pursuit shows more sharply that, as $\alpha\to0$, the gradient flow of a depth-$p$ DLN approximates minimizers of
$$
U(A,y)=\operatorname{Arg\,min}_{z\in\mathbb R^N}\|z\|_1\quad\text{subject to }Az=y,
$$
not merely the basis-pursuit objective value [2307.07410].

The finite-$\alpha$ limit is not arbitrary. For every fixed $A,y$ and $p\ge2$, there exist constants $C_1(A,p),C_2(A,p)>0$ such that
$$
\|\psi_\alpha(\infty)-W_p(A,y)\|_2
\le
C_1\|y\|_2\Bigl(\frac{\alpha^p}{\|y\|_2}\Bigr)^{C_2},
$$
and for $p>2$ one has $C_2=1$, hence
$$
\|\psi_\alpha(\infty)-W_p\|_2=O(\alpha^p).
$$
The selected $\ell_1$ minimizer depends on depth: for $p=2$, $W_2(A,y)=\operatorname{Arg\,max}_{z\in U(A,y)}H(z)$ with $H(z)=-\sum_i|z_i|\ln|z_i|$, whereas for $p>2$, $W_p(A,y)=\operatorname{Arg\,max}_{z\in U(A,y)}\|z\|_{2/p}$ [2307.07410].

This depth dependence corrects a common oversimplification. Tiny-initialization DLNs do not merely “solve $\ell_1$ minimization” in a universal sense; when the basis-pursuit minimizer is nonunique, the particular minimizer depends on the network depth through the selection rule $W_p$ [2307.07410].

Initialization scale controls a second, broader interpolation between dense and sparse bias. In the proportional high-dimensional limit with $\lambda=0$ and $\delta<1$, the long-time limit minimizes a data-dependent norm
$$
J_\alpha(w)=\alpha^2\sum\bigl[x\,\operatorname{arcsinh}(x)-\sqrt{1+x^2}+1\bigr]_{x=w_i/\alpha^2}.
$$
As $\alpha\to\infty$, this norm tends to $\|w\|_2$; as $\alpha\to0$, it tends to $\mathrm{const}\cdot\|w\|_1$ [2510.01930]. The deep-DLN optimization perspective states the same dichotomy as a kernel/lazy versus rich regime: large initialization produces an $M(t)\approx I$ behavior associated with minimum-$\ell_2$ bias, whereas small initialization makes the mirror map $\mathcal Q$ $\ell_1$-like but slows convergence because the rate constant depends on initialization gaps [2412.16765].

A plausible implication is that DLNs are best viewed not as single-bias models, but as a family of initialization- and depth-indexed implicit regularizers whose convex surrogates become increasingly sparse as scale shrinks or homogeneity increases.

## 4. Trajectory geometry: incremental learning, saddles, and the Lasso path

The full training trajectory of DLNs is now understood in considerably more detail than the endpoint alone. An early rigorous result, under the underparameterized regime and anti-correlated features $M_{ij}\le0$ for $i\neq j$, showed that coordinates are successively activated as initialization tends to zero. In that setting, there is a piecewise-constant activation set $I(s)$ such that
$$
\theta(s\log(1/\varepsilon))\to \theta_*^{I(s)},
$$
where $\theta_*^I$ is the unique fixed point supported on $I$, and the time-average recovers
$$
\mu(s)=\arg\min_{\theta\ge0}\Bigl\{f(\theta)+\frac1s\langle r,\theta\rangle\Bigr\}.
$$
This identifies incremental learning with a decaying weighted $\ell_1$ penalty $\lambda(s)=1/s$ [2208.14673].

A later and more general analysis removed the need for monotone support growth. For any index set $A\subset\{1,\dots,d\}$, a saddle of the training loss maps to a vector $\beta_c$ satisfying
$$
|\beta_c|\odot \nabla L(\beta_c)=0
\quad\Longleftrightarrow\quad
\beta_c\in\arg\min_{\beta:\,\beta_i=0\,(i\notin A)}L(\beta),
\qquad A=\operatorname{supp}(\beta_c).
$$
Thus every visited saddle is the exact minimizer of the least-squares loss restricted to its support. After the $\ln(1/\alpha)$ time renormalization, the vanishing-initialization flow converges to a limiting piecewise-constant process that sticks at saddles $\beta_0,\beta_1,\dots,\beta_p$ and jumps instantaneously between them along heteroclinic orbits. The visited saddles and jump times are characterized by a recursive algorithm reminiscent of the LARS algorithm used for the Lasso path, and the recursion terminates after at most $\min(2^d,\sum_{k=0}^n\binom d k)$ steps at the minimal-$\ell_1$ solution [2304.00488].

One notable consequence is that active-set size need not be monotone. The general theory explicitly allows both activations and deactivations, so coordinates can leave the active set later in training. This is the opposite of the monotone support-growth picture that one might infer from simpler sparse-path analogies [2304.00488].

A further refinement identifies an explicit relationship to the Lasso regularization path. For the averaged trajectory $\overline x^\varepsilon(s)$ under the rescaling
$$
s=\frac{2}{\log(1/\varepsilon)}\,t,
$$
one has
$$
\overline x^\varepsilon(s)\approx \arg\min_x\{\ell(x)+(1/s)\|x\|_1\},
$$
so training time plays the role of an inverse regularization parameter. If the parametric Lasso path satisfies the monotonicity condition that $z(s)=s\,x(s)$ is coordinate-wise monotone, then the match is exact at the level of the Lasso objective value; without monotonicity, the gap is controlled by the downward variation $z^\downarrow(s)$ and remains small whenever $z^\downarrow(s)$ is small [2509.18766].

Taken together, these results make DLNs one of the rare nonconvex neural models for which the entire optimization trajectory can be interpreted as a stagewise sparse path rather than merely as convergence to a sparse endpoint.

## 5. Stochasticity, momentum, stepsize, and sharpness-aware perturbations

Stochasticity changes the implicit bias in a quantitatively analyzable way. In the two-layer difference-of-squares model, stochastic gradient flow satisfies a stochastic mirror-descent equation with a time-varying hyperbolic-entropy potential. If the trajectory converges, the limit solves
$$
\beta_\infty=\arg\min_{X\beta=y}\phi_{\alpha_{\mathrm{eff}}}(\beta),
$$
with
$$
\alpha_{\mathrm{eff}}
=
\alpha\odot
\exp\!\Bigl(-2\gamma\,\operatorname{Diag}(X^\top X/n)\int_0^\infty L(\beta_s)\,ds\Bigr).
$$
Since $\alpha_{\mathrm{eff}}\le\alpha$ coordinate-wise, the stochastic flow is strictly more $\ell_1$-biased than batch gradient flow. The same analysis shows that slower loss convergence yields stronger bias because the biasing effect is controlled by $\int_0^\infty L(\beta_s)\,ds$ [2106.09524].

Constant stepsizes also matter. For GD and SGD with macroscopic stepsizes in the overparameterized regression setting, the limit is still characterized as the unique minimizer of a hypentropy potential $\psi_{\alpha_\infty}$ over the interpolation set, but the effective scale $\alpha_\infty$ depends on the entire trajectory and on the stepsize. The resulting qualitative picture is asymmetric: large stepsizes consistently benefit SGD for sparse regression problems, while they can hinder the recovery of sparse solutions for GD, and these effects are magnified in the edge-of-stability regime just below the divergence threshold [2302.08982].

Momentum introduces a second-order continuous-time dynamics
$$
\lambda \ddot x_t+\dot x_t+\nabla F(x_t)=0,
\qquad
\lambda=\frac{\gamma}{(1-\beta)^2}.
$$
In the two-layer diagonal network, $\lambda$ is the intrinsic parameter that uniquely defines the optimization path up to time-reparameterization. The recovered solution is characterized through an implicit regularization problem involving the Bregman divergence of a hyperbolic entropy $\psi_{\Delta_\infty}$, and small values of $\lambda$ help to recover sparse solutions. Comparable, though weaker, results are obtained for stochastic momentum gradient descent [2403.05293].

Noise added directly to the factors yields another distinct regularization mechanism. In stochastic sharpness-aware minimization for depth-$L$ DLNs, isotropic Gaussian perturbations transform the expected objective into $L_R=L+R$, where
$$
R(W_1,\dots,W_L)=\operatorname{Tr}\!\Bigl(\prod_{\ell=1}^L(W_\ell^2+\eta^2I_d)-\prod_{\ell=1}^LW_\ell^2\Bigr).
$$
Every critical point of $L_R$ is balanced in the sense that $W_\ell^2=W_{\ell+1}^2$, and gradient flow or descent on $L_R$ enforces balancing at an exponential or geometric rate. Among factorizations of a fixed end-to-end matrix, the balanced one uniquely minimizes both the induced average-sharpness penalty and the trace of the Hessian. The resulting stationary condition yields coordinatewise shrinkage-thresholding of the teacher parameter, with the noise level regulating both the shrinkage factor and the threshold [2503.11891].

These variants show that DLNs are not tied to a single optimization story. Stochasticity, inertial terms, and noise-based sharpness penalties all alter the effective convex potential seen in predictor space, yet they do so in ways that remain analytically explicit.

## 6. High-dimensional theories, algorithmic links, and broader scope

In the proportional high-dimensional limit $n,d\to\infty$ with $n/d\to\delta$, DLNs admit a unifying Dynamical Mean-Field Theory reduction. For the symmetric parameterization
$$
f(x;u,v)=w^\top x,\qquad w=\tfrac12(u\odot u-v\odot v),
$$
the gradient-flow dynamics concentrate onto a scalar stochastic integro-differential effective process coupled to kernels $C_w,C_f,R_w,R_f$. This reduction yields fixed-point laws, convergence rates, phase-specific timescales, and an explicit speed–generalization trade-off. Among the reported consequences are: for $\lambda>0$, the limit solves $\ell_1$-regularized regression; for $\lambda=0$ and $\delta>1$, it is the unique ridgeless least-squares solution; and for $\lambda=0$ and $\delta<1$, it is an interpolator minimizing the data-dependent norm $J_\alpha$. The same framework reproduces lazy versus rich training, search versus descent in the small-$\alpha$ regime, coordinate-wise activation times, and the monotonic dependence of the convergence-rate parameter $\gamma(\alpha)$ on the initialization scale [2510.01930].

A complementary high-dimensional program analyzes fixed-stepsize SGD directly. In that setting, the iterates are approximated by an SDE whose drift and gradient-noise terms are explicitly decoupled, and a deterministic nonlinear PDE propagates observable statistics such as risk and curvature. In the isotropic squared parametrization with constant stepsize below a threshold, the resulting SDE is globally well posed and the risk decays exponentially fast to zero with high probability [2605.17177].

DLNs also serve as algorithmic solvers for classical optimization problems. For linear programming, the nonnegative parameterization $x=u\circ u$ combined with the slack-penalized objective $\frac12\|Ax-b\|_2^2$ leads, under gradient flow, to the unique minimizer of the entropically regularized linear program
$$
\min_{x\ge0}\; c^\top x+\varepsilon\sum_{i=1}^n[x_i\log x_i-x_i]
\quad\text{subject to }Ax=b,
$$
with the regularization strength controlled by the initialization. In discrete time, under strict feasibility and full-row-rank assumptions, the same framework gives global linear convergence and an “almost” entropic LP in which the extra term vanishes as the stepsize tends to zero. Basis pursuit and optimal transport are explicit examples [2310.02535].

Another algorithmic line connects DLNs to reweighted least-squares. In a batched Gaussian design model, a unified family encompassing IRLS, the lin-RFM algorithm, and alternating minimization over a linear diagonal network admits a precise state-evolution recursion for the joint empirical law of the iterates. This theory supports optimized reweighting policies, shows that only a handful of iterations can suffice asymptotically, and extends naturally to group-sparse recovery, where group-aware reweighting improves test error relative to coordinate-wise reweighting [2406.02769].

Recent work also calibrates the DLN initialization scale $\alpha$ to an effective exponent $p_{\mathrm{eff}}(\alpha)$ by matching the separable DLN potential to explicit minimum-$\ell_p$ interpolation. In that formulation, the family of norms $\{\|\widehat w\|_r\}_{r\in[1,p]}$ exhibits a data-dependent elbow $n_\star$ and a universal threshold $r_\star=2(p-1)$ separating norms that plateau from norms that continue to grow with sample size, and empirical DLN experiments are reported to inherit the same elbow/threshold laws once $p$ is replaced by $p_{\mathrm{eff}}(\alpha)$ [2509.21181].

The cumulative picture is that DLNs constitute a compact analytical laboratory for nonconvex learning dynamics. They connect mirror descent, sparse interpolation, heteroclinic optimization paths, stochastic regularization, mean-field limits, and classical optimization algorithms within a single diagonal framework.

Source: https://www.emergentmind.com/topics/diagonal-linear-networks