Training Dynamics & Gradient Flows

• Training Dynamics and Gradient Flows are the differential processes governing iterative optimization, characterized by homogeneous equations and finite-time extinction.
• The analytic framework distinguishes linear, nonlinear, and adapted regimes using modal decomposition techniques to capture distinct decay behaviors.
• Applications in machine learning include enhancing model interpretability, guiding regularization methods, and informing adaptive learning rate designs.

Training dynamics and gradient flows are central to the analysis and understanding of optimization processes in machine learning, signal processing, and applied mathematics. These concepts provide a differential perspective on how parameters evolve during iterative algorithms such as gradient descent, with direct implications for the interpretation of learned models, implicit bias, convergence, and the emergence of latent data structures.

1. Analytic Framework for Homogeneous Gradient Flows

The paper of homogeneous gradient flows focuses on evolution equations of the form

$\psi_t = P(\psi)$

where $P$ is a nonlinear homogeneous operator, often deriving from the negative gradient of a homogeneous energy functional $R$ (Cohen et al., 2020). Homogeneity implies that for $\alpha > 0$, $P(\alpha \psi) = \alpha^{p-1} P(\psi)$ for some degree $p > 1$.

When the initial datum is an eigenfunction of $P$, i.e., $P(f) = \lambda f$, the solution admits a scaling form

$$\psi(t) = a(t) f, \qquad a(t) = \left[1 + (2-p)\lambda t \right]^{\frac{1}{2-p}}$$

which exhibits finite-time extinction when $p \ne 2$. This explicit solution elucidates the closed-form decay of homogeneous flows, with extinction time

$T_{ext} = -\frac{1}{\lambda(2-p)}$

demonstrating that the decay is polynomial for $p \ne 2$ and exponential (linear case) when $p = 2$. This framework sets the foundation for spectral analysis of gradient flows.

2. Spectral Modes and Nonlinear Decomposition

The evolving state under a homogeneous flow can, in general, be decomposed into nonlinear "modes", defined as components that approximately satisfy a nonlinear eigenvalue problem: $P(\phi_i) \approx \lambda_i \phi_i$ When sampled at discrete times, the solution becomes a sum of such modal components, each with its own decay rate. This approach generalizes classic linear modal decomposition—such as Fourier or singular value decomposition—to nonlinear and non-exponential decay regimes that arise in regularization and deep learning. In the eigenfunction case, the modal decomposition collapses to a single mode corresponding to $f$ itself.

3. Dynamic Mode Decomposition and Its Limitations

Dynamic Mode Decomposition (DMD) is a data-driven method from dynamical systems theory widely adopted for uncovering latent linear structure in times series data. In its classical form, DMD fits an evolution operator between consecutive snapshots and identifies its eigenvalues (decay rates) and eigenvectors (modes) (Cohen et al., 2020).

When applied to homogeneous gradient flows:

• DMD produces an effective mode and decay rate by fitting exponentials to the trajectory data.
• For polynomial decays ($p \ne 2$), this linear fit becomes fundamentally mismatched; DMD may display vanishing least-squares error as the sampling becomes dense, yet the error in reconstructing the nonlinear dynamics remains bounded. This is the "DMD paradox."
• Only in the linear case ($p = 2$) does DMD provide an exact representation, as the dynamics are truly linear/exponential.

To resolve this, adaptive time sampling is introduced. By reparameterizing time—or equivalently, adaptively modulating the sample spacing—the non-exponential decay is transformed into an exponential one, to which DMD is again well-suited. Specifically, the time-rescaling is chosen so that the sampled $\psi_k$ forms a geometric progression, enabling exact recovery by DMD with zero error in the eigenfunction case.

An additional refinement is the use of Symmetric DMD (SDMD), enforcing the fitted evolution operator to be symmetric, thereby ensuring a real spectrum and better capturing non-oscillatory flows typical in smoothing or regularization.

4. Orthogonal Nonlinear Spectral Decomposition (OrthoNS)

The Orthogonal Nonlinear Spectral Decomposition (OrthoNS) framework generalizes classical spectral analysis to nonlinear flows by leveraging time-rescaled DMD and SDMD. The main steps are:

• Apply time-rescaled SDMD to obtain a set of orthogonal modes $\{\phi_i\}$ and associated decay parameters.
• Each mode approximates a nonlinear eigenfunction:

$$P(\phi_i) \approx \left[ \frac{1-\mu_i}{\delta} \cdot \frac{||P(\phi_i)||^2}{\langle P(\phi_i), \phi_i\rangle} \right] \phi_i$$

• The solution is reconstructed as:

$$\hat\psi(t) = \sum_i \alpha_i \phi_i [1 + \lambda_i(2-p)t]^{\frac{1}{2-p}}$$

• The spectral information comprises pairs $(T_i, |\alpha_i|^2)$, with $T_i$ (the mode extinction time) and $|\alpha_i|^2$ (mode energy).
• Spectral filtering is analogized to classical filtering by modulation of modal coefficients.

A Parseval-type identity holds: $||\hat f||^2 = \sum_i |\alpha_i|^2$ ensuring that the energy is preserved in the nonlinear modal decomposition.

5. Implications for Gradient-Based Machine Learning

Homogeneous gradient flows and their spectral analysis provide a theoretical lens for understanding training dynamics in modern machine learning:

• Standard neural network training via gradient descent can be viewed as a discretized homogeneous gradient flow, especially when regularizers or certain loss functions induce nonlinear homogeneity.
• The modal decomposition of the training trajectory enables interpretability: one may attribute the network's evolution to latent “modes” (eigenfeatures) and their corresponding decay rates.
• The identification of dominant modes offers a principled basis for early stopping, regularization, and potentially for network pruning or model compression.
• Adaptive time-rescaling aligns with the notion of learning rate schedules; by effectively “homogenizing” the dynamics, one can enforce more stable and interpretable convergence.

The spectral normalization obtained by adaptive sampling could inspire new algorithms for adaptive step size selection in deep networks, targeting a regime where effective dynamics are simple and well-structured.

6. Summary Table: Classical vs. Homogeneous vs. Adapted Modal Analysis

Regime	Decay Law (a(t))	DMD Accuracy	Modal Representation
Linear ($p=2$)	$e^{\lambda t}$	Exact	Standard DMD
Homogeneous ($p\ne 2$)	$[1 + (2-p)\lambda t]^{1/(2-p)}$	Inaccurate	Nonlinear, requires OrthoNS
Adapted (rescaled)	Geometric progression w/ time rescaling	Exact	SDMD/OrthoNS

This table clarifies the relationship between decay profile, DMD performance, and the appropriate spectral representation for each regime.

7. Outlook and Research Directions

The analytic and spectral approach to gradient flows outlined here extends naturally to:

• Nonlinear energy regularization (e.g., total variation, $p$-Dirichlet),
• Variational perspectives on representation learning,
• Development of adaptive normalization and step-size schedules,
• Interpretability and unsupervised structure extraction from deep model evolution.

These directions bridge spectral theory, dynamical systems, and machine learning, advancing theoretical understanding while motivating practical algorithmic innovation (Cohen et al., 2020).