Neural Network Training Dynamics

• Neural Network Training Dynamics (NNTD) is the study of how neural networks evolve during optimization using multi-faceted state-space lenses.
• It combines observations from prediction, parameter, activation, graph, and function spaces to characterize uncertainty, instability, and regime changes.
• These insights enable practical applications like improved uncertainty estimation, selective prediction, early performance forecasting, and model compression.

Neural Network Training Dynamics (NNTD) denotes the study of how neural networks evolve during optimization, but the term does not refer to a single canonical object. In the recent literature, it can mean per-sample prediction trajectories across epochs, sequences of intermediate checkpoints, trajectories in parameter space, temporal graphs induced by evolving weights, activation-pattern trajectories in piecewise-linear subnetworks, low-dimensional scalar embeddings of those trajectories, or population-level coupled dynamics of parameters and hyperparameters (Gu et al., 15 Sep 2025, Rabanser et al., 2022, Vahedian et al., 2021, Borghi et al., 20 Mar 2026, Pérez-Corral et al., 9 Feb 2026). Across these formulations, NNTD is used both descriptively—to characterize stability, feature emergence, regime changes, invariant structure, and chaos—and operationally, to improve uncertainty estimation, selective prediction, early performance prediction, convergence forecasting, and statistical testing (Khurana et al., 2024, Shou et al., 26 May 2025).

1. Core representations of training dynamics

A useful way to organize the literature is by several "state-space lenses" (Editor's term). In a prediction-space lens, NNTD is the sequence of per-epoch logits or probabilities assigned to each sample. In "Pseudo-D," for each training example $x_i$, the logits $f_t(x_i)\in\mathbb{R}^C$ are recorded at epochs $t=1,\dots,T$, converted to softmax probabilities $\sigma(f_t(x_i))$, and then aggregated to characterize whether the sample is easy, ambiguous, or persistently uncertain (Gu et al., 15 Sep 2025). In a checkpoint lens, NNTD is the ordered family of intermediate models $\{f_t\}_{t=1}^T$, and late-stage disagreement with the final predictor $f_T$ becomes a test-time signal for selective prediction (Rabanser et al., 2022).

A parameter-space lens treats training as a trajectory $\{\theta_t\}$ or $\{W_t\}$ in a high-dimensional parameter manifold. This is the central viewpoint in analyses of early training, edge-of-chaos behavior, and dynamical stability, where gradient descent or Langevin-type updates define the temporal evolution of weights and biases (Frankle et al., 2020, Zhang et al., 2021, Danovski et al., 2024). A related activation-space lens, specialized to ReLU networks, records binary activation patterns $a_\ell(x;\theta)\in\{0,1\}^{n_\ell}$ induced by fixed inputs. Because a fixed activation pattern determines a piecewise-linear region in which the network is affine, changes in these patterns provide a direct marker of whether training is still crossing region boundaries or has entered a within-region refinement phase (Pérez-Corral et al., 9 Feb 2026).

Other works recast NNTD in explicitly structural terms. In a graph-space lens, each training epoch yields a graph snapshot $G^{(\tau)}$ whose weighted adjacency matrix is induced by the current network parameters; training then becomes a temporal graph process, and graph centralities summarize how influence is redistributed over time (Vahedian et al., 2021). In a function-space lens, the focus is not individual parameters but the terminal hidden-layer neuron functions $f_t(x_i)\in\mathbb{R}^C$0; their Gram matrix $f_t(x_i)\in\mathbb{R}^C$1 defines an $f_t(x_i)\in\mathbb{R}^C$2-rank that measures effective feature diversity during training (Yang et al., 2024). Recent work also introduces scalar embeddings of temporal network trajectories, producing a one-dimensional time series $f_t(x_i)\in\mathbb{R}^C$3 that preserves pairwise distances between snapshots and can recover Lyapunov-style dynamical features (Jiménez-González et al., 29 Jun 2026).

These representations are not interchangeable. Some are sample-centric, some model-centric, and some population-centric. The literature therefore treats NNTD less as a singular statistic than as a family of observables tailored to different questions: uncertainty, selectivity, optimization phase structure, architectural invariants, or low-dimensional dynamical summaries.

2. Observables, equations, and reduced descriptions

In sample-level studies, the basic NNTD objects are trajectory statistics. "Pseudo-D" defines the mean trajectory $f_t(x_i)\in\mathbb{R}^C$4, variability $f_t(x_i)\in\mathbb{R}^C$5, and entropy of the mean $f_t(x_i)\in\mathbb{R}^C$6, where $f_t(x_i)\in\mathbb{R}^C$7. Its main aggregation, however, is the mean of logits,

$f_t(x_i)\in\mathbb{R}^C$8

which is then calibrated on validation data either by a global temperature $f_t(x_i)\in\mathbb{R}^C$9 or by class-wise Dirichlet calibration $t=1,\dots,T$0 (Gu et al., 15 Sep 2025). This work also evaluates calibration with Expected Calibration Error,

$t=1,\dots,T$1

making explicit that NNTD can be turned into a supervisory signal rather than used only for post hoc analysis (Gu et al., 15 Sep 2025).

Selective-prediction work defines a different class of observables. Let $t=1,\dots,T$2 be the final prediction and $t=1,\dots,T$3 the disagreement indicator at checkpoint $t=1,\dots,T$4. The two central scores are

$t=1,\dots,T$5

with convex decreasing checkpoint weights $t=1,\dots,T$6 on a normalized time axis. These scores formalize the intuition that persistent late-stage disagreement is predictive of future error (Rabanser et al., 2022).

Graph-based NNTD replaces logits or checkpoints with structural descriptors of evolving network graphs. For a temporal sequence $t=1,\dots,T$7, node-level weighted degree is

$t=1,\dots,T$8

and eigenvector centrality satisfies

$t=1,\dots,T$9

Per epoch, the node centralities are summarized by median, mean, standard deviation, skewness, and kurtosis, then concatenated into a temporal signature $\sigma(f_t(x_i))$0 (Vahedian et al., 2021). This yields a compact descriptor of early structural evolution.

At the level of function-space training dynamics, $\sigma(f_t(x_i))$1-rank is defined from the Gram matrix $\sigma(f_t(x_i))$2 of terminal hidden-layer neuron functions. If $\sigma(f_t(x_i))$3 denotes its eigenvalues, then

$\sigma(f_t(x_i))$4

Equivalent sampled formulations use the singular values of a feature matrix $\sigma(f_t(x_i))$5, so that $\sigma(f_t(x_i))$6 (Yang et al., 2024). This quantity is explicitly computable and is used to connect low loss to effective feature diversity.

A different reduced description arises in finite-width kernel analyses. The Neural Tangent Hierarchy (NTH) starts from the output dynamics

$\sigma(f_t(x_i))$7

and extends it to higher-order tangent kernels through

$\sigma(f_t(x_i))$8

This hierarchy quantifies how the Neural Tangent Kernel changes during training and shows that finite-width feature learning appears as a controlled departure from the frozen infinite-width kernel regime (Huang et al., 2019).

Taken together, these observables illustrate a central fact: NNTD is often made useful by reducing extremely high-dimensional trajectories to a few statistics—entropy, disagreement, centrality, $\sigma(f_t(x_i))$9-rank, kernel evolution, or scalar embeddings—that retain the dynamical phenomenon of interest.

3. Early phases, instability, and regime changes

Several papers identify a distinct early training regime. "The Early Phase of Neural Network Training" studies the first $\{f_t\}_{t=1}^T$0 iterations, approximately $\{f_t\}_{t=1}^T$1 epochs on CIFAR-10 for the architectures considered, and separates this interval into an initial burst with extremely large gradients and rapid motion, a fast-improvement phase through approximately $\{f_t\}_{t=1}^T$2 iterations, and a decelerating phase thereafter. For ResNet-20, accuracy reaches approximately $\{f_t\}_{t=1}^T$3 by $\{f_t\}_{t=1}^T$4 iterations and approximately $\{f_t\}_{t=1}^T$5 by $\{f_t\}_{t=1}^T$6 iterations, while the benefits of rewinding in iterative magnitude pruning saturate between roughly $\{f_t\}_{t=1}^T$7 and $\{f_t\}_{t=1}^T$8 iterations (Frankle et al., 2020). That study also shows that early changes are highly structured: sign-preserving reinitialization of magnitudes is strongly damaging at $\{f_t\}_{t=1}^T$9, and global or layerwise weight permutations collapse performance to near-initialization levels, indicating pronounced non-i.i.d. structure already after a few hundred iterations (Frankle et al., 2020).

The relation between optimization and phase behavior is made explicit in the edge-of-chaos framework. For a single-hidden-layer tanh network, the order parameter is the normalized Jacobian Frobenius norm, and the edge-of-chaos condition is

$f_T$0

In the ordered phase, the hidden-layer variance scale obeys a linear law

$f_T$1

with $f_T$2 the learning rate, $f_T$3 the momentum coefficient, and $f_T$4 the batch size (Zhang et al., 2021). This scaling fails in the chaotic phase. The same work reports that model generalization peaks when training saturates at the edge of chaos and that weight decay effectively pushes the model toward the ordered phase (Zhang et al., 2021).

A different dynamical-systems study directly probes orbital instability in shallow networks. With full-batch gradient descent, $f_T$5 yields regular learning but not orbital stability, $f_T$6 produces early exponential divergence with a network maximal Lyapunov exponent of approximately $f_T$7, and $f_T$8 yields irregular, intermittent-like behavior with mixed evidence for chaos in loss-space projections (Danovski et al., 2024). This is an important corrective to a common simplification: monotone or successful loss reduction does not imply stable trajectories in parameter space.

Recent work on ReLU networks formulates a specific regime-change hypothesis. Because activation patterns are locally stable away from measure-zero boundaries, late training can proceed within relatively stable activation regions. Empirically, activation-pattern changes decay $f_T$9 times earlier than weight-update magnitudes across the evaluated settings, supporting a two-timescale picture in which early training explores activation regimes and late training mostly refines parameters within them (Pérez-Corral et al., 9 Feb 2026). This does not appear uniformly across all settings: the paper reports that the MLP on MNIST is a clear exception, where activation changes converged slightly slower than weights (Pérez-Corral et al., 9 Feb 2026).

The earliest phase is also highly sensitive to loss design. "Effects of Initialization Biases on Deep Neural Network Training Dynamics" identifies Initial Guessing Bias (IGB): randomly initialized deep networks can strongly favor a small subset of classes. On CIFAR-10 with ResNet-50, cross-entropy rapidly rebalances class probabilities, whereas Blurry loss with $\{\theta_t\}$0 is slower and Piecewise-zero loss with cutoff $\{\theta_t\}$1 can fail to steer the model away from the favored class (Pellegrino et al., 25 Nov 2025). This directly contradicts the misconception that robustness-oriented losses are neutral in the earliest phase; the paper shows that their interaction with IGB can dramatically reshape, or even stall, the initial corrective dynamics.

A plausible implication is that "early training" is not a single universal transient. The cited works instead distinguish several non-equivalent phenomena: fast formation of sparse trainable structure, growth of weight variance toward criticality, loss-landscape instability, stabilization of activation patterns, and sensitivity of class rebalancing to the loss.

4. Architecture, symmetry, and population structure

One line of work argues that part of NNTD is determined by architecture prior to any data-dependent effects. "Architecture Induces Structural Invariant Manifolds of Neural Network Training Dynamics" defines a Structural Invariant Manifold (SIM) as an immersed submanifold of parameter space invariant under gradient flow for every real-analytic loss and every dataset. Its main characterization states that SIMs are unions of orbits of the induced vector-field family $\{\theta_t\}$2, and, for fully connected networks, permutation and sign symmetries generate explicit hierarchies of such invariant leaves (Zhao et al., 10 Oct 2025). On condensation leaves, the dynamics is equivalent to training a reduced-width model, making neuron condensation a genuine dynamical equivalence rather than only a heuristic redundancy argument (Zhao et al., 10 Oct 2025). For generic two-layer networks, the paper proves that all SIMs are symmetry-induced.

A second structural viewpoint treats NNTD as a coupled population process over parameters and hyperparameters. In the two-time-scale framework, a population of networks $\{\theta_t\}$3 evolves by fast SGD/Langevin dynamics in $\{\theta_t\}$4,

$\{\theta_t\}$5

and slower selection–mutation dynamics in $\{\theta_t\}$6 (Borghi et al., 20 Mar 2026). Under strong time-scale separation $\{\theta_t\}$7, the fast parameter dynamics relaxes to a Gibbs law

$\{\theta_t\}$8

which induces an effective fitness

$\{\theta_t\}$9

The hyperparameter density then obeys a reduced selection–mutation equation $\{W_t\}$0, and, under slow mutation scaling, a replicator–mutator PDE (Borghi et al., 20 Mar 2026). Here NNTD is neither a single-network trajectory nor a fixed-kernel approximation but a mean-field kinetic object coupling optimization and hyperparameter adaptation.

A third theoretical strand formulates training as transportation-map estimation in infinite-dimensional function space. The core stochastic process is the Hilbert-space Langevin dynamic

$\{W_t\}$1

whose invariant measure is a Gibbs posterior over transport maps (Suzuki, 2020). The framework yields dimension-free geometric ergodicity for both narrow and wide networks, together with PAC-Bayes generalization bounds and fast excess-risk rates under additional assumptions (Suzuki, 2020). This suggests that NNTD can be analyzed as a stochastic flow on function-space maps rather than only as a finite-dimensional weight trajectory.

These theories address different invariants. SIMs isolate architectural confinement of gradient flow. Two-time-scale population dynamics isolates fast–slow interaction between optimization and hyperparameter selection. Infinite-dimensional Langevin analysis isolates stochastic convergence and generalization of transport maps. Their coexistence indicates that "training dynamics" is stratified: some structure is induced by architecture, some by optimization noise and regularization, and some by collective adaptation across models.

5. NNTD as a supervisory and decision signal

The most explicit operationalization of NNTD is the construction of uncertainty-aware supervisory targets. "Pseudo-D" uses per-epoch logits to generate soft pseudo-labels whose entropy reflects sample-specific difficulty and whose confidence is aligned to validation accuracy by class-wise Dirichlet calibration (Gu et al., 15 Sep 2025). The pipeline is a pair of training passes with calibration in between: first collect logits at every epoch, then average them into $\{W_t\}$2, calibrate $\{W_t\}$3 by either a global temperature or a class-wise affine transform $\{W_t\}$4, and finally retrain with cross-entropy on soft labels. On a benchmark of $\{W_t\}$5 retrospective echocardiography studies comprising $\{W_t\}$6 videos, Pseudo-D achieves balanced video-level ECE $\{W_t\}$7 versus Vanilla $\{W_t\}$8, and balanced study-level ECE $\{W_t\}$9, the best among reported methods after fusion. Its balanced study-level AURC is $a_\ell(x;\theta)\in\{0,1\}^{n_\ell}$0, also the best reported value (Gu et al., 15 Sep 2025). An important clarification made by the paper is that its "Dirichlet calibration" is not a Dirichlet distribution over labels; it is a class-wise logit transformation followed by softmax (Gu et al., 15 Sep 2025).

NNTD is also used as a purely post hoc reject option. The selective-prediction framework based on checkpoint disagreement requires no architectural or train-time modification. The best-performing default is $a_\ell(x;\theta)\in\{0,1\}^{n_\ell}$1 with convex weighting $a_\ell(x;\theta)\in\{0,1\}^{n_\ell}$2 and $a_\ell(x;\theta)\in\{0,1\}^{n_\ell}$3, using $a_\ell(x;\theta)\in\{0,1\}^{n_\ell}$4–$a_\ell(x;\theta)\in\{0,1\}^{n_\ell}$5 checkpoints (Rabanser et al., 2022). On CIFAR-10 at $a_\ell(x;\theta)\in\{0,1\}^{n_\ell}$6 coverage, it attains $a_\ell(x;\theta)\in\{0,1\}^{n_\ell}$7 error versus $a_\ell(x;\theta)\in\{0,1\}^{n_\ell}$8 for SR, $a_\ell(x;\theta)\in\{0,1\}^{n_\ell}$9 for SelectiveNet, and $G^{(\tau)}$0 for Deep Gamblers; at $G^{(\tau)}$1 target error, coverage is $G^{(\tau)}$2 versus $G^{(\tau)}$3 for SR (Rabanser et al., 2022). The method interprets persistent late-stage disagreement with the final prediction as an instability signal. A common misconception is that selective prediction necessarily requires abstention-specific architectures or losses; this work shows that NNTD alone can reach state-of-the-art operating points.

A third use is hypothesis testing. The NTK-based two-sample test analyzes how a classifier-based statistic grows with training time. In the zero-time NTK approximation,

$G^{(\tau)}$4

so detection times are governed by the spectrum of the initial kernel and the projection of the signal $G^{(\tau)}$5 onto its eigenfunctions (Khurana et al., 2024). The paper derives a minimum training time $G^{(\tau)}$6 ensuring $G^{(\tau)}$7, a maximum undetectable time $G^{(\tau)}$8 under the null, and shows that, with growing train and test sample sizes, statistical power goes to $G^{(\tau)}$9 in the small-time regime (Khurana et al., 2024). Here NNTD functions as a time-resolved testing mechanism rather than as an optimizer diagnostic.

Across these applications, the same theme recurs: dynamics are distilled into signals that cannot be recovered from the final model alone—sample difficulty, prediction instability, or detection time—and then used to reshape supervision, abstention, or decision thresholds.

6. Compact modeling, forecasting, and open directions

A large body of work aims to compress NNTD without discarding its predictive structure. In temporal-graph analysis, early graph signatures computed from weighted degree or eigenvector centrality over fewer than $f_t(x_i)\in\mathbb{R}^C$00 epochs can already predict whether a model will end in a high- or low-performance regime. The reported abstract-level claim is $f_t(x_i)\in\mathbb{R}^C$01 classification accuracy with temporal summaries over less than $f_t(x_i)\in\mathbb{R}^C$02 epochs, and specific settings such as VGG and AlexNet on CIFAR-10 reach approximately $f_t(x_i)\in\mathbb{R}^C$03 accuracy using the first $f_t(x_i)\in\mathbb{R}^C$04 epochs (Vahedian et al., 2021). The same work shows that these signatures transfer from smaller-width to larger-width ResNets, indicating a scale-stable description of structural dynamics (Vahedian et al., 2021).

Correlation Mode Decomposition (CMD) compresses parameter trajectories themselves. The 2022 formulation models a parameter trajectory $f_t(x_i)\in\mathbb{R}^C$05 inside a mode $f_t(x_i)\in\mathbb{R}^C$06 as

$f_t(x_i)\in\mathbb{R}^C$07

where $f_t(x_i)\in\mathbb{R}^C$08 is a reference trajectory shared by a highly correlated cluster of parameters (Turjeman et al., 2022). The 2023 extension introduces an online variant and embedded training strategy. On CIFAR-10, ResNet-18 improves from $f_t(x_i)\in\mathbb{R}^C$09 baseline accuracy to $f_t(x_i)\in\mathbb{R}^C$10 with Online CMD, while ViT-b-16 improves from $f_t(x_i)\in\mathbb{R}^C$11 to $f_t(x_i)\in\mathbb{R}^C$12; in federated learning with ResNet-18 over $f_t(x_i)\in\mathbb{R}^C$13 non-IID clients, CMD reduces average communication volume to $f_t(x_i)\in\mathbb{R}^C$14 of baseline while slightly improving accuracy from $f_t(x_i)\in\mathbb{R}^C$15 to $f_t(x_i)\in\mathbb{R}^C$16 (Brokman et al., 2023). These results suggest that many networks traverse a low number of synchronized directions even when their raw parameter counts are very large.

Gradient Flow Matching (GFM) takes a forecasting perspective. It treats weight evolution as a continuous-time ODE

$f_t(x_i)\in\mathbb{R}^C$17

and learns the vector field $f_t(x_i)\in\mathbb{R}^C$18 from partial training sequences using conditional flow matching, finite-difference targets, and a midpoint-consistency penalty (Shou et al., 26 May 2025). From prefixes of length $f_t(x_i)\in\mathbb{R}^C$19, it forecasts $f_t(x_i)\in\mathbb{R}^C$20 across SGD, Adam, RMSprop, AdamW, and Adagrad; on synthetic tasks it is typically best or second-best in MSE, and on Transformer trajectories on CIFAR-10 it achieves the best reported task alignment with $f_t(x_i)\in\mathbb{R}^C$21 (Shou et al., 26 May 2025). This is a different use of NNTD from CMD: the objective is not compression or mode discovery but extrapolation of the training flow itself.

Recent scalar-embedding work goes further in dimensionality reduction. By building a distance matrix between epoch-wise adjacency matrices and applying classical multidimensional scaling, training can be represented by a scalar trajectory $f_t(x_i)\in\mathbb{R}^C$22. For an MLP on MNIST, the scalar embedding preserves the transition to sensitivity to initial conditions, reconstructs the maximum Lyapunov exponent, and yields an embedded decorrelation time $f_t(x_i)\in\mathbb{R}^C$23 that correlates with an accuracy-based decorrelation time with Pearson $f_t(x_i)\in\mathbb{R}^C$24 and $f_t(x_i)\in\mathbb{R}^C$25 (Jiménez-González et al., 29 Jun 2026). The same study reports that rescaled asymptotic spacing distributions in the embedded space are compatible with a skew lognormal distribution (Jiménez-González et al., 29 Jun 2026).

Finally, $f_t(x_i)\in\mathbb{R}^C$26-rank introduces a function-space notion of feature capacity that tracks a staircase phenomenon in loss reduction. The paper proves

$f_t(x_i)\in\mathbb{R}^C$27

with $f_t(x_i)\in\mathbb{R}^C$28, showing that substantial loss reduction requires larger $f_t(x_i)\in\mathbb{R}^C$29-rank (Yang et al., 2024). Empirically, the loss drops in stages as $f_t(x_i)\in\mathbb{R}^C$30-rank jumps, and a pre-training strategy on the initial hidden layer that increases terminal-layer $f_t(x_i)\in\mathbb{R}^C$31-rank reduces training time and improves accuracy across the reported tasks (Yang et al., 2024).

These compact models also clarify current limitations. Static graph snapshots are insufficient for temporal prediction (Vahedian et al., 2021). DMD-style exponential temporal models can fail under common augmentation-heavy training regimes (Brokman et al., 2023). Activation-pattern instrumentation is specialized to piecewise-linear networks and does not directly extend to smooth activations (Pérez-Corral et al., 9 Feb 2026). Calibration based on validation dynamics depends on representative validation data (Gu et al., 15 Sep 2025). Population-level and infinite-dimensional theories impose regularity assumptions that are stronger than ordinary engineering practice (Borghi et al., 20 Mar 2026, Suzuki, 2020). A plausible synthesis is that NNTD is becoming a family of reduced-order models, each reliable only for particular dynamical questions.

What emerges across the cited work is not a single general law of training but a layered taxonomy. Prediction trajectories expose uncertainty and selectivity. Early-phase and edge-of-chaos analyses expose critical windows and instability. Symmetry and population theories expose invariant and effective dynamics. Compression and forecasting methods expose low-dimensional structure in seemingly high-dimensional optimization. This suggests that NNTD is best understood not as one theory, but as a research program for extracting task-relevant dynamical structure from training itself.