Absolute Gradient Entropy (AGE)

• AGE is an entropy-based metric that quantifies the dispersion of absolute gradient values to distinguish memorization from generalization phases in neural network training.
• It computes an entropy-like sum over gradient magnitudes at each training step, providing insights into learning dynamics beyond traditional norm-based methods.
• Empirical observations reveal that AGE peaks during the memorization phase and declines as models simplify, closely aligning with breakthroughs in test accuracy.

Absolute Gradient Entropy (AGE) is an information-theoretic metric designed to quantify the instantaneous complexity of gradient updates during neural network training. Introduced in the context of grokking analysis for transformer-based models, AGE enables precise tracking of the transition from memorization to generalization by measuring the entropy of absolute gradient magnitudes across parameters. This approach reveals complexity dynamics that are not captured by traditional norm-based regularization techniques, providing unique insights into learning phases and model compression under various training regimens (Zhou et al., 24 Apr 2025).

1. Formal Definition

Let $\mathcal{G} \in \mathbb{R}^n$ denote the vector of all current gradient entries $g_1,\ldots,g_n$ of a model or of a specific layer/block. The Absolute Gradient Entropy is defined by:

$$H(\mathcal{G}) = -\sum_{i=1}^n |g_i| \cdot \ln |g_i|$$

This definition is directly analogous to the previously established Absolute Weight Entropy (AWE), where gradient entries $g_i$ replace weight entries $w_i$:

$$H(\mathcal{W}) = -\sum_{i=1}^m |w_i| \cdot \ln |w_i|$$

The key quantity in AGE is the entropy-like sum over the absolute gradient magnitudes, with no explicit normalization in the original presentation. However, one may optionally normalize the absolute values to sum to unity—the empirical work does not enforce this.

2. Computation Procedure

The calculation of AGE occurs at each training step and consists of the following steps:

1. Collect and flatten all model parameter gradients into a vector $\mathcal{G} = [g_1,\ldots,g_n]$.
2. Take elementwise absolute values: $a_i = |g_i|$ for $i=1,\ldots,n$.
3. (Optionally) Normalize the values to obtain $p_i = \frac{a_i}{\sum_j a_j}$; the original deployment does not mandate normalization.
4. Compute AGE using $g_1,\ldots,g_n$0 or (if normalized) $g_1,\ldots,g_n$1
5. Log or plot $g_1,\ldots,g_n$2 in parallel with training and test accuracy metrics to track complexity dynamics.

This process is typically applied at every iteration or epoch and is designed to elucidate the evolution of learning complexity over the entire optimization trajectory.

3. Intuitive and Theoretical Rationale

AGE measures the entropy of the absolute values of gradient entries, encoding how dispersed or concentrated the update signal is across model parameters.

• High AGE: Indicates that many parameters receive substantial, diverse updates and usually corresponds to periods when the model encodes a highly entangled, complex (often memorizing) hypothesis.
• Low AGE: Implies the update signal is focused in fewer directions, reflecting a condensed and sharper gradient distribution, characteristic of a model that has discovered a simpler, more robust (generalizing) representation.
• Information-Theoretic Linkage: AGE directly implements an entropy-based view of dynamic complexity, providing an instantaneous proxy for the structural “compression” or “distillation” the model undergoes as it transitions from overfitting to generalization.

This interpretation is supported by empirical observations showing that models exhibit high AGE during the memorization phase, which rapidly declines as they transition into generalization—the central phenomenon in grokking (Zhou et al., 24 Apr 2025).

4. Empirical Observations and Grokking Dynamics

AGE traces several distinct phases of training, particularly when analyzing the grokking phenomenon:

• Standard and GrokFast-MA Training:
  • AGE rises steadily during memorization, where training accuracy increases but test accuracy remains low.
  • It peaks just before the grokking transition, then declines as the model’s test accuracy climbs and the configuration generalizes.
• NeuralGrok Gradient Transformation:
  • Both AGE and Absolute Weight Entropy are consistently lower.
  • The rise-and-fall signature of AGE is preserved but the transition is compressed temporally.
  • Spikes in AGE are dampened, allowing more stable and monotonic generalization.
• Quantitative Example: On the $g_1,\ldots,g_n$3 arithmetic task, AGE tracks a rise from initialization through memorization, reaching a plateau, and then a synchronized decline with the leap in test accuracy to perfect generalization.
• Visualization: Figures in (Zhou et al., 24 Apr 2025) display AGE alongside accuracy, marking memorization (“red” window) and generalization (“green” window) epochs.

5. Theoretical Properties and Limitations

No formal convergence guarantees, tight analytic bounds, or inference idempotency results are derived for AGE. The measure is motivated and validated through empirical behaviors rather than theoretical proofs.

• Given normalized absolute gradients ($g_1,\ldots,g_n$4), simple extremal bounds include $g_1,\ldots,g_n$5.
• Sensitive dependence on the handling of zero or near-zero gradient magnitudes, due to the singularity of $g_1,\ldots,g_n$6 at zero.
• The metric incurs practical costs associated with collecting and storing absolute gradients at each step, particularly for large-scale models.
• The analysis in (Zhou et al., 24 Apr 2025) restricts its claims to empirically observed behavior under studied training workflows; further theoretical exploration remains open.

6. Comparison to Norm-Based and Traditional Regularization Metrics

AGE provides qualitative information unavailable from standard L1/L2-based metrics:

Metric	What it Controls	Sensitivity to Distribution	Empirical Marker of Grokking
L2 Gradient/Weight Norm	Magnitude (Total Energy)	None	Weak
Weight Decay	Magnitude (Shrinkage)	None	No
AGE	Entropy (Shape and Dispersion)	High	Yes

• Weight Decay & Gradient-Norm Regularization: Control only the overall scale of parameter updates, insensitive to how this magnitude is distributed across the parameter space.
• AGE: Remains constant under uniform rescaling (e.g., learning rate changes), but responds acutely to shifts in the distribution's shape and focus.
• Interpretive Value: The moment at which AGE sharply decreases closely corresponds to the model’s transition to generalization, aligning with empirical breakthroughs in test accuracy. L2 norms may decrease without a corresponding gain in generalization, as uniform shrinkage does not ensure representational compression.

AGE’s sensitivity to the allocation and overlap of the gradients informs a more nuanced view of model complexity reduction and the emergence of minimal sufficient representations.

7. Practical Considerations and Usage

• Computing AGE during training, and logging it alongside accuracy metrics, enables quantitative tracking of model complexity and the grokking phase transition.
• The measure functions as a lightweight, information-theoretic tool to supplement standard diagnostics.
• Limitations include explicit dependence on the treatment of zero gradients and resource overhead for high-dimensional parameterizations.
• No formal guarantees on its optimization, but its empirical correlation with generalization transitions is robust across diverse transformer-based tasks.

By charting the evolution of entropy in the update signals themselves, AGE provides a unique lens for studying generalization, model compression, and the grokking transition—phenomena not amenable to characterization via conventional parameter norms alone (Zhou et al., 24 Apr 2025).