Directional Consistency as a Complementary Optimization Signal: The GONO Framework

Abstract: We identify and formalize an underexplored phenomenon in deep learning optimization: directional alignment and loss convergence can be decoupled. An optimizer can exhibit near-perfect directional consistency (cc_t -> 1, measured via consecutive gradient cosine similarity) while the loss remains high or decreases slowly. This observation reveals that existing optimizers such as Adam, SGD, and RMSprop lack explicit mechanisms to exploit temporal consistency in gradient directions, relying instead on magnitude-based signals that fail to distinguish plateaus, saddle points, and genuine convergence. Motivated by this, we introduce GONO (Gradient-Oriented Norm-Adaptive Optimizer), which adapts Adam's momentum coefficient beta_1 based on cc_t: amplifying momentum under directional consistency and suppressing it during oscillation. We prove GONO matches Adam's O(1/sqrt(T)) convergence rate and reduces exactly to Adam when the signal is uninformative. Empirically, cc_t achieves oscillation detection with F1=1.00 (vs. 0.45 for gradient norm), and GONO remains competitive with AdamW on MNIST (98.15%), CIFAR-10 (43.14%), and ResNet-18 (75.44%), establishing directional alignment as a theoretically grounded, practically actionable optimization signal. Code: https://github.com/victordaniel/gono-optimizer

• The paper introduces the GONO framework that adaptively adjusts momentum based on consecutive gradient cosine similarity to enhance training regimes.
• It establishes that gradient alignment can decouple from loss reduction, impacting first-order optimizer design with rigorous theoretical proofs and guarantees.
• Empirical evaluations on benchmarks like MNIST, CIFAR-10, and the Rosenbrock function confirm GONO’s efficiency in mitigating oscillations and improving performance.

Directional Consistency as a Complementary Optimization Signal: The GONO Framework

Introduction and Conceptual Foundation

This work identifies and formalizes a critical and underexplored aspect of deep learning optimization—the decoupling of directional consistency and loss convergence. While state-of-the-art optimizers such as Adam, SGD, and RMSprop employ fixed momentum mechanisms focused on gradient magnitude, this study highlights that directional alignment (measured via consecutive cosine similarity, $\mathrm{cc}_t$) and convergence in objective value are not inherently linked. Specifically, the optimizer can converge to nearly perfect consecutive gradient alignment ($\mathrm{cc}_t \to 1$) while the loss remains stagnant or decreases slowly. Empirical results underscore this phenomenon, which has substantial implications for both the design and theoretical understanding of first-order optimizers.

The authors propose the GONO (Gradient-Oriented Norm-Adaptive Optimizer) framework as a principled extension to the Adam optimizer. GONO introduces a real-time adaptation of Adam's momentum parameter, $\beta_1$, based on the signal provided by $\mathrm{cc}_t$. The core hypothesis is that strong directional consistency should increase momentum for accelerated progress on plateaus, while oscillatory behavior should decrease momentum and mitigate overshoot. The approach is both theoretically grounded (retaining the $\mathcal{O}(1/\sqrt{T})$ convergence rate) and computationally efficient, requiring only one additional dot product per iteration.

Direction-Loss Decoupling and Theoretical Results

The central theoretical contribution is a formal proof (Theorem 1) that gradient agreement between consecutive steps is insufficient for loss minimization. The construction shows that it is possible for $\mathrm{cc}_t \to 1$ while the parameters traverse a flat plateau, with the objective value remaining significantly above the global minimum for exponentially many steps. This demonstrates that gradient direction information and gradient magnitude evolve on different timescales and have distinct convergence implications.

Figure 1: Three signals during a 300-epoch MLP training run: MSE loss decreases monotonically, but the consecutive cosine similarity and gradient norm stabilize well before convergence, evidencing direction-loss decoupling.

Additionally, the study presents a theoretical guarantee (Theorem 2) that GONO matches Adam’s convergence rate under standard smoothness and boundedness assumptions. The adoption of an adaptive $\beta_1$ is analytically shown to introduce no degradation and to strictly recover Adam when $\lambda=0$. Furthermore, the authors provide a rigorous justification for adopting $\mathrm{cc}_t$ as an oscillation detector: negative consecutive cosine reliably signals oscillatory updates, whereas gradient norm-based detectors perform poorly in this regime.

GONO Algorithm and Practical Implementation

The GONO optimizer leverages a minimal modification to the Adam update:

$$\beta_{1,t} = \mathrm{clip}\left(\beta_1(1+\lambda\,\mathrm{cc}_t), \beta_{1,\min}, \beta_{1,\max}\right),$$

with typical bounds $\mathrm{cc}_t \to 1$0 and $\mathrm{cc}_t \to 1$1 and a sensitivity hyperparameter $\mathrm{cc}_t \to 1$2. This adaptation is applied at each iteration, requiring only the storage of the previous gradient and a scalar cosine computation.

The practical advantages are as follows:

• Computational Efficiency: The only overhead is a dot product for $\mathrm{cc}_t \to 1$3.
• Interpretability: $\mathrm{cc}_t \to 1$4 signals stable descent appropriate for high momentum; $\mathrm{cc}_t \to 1$5 detects oscillation, requiring momentum reduction.
• Plug-and-Play: When the directional signal is uninformative, GONO defaults seamlessly to Adam.

Empirical Evaluation

Oscillation Detection

GONO’s use of the consecutive cosine similarity as a regime detector is strongly supported by experimental evidence. In quadratic test environments with forced oscillation, the $\mathrm{cc}_t \to 1$6 threshold flags all oscillatory steps with F1=1.0, whereas norm-based detectors miss over half of oscillatory steps.

Figure 2: Oscillation detection in a synthetic quadratic; cosine-based detection achieves perfect F1, whereas gradient norm fails to identify most oscillations.

Rosenbrock Optimization

On the Rosenbrock function, a non-convex benchmark with curved valleys, GONO matches Adam in convergence while SGD with momentum fails to approach the optimum within 3000 steps due to inappropriate momentum persistence during oscillatory regimes.

Figure 3: Loss and convergence analysis on Rosenbrock; GONO and Adam reach the objective, whereas SGD-momentum stagnates.

Standard Deep Learning Benchmarks

GONO is evaluated on MNIST and CIFAR-10 with both MLP and ResNet-18 architectures. On MNIST, GONO (98.15% test accuracy) matches AdamW and consistently outperforms Adam and SGD. For CIFAR-10, GONO and AdamW perform comparably on both MLP (43.14% vs. 43.22%) and ResNet-18 (75.44% vs. 76.88%), confirming that the introduction of directional adaptation does not degrade baseline performance even on large-scale tasks.

Figure 4: Adaptive momentum behavior during MNIST training. GONO increases or decreases momentum adaptively according to $\mathrm{cc}_t \to 1$7, with 5.4% steps detected as oscillating.

Figure 5: ResNet-18 on CIFAR-10; GONO closely matches AdamW's learning curve and test accuracy, outperforming SGD-momentum.

Comparative Analysis and Positioning in Literature

The study positions GONO in the context of other adaptive optimizers:

• Adam, AdamW, RMSprop, AMSGrad: All adapt magnitude per-parameter but maintain a fixed momentum. None explicitly utilize directional information over time.
• HGM, ZetA: Contemporary methods leveraging longer-horizon angular information; GONO is unique in focusing on consecutive step-local alignment for regime detection.
• Lion and RAdam: Explore orthogonal approaches (sign-based updates, variance rectification) but do not adapt momentum via directional consistency.
• Multi-task Optimizers (PCGrad): Use cosine signals for gradient projection across tasks; GONO’s signal is cross-step for single-task trajectories.

GONO's design achieves precise control over training regime without extensive hyperparameter tuning or loss of Adam compatibility.

Implications and Future Directions

The introduction of directional agreement as an independent, first-class optimization signal has several key practical and theoretical implications:

• Optimization Landscape Diagnostics: $\mathrm{cc}_t \to 1$8 is a robust regime indicator and can be monitored alongside (or in place of) gradient norm for dynamic optimizer control.
• Robustness to Instability: Under high learning rates or sharp curvature, GONO’s adaptive damping mitigates catastrophic divergence events.
• Algorithmic Flexibility: The consecutive cosine mechanism can be integrated into a wide range of adaptively-tuned optimizers with minimal disruption.

Future directions include (1) refined integration of directionality signals with schedule learning rate adaptation, (2) exploration of multi-step or windowed $\mathrm{cc}_t \to 1$9 statistics, and (3) expansion to distributed and federated optimization environments.

Conclusion

This work establishes that directional consistency and magnitude should be independently monitored and controlled in first-order optimizers. The GONO framework provides robust momentum adaptation on the basis of consecutive gradient alignment, without compromising theoretical guarantees or computational efficiency. Empirical evaluation demonstrates competitive or improved performance relative to state-of-the-art variants on canonical optimization benchmarks and deep learning tasks. These findings motivate further investigation into directional-consistency-aware optimization algorithms, and suggest broad applicability for training monitoring, optimizer design, and theoretical analysis of non-convex optimization dynamics.

For detailed algorithmic and implementation information, refer to the original work "Directional Consistency as a Complementary Optimization Signal: The GONO Framework" (2605.06575).