AGD: an Auto-switchable Optimizer using Stepwise Gradient Difference for Preconditioning Matrix (2312.01658v2)

Abstract: Adaptive optimizers, such as Adam, have achieved remarkable success in deep learning. A key component of these optimizers is the so-called preconditioning matrix, providing enhanced gradient information and regulating the step size of each gradient direction. In this paper, we propose a novel approach to designing the preconditioning matrix by utilizing the gradient difference between two successive steps as the diagonal elements. These diagonal elements are closely related to the Hessian and can be perceived as an approximation of the inner product between the Hessian row vectors and difference of the adjacent parameter vectors. Additionally, we introduce an auto-switching function that enables the preconditioning matrix to switch dynamically between Stochastic Gradient Descent (SGD) and the adaptive optimizer. Based on these two techniques, we develop a new optimizer named AGD that enhances the generalization performance. We evaluate AGD on public datasets of NLP, Computer Vision (CV), and Recommendation Systems (RecSys). Our experimental results demonstrate that AGD outperforms the state-of-the-art (SOTA) optimizers, achieving highly competitive or significantly better predictive performance. Furthermore, we analyze how AGD is able to switch automatically between SGD and the adaptive optimizer and its actual effects on various scenarios. The code is available at https://github.com/intelligent-machine-learning/atorch/tree/main/atorch/optimizers.

• The paper proposes AGD with an auto-switch mechanism that dynamically switches between SGD and adaptive methods based on gradient differences.
• It leverages stepwise gradient differences to approximate Hessian information, thereby enhancing convergence and generalization in deep learning models.
• Experimental results show AGD outperforms state-of-the-art optimizers in NLP, CV, and RecSys tasks with improved accuracy and efficiency.

An Analysis of AGD: An Auto-switchable Optimizer

In this paper, the authors introduce AGD, a novel optimizer aimed at enhancing the generalization performance of adaptive optimization algorithms commonly used in deep learning. Their approach introduces a method for designing the preconditioning matrix by leveraging the gradient differences between successive steps. This is distinguished by the incorporation of an auto-switching mechanism that dynamically transitions between Stochastic Gradient Descent (SGD) and adaptive optimization methods.

Design and Methodology

AGD's foundation is built upon two primary innovations:

1. Gradient Difference for Preconditioning: The optimizer employs the gradient difference between consecutive steps as diagonal elements in the preconditioning matrix. This design is a computational shortcut to approximate the Hessian's information, specifically through the lens of the inner product between the Hessian row vectors and parameter vector differences. Unlike methods requiring direct Hessian computations, AGD derives efficiency by focusing on these gradient differences.
2. Auto-Switch Mechanism: AGD can dynamically switch its update strategy between SGD and an adaptive optimizer, guided by a threshold parameter, $\delta$. If the preconditioned gradient surpasses $\delta$, the optimizer takes an adaptive step; otherwise, it defaults to an SGD-like behavior. This mechanism acknowledges the benefits of both optimization strategies, aiming to harness their strengths in varying training contexts.

Experimental Results

The authors validate AGD across multiple benchmarks including tasks in NLP, CV, and RecSys. The results demonstrate AGD's proficiency in outperforming state-of-the-art optimizers such as Adam, AdaBelief, and AdaHessian in most scenarios. The paper highlights certain key performance metrics:

• NLP Tasks: AGD achieved the lowest perplexity scores in LLMing tasks and competitive BLEU scores in translation tasks.
• CV Tasks: In image classification on Cifar10 and ImageNet, AGD exhibited better top-1 accuracy compared to other optimizers.
• RecSys Tasks: For CTR prediction tasks, AGD showed a significant improvement in AUC scores, highlighting its capacity to generalize effectively.

Theoretical Analysis

The paper provides theoretical backing for AGD's convergence properties. In non-convex settings, the optimizer demonstrates a convergence rate of $O(\log{T} / \sqrt{T})$, and in convex settings, it shows a regret bound of $O(1/\sqrt{T})$. These analyses support the optimizer’s ability to adapt efficiently across diverse training landscapes.

Implications and Future Directions

The introduction of AGD presents practical implications for training large-scale deep learning models. Its ability to switch between optimization modes aligns well with varying phases of training, offering potentially better generalization and convergence speeds.

Future developments might explore:

• Tuning the Switching Parameter ($\delta$): Investigating adaptive methods for tuning $\delta$ could further automate and refine the optimizer's performance.
• Extending AGD to Other Domains: Although tested on NLP, CV, and RecSys, applying AGD to other specific domains like reinforcement learning could yield interesting insights.
• Comparative Analyses: Longitudinal studies that compare AGD with emerging optimization algorithms will be integral to understanding its position in the optimizer landscape.

In conclusion, AGD represents a significant contribution to adaptive optimization techniques by integrating gradient-based preconditioning with dynamic mode switching. Its performance metrics and theoretical guarantees suggest that AGD holds promise for improving optimization strategies in deep learning.