AdaCubic: An Adaptive Cubic Regularization Optimizer for Deep Learning

Abstract: A novel regularization technique, AdaCubic, is proposed that adapts the weight of the cubic term. The heart of AdaCubic is an auxiliary optimization problem with cubic constraints that dynamically adjusts the weight of the cubic term in Newton's cubic regularized method. We use Hutchinson's method to approximate the Hessian matrix, thereby reducing computational cost. We demonstrate that AdaCubic inherits the cubically regularized Newton method's local convergence guarantees. Our experiments in Computer Vision, Natural Language Processing, and Signal Processing tasks demonstrate that AdaCubic outperforms or competes with several widely used optimizers. Unlike other adaptive algorithms that require hyperparameter fine-tuning, AdaCubic is evaluated with a fixed set of hyperparameters, rendering it a highly attractive optimizer in settings where fine-tuning is infeasible. This makes AdaCubic an attractive option for researchers and practitioners alike. To our knowledge, AdaCubic is the first optimizer to leverage cubic regularization in scalable deep learning applications.

• The paper proposes an adaptive cubic regularization optimizer that leverages a stochastic diagonal Hessian approximation to scale second-order methods for deep learning.
• It introduces a dual-augmented constrained formulation to automatically adjust the cubic parameter, ensuring robust convergence without manual tuning.
• Empirical results demonstrate that AdaCubic achieves competitive accuracy across computer vision, language understanding, and modeling tasks while minimizing hyperparameter sensitivity.

AdaCubic: An Adaptive Cubic Regularization Optimizer for Deep Learning (2604.09437)

Introduction and Motivation

The "AdaCubic" optimizer addresses the limitations of existing first- and second-order deep learning optimizers by efficiently incorporating adaptive cubic regularization within scalable training regimes. The approach builds on classical Cubic Regularization (CR) of Newton’s method, which is well established for its provable avoidance of saddle points in non-convex optimization. However, existing CR-based optimizers are not practical for large-scale deep learning due to the computational cost of full Hessian evaluations and the need to set appropriate regularization strengths. AdaCubic overcomes these by (a) using stochastic diagonal Hessian approximation via Hutchinson’s estimator, and (b) introducing a principled mechanism for optimally and adaptively adjusting the cubic regularization parameter during training through a dual-augmented constrained formulation.

Formulation and Algorithmic Framework

AdaCubic exploits a minimization problem with a dynamically constrained cubic regularizer. Rather than a fixed cubic parameter, it introduces an auxiliary Lagrangian dual problem where the regularization weight $M$ is obtained adaptively as an optimal dual variable. The cubic subproblem at each step is

$$\arg\min_\mathbf{s} \left(\nabla f(\mathbf{x}_k)^\top \mathbf{s} + \frac{1}{2} \mathbf{s}^\top H(\mathbf{x}_k)\mathbf{s} + \frac{M}{6}\|\mathbf{s}\|^3_2 \right),$$

and AdaCubic replaces $M$ with a dual variable determined by the solution of a coupled root-finding procedure on the constrained Lagrangian dual. The resulting direction can be computed efficiently using only diagonal Hessian approximations, facilitating scale to large DNNs.

Key derivations include:

• Strong duality and equivalence between cubic regularized and constrained problems: Theoretical results show that the two formulations yield the same step when $M$ is selected as the optimal dual variable for the cubic constraint.
• Hutchinson’s estimator: A minimal-memory stochastic scheme approximates the diagonal of the Hessian, supporting efficient computation of the Newton step in ultra-high dimensions without full Hessian storage.
• Hyperparameter-independence: Universally stable behavior is achieved using batch statistics and Trust Region theory, eliminating the need for dataset/architecture-dependent learning rate schedules.

Theoretical Guarantees

The convergence analysis combines classical results in non-convex optimization with new bounds arising from adaptive cubic regularization and inexact stochastic approximations. The main results include:

• Iteration complexity: AdaCubic preserves the $\mathcal{O}(\epsilon^{-3/2})$ iteration complexity for finding $\epsilon$-stationary points, matching the best-known rates for stochastic cubic regularized Newton methods.
• High-probability deviation control: Given approximate (sub-sampled) gradients and Hessians, vector and matrix concentration inequalities yield explicit sampling schemes ensuring empirical estimates track the true descent directions with high probability. This holds for both the gradient and for the diagonal elements of the Hessian.
• Curvature adaptation: The dual formulation enforces an automatic transition between regimes requiring high regularization (e.g., near saddle points) and low regularization (e.g., in well-conditioned basins), which is critical for efficient large-scale optimization.

Empirical Results

AdaCubic is benchmarked across three canonical domains: Computer Vision (CV), Natural Language Understanding (NLU), and Language Modeling (LM), as well as Camera Model Identification (CMI) in signal processing. Performance is compared with SGD, Adam, and AdaHessian.

Key findings:

• CV (CIFAR-10/100, ResNet): AdaCubic matches or marginally trails AdaHessian and surpasses standard SGD and Adam in test accuracy. Notably, AdaCubic achieves competitive loss-minimization dynamics using a single fixed hyperparameter set, while other optimizers require domain-specific learning rate tuning.
• NLU (GLUE, SqueezeBERT): AdaCubic consistently scores either best or second-best on all tasks, with performance gaps to first place being statistically negligible, again with zero hyperparameter tuning.
• LM (RoBERTa, BERT, DistilBERT): Across both WikiText-2 and PTB, AdaCubic outperforms AdaHessian and matches SGD when SGD is heavily tuned; in some cases, AdaCubic yields strictly lower perplexity.
• CMI (ResNet18): AdaCubic outperforms Adam and demonstrates reduced variance in accuracy across cross-validation folds.

Strong claim: AdaCubic is the first optimizer to utilize cubic regularization with adaptive strength in deep learning at scale, exhibiting high empirical robustness without learning rate selection, a persistent weakness in all prior adaptive methods.

Computational Complexity and Practical Considerations

AdaCubic’s per-iteration cost is $\mathcal{O}(d)$ in memory (storing gradient and diagonal Hessian approximation) and increases wallclock time relative to first-order methods due to additional backward passes for Hutchinson’s estimator. However, it improves epoch convergence and robustness, achieving target training losses in substantially fewer epochs than fine-tuned SGD or AdaHessian. The diagonal Hessian approximation does not capture full curvature but is shown empirically adequate for a large set of deep models.

A notable practical value is hyperparameter insensitivity: AdaCubic maintains performance across tasks, architectures, and datasets using a single Trust Region–motivated parameter set, whereas SGD, Adam, and AdaHessian all require dataset-specific tuning.

Implications and Future Perspectives

The AdaCubic framework demonstrates that efficient, adaptive cubic regularization can yield robust, high-performing second-order optimization for large-scale deep models, without the burdensome tuning that hampers deployment of both first-order and classical adaptive second-order optimizers. Theoretically, this indicates strong potential for more principled, automatic curvature adaptation across diverse non-convex landscapes.

Limitations include the restriction to diagonal Hessian approximations, which may be insufficient for highly structured models with substantial block-wise parameter coupling (as recent Hessian analyses of transformers suggest [zhang2024transformers]), and the potential benefit of extending cubic adaptation to richer preconditioning schemes.

Future work could involve:

• Block-diagonal or Kronecker-factored cubic regularization schemes, maintaining scalability while better modeling parameter interactions.
• Integration with efficient Hessian-vector product schemes to go beyond the diagonal and approach true Newton steps.
• Exploration of adaptive cubic regularization behavior in foundation model scale and non-standard architectures (GNNs, diffusion models, etc.).
• Theoretical refinement of generalization bounds for stochastic adaptive cubic methods in overparameterized regimes.

Conclusion

AdaCubic advances the field of scalable second-order optimizers by leveraging an adaptive, dual-based cubic regularization framework with diagonal Hessian approximations. It provides high empirical performance and robustness with minimal tuning and strong theoretical convergence, serving as an attractive paradigm for future research in adaptive second-order optimization for deep learning.

Reference: "AdaCubic: An Adaptive Cubic Regularization Optimizer for Deep Learning" (2604.09437).