AdaCubic: Adaptive Cubic Optimizer

• AdaCubic is an adaptive cubic-regularization based optimizer designed for scalable deep learning, using a dual formulation to automate regularization weight selection.
• It employs Hutchinson-based diagonal Hessian approximations to reduce memory costs and computational overhead while ensuring second-order convergence guarantees.
• Empirical results show that AdaCubic achieves competitive performance on vision and NLP tasks, often outperforming optimizers like SGD and Adam.

Searching arXiv for AdaCubic and related optimizer/context papers. arXiv search query: AdaCubic optimizer cubic regularization deep learning AdaCubic is a novel regularization technique that adapts the weight of the cubic term and an adaptive, cubic-regularization–based second-order optimizer specifically engineered to be practical for deep learning (Tsingalis et al., 10 Apr 2026). It starts from cubic-regularized Newton, replaces the fixed cubic weight by an automatically learned one, and uses a Hutchinson-based diagonal Hessian approximation to reduce computational cost and memory. The method is positioned for non-convex empirical risk minimization in large neural networks, with reported evaluations in Computer Vision, Natural Language Processing, Language Modeling, and Signal Processing. A broader, noncanonical use of the label is also suggested by explanatory treatments of cubic equations and cubic Diophantine problems, where “AdaCubic” denotes an adaptive or geometry-aware cubic procedure rather than the 2026 optimizer (Kalantari, 2014, Brucal-Hallare et al., 2021).

1. Historical placement and naming

In its primary arXiv usage, AdaCubic denotes a deep-learning optimizer built around cubic regularization (Tsingalis et al., 10 Apr 2026). Its conceptual starting point is the classical cubic-regularized Newton model of Nesterov & Polyak,

$$m_M(s) = f(x_k) + \nabla f(x_k)^\top s + \tfrac12 s^\top H_k s + \tfrac{M}{6} \|s\|^3,$$

where the cubic term regularizes the local second-order model and supports convergence to an $(\epsilon_g,\epsilon_H)$-stationary point with iteration complexity $O(\epsilon^{-3/2})$. The motivation for AdaCubic is that direct deployment of cubic-regularized Newton in deep learning is obstructed by the prohibitive cost of computing and storing $H_k$, and by the need to choose a robust value of $M$.

AdaCubic addresses both issues simultaneously. It uses Hutchinson’s method to approximate only the diagonal of the Hessian from Hessian–vector products, giving an $O(d)$-memory second-order method, and it introduces an auxiliary constrained problem that turns the cubic term into a constraint and recovers the cubic weight $M$ as a dual variable $\nu_k$. The optimizer is therefore not a fixed-$M$ cubic method, but a trust-region–style cubic method in which the regularization weight is adapted per iteration (Tsingalis et al., 10 Apr 2026).

A recurring source of terminological ambiguity is that “AdaCubic” has also been used informally in explanatory discussions outside optimization. Kalantari’s work on solving cubic equations by the quadratic formula uses critical points, Voronoi geometry, and the basic family $B_m$ to obtain a root of a cubic from quadratic-level computations (Kalantari, 2014). A separate line of work on $(\epsilon_g,\epsilon_H)$0 organizes solutions through 2-adic valuation trees and has been described from an “AdaCubic” perspective because of its adaptive tree traversal and congruence-structured solution generation (Brucal-Hallare et al., 2021). These latter usages are interpretive rather than canonical titles.

2. Core mathematical formulation

AdaCubic targets minimization of a non-convex empirical risk

$(\epsilon_g,\epsilon_H)$1

Second-order stationarity is measured by

$(\epsilon_g,\epsilon_H)$2

The central reformulation replaces the unconstrained cubic model by the constrained problem

$(\epsilon_g,\epsilon_H)$3

with Lagrangian

$(\epsilon_g,\epsilon_H)$4

The KKT system yields three conditions: a first-order equation,

$(\epsilon_g,\epsilon_H)$5

a positive-semidefinite condition,

$(\epsilon_g,\epsilon_H)$6

and complementary slackness,

$(\epsilon_g,\epsilon_H)$7

The dual variable $(\epsilon_g,\epsilon_H)$8 therefore plays exactly the role of the cubic regularization weight (Tsingalis et al., 10 Apr 2026).

The dual formulation introduces

$(\epsilon_g,\epsilon_H)$9

and a scalar function $O(\epsilon^{-3/2})$0. The maximizing pair $O(\epsilon^{-3/2})$1 satisfies $O(\epsilon^{-3/2})$2, and $O(\epsilon^{-3/2})$3 solves the scalar equation

$O(\epsilon^{-3/2})$4

The paper’s equivalence theorem states that the constrained formulation and the cubic-regularized step coincide when the cubic weight $O(\epsilon^{-3/2})$5 is set to the optimal Lagrange multiplier $O(\epsilon^{-3/2})$6. In this sense, AdaCubic replaces heuristic tuning of $O(\epsilon^{-3/2})$7 by solving for an iteration-dependent dual optimum (Tsingalis et al., 10 Apr 2026).

3. Algorithmic mechanism and trust-region updates

In practice, AdaCubic uses a batch-based approximate model,

$O(\epsilon^{-3/2})$8

where $O(\epsilon^{-3/2})$9 and $H_k$0 are minibatch estimates and $H_k$1 is a diagonal Hessian approximation. The inner solver, denoted RootFinder, computes $H_k$2 as the minimizer of the constrained problem in this approximate model.

The outer iteration follows a trust-region logic in the cubic norm. At each step, the method computes a trial step, evaluates the ratio

$H_k$3

and accepts the step when $H_k$4. The quantity $H_k$5 acts as a trust-region radius parameter through the constraint $H_k$6. The update policy is:

• if $H_k$7, the iteration is very successful and

$H_k$8

• if $H_k$9, the iteration is successful and $M$0;
• if $M$1, the iteration is unsuccessful and

$M$2

The inner RootFinder sets $M$3, forms

$M$4

and solves for $M$5 so that the corresponding step

$M$6

satisfies $M$7. Because $M$8 is diagonal, the linear algebra reduces to cheap elementwise operations. The one-dimensional equation $M$9 is handled by safeguarded Newton iterations, exploiting the monotonicity and concavity properties reported for $O(d)$0 (Tsingalis et al., 10 Apr 2026).

The hyperparameters are taken directly from trust-region literature and fixed across all experiments: $O(d)$1 This fixed configuration is central to the paper’s positioning of AdaCubic as an optimizer for settings where fine-tuning is infeasible (Tsingalis et al., 10 Apr 2026).

4. Hessian approximation, complexity, and convergence theory

A defining scalability feature is that AdaCubic estimates only the diagonal of the Hessian. If $O(d)$2 denotes a Hessian–vector product, then Hutchinson’s method forms

$O(d)$3

using Rademacher probe vectors $O(d)$4, and sets

$O(d)$5

The estimator is unbiased for $O(d)$6, requires one Hessian–vector product per probe, and yields $O(d)$7 memory rather than the storage cost of a dense $O(d)$8 Hessian (Tsingalis et al., 10 Apr 2026).

The deterministic theory assumes that each $O(d)$9 is twice continuously differentiable and bounded below, and that $M$0, $M$1, and $M$2 are Lipschitz continuous on a closed convex set $M$3. With the exact diagonal Hessian $M$4, the resulting diagonal cubic-regularized iteration satisfies

$M$5

and therefore reaches $M$6 in

$M$7

iterations. The paper presents this as the classical cubic-regularization iteration complexity inherited in the diagonal approximation setting (Tsingalis et al., 10 Apr 2026).

For the stochastic case, the analysis imposes inexact-model bounds

$M$8

Probabilistic deviation bounds are then derived with vector and matrix Bernstein inequalities. The gradient bound holds with probability $M$9 provided

$\nu_k$0

and the Hessian-diagonal bound holds with probability $\nu_k$1 provided

$\nu_k$2

The stated implication is asymptotic: as $\nu_k$3, batch sizes must become large, approaching full-data behavior, and the deterministic cubic-regularization guarantees apply locally (Tsingalis et al., 10 Apr 2026).

5. Empirical performance across application domains

The experimental program covers Computer Vision, GLUE natural language understanding, Language Modeling, and audio-based Camera Model Identification (Tsingalis et al., 10 Apr 2026). In all experiments AdaCubic uses the same universal hyperparameters, whereas the baselines are tuned.

On CIFAR-10, with ResNet20 and ResNet32, AdaCubic clearly outperforms SGD and Adam and remains close to AdaHessian. For ResNet20, the reported accuracies are AdaHessian $\nu_k$4, AdaCubic $\nu_k$5, Adam $\nu_k$6, and SGD $\nu_k$7. For ResNet32, the reported values are AdaHessian $\nu_k$8, AdaCubic $\nu_k$9, Adam $M$0, and SGD $M$1. On CIFAR-100 with ResNet18, AdaCubic is slightly behind tuned baselines: without spatial averaging, SGD $M$2, Adam $M$3, AdaHessian $M$4, and AdaCubic $M$5; with spatial averaging, AdaHessian $M$6 and AdaCubic $M$7.

On GLUE with SqueezeBERT, AdaCubic is reported as either best or second-best across tasks despite the absence of per-task tuning. Across SST-2, QNLI, RTE, and WNLI, average accuracy is SGD $M$8, AdaCubic $M$9, and AdaHessian $B_m$0. On MRPC, AdaCubic reaches F1 $B_m$1 and accuracy $B_m$2, compared with SGD F1 $B_m$3, accuracy $B_m$4, and AdaHessian F1 $B_m$5, accuracy $B_m$6. On QQP, AdaCubic reports F1 $B_m$7 and accuracy $B_m$8, against SGD F1 $B_m$9, accuracy $(\epsilon_g,\epsilon_H)$00, and AdaHessian F1 $(\epsilon_g,\epsilon_H)$01, accuracy $(\epsilon_g,\epsilon_H)$02. On MNLI, AdaCubic reports $(\epsilon_g,\epsilon_H)$03 for matched/mismatched, compared with SGD $(\epsilon_g,\epsilon_H)$04 and AdaHessian $(\epsilon_g,\epsilon_H)$05.

On language modeling, the optimizer is often much better than AdaHessian and sometimes better than SGD. On WikiText-2, perplexity is reported as $(\epsilon_g,\epsilon_H)$06 for AdaCubic and $(\epsilon_g,\epsilon_H)$07 for AdaHessian on RoBERTa, $(\epsilon_g,\epsilon_H)$08 for AdaCubic and $(\epsilon_g,\epsilon_H)$09 for AdaHessian on BERT, and $(\epsilon_g,\epsilon_H)$10 for AdaCubic and $(\epsilon_g,\epsilon_H)$11 for AdaHessian on DistilBERT. On PTB, the reported perplexities are $(\epsilon_g,\epsilon_H)$12 for AdaCubic and $(\epsilon_g,\epsilon_H)$13 for AdaHessian on RoBERTa, $(\epsilon_g,\epsilon_H)$14 for AdaCubic and $(\epsilon_g,\epsilon_H)$15 for AdaHessian on BERT, and $(\epsilon_g,\epsilon_H)$16 for AdaCubic and $(\epsilon_g,\epsilon_H)$17 for AdaHessian on DistilBERT.

In audio-based Camera Model Identification on the VISION dataset with ResNet18, AdaCubic is consistently more accurate and more stable than Adam. The reported mean accuracies over 5 folds are $(\epsilon_g,\epsilon_H)$18 versus $(\epsilon_g,\epsilon_H)$19 on native videos, $(\epsilon_g,\epsilon_H)$20 versus $(\epsilon_g,\epsilon_H)$21 on WhatsApp, and $(\epsilon_g,\epsilon_H)$22 versus $(\epsilon_g,\epsilon_H)$23 on YouTube.

Wall-clock comparisons indicate the expected trade-off between iteration cost and training progress. On ResNet20/CIFAR-10, to reach target loss $(\epsilon_g,\epsilon_H)$24, the reported figures are 83 epochs and 35.16 minutes for SGD, 81 epochs and 61.85 minutes for AdaHessian, and 55 epochs and 42.40 minutes for AdaCubic (Tsingalis et al., 10 Apr 2026).

6. Limitations, misconceptions, and related usages beyond deep learning

AdaCubic is a second-order method, but it is not a full-Hessian cubic optimizer. Its curvature model is diagonal, so off-diagonal structure is ignored. The paper identifies this as a plausible explanation for weaker performance on CIFAR-100 and for small gaps versus tuned SGD on some GLUE tasks (Tsingalis et al., 10 Apr 2026). The method also incurs additional cost: Hutchinson estimation requires an extra backward pass, or a few, per iteration. Its practical memory overhead is described as around $(\epsilon_g,\epsilon_H)$25 versus SGD’s $(\epsilon_g,\epsilon_H)$26, and below AdaHessian’s $(\epsilon_g,\epsilon_H)$27. Theoretical local convergence in the stochastic setting is asymptotic in the sense that batch sizes must become large as $(\epsilon_g,\epsilon_H)$28.

A further misconception is terminological. Outside the 2026 optimizer, the label has been used loosely for cubic procedures whose adaptivity is geometric or arithmetic rather than optimization-theoretic. In Kalantari’s “Solving Cubic Equations By the Quadratic Formula,” a monic cubic with distinct roots and distinct critical points has at least one critical point with the Voronoi property, meaning that the critical point lies in the Voronoi cell of some root (Kalantari, 2014). For any $(\epsilon_g,\epsilon_H)$29, the basic sequence

$(\epsilon_g,\epsilon_H)$30

converges to $(\epsilon_g,\epsilon_H)$31, where

$(\epsilon_g,\epsilon_H)$32

Because at least one critical point lies in a root’s Voronoi cell, solving $(\epsilon_g,\epsilon_H)$33 by the quadratic formula supplies a guaranteed-convergence starting point for a root of $(\epsilon_g,\epsilon_H)$34. The rate of convergence is proportional to the ratio between the distance from the starting point to the nearest root and the distance to the second-closest root. This suggests an adaptive cubic procedure in a geometric sense, but it is numerically and conceptually distinct from the deep-learning optimizer.

An arithmetic analogue appears in work on $(\epsilon_g,\epsilon_H)$35 with $(\epsilon_g,\epsilon_H)$36 odd, where 2-adic valuation trees classify the possible values of $(\epsilon_g,\epsilon_H)$37 and the congruence classes of $(\epsilon_g,\epsilon_H)$38 that realize them (Brucal-Hallare et al., 2021). The central cubic theorem states that if $(\epsilon_g,\epsilon_H)$39, then the valuation tree of $(\epsilon_g,\epsilon_H)$40 is infinite and $(\epsilon_g,\epsilon_H)$41 has nontrivial solutions for all $(\epsilon_g,\epsilon_H)$42 except finitely many values; if $(\epsilon_g,\epsilon_H)$43, only finitely many exponents $(\epsilon_g,\epsilon_H)$44 occur. For the example $(\epsilon_g,\epsilon_H)$45,

$(\epsilon_g,\epsilon_H)$46

This line of work has likewise been described from an “AdaCubic” perspective because the solution space is traversed adaptively through a valuation tree, but it is a 2-adic Diophantine framework rather than an optimizer.

Taken together, these usages identify two distinct meanings. In the strict arXiv sense, AdaCubic is the adaptive cubic-regularization optimizer introduced in 2026 for scalable deep learning (Tsingalis et al., 10 Apr 2026). In a broader interpretive sense, the name also evokes cubic procedures whose adaptivity is governed by Voronoi geometry or valuation-tree structure rather than by a dual trust-region mechanism (Kalantari, 2014, Brucal-Hallare et al., 2021).