---
title: Stochastic Cubic Regularized Newton (SCRN)
url: https://www.emergentmind.com/topics/stochastic-cubic-regularized-newton-scrn
type: topic
---

# Stochastic Cubic Regularized Newton (SCRN)

Stochastic Cubic Regularized Newton (SCRN), also termed stochastic Cubic Newton or SCN in parts of the literature, is a class of stochastic second-order methods that extends cubic-regularized Newton to settings in which only noisy, subsampled, or otherwise approximate gradients and Hessians are available. The method is studied for both stochastic expectation problems \(f(x)=\mathbb E_\xi[f(x;\xi)]\) and finite-sum objectives \(f(x)=\frac1n\sum_{i=1}^n f_i(x)\), and in general nonconvex settings its target is typically an approximate local minimum or second-order stationary point rather than a global minimizer [1711.02838][2302.11962].

## 1. Optimization model and stationarity notions

SCRN appears in two closely related optimization models. One is stochastic approximation, in which \(f(x)=\mathbb E_{\xi\sim\mathcal D}[f(x;\xi)]\), and the other is finite-sum optimization, in which \(f(x)=\frac1n\sum_{i=1}^n f_i(x)\). The first model is prominent in the original stochastic cubic regularization analysis, whereas the second underlies variance-reduced, memory-based, and helper-based variants [1711.02838][1802.04796].

The standard target in nonconvex settings is an approximate second-order stationary point. One formulation requires
\[
\|\nabla f(x)\|\le \epsilon
\quad\text{and}\quad
\lambda_{\min}(\nabla^2 f(x))\ge -\sqrt{\rho\epsilon},
\]
which is the definition used in the stochastic cubic regularization analysis for smooth nonconvex objectives with \(\rho\)-Lipschitz Hessian [1711.02838]. Finite-sum cubic-regularization papers often state the goal as an \((\epsilon,\sqrt{\epsilon})\)-approximate local solution, meaning small gradient norm together with a Hessian lower bound of order \(-\sqrt{\epsilon}\) [1811.11637][1802.04796].

Several later papers compress first- and second-order conditions into a single scalar stationarity measure. One representative definition is
\[
\mu_M(x):=\max\Bigl(\|\nabla f(x)\|^{3/2},\; \frac{-\lambda_{\min}(\nabla^2 f(x))^3}{M^{3/2}}\Bigr),
\]
or a closely related variant with \([-\lambda_{\min}(\nabla^2 f(x))]_+^3\). Small \(\mu_M(x)\) means both small gradient and near-positive-semidefinite Hessian [2302.11962][2410.19644]. This suggests that the cubic regularization literature treats second-order stationarity not as an auxiliary condition but as the primary termination criterion.

Across the literature, the regularity assumptions are centered on lower boundedness of \(f\), Lipschitz continuity of the gradient, and especially Lipschitz continuity of the Hessian. The Hessian-Lipschitz property is the condition that enables third-order Taylor control and justifies the cubic term in the local model [1711.02838][1912.01597].

## 2. Cubic model and step computation

At each iterate \(x_t\), SCRN forms a cubic model using approximate first- and second-order information. A representative update computes
\[
s_t \in \arg\min_{s\in\mathbb R^d}
\left\{
g_t^\top s+\frac12 s^\top H_t s+\frac{M}{6}\|s\|^3
\right\},
\qquad
x_{t+1}=x_t+s_t,
\]
where \(g_t\) and \(H_t\) approximate \(\nabla f(x_t)\) and \(\nabla^2 f(x_t)\), and \(M\) is the cubic regularization parameter [2302.11962][1711.02838].

The cubic term is the defining difference from a stochastic Newton step. Under Hessian Lipschitzness, it stabilizes the model in directions of negative curvature and can make the model a valid upper bound. In the strongly convex finite-sum setting, the same rationale is expressed through Taylor remainder bounds controlled by the Hessian-Lipschitz constant \(H\), with \(M\ge H\) ensuring a valid cubic upper model [1912.01597].

For an exact solution of the cubic subproblem, the standard optimality conditions are
\[
g + Hs + \frac{M}{2}\|s\|s = 0,
\qquad
H+\frac{M}{2}\|s\|I \succeq 0.
\]
These identities recur throughout the theory because they connect the step norm, local curvature, and stationarity of the next iterate [1811.11637][2009.13016]. In the interpolation-based SCRN analysis, they yield the model decrease bound
\[
m_t(x_t+h_t^*)-m_t(x_t)\le -\frac{M}{12}\|h_t^*\|^3,
\]
which is the basic cubic decrease mechanism [2009.13016].

A recurring analytical pattern is that if the iterate is not yet approximately second-order stationary, then the cubic subproblem admits a step of nontrivial norm, and the model decrease transfers to true objective decrease. Conversely, if the cubic step is sufficiently small, then the iterate is already near second-order stationarity. This “large step implies descent, small step implies stationarity” dichotomy underlies both the original stochastic analysis and later inexact, variance-reduced, and adaptive variants [1711.02838][2308.13161].

## 3. Estimator design and algorithmic families

The simplest SCRN instantiation uses mini-batch stochastic estimators. In the original stochastic cubic regularization method, the algorithm samples independent mini-batches \(S_1\) and \(S_2\), forms
\[
g_t=\frac1{|S_1|}\sum_{\xi_i\in S_1}\nabla f(x_t;\xi_i),
\qquad
B_t=\frac1{|S_2|}\sum_{\xi_i\in S_2}\nabla^2 f(x_t;\xi_i),
\]
and solves the cubic model using \(g_t\) and the Hessian operator \(B_t[\cdot]\). The method is explicitly designed to rely only on stochastic gradients and stochastic Hessian-vector products, with the latter used in place of explicitly forming a dense Hessian matrix [1711.02838].

A major finite-sum direction is variance reduction. The paper "Stochastic Variance-Reduced Cubic Regularized Newton Method" introduced a semi-stochastic gradient together with a semi-stochastic Hessian specifically designed for cubic regularization, and established a complexity improvement over earlier cubic regularization schemes [1802.04796]. A more explicit stagewise design appears in adaptive SVRC, where a reference point \(\tilde x\) is used to build estimators
\[
g_t^k=\frac1{|S_k|}\sum_{i\in S_k}\left(\nabla f_i(x_t^k)-\nabla f_i(\tilde x)\right)+\nabla F(\tilde x),
\]
\[
H_t^k=\frac1{|B_k|}\sum_{i\in B_k}\left(\nabla^2 f_i(x_t^k)-\nabla^2 f_i(\tilde x)\right)+\nabla^2F(\tilde x),
\]
with adaptive sample sizes based on the previous step norm \(\|\xi^{k-1}\|\) rather than the unknown current step [1811.11637].

A distinct finite-sum formulation is the memory-based stochastic cubic Newton method for strongly convex objectives. It stores points \(w_i^k\) for each component, builds second-order Taylor models around those stored points, and updates only a random subset \(S^k\) of the memories after each iteration. In its simplest form, each iteration computes the gradient and Hessian of a single randomly selected function only. The method is biased rather than unbiased, and its analysis is based on a tailored Lyapunov function rather than estimator unbiasedness [1912.01597].

The helper framework generalizes these constructions further. It represents \(f\) as \(f=h+(f-h)\), models the cheaper helper \(h\) locally, and handles noisy and possibly biased gradient and Hessian estimates, arbitrary batch sizes, variance reduction, and lazy Hessian updates in one analysis. This framework is presented as a unifying theory for stochastic and variance-reduced cubic Newton methods, and also applies to auxiliary learning [2302.11962].

Recent work adds momentum to the estimator design. One line uses transported gradient momentum together with heavy-ball Hessian momentum and proves global convergence for arbitrary batch size, including one stochastic sample per iteration [2410.19644]. Another introduces Polyak momentum and recursive momentum for Hessian estimation, combined with a small-error stochastic gradient estimator and a potential function of the form
\[
\mathcal P_k = f(x^k) + p_k\|M_k-\nabla^2 f(x^k)\|_F^3,
\]
to analyze stochastic second-order stationarity in moment form [2507.13003].

## 4. Complexity landscape

Reported complexity guarantees depend strongly on the oracle model: some papers count stochastic gradients and Hessian-vector products, some count higher-order oracle calls, some count Hessian samples, and some separate arithmetic complexity from oracle complexity. The principal claims therefore require direct comparison only within matching oracle models.

| Setting | Reported guarantee | Paper |
|---|---|---|
| General smooth nonconvex SCRN with stochastic gradients and Hessian-vector products | \(\tilde O(\epsilon^{-3.5})\) stochastic gradient and stochastic Hessian-vector product evaluations to find an \(\epsilon\)-approximate local minimum | [1711.02838] |
| Finite-sum stochastic variance-reduced cubic regularized Newton | \(\tilde O(n^{4/5}/\epsilon^{3/2})\) second-order oracle calls to an \((\epsilon,\sqrt{\epsilon})\)-approximately local minimum | [1802.04796] |
| SCRN under interpolation-like conditions | \(\tilde O(1/\epsilon^{2.5})\) higher-order oracle calls to an \(\epsilon\)-local-minimizer | [2009.13016] |
| Adaptive variance-reduced subsampled Newton with cubic regularization | \(O(N+N^{2/3}\epsilon^{-3/2})\) expected Hessian sample complexity | [1811.11637] |
| Finite-sum EMA/SARAH cubic Newton | \(n+\widetilde O(n^{1/2}\epsilon^{-3/2})\) total stochastic oracle calls to an \((\varepsilon,\sqrt{L_2\varepsilon})\)-SOSP | [2510.08714] |

The 2018 variance-reduced method explicitly states that \(\tilde O(n^{4/5}/\epsilon^{3/2})\) second-order oracle complexity outperforms the state-of-the-art cubic regularization algorithms including subsampled cubic regularization [1802.04796]. The interpolation-based analysis is positioned between generic stochastic and deterministic cubic Newton: without interpolation-like assumptions, the cited stochastic complexity is \(\tilde O(1/\epsilon^{3.5})\); with the Strong Growth Condition it improves to \(\tilde O(1/\epsilon^{2.5})\); the deterministic cubic-regularized Newton benchmark remains \(\tilde O(1/\epsilon^{1.5})\) [2009.13016].

Momentum-based analyses introduce a different stationarity regime. With stochastic second-order stationarity defined through moments,
\[
\mathbb E\big[\|\nabla f(x)\|^{3/2}\big]\le \epsilon_g^{3/2},
\qquad
\mathbb E\big[\lambda_{\min}(\nabla^2 f(x))^3\big]\ge -\epsilon_H^3,
\]
SCRN-PM attains complexity
\[
K = \mathcal O\!\left(\max\{\epsilon_g^{-7/4},\epsilon_H^{-7}\}\right),
\]
while SCRN-RM improves this to
\[
K = \mathcal O\!\left(\max\{\epsilon_g^{-5/3},\epsilon_H^{-5}\}\right)
\]
under an additional mean-cubed smoothness condition on the stochastic Hessian [2507.13003].

Another recent development is the proof of global convergence for nonconvex stochastic cubic Newton with arbitrary batch size. The momentum-based analysis in this line does not present the result as a single classical \(\epsilon\)-oracle complexity bound, but it does establish convergence to second-order stationary points even when only one stochastic data sample is used per iteration [2410.19644].

## 5. Structural regimes, local behavior, and improved rates

SCRN is sensitive to additional structure in the objective class. In sufficiently smooth and strongly convex finite-sum problems, the memory-based stochastic Newton and stochastic cubically regularized Newton methods achieve local linear-quadratic behavior. The SCN analysis uses the Lyapunov quantity
\[
V^k = \frac1n \sum_{i=1}^n \left(f(w_i^k)-f(x^\star)\right)^{3/2},
\]
and proves a local contraction
\[
\mathbb E[V^{k+1}] \le \left(1-\frac{\tau}{2n}\right)V^k
\]
under a suitable local initialization condition. When \(\tau=n\), the scheme reduces to the deterministic cubic Newton setting and the rate becomes locally superlinear or quadratic-like [1912.01597].

In over-parameterized or interpolation-like regimes, the Strong Growth Condition
\[
\mathbb E_\xi\|\nabla F(x,\xi)\|^2 \le \rho \|\nabla f(x)\|^2
\]
causes stochastic gradient noise to vanish near stationary points. Under this assumption, SCRN improves from the cited generic stochastic complexity \(\tilde O(1/\epsilon^{3.5})\) to \(\tilde O(1/\epsilon^{2.5})\), but still does not match the deterministic \(\tilde O(1/\epsilon^{1.5})\) rate. The same work explicitly argues that gradient-level interpolation is not sufficient to close the gap and suggests that Hessian interpolation-like assumptions may be necessary [2009.13016]. This is one of the clearest articulated limitations in the SCRN literature.

Another important structural regime is gradient dominance. For functions satisfying
\[
F(x)-F(x^*) \le \tau_F \|\nabla F(x)\|^\alpha,
\qquad 1\le \alpha\le 2,
\]
SCRN improves the best-known sample complexity of stochastic gradient descent. The stated total sample complexity is \(\mathcal O(\epsilon^{-7/(2\alpha)+1})\) for \(1\le\alpha<3/2\) and \(\tilde{\mathcal O}(\epsilon^{-2/\alpha})\) for \(3/2\le\alpha\le 2\) [2205.12856]. For \(\alpha=1\), a variance-reduced SCRN with time-varying batch sizes reduces the average sample complexity to \(\mathcal O(\epsilon^{-2})\) for stochastic gradients and \(\mathcal O(\epsilon^{-1})\) for stochastic Hessians [2205.12856].

The same paper extends this perspective to policy-based reinforcement learning under a weak gradient dominance condition of the form
\[
\epsilon' + \tau_J\|\nabla J(\theta)\|\ge J^*-J(\theta),
\]
and reports that SCRN improves over stochastic policy gradient by a factor of \(\tilde{\mathcal O}(\epsilon^{-1/2})\) in the \(\epsilon'=0\) case [2205.12856]. This suggests that cubic regularization can exploit global error-bound structure, not merely local negative-curvature information.

## 6. Extensions, implementations, and unresolved technical issues

The SCRN paradigm has been extended well beyond the Euclidean, single-machine baseline. On embedded Riemannian submanifolds, the Riemannian stochastic variance-reduced cubic regularized Newton method transports gradient and Hessian information between tangent spaces, solves a cubic model in the tangent space, and achieves \(O(\epsilon^{-3/2})\) iteration complexity for an \((\epsilon,\sqrt{\epsilon})\)-second-order stationary point. With the choice \(T=N^{1/5}\), its total second-order oracle complexity is
\[
\tilde O\!\left(N + L_H^{1/2}\Delta_F\, N^{4/5}\epsilon^{-3/2}\right),
\]
and the same rate is preserved under an inexact cubic subproblem solve [2010.03785].

Distributed and nested formulations have also been studied. In Distributed Stochastic Cubic-Regularized Newton (DiSCRN), the outer model is
\[
m_S^k(\mathbf x)=F^S(\mathbf x^k)+(\mathbf x-\mathbf x^k)^\top g^k+\frac12(\mathbf x-\mathbf x^k)^\top H^k(\mathbf x-\mathbf x^k)
+\sum_{i=1}^n \frac{\rho_i}{6}\|x_i-x_i^k\|^3,
\]
with a consensus constraint \((L\otimes I_d)\mathbf x=0\). A local stopping criterion for the inner problem guarantees \(\|p^+-p^\star\|\le \Delta\), and the outer objective is shown to decrease monotonically with high probability under the stated assumptions [2008.13291].

A different extension replaces the full ambient space by random low-dimensional subspaces. The Stochastic Subspace Cubic Newton method is convex rather than nonconvex, but it is explicitly described as both a stochastic extension of cubically regularized Newton and a second-order enhancement of stochastic subspace descent. As the minibatch or subspace dimension varies, its global rate interpolates between stochastic coordinate descent and cubic regularized Newton [2002.09526].

High-probability adaptive regularization with cubics provides a complementary route. In this line, the method uses noisy function-value, gradient, and Hessian oracles together with an acceptance ratio
\[
\rho_k= \frac{f(x_k)-f(x_k^+)+2\epsilon_f'}{m_k(x_k)-m_k(x_k^+)},
\]
and obtains a high-probability iteration bound of order \(O(\epsilon^{-3/2})\), matching deterministic ARC/SCRN at the iteration level [2308.13161].

Implementation remains a central technical issue. The original stochastic cubic regularization method emphasizes that stochastic Hessian-vector products can be computed as efficiently as stochastic gradients, which avoids explicit Hessian formation [1711.02838]. The helper framework introduces lazy Hessian reuse to reduce arithmetic complexity in large dimension [2302.11962]. Re\(^3\)MCN develops a matrix-free Hutchinson variant for Hessian estimation and a fast inner solver based on a secular equation and conjugate gradients, again targeting large-scale finite-sum problems where full Hessians are impractical [2510.08714].

A persistent unresolved issue is the gap between the best deterministic and stochastic second-order rates. Interpolation-like gradient assumptions improve SCRN substantially but do not recover deterministic cubic Newton complexity, and the literature explicitly points to missing Hessian-level structure as a plausible reason [2009.13016]. The current trajectory of the field therefore combines three themes: more aggressive control of stochastic Hessian error, cheaper matrix-free or lazy implementations, and analytical frameworks that can accommodate biased, momentum-based, or auxiliary-information-driven estimators without losing second-order guarantees [2302.11962][2410.19644].

Source: https://www.emergentmind.com/topics/stochastic-cubic-regularized-newton-scrn