Parameter-Free Cubic-Regularized Newton Method: Sharp Complexity and Generalized Smoothness

Abstract: We analyze a variant of the cubic-regularized Newton method for nonconvex optimization. This variant is parameter-free in that it requires no prior knowledge of problem-dependent parameters. Under the generalized smoothness condition $$|\nabla³ f(x)| \leq L_0 + L_1 |\nabla f(x)|$$, we derive an oracle complexity bound for finding an $(\varepsilon, δ)$-second-order stationary point. This assumption is weaker than the generalized smoothness conditions used in existing analyses of second-order methods, while the complexity bound improves upon existing guarantees for parameter-free second-order methods. In particular, when $L_1 = 0$, the bound matches the optimal dependence on $L_0$ as well as on $\varepsilon$, $δ$, and the initial function value gap, up to additive logarithmic terms. To establish this bound, we derive Taylor-type inequalities and prove their equivalence to the generalized smoothness condition.

• The paper introduces a parameter-free CRN method that attains sharp oracle complexity for approximate second-order stationary points under a pointwise generalized smoothness condition.
• It employs an adaptive backtracking strategy to adjust the cubic regularization parameter, ensuring efficient convergence without dependence on fixed smoothness parameters.
• The analysis rigorously extends traditional Hessian Lipschitz conditions, demonstrating universal performance improvements in nonconvex optimization settings.

Parameter-Free Cubic-Regularized Newton Under Pointwise Generalized Smoothness

Introduction

The paper "Parameter-Free Cubic-Regularized Newton Method: Sharp Complexity and Generalized Smoothness" (2607.10741) introduces a parameter-free variant of the Cubic-Regularized Newton (CRN) method for nonconvex optimization, achieving sharp oracle complexity for finding approximate second-order stationary points under a broad, pointwise generalized smoothness condition on the third derivative. This work addresses a longstanding limitation in the parameter-free second-order optimization literature: the suboptimal dependence on problem-specific smoothness parameters in oracle complexity bounds.

Generalized Smoothness: Motivation and Formalization

Second-order methods for nonconvex optimization are often analyzed under Lipschitz continuity of the Hessian:

$$\|\nabla^2 f(y) - \nabla^2 f(x)\| \leq L_0 \|y - x\|.$$

CRN achieves optimal complexity in this setting. However, many practical problems exhibit non-Lipschitz behavior, motivating more flexible smoothness constraints. This work proposes a pointwise third-derivative bound

$\|\nabla^3 f(x)\| \leq L_0 + L_1 \|\nabla f(x)\|$

which is strictly weaker than two-point conditions prevalent in the literature. Importantly, this broadens the class of functions for which optimal convergence guarantees can be established, and is shown to strictly include functions excluded by standard or “global” generalized smoothness.

The value of this generalization is empirically illustrated with canonical nonconvex functions, such as the Rastrigin function.

Figure 1: Rastrigin function visualization and scatter plot of $\|\nabla f(x)\|$ vs. $\|\nabla^3 f(x)\|$, illustrating the appropriateness of the affine smoothness model for parameter selection.

Algorithmic Design: Parameter-Free CRN with Adaptive Backtracking

The proposed algorithm is a CRN variant with a crucial backtracking procedure for the regularization parameter $\sigma$, independent of $L_0$ or $L_1$. Each iteration constructs an approximate minimizer of the model

$$m(s) = \langle \nabla f(x_k), s \rangle + \frac{1}{2} s^\top \nabla^2 f(x_k) s + \frac{\sigma}{3} \|s\|^3$$

and accepts a trial step only if both a sufficient decrease in the objective and a gradient norm reduction at the next step are observed. The regularization parameter is adaptively doubled until these conditions are met, then halved before the next iteration, ensuring efficiency even when gradients are large due to the $L_1 \|\nabla f(x)\|$ term. This halving mechanism is shown to be essential for the improved complexity guarantees under pointwise generalized smoothness.

The algorithm requires only zero-respecting access to $f$, $\|\nabla^3 f(x)\| \leq L_0 + L_1 \|\nabla f(x)\|$0, and $\|\nabla^3 f(x)\| \leq L_0 + L_1 \|\nabla f(x)\|$1, and is robust to inexact solutions of the cubic model subproblem.

Theoretical Contributions

Sharp Complexity Bound

Under the assumption

$\|\nabla^3 f(x)\| \leq L_0 + L_1 \|\nabla f(x)\|$2

the method provably computes an $\|\nabla^3 f(x)\| \leq L_0 + L_1 \|\nabla f(x)\|$3-second-order stationary point in

$\|\nabla^3 f(x)\| \leq L_0 + L_1 \|\nabla f(x)\|$4

oracle calls, up to additive logarithmic factors, where $\|\nabla^3 f(x)\| \leq L_0 + L_1 \|\nabla f(x)\|$5.

Notably:

• When $\|\nabla^3 f(x)\| \leq L_0 + L_1 \|\nabla f(x)\|$6 (Lipschitz Hessian), the bound matches the known-optimal O(Δ(L₀^{1/2}ε^{-3/2} + L₀^2δ^{-3})) up to log terms.
• For general $\|\nabla^3 f(x)\| \leq L_0 + L_1 \|\nabla f(x)\|$7, the established rates are novel, and the universality of the guarantee allows the user to optimize over admissible $\|\nabla^3 f(x)\| \leq L_0 + L_1 \|\nabla f(x)\|$8 pairs for tighter complexity.

Technical Insights

• Taylor-type inequalities: The work proves that the pointwise condition $\|\nabla^3 f(x)\| \leq L_0 + L_1 \|\nabla f(x)\|$9 is equivalent to specific, hyperbolic-function-weighted Taylor remainder bounds for $\|\nabla f(x)\|$0, $\|\nabla f(x)\|$1, and $\|\nabla f(x)\|$2. This exact equivalence is leveraged to rigorously extend the standard CRN analysis into this much more general regime.
• Backtracking analysis: A careful characterization of when the acceptance conditions become satisfied is provided, crucially making the overall complexity nearly independent of $\|\nabla f(x)\|$3 when this parameter is small. The halving of $\|\nabla f(x)\|$4 at each iteration is shown to be necessary to avoid redundant large regularization, in contrast to earlier analyses.
• Universality: The complexity bound holds uniformly for all admissible $\|\nabla f(x)\|$5 pairs, so the method’s performance is never bottlenecked by poor internal parameter choices.

Empirical and Analytical Examples

Through worked-out examples (quartic, smooth-abs, Rastrigin, and camel functions), the paper demonstrates scenarios where the pointwise third-derivative model yields dramatically tighter parameterization than either the Hessian Lipschitz or two-point generalized smoothness.

Figure 1: Scatter plot for the Rastrigin function, verifying that a relatively modest $\|\nabla f(x)\|$6 suffices compared to the excessively large $\|\nabla f(x)\|$7 required by global Hessian Lipschitz continuity.

Implications and Future Directions

The analysis:

• Closes a major open question regarding sharp oracle complexity for parameter-free CRN at second-order stationarity (previously resolved only for first-order stationarity [hamad2022consistently]).
• Demonstrates that parameter-freeness does not entail an optimality penalty (beyond log factors) for second-order methods in nonconvex optimization, provided pointwise generalized smoothness.
• Suggests that practical parameter tuning, which can leverage empirical $\|\nabla f(x)\|$8 estimation, can benefit directly from the universal guarantee.

On the theoretical side, it remains undetermined whether the terms $\|\nabla f(x)\|$9 and $\|\nabla^3 f(x)\|$0 are minimax sharp, or artifacts of the proof strategy. Extending the analysis to higher-order methods and more structured nonconvex settings under analogous pointwise smoothness assumptions is a promising direction, with links to recent advances in high-order adaptive optimization.

Conclusion

This work establishes that a minor yet technically involved variant of the CRN method achieves optimal (up to logs) oracle complexity for finding approximate second-order stationary points in nonconvex optimization, under a strictly weaker smoothness assumption than prior state-of-the-art. The universality and sharpness of the complexity analysis mark a significant advance in parameter-free adaptive second-order optimization theory, with clear algorithmic implications for both theoretical analysis and practical large-scale nonconvex optimization.

Reference: "Parameter-Free Cubic-Regularized Newton Method: Sharp Complexity and Generalized Smoothness" (2607.10741)