Restricted Newton Descent

• Restricted Newton Descent is a framework that computes Newton-type updates under explicit structural restrictions, ensuring controlled second-order descent.
• It employs cosine angle and weak gradient edge measures to quantify the alignment between the restricted update and the exact Newton step, thereby improving convergence guarantees.
• The method unifies Newton boosting with Gradient Boosting Decision Trees and extends to contexts like Nash equilibria and Riemannian optimization, highlighting its broad applicability.

Searching arXiv for papers on Restricted Newton Descent and closely related Newton-restriction interpretations. Restricted Newton Descent denotes a framework in which Newton-type updates are computed under explicit restrictions on the admissible descent direction rather than by taking the exact Newton step in the ambient space. In its formal introduction for boosting, it studies convex optimization with Newton’s method on Hilbert spaces with inexact iterates, based on the concepts of cosine angle and weak gradient edge; within this formulation, Newton boosting with Gradient Boosting Decision Trees (GBDTs) and classical finite-dimensional Newton theory appear as special cases (Zozoulenko et al., 1 May 2026). In a broader research sense, closely related “restricted Newton” interpretations also arise when Newton directions are filtered by predicted-response descent tests in Nash equilibrium problems, by Krylov and curvature tests in nonconvex optimization, by tangent-space projections in function space, or by restricted-memory bundle aggregation on Riemannian manifolds. This broader usage suggests that the unifying theme is not merely approximation, but controlled second-order descent under structural constraints.

1. Hilbert-space formulation and exact versus restricted Newton steps

The formal Restricted Newton Descent framework is posed on a Hilbert space $\mathcal H$, typically $\mathcal L^2(\widehat\nu_N)$ for the empirical distribution, together with a weak learner family $\mathcal F\subset \mathcal H$. At iteration $k$, with current ensemble $F_k$, the local quadratic model is

$$Q_k(f)=\left\langle f,\, g_k+\frac12 H_k[f]\right\rangle, \qquad g_k=\nabla L(F_k),\quad H_k=\nabla^2L(F_k).$$

The exact Newton step is

$$f_{k+1}=-H_k^{-1}g_k=\arg\min_{f\in\mathcal H}Q_k(f),$$

whereas the restricted Newton update replaces the ambient minimization by

$$f_{k+1}^w\in \arg\min_{f\in\mathcal F} Q_k(f), \qquad F_{k+1}=F_k+\eta f_{k+1}^w.$$

Thus the method studies Newton updates with an “inexact iterate” viewpoint: the local quadratic model is still Newtonian, but the update direction is constrained by the weak learner class rather than taken exactly in $\mathcal H$ (Zozoulenko et al., 1 May 2026).

This formulation interpolates between familiar cases. When $\mathcal F=\mathcal H$, the restricted minimizer coincides with the exact Newton step, so vanilla Newton is recovered. When $\mathcal L^2(\widehat\nu_N)$0 is the class of decision trees of bounded depth and $\mathcal L^2(\widehat\nu_N)$1, the same construction recovers standard Newton boosting. The conceptual point is that restriction is built into the model class rather than added afterward as an external damping heuristic.

2. Alignment measures: cosine angle and weak gradient edge

A central issue is how well the restricted weak learner approximates the exact Newton direction. For vanilla restricted Newton, the relevant geometry is the Hessian-induced inner product

$\mathcal L^2(\widehat\nu_N)$2

In this geometry, minimizing $\mathcal L^2(\widehat\nu_N)$3 over $\mathcal L^2(\widehat\nu_N)$4 is equivalent to approximating the exact Newton step as closely as possible: $\mathcal L^2(\widehat\nu_N)$5 The paper therefore defines a cosine angle $\mathcal L^2(\widehat\nu_N)$6 in the Hessian metric, with $\mathcal L^2(\widehat\nu_N)$7, as the alignment measure between the exact Newton step and the restricted minimizer. Because $\mathcal L^2(\widehat\nu_N)$8 is assumed closed under scalar multiplication, the framework derives the identities

$\mathcal L^2(\widehat\nu_N)$9

$\mathcal F\subset \mathcal H$0

and

$\mathcal F\subset \mathcal H$1

These relations make the restriction quantitatively explicit: $\mathcal F\subset \mathcal H$2 measures how much of the exact Newton geometry is retained by the weak learner (Zozoulenko et al., 1 May 2026).

For the gradient-regularized scheme, the paper uses a different approximation notion. With regularized quadratic

$\mathcal F\subset \mathcal H$3

the exact regularized step is

$\mathcal F\subset \mathcal H$4

The restricted learner induces the “weak gradient”

$\mathcal F\subset \mathcal H$5

and the approximation requirement is a weak gradient edge $\mathcal F\subset \mathcal H$6: $\mathcal F\subset \mathcal H$7 Where $\mathcal F\subset \mathcal H$8 quantifies Newton-direction alignment in Hessian geometry, $\mathcal F\subset \mathcal H$9 quantifies how much of the true gradient is captured through the regularized Newton system.

3. Convergence regimes and regularization theory

For the unregularized restricted Newton scheme, the main global theory is linear convergence under smoothness, strong convexity, and Hessian dominance. If $k$0 is $k$1-smooth, $k$2-strongly convex, and $k$3 is closed under scalar multiplication, then for any learning rate

$k$4

the one-step decrease satisfies

$k$5

If, in addition, the Hessian-dominance condition

$k$6

holds for some $k$7, then

$k$8

The paper notes that this Hessian-dominance condition holds for binary and categorical cross entropy with $k$9, after the standard gauge fix in the multiclass case (Zozoulenko et al., 1 May 2026).

The second regime is gradient-regularized restricted Newton. The adaptive regularization is

$F_k$0

under the assumption that the loss has $F_k$1-Lipschitz Hessian. This $F_k$2-regularization is designed to stabilize Newton in large-gradient regions while vanishing near optimality. Under convexity, $F_k$3-Lipschitz Hessian, finite sublevel sets with diameter $F_k$4, weak gradient edge $F_k$5, and $F_k$6, the global theorem establishes

$F_k$7

Accordingly, the method has a global $F_k$8 rate, together with an additional linear regime when the gradient contracts often enough (Zozoulenko et al., 1 May 2026).

The same paper states a local linear convergence result under strong convexity; near the optimum, the rate becomes linear, and in the exact case $F_k$9 the classical superlinear $$Q_k(f)=\left\langle f,\, g_k+\frac12 H_k[f]\right\rangle, \qquad g_k=\nabla L(F_k),\quad H_k=\nabla^2L(F_k).$$0-type behavior is recovered from the underlying gradient-regularized Newton theory. The framework therefore distinguishes two different roles for restriction: in the unregularized case, it is controlled by Newton-angle alignment; in the regularized case, it is controlled by a weak learning condition on the induced gradient.

4. Newton boosting with GBDTs

Restricted Newton Descent was introduced in part to make Newton boosting mathematically explicit. For empirical risk

$$Q_k(f)=\left\langle f,\, g_k+\frac12 H_k[f]\right\rangle, \qquad g_k=\nabla L(F_k),\quad H_k=\nabla^2L(F_k).$$1

the pointwise gradient and Hessian blocks are

$$Q_k(f)=\left\langle f,\, g_k+\frac12 H_k[f]\right\rangle, \qquad g_k=\nabla L(F_k),\quad H_k=\nabla^2L(F_k).$$2

and the second-order model becomes

$$Q_k(f)=\left\langle f,\, g_k+\frac12 H_k[f]\right\rangle, \qquad g_k=\nabla L(F_k),\quad H_k=\nabla^2L(F_k).$$3

If $$Q_k(f)=\left\langle f,\, g_k+\frac12 H_k[f]\right\rangle, \qquad g_k=\nabla L(F_k),\quad H_k=\nabla^2L(F_k).$$4 is a tree with leaf values $$Q_k(f)=\left\langle f,\, g_k+\frac12 H_k[f]\right\rangle, \qquad g_k=\nabla L(F_k),\quad H_k=\nabla^2L(F_k).$$5, then the restricted minimizer yields the familiar XGBoost leaf formula

$$Q_k(f)=\left\langle f,\, g_k+\frac12 H_k[f]\right\rangle, \qquad g_k=\nabla L(F_k),\quad H_k=\nabla^2L(F_k).$$6

The paper’s point is therefore not merely analogical: the standard Newton-boosting leaf update is derived directly as the restricted minimizer of the Hilbert-space quadratic model (Zozoulenko et al., 1 May 2026).

The gradient-regularized variant minimally modifies this construction by adding

$$Q_k(f)=\left\langle f,\, g_k+\frac12 H_k[f]\right\rangle, \qquad g_k=\nabla L(F_k),\quad H_k=\nabla^2L(F_k).$$7

to the local quadratic. In the boosting implementation, split finding and leaf fitting remain identical, but the leaf Hessian is effectively shifted by the adaptive $$Q_k(f)=\left\langle f,\, g_k+\frac12 H_k[f]\right\rangle, \qquad g_k=\nabla L(F_k),\quad H_k=\nabla^2L(F_k).$$8. The regularization is “minimal” in the sense that it does not replace the Newton step by a different subproblem; it adds only a scalar ridge term, which controls third-order Taylor remainder effects, prevents unstable Newton steps far from the optimum, and vanishes as $$Q_k(f)=\left\langle f,\, g_k+\frac12 H_k[f]\right\rangle, \qquad g_k=\nabla L(F_k),\quad H_k=\nabla^2L(F_k).$$9.

The experiments are designed to show why this modification matters. On the Wine Quality dataset with depth-4 trees and the Charbonnier loss

$$f_{k+1}=-H_k^{-1}g_k=\arg\min_{f\in\mathcal H}Q_k(f),$$0

vanilla Newton boosting can diverge at learning rate $$f_{k+1}=-H_k^{-1}g_k=\arg\min_{f\in\mathcal H}Q_k(f),$$1, whereas the gradient-regularized Newton scheme converges and typically decreases the loss fastest. For smaller $$f_{k+1}=-H_k^{-1}g_k=\arg\min_{f\in\mathcal H}Q_k(f),$$2, the regularized scheme behaves like a lightly damped Newton method initially and then improves as the gradient norm shrinks, whereas a fixed $$f_{k+1}=-H_k^{-1}g_k=\arg\min_{f\in\mathcal H}Q_k(f),$$3-regularized model converges to a regularized optimum rather than the original objective. On the HIGGS dataset with binary cross entropy, both the cosine angle $$f_{k+1}=-H_k^{-1}g_k=\arg\min_{f\in\mathcal H}Q_k(f),$$4 and the weak gradient edge $$f_{k+1}=-H_k^{-1}g_k=\arg\min_{f\in\mathcal H}Q_k(f),$$5 increase with tree depth, are high early in training, and then plateau (Zozoulenko et al., 1 May 2026).

5. Broader restricted-Newton interpretations in adjacent literature

The term Restricted Newton Descent is formalized in the boosting setting, but several neighboring papers use closely aligned restricted-Newton interpretations.

Context	Restriction mechanism	Paper
Two-player Nash equilibrium problems	Predicted opponent action and descent-certified backtracking	(Kolossoski et al., 2022)
Unconstrained nonconvex optimization	MINRES truncation, curvature test, SOL/NPC/GD switching	(Zeng et al., 4 Jan 2026)
Function-space training dynamics	Projection onto the model tangent space $$f_{k+1}=-H_k^{-1}g_k=\arg\min_{f\in\mathcal H}Q_k(f),$$6	(McKay et al., 11 May 2026)
Nonsmooth optimization on manifolds	Restricted-memory bundle aggregation and quasi-Newton transport	(Tang et al., 2024)

In the two-player Nash equilibrium setting, the proposed method is best understood as a Jacobi-type Newton descent scheme. Each player computes a Newton-like step for their own objective, but it is built around a prediction of the other player’s move rather than the other player’s current iterate. A simultaneous backtracking procedure on a common step size $$f_{k+1}=-H_k^{-1}g_k=\arg\min_{f\in\mathcal H}Q_k(f),$$7 rescales the mixed Hessian blocks, recomputes the direction, and accepts it only when Armijo-type sufficient decrease and angle conditions hold for the player-specific objectives evaluated at predicted opponent actions. The method is explicitly motivated by the problem that standard Newton for the stationarity system can converge to maximizers or saddle points rather than true minimizers in nonconvex games.

In unconstrained nonconvex optimization, a MINRES-based Newton-type algorithm develops a related restricted philosophy. A regularized system

$$f_{k+1}=-H_k^{-1}g_k=\arg\min_{f\in\mathcal H}Q_k(f),$$8

is solved approximately by MINRES, which can return either an inexact regularized Newton step, a non-positive-curvature direction extracted from a residual, or a fallback gradient direction if a curvature test fails. The algorithm then uses ordinary Armijo for solution and gradient directions, and a modified forward/backward linesearch for non-positive-curvature directions. Under the Kurdyka–Łojasiewicz inequality and an NPC-detectability condition, the method can avoid strict saddle points and converge to second-order critical points.

In function-space optimization, the generalized Gauss–Newton matrix is interpreted as a restricted Newton method in function space. The restriction is geometric: parameter updates can move the model output only inside the tangent space $$f_{k+1}=-H_k^{-1}g_k=\arg\min_{f\in\mathcal H}Q_k(f),$$9. In this view, the Gauss–Newton flow is the $$f_{k+1}^w\in \arg\min_{f\in\mathcal F} Q_k(f), \qquad F_{k+1}=F_k+\eta f_{k+1}^w.$$0-weighted projection of the Newton direction in function space onto the reachable tangent space. A Jacobian-only variant projects the loss gradient onto the same tangent space. The paper terms the resulting removal of Jacobian singular-value distortion “error whitening,” and uses it to explain why Gauss–Newton descent can outperform full Newton.

On Riemannian manifolds, a restricted-memory quasi-Newton bundle method provides another Newton-type restriction. The candidate descent direction has the form

$$f_{k+1}^w\in \arg\min_{f\in\mathcal F} Q_k(f), \qquad F_{k+1}=F_k+\eta f_{k+1}^w.$$1

where $$f_{k+1}^w\in \arg\min_{f\in\mathcal F} Q_k(f), \qquad F_{k+1}=F_k+\eta f_{k+1}^w.$$2 is a symmetric positive definite inverse-Hessian approximation and $$f_{k+1}^w\in \arg\min_{f\in\mathcal F} Q_k(f), \qquad F_{k+1}=F_k+\eta f_{k+1}^w.$$3 is an aggregate subgradient. The “restricted-memory” aspect comes from compressing bundle information into a 3-variable quadratic subproblem rather than maintaining a full bundle QP. Retractions, vector transport, serious/null steps, and a Riemannian semismooth line search replace their Euclidean counterparts.

6. Conceptual scope, limitations, and recurring misconceptions

A common misconception is that restricted Newton descent is simply damped Newton or Newton with line search. In the formal boosting framework, the decisive restriction is the admissible update set $$f_{k+1}^w\in \arg\min_{f\in\mathcal F} Q_k(f), \qquad F_{k+1}=F_k+\eta f_{k+1}^w.$$4, not merely a scalar step-size reduction; the method is built around inexact Newton iterates in a weak learner class, together with geometry-based measures of approximation quality (Zozoulenko et al., 1 May 2026). This suggests that “restricted” refers to the structure of the allowable second-order direction, not only to globalization.

Another misconception is that restriction is merely a computational concession. The motivating argument is stronger: classical Newton’s method is not globally convergent for general strictly convex losses, even with damping or line search, whereas a theory that matches practical boosting systems must account for approximation and restriction explicitly. In the gradient-regularized scheme, the adaptive regularizer

$$f_{k+1}^w\in \arg\min_{f\in\mathcal F} Q_k(f), \qquad F_{k+1}=F_k+\eta f_{k+1}^w.$$5

is precisely the device that stabilizes Newton globally while preserving asymptotically Newton-like behavior near optimality. The corresponding theory does not assert generic quadratic convergence; instead it gives a global $$f_{k+1}^w\in \arg\min_{f\in\mathcal F} Q_k(f), \qquad F_{k+1}=F_k+\eta f_{k+1}^w.$$6 rate for convex losses with Lipschitz Hessians, while vanilla restricted Newton yields linear convergence under stronger smoothness, strong convexity, and Hessian-dominance assumptions (Zozoulenko et al., 1 May 2026).

A further misconception is that all restricted-Newton methods are variants of the same algorithm. The surrounding literature shows otherwise. In some settings, restriction means descent certification against predicted objectives; in others it means Krylov truncation with non-positive-curvature detection, tangent-space projection in function space, or restricted-memory aggregation in bundle methods. The family resemblance is real, but the mechanisms differ. The most defensible common description is therefore methodological rather than procedural: Restricted Newton Descent names a class of Newton-type strategies that retain second-order local modeling while constraining, filtering, or projecting the realized update so that it respects the geometry, model class, or descent structure of the underlying problem.