Newton Boosting Techniques

• Newton boosting is a second-order variant of boosting that uses both gradients and Hessians to guide stagewise updates in additive models.
• It transforms tree boosting into a weighted least-squares problem by optimizing leaf values with Newton pseudo-responses and Hessian weights.
• Empirical studies indicate that Newton boosting achieves faster convergence and improved predictive accuracy compared to first-order methods.

Newton boosting is the second-order variant of boosting in which the stagewise update is derived from a Newton step in function space rather than from a pure gradient step. In the standard additive model $F_m(x)=F_{m-1}(x)+f_m(x)$, gradient boosting fits $f_m$ to negative gradients of the loss, whereas Newton boosting fits $f_m$ to a curvature-rescaled target using both first and second derivatives of the loss with respect to current predictions. In tree-based implementations, this yields weighted least-squares tree fitting, leaf values of the form $-\sum g/\sum h$, and the second-order procedures used by systems such as XGBoost and LightGBM (Sigrist, 2018, Parnell et al., 2020).

1. Functional optimization viewpoint

Newton boosting is formulated as empirical risk minimization over an additive span of base learners. The underlying objective is an expected or empirical risk

$$R(F)=E_{Y,X}[L(Y,F(X))], \qquad R^e(F)=\frac1n\sum_{i=1}^n L(y_i,F(x_i)),$$

and the estimator is built stagewise through

$F_m(x)=F_{m-1}(x)+f_m(x), \qquad f_m\in\mathcal S,$

with $\mathcal S$ typically a class of regression trees (Sigrist, 2018). In the decision-tree exposition of boosting, the same structure appears as an additive model over trees,

$F_M(x)=\sum_{m=1}^M \alpha_m T_m(x),$

or, in shrinkage notation,

$F_M(x)=\sum_{m=1}^M \nu f_m(x),$

together with an iterative loop that trains a classifier $T_k$, assigns it a weight $f_m$0, and reweights the training sample for the next iteration (Coadou, 2022).

The central distinction is the order of the local approximation used to choose $f_m$1. Gradient boosting uses a first-order approximation and hence follows functional gradient descent. Newton boosting uses a second-order Taylor approximation of the risk around $f_m$2, which rescales the descent direction by local curvature. In the wording of the heterogeneous-boosting literature, modern boosting frameworks “implement Newton descent in a functional space,” and at each iteration try to find the base hypothesis closest to the Newton descent direction in a Euclidean sense (Parnell et al., 2020).

This functional viewpoint also clarifies a common source of terminological ambiguity. In tree boosting, “Newton boosting” refers to second-order descent in prediction space with stagewise addition of weak learners. By contrast, a “Pseudo-Newton Boosting layer” in a hybrid CNN has been used to denote a Newton-Raphson-type feature-refinement layer inside a neural network, with an approximate Hessian and no stagewise additive ensemble of trees; that usage is Newton-style optimization, but not classical tree-based Newton boosting (V et al., 17 Nov 2025).

2. Gradient, Newton, and hybrid stagewise updates

The unified formulation in the comparative analysis of boosting algorithms distinguishes three stagewise procedures (Sigrist, 2018). For a differentiable loss, define

$f_m$3

Gradient boosting fits a base learner to pseudo-residuals

$f_m$4

equivalently solving

$f_m$5

This is the first-order procedure described in the classical boosting overview, where residuals are interpreted as negative gradients; for mean squared error,

$f_m$6

(Coadou, 2022).

Newton boosting instead uses the quadratic expansion

$f_m$7

which yields the sample-level optimization

$f_m$8

Thus the learner is fit to the Newton pseudo-response

$f_m$9

with weights

$f_m$0

Hybrid gradient-Newton boosting lies between the two: gradient information chooses tree structure, while Newton information determines leaf values conditional on that structure (Sigrist, 2018).

A useful limiting case is squared loss. When $f_m$1 is constant, Newton and gradient boosting coincide up to a multiplicative constant, so Newton boosting reduces to gradient boosting (Sigrist, 2018). This is consistent with the tree-based derivation in the decision-tree survey, where the mean squared error example produces residual fitting directly and serves as the first-order foundation from which Newton boosting is obtained by adding second derivatives (Coadou, 2022).

3. Tree-based mechanics, leaf formulas, and loss functions

For regression trees of the form

$f_m$2

Newton boosting becomes a weighted least-squares tree-fitting problem. Given a fixed tree structure with leaves $f_m$3, the optimal leaf value in leaf $f_m$4 is

$f_m$5

which is the standard Newton leaf update used in tree-boosting systems (Sigrist, 2018). In the Hilbert-space treatment of gradient-regularized Newton trees, the same result appears leafwise as

$f_m$6

with $f_m$7 and $f_m$8 (Zozoulenko et al., 1 May 2026).

The required derivatives depend on the loss. For binary logistic loss,

$f_m$9

the derivatives are

$-\sum g/\sum h$0

so Newton boosting uses

$-\sum g/\sum h$1

The same framework extends to multiclass softmax loss, Poisson regression, Gamma regression, Tobit regression, and mean-scale Gaussian regression, with explicit $-\sum g/\sum h$2 and $-\sum g/\sum h$3 formulas available for each case (Sigrist, 2018).

The broader tree literature also makes clear that the second-order ingredients belong to the global boosting loss $-\sum g/\sum h$4, not to the impurity criteria used inside a single tree. Misclassification error, cross-entropy, and Gini index are node impurity measures; Newton boosting changes how trees are fit to gradients and Hessians rather than replacing impurity-based split evaluation in principle (Coadou, 2022).

This tree-based Newton view underlies modern implementations. XGBoost, LightGBM, and CatBoost are described as systems based on Newton boosting, that is, “a second-order descent step in the space of decision trees,” with split and leaf calculations derived from the quadratic model of the loss (Zozoulenko et al., 1 May 2026). The heterogeneous-boosting analysis treats XGBoost and LightGBM as homogeneous Newton boosting machines whose base class is fixed to trees of bounded depth (Parnell et al., 2020).

4. Empirical behavior and tuning

A central empirical result is that Newton boosting is not merely a reformulation of gradient boosting. In a unified comparison over trees, various datasets, and various loss functions, Newton boosting outperformed gradient and hybrid gradient-Newton boosting in predictive accuracy on the majority of datasets (Sigrist, 2018). The reported average ranks on real data where all four methods applied were 1.6 for Newton, 2.2 for Hybrid, 2.93 for XGBoost’s Newton variant, and 3.27 for Gradient; on simulated data the average ranks were 1.25 for Newton, 2.5 for Hybrid, 2.75 for XGBoost, and 3.5 for Gradient (Sigrist, 2018).

That study also argues that the advantage is not simply faster convergence. Newton and hybrid often reduced training loss faster than gradient boosting, but the improvement in test performance persisted across iterations and was not explained solely by reaching a given training loss earlier (Sigrist, 2018). This suggests that second-order fitting changes the effective solution path in function space rather than only its speed.

Tree-based Newton boosting introduces a regularization parameter that differs from the usual minimum number of samples per leaf. Because fitting is weighted by Hessians, the relevant complexity control is the total normalized Hessian weight in a leaf. The proposed tuning parameter is the minimum equivalent sample size per leaf $-\sum g/\sum h$5, based on normalized weights

$-\sum g/\sum h$6

with the constraint

$-\sum g/\sum h$7

For Gaussian regression this reduces exactly to the ordinary minimum number of samples per leaf, because the Hessian is constant (Sigrist, 2018). The same analysis contrasts this normalized ESS parameter with XGBoost’s unnormalized $-\sum g/\sum h$8, arguing that the latter is harder to interpret and tune because it depends on the scale of the Hessian (Sigrist, 2018).

From the older boosted-tree survey, the practical controls surrounding Newton-style methods remain the familiar ones: number of trees, tree depth or size, shrinkage, early stopping, and evaluation through ROC curves, AUC, and significance measures such as $-\sum g/\sum h$9 or AMS in high-energy physics (Coadou, 2022). This suggests continuity rather than discontinuity at the engineering level: Newton boosting changes the training step, but model selection still relies on the same structural hyperparameters and validation criteria.

5. Convergence theory and gradient regularization

The optimization theory of Newton boosting has developed more slowly than its empirical practice. The heterogeneous Newton boosting analysis establishes a global linear convergence rate under strong convexity and smoothness assumptions for a randomized setting in which the base hypothesis class may vary across iterations (Parnell et al., 2020). There the rate depends on a minimum cosine angle $$R(F)=E_{Y,X}[L(Y,F(X))], \qquad R^e(F)=\frac1n\sum_{i=1}^n L(y_i,F(x_i)),$$0 that measures how well the chosen hypothesis classes align with the Newton direction; when $$R(F)=E_{Y,X}[L(Y,F(X))], \qquad R^e(F)=\frac1n\sum_{i=1}^n L(y_i,F(x_i)),$$1, the rate matches existing global rates for Newton’s method under the same assumptions (Parnell et al., 2020).

A later Hilbert-space treatment specializes this line of reasoning to tree boosting and clarifies when vanilla Newton boosting is globally convergent (Zozoulenko et al., 1 May 2026). In that framework, the exact Newton step at iteration $$R(F)=E_{Y,X}[L(Y,F(X))], \qquad R^e(F)=\frac1n\sum_{i=1}^n L(y_i,F(x_i)),$$2 is

$$R(F)=E_{Y,X}[L(Y,F(X))], \qquad R^e(F)=\frac1n\sum_{i=1}^n L(y_i,F(x_i)),$$3

while the weak learner step is the restricted minimizer of the quadratic model over the tree class $$R(F)=E_{Y,X}[L(Y,F(X))], \qquad R^e(F)=\frac1n\sum_{i=1}^n L(y_i,F(x_i)),$$4. The quality of this approximation is quantified by a Hessian-cosine angle

$$R(F)=E_{Y,X}[L(Y,F(X))], \qquad R^e(F)=\frac1n\sum_{i=1}^n L(y_i,F(x_i)),$$5

Under $$R(F)=E_{Y,X}[L(Y,F(X))], \qquad R^e(F)=\frac1n\sum_{i=1}^n L(y_i,F(x_i)),$$6-smoothness, $$R(F)=E_{Y,X}[L(Y,F(X))], \qquad R^e(F)=\frac1n\sum_{i=1}^n L(y_i,F(x_i)),$$7-strong convexity, scalability of the weak-learner class, and a Hessian-dominance condition, vanilla Newton boosting achieves a linear rate

$$R(F)=E_{Y,X}[L(Y,F(X))], \qquad R^e(F)=\frac1n\sum_{i=1}^n L(y_i,F(x_i)),$$8

for $$R(F)=E_{Y,X}[L(Y,F(X))], \qquad R^e(F)=\frac1n\sum_{i=1}^n L(y_i,F(x_i)),$$9 (Zozoulenko et al., 1 May 2026).

The same work also shows why global convergence is delicate. For general convex losses with Lipschitz Hessians, vanilla Newton boosting may diverge, and numerical experiments demonstrate divergence on the Wine Quality dataset under Charbonnier loss for learning rate $F_m(x)=F_{m-1}(x)+f_m(x), \qquad f_m\in\mathcal S,$0 (Zozoulenko et al., 1 May 2026). To address this, the paper introduces Gradient Regularized Newton (GRN), which modifies the classical quadratic model by adding an adaptive $F_m(x)=F_{m-1}(x)+f_m(x), \qquad f_m\in\mathcal S,$1-regularization term: $F_m(x)=F_{m-1}(x)+f_m(x), \qquad f_m\in\mathcal S,$2 In practice this is implemented as

$F_m(x)=F_{m-1}(x)+f_m(x), \qquad f_m\in\mathcal S,$3

Under convexity, Lipschitz Hessian, finite sublevel sets, and a weak gradient edge condition, the resulting restricted Newton algorithm achieves

$F_m(x)=F_{m-1}(x)+f_m(x), \qquad f_m\in\mathcal S,$4

matching the rate of first-order boosting with Nesterov momentum, while experiments indicate that GRN converges in cases where vanilla Newton boosting may diverge (Zozoulenko et al., 1 May 2026).

This combination of results establishes a nuanced picture. Newton boosting can enjoy global linear convergence for smooth, strongly convex, Hessian-dominated losses, but broader global guarantees may require adaptive gradient regularization. A plausible implication is that the static $F_m(x)=F_{m-1}(x)+f_m(x), \qquad f_m\in\mathcal S,$5-regularization already present in practical libraries is not, by itself, a complete convergence theory for second-order tree boosting.

6. Extensions, accelerations, and related variants

Newton boosting has been generalized in several directions beyond homogeneous tree ensembles. A heterogeneous Newton boosting machine allows the base hypothesis class to vary across boosting iterations, sampling from a fixed set of subclasses according to a probability distribution. SnapBoost instantiates this idea by choosing between a decision tree of variable depth and a linear regressor with random Fourier features, while still fitting the Newton direction through weighted least squares (Parnell et al., 2020). On 10 OpenML binary classification datasets, SnapBoost achieved average test-loss rank 1.2 versus 2.2 for LightGBM and 2.6 for XGBoost, with significant differences reported against both competitors (Parnell et al., 2020).

Computational acceleration has also been studied at the level of the Newton step. A fast sampling framework combines Newton-style gradient tree boosting with importance sampling, using probabilities proportional to $F_m(x)=F_{m-1}(x)+f_m(x), \qquad f_m\in\mathcal S,$6 or $F_m(x)=F_{m-1}(x)+f_m(x), \qquad f_m\in\mathcal S,$7, and introduces a regularizer intended to improve the diagonal approximation in the Newton step (Zhou et al., 2019). For logistic loss, the paper proves linear convergence of the importance-sampled method under standard weak learnability conditions, and empirically reports $F_m(x)=F_{m-1}(x)+f_m(x), \qquad f_m\in\mathcal S,$8–$F_m(x)=F_{m-1}(x)+f_m(x), \qquad f_m\in\mathcal S,$9 acceleration on LogitBoost and LambdaMART “without appreciable performance loss” (Zhou et al., 2019).

The second-order viewpoint is not restricted to supervised tabular prediction. In univariate nonparametric maximum likelihood estimation, boosting has been derived from a second-order approximation of a surrogate log-likelihood for log-density estimation. There the Newton pseudo-response at point $\mathcal S$0 is

$\mathcal S$1

and each boosting step solves a weighted least-squares problem in a smooth weak-learner class such as Gaussian kernels or smoothing splines (Li et al., 2021). This is functionally the same Newton principle—fit weak learners to a curvature-weighted Newton direction—but applied to density estimation rather than classification or regression.

Related recent work also shows that Newton-style optimization may enter boosting at a different level from classical tree boosting. HRT-Boost uses Newton-optimized oblique trees as base learners, where each node is trained by a damped Newton or Gauss-Newton method, but the ensemble itself is updated by stagewise functional gradient descent under squared loss (Li et al., 22 May 2026). That design is Newton-style at the node level, not a full Newton boosting method in the XGBoost sense. Likewise, the CNN-based “Pseudo-Newton Boosting layer” is a second-order feature-weight refinement module integrated into a neural architecture, rather than a sequence of weak learners fitted to residuals (V et al., 17 Nov 2025).

Taken together, these variants indicate that “Newton boosting” has a narrow and a broad usage. In the narrow sense, it denotes second-order stagewise boosting of additive models, especially trees, using Newton pseudo-responses and Hessian weights. In the broader literature, it can denote any boosting or boosting-like procedure whose update is derived from a second-order local model. The tree-based definition remains the canonical one for tabular machine learning and for the systems most often associated with the term (Sigrist, 2018, Zozoulenko et al., 1 May 2026).