Gradient Regularized Newton Boosting Trees with Global Convergence

Abstract: Gradient Boosting Decision Trees (GBDTs) dominate tabular machine learning, with modern implementations like XGBoost, LightGBM, and CatBoost being based on Newton boosting: a second-order descent step in the space of decision trees. Despite its empirical success, the global convergence of Newton boosting is poorly understood compared to first-order boosting. In this paper, we introduce Restricted Newton Descent, which studies convex optimization with Newton's method on Hilbert spaces with inexact iterates, based on the concepts of cosine angle and weak gradient edge. Within this framework, we recover Newton boosting with GBDTs and classical finite-dimensional theory as special cases. We first prove that vanilla Newton boosting achieves a linear rate of convergence for smooth, strongly convex losses that satisfy a Hessian-dominance condition. To handle general convex losses with Lipschitz Hessians, we extend a recent gradient regularized Newton scheme to the restricted weak learner setting. This scheme minimally modifies the classical algorithm by introducing an adaptive $\ell_2$-regularization term proportional to the square root of the gradient norm at each iteration. We establish a $\mathcal{O}(\frac{1}{k^2})$ rate for this scheme, thereby obtaining a globally convergent second-order GBDT algorithm with a rate matching that of first-order boosting with Nesterov momentum. In numerical experiments, we show that our scheme converges while vanilla Newton boosting may diverge.

• The paper presents a novel framework that guarantees global convergence for Newton boosting by leveraging a restricted Newton descent paradigm over Hilbert spaces.
• It introduces an adaptive gradient regularization technique that achieves an O(1/k²) convergence rate for general convex losses and robust GBDT performance.
• Empirical validations demonstrate that GRN boosting outperforms vanilla Newton boosting in stability and loss minimization, even with minimal hyperparameter tuning.

Gradient Regularized Newton Boosting Trees: Theory and Practice

Introduction

The paper "Gradient Regularized Newton Boosting Trees with Global Convergence" (2605.00581) presents a comprehensive theoretical framework for analyzing and improving the convergence properties of Gradient Boosting Decision Trees (GBDTs) under second-order (Newton) boosting. By rigorously formulating boosting as convex optimization on Hilbert spaces and introducing the Restricted Newton Descent paradigm, the authors bridge empirical machine learning practices with advanced convex optimization theory. The central contribution is to guarantee global convergence rates for Newton boosting algorithms, which have previously lacked such guarantees, especially in settings beyond strongly convex losses.

Restricted Newton Descent Framework

The central theoretical construct is Restricted Newton Descent, which generalizes classical Newton's method to Hilbert spaces and accommodates weak learners (such as decision trees) in lieu of exact Newton directions. Optimization proceeds by minimizing a second-order Taylor approximation of the loss, with updates restricted to a family $\mathcal{F}$ of weak learners closed under scalar multiplication.

This abstraction encompasses algorithms like XGBoost and LightGBM as special cases, with the functional boosting space $\mathcal{L}^2(\nu_X)$ mapped onto the empirical risk minimization paradigm. The update for each weak learner is defined by minimizing a local quadratic $Q_k(f)$ over $\mathcal{F}$, leading to tree fitting formulas that recover standard Newton boosting updates (e.g., for leaf values in XGBoost).

Convergence Guarantees for Newton Boosting

Vanilla Newton Boosting

For smooth, strongly convex losses with Hessian-dominance, the analysis proves a global linear convergence rate for vanilla Newton boosting. The critical factor is the Hessian-induced cosine angle $\Theta_k$ measuring alignment between the exact and weak Newton directions at each boosting step. The rate depends on $\Theta^2$ and the Hessian-dominance constant $c$.

This result subsumes previous analyses for logistic loss and extends them to general convex losses under mild structural conditions. Importantly, practical losses such as cross entropy (with $\ell_2$ regularization or probability clipping) satisfy these conditions, ensuring theoretical convergence for real-world boosting schemes.

Gradient Regularized Newton (GRN) Boosting

Motivated by the lack of global convergence for Newton's method even with fixed damping, the authors extend the gradient regularized Newton (GRN) scheme to restricted, weak learner settings. GRN modifies the classical update by adaptively adding an $\ell_2$-regularization term proportional to $\sqrt{\|\nabla L(\cdot)\|}$, avoiding expensive cubic regularization subproblems.

The analysis establishes a global $\mathcal{L}^2(\nu_X)$0 convergence rate for GRN boosting schemes on general convex losses with $\mathcal{L}^2(\nu_X)$1-Lipschitz continuous Hessians, matching the rate of Nesterov-momentum accelerated first-order boosting. Theoretical bounds depend on both the weak gradient edge $\mathcal{L}^2(\nu_X)$2 (quantifying alignment of the weak gradient with the true gradient) and problem constants. Locally, GRN achieves a linear rate under strong convexity and, for ideal $\mathcal{L}^2(\nu_X)$3 and $\mathcal{L}^2(\nu_X)$4, can exhibit superlinear behavior.

Empirical Validation

Numerical experiments corroborate the theoretical claims. On the Wine Quality UCI dataset (Charbonnier loss), vanilla Newton boosting diverges for $\mathcal{L}^2(\nu_X)$5, while GRN converges robustly (Figure 1). For small learning rates, vanilla Newton is efficient but unstable, whereas GRN maintains convergence with minimal sensitivity to hyperparameters. Notably, GRN achieves loss minimization for the original (unregularized) loss, unlike static $\mathcal{L}^2(\nu_X)$6-regularized schemes that converge only to a regularized optimum.

Figure 1: Charbonnier train loss on the Wine Quality UCI dataset, for different gradient boosting schemes and learning rates $\mathcal{L}^2(\nu_X)$7. Vanilla Newton boosting diverges for $\mathcal{L}^2(\nu_X)$8.

The study also demonstrates the effect of tree depth on weak learner strength in BCE loss boosting (Figure 2). Larger tree depths yield higher cosine angles $\mathcal{L}^2(\nu_X)$9 and weak gradient edges $Q_k(f)$0, indicating better alignment and convergence rates. These metrics stabilize early in training, and their behavior mirrors theoretical bounds, reinforcing the relevance of the restricted update analysis.

Figure 2: Binary cross entropy, cosine angle $Q_k(f)$1, and weak gradient edge $Q_k(f)$2 computed at each boosting iteration $Q_k(f)$3, for varying tree depths $Q_k(f)$4. The exact values are noisy between iterations, but the rolling window average (highlighted in bold) remains stable.

Practical Implementation and Regularity

The GRN scheme is trivial to implement in GBDT frameworks: it requires only the addition of an adaptive $Q_k(f)$5 regularizer based on the gradient norm per iteration. All other tree fitting and split finding calculations remain unchanged. Crucially, the regularity conditions (smoothness, strong convexity, Lipschitz Hessians) required for theoretical guarantees are inherited directly from the per-sample loss used in boosting, ensuring practicality across standard choices (MSE, BCE, CCE, Charbonnier).

Implications and Future Directions

The results have several practical and theoretical implications:

• Robust Global Convergence: Algorithms such as XGBoost and LightGBM can be augmented with GRN-type updates to guarantee global convergence for a wider class of losses and faster rates.
• Accelerated Boosting: Achieving $Q_k(f)$6 rates with second-order methods, without requiring momentum or multiple trees per iteration, advances boosting optimization theory and practice.
• Analysis of Weak Learner Restrictions: The formalization of the restriction operation under various optimization schemes opens the door for improved analysis of boosting in other settings, including neural and random feature models.
• Further Acceleration: The possibility of $Q_k(f)$7 rates (as obtained in cubic regularized Newton in finite-dimensional settings), adaptation to nonconvex losses, generalization bounds, and improved dependence on $Q_k(f)$8, $Q_k(f)$9 are promising research avenues.

A notable technical observation is that while vanilla Newton possesses quadratic local convergence, global convergence is not guaranteed for generic convex losses—even with damping—unless adaptive regularization is used. The presented drift-to-loss generator for one-dimensional Newton counterexamples provides explicit demonstration.

Conclusion

This paper delivers a rigorous theoretical foundation and practical solution for globally convergent Newton boosting, via adaptive gradient regularization and Hilbert space-based analysis. Empirical and theoretical evidence demonstrates the superiority and robustness of GRN boosting, representing an important step toward reliable, efficient, and principled ensemble methods in tabular machine learning. Future research may extend these convergence guarantees to more general weak learner settings, deeper optimization schemes, and generalization analyses.