Gradient Boosting: Concepts & Advances

• Gradient boosting is a stagewise ensemble method that adds weak learners sequentially to minimize a differentiable loss function.
• It employs both first- and second-order optimization techniques, often using decision trees, to effectively reduce prediction errors.
• The approach is versatile, extending from traditional tree-based models to deep and probabilistic variants for tasks like regression and classification.

Searching arXiv for recent and foundational gradient boosting papers to ground the article. Gradient boosting (GB) is a stagewise ensemble-learning methodology that constructs an additive predictor by sequentially adding weak learners so as to reduce a differentiable objective. In its most general form, the predictor is written as $f_M(x)=\sum_{m=1}^{M}\beta_m\,h_m(x;\Theta_m)$, and each stage fits a new component to the negative gradient, or pseudo-residual, induced by the current model. Decision trees are the canonical weak learners in practical gradient boosting machines (GBMs), but the same functional template has been instantiated with symmetric trees, piece-wise linear regression trees, differentiable soft trees, widened multi-output embeddings, deep-network heads, and ensembles of PINN subnetworks (Chevalier et al., 2024, Feng et al., 2020, Emami et al., 2023, Fang et al., 2023).

1. Functional formulation

In the standard supervised setting, GB minimizes an empirical risk such as

$L(F)=\frac{1}{n}\sum_{i=1}^n l(y_i,F(x_i)),$

or, equivalently in several formulations, $L(f)=\sum_{i=1}^n \ell(y_i,f(x_i))$. The model is built additively, for example as $F_T(x)=\sum_{t=1}^T f_t(x)$ or $f^m(x)=f^{m-1}(x)+\lambda\,h_m(x)$, with $\lambda$ or $\nu$ acting as a shrinkage parameter (Luo et al., 2022, Chevalier et al., 2024).

The core computational object is the pseudo-residual. In a first-order formulation it is the negative derivative of the loss with respect to the current prediction,

$$g_i^{(m)}=-\left.\frac{\partial \mathcal{L}(y_i,b)}{\partial b}\right|_{b=f^{m-1}(x_i)},$$

or, in equivalent notation,

$$r_i^{(m)}=-\left.\frac{\partial}{\partial \hat y}\ell(y_i,\hat y)\right|_{\hat y=f_{m-1}(x_i)}.$$

A new weak learner is fitted to these targets, usually by least squares in the weak-learner class, and then added to the current ensemble (Chevalier et al., 2024, Lu et al., 2018).

Tree-based implementations often refine this stagewise step with leaf-wise optimization. In the generic first-order tree construction, one fits a small tree to the pseudo-residuals and then computes a leaf update

$$\hat b_j^m=\arg\min_b \sum_{i:x_i\in R_j^m}\mathcal{L}\bigl(y_i,f^{m-1}(x_i)+b\bigr),$$

yielding an update of the form

$L(F)=\frac{1}{n}\sum_{i=1}^n l(y_i,F(x_i)),$0

Second-order variants incorporate both gradients and Hessians. In the XGBoost-style formulation, the approximate objective uses

$L(F)=\frac{1}{n}\sum_{i=1}^n l(y_i,F(x_i)),$1

and the optimal leaf value becomes

$L(F)=\frac{1}{n}\sum_{i=1}^n l(y_i,F(x_i)),$2

This makes explicit that GB is not a single algorithm but a family of stagewise functional optimization procedures parameterized by the weak-learner class, the order of local approximation, and the regularization used at each step (Chevalier et al., 2024).

2. Optimization viewpoints and convergence theory

A substantial theoretical literature treats GB as optimization in function space. “Optimization by gradient boosting” formulates the problem in the Hilbert space $L(F)=\frac{1}{n}\sum_{i=1}^n l(y_i,F(x_i)),$3, studies two widespread versions of boosting, and proves that the empirical risk decreases monotonically toward the infimum over the linear span of the weak learners. When the loss is not already strongly convex in its first argument, the paper adds an $L(F)=\frac{1}{n}\sum_{i=1}^n l(y_i,F(x_i)),$4 penalty,

$L(F)=\frac{1}{n}\sum_{i=1}^n l(y_i,F(x_i)),$5

and shows that strong convexity yields a unique minimizer; in that framework, statistical regularization is handled by penalization rather than by early stopping (Biau et al., 2017).

The basic first-order rate can itself be improved. “Accelerating Gradient Boosting Machine” introduces Nesterov-style acceleration through three coupled ensembles $L(F)=\frac{1}{n}\sum_{i=1}^n l(y_i,F(x_i)),$6, a corrected pseudo-residual, and a dual update scheme. Under $L(F)=\frac{1}{n}\sum_{i=1}^n l(y_i,F(x_i)),$7-smoothness, a suitable step size, and a bound on the momentum parameter $L(F)=\frac{1}{n}\sum_{i=1}^n l(y_i,F(x_i)),$8, the method achieves

$L(F)=\frac{1}{n}\sum_{i=1}^n l(y_i,F(x_i)),$9

that is, an $L(f)=\sum_{i=1}^n \ell(y_i,f(x_i))$0 rate rather than the usual $L(f)=\sum_{i=1}^n \ell(y_i,f(x_i))$1 rate (Lu et al., 2019).

The geometry of the weak-learner dictionary also matters. “Randomized Gradient Boosting Machine” interprets GBM as greedy coordinate descent on the coefficient vector of the weak learners and RGBM as random-then-greedy coordinate descent. Its convergence rate depends on the Minimal Cosine Angle,

$L(f)=\sum_{i=1}^n \ell(y_i,f(x_i))$2

which measures the density of weak learners in prediction space. Under $L(f)=\sum_{i=1}^n \ell(y_i,f(x_i))$3-strong convexity and $L(f)=\sum_{i=1}^n \ell(y_i,f(x_i))$4-smoothness, the expected suboptimality contracts linearly as

$L(f)=\sum_{i=1}^n \ell(y_i,f(x_i))$5

The same paper also gives a fixed step-size rule that avoids line search under $L(f)=\sum_{i=1}^n \ell(y_i,f(x_i))$6-smoothness (Lu et al., 2018).

Trust-region methods furnish another route beyond ordinary first- and second-order boosting. TRBoost defines targets

$L(f)=\sum_{i=1}^n \ell(y_i,f(x_i))$7

where the adaptive barrier $L(f)=\sum_{i=1}^n \ell(y_i,f(x_i))$8 plays the role of the inverse trust-region radius. Because the trust region is compact, the quadratic model remains well defined even when Hessians are negative or zero. The convergence analysis recovers $L(f)=\sum_{i=1}^n \ell(y_i,f(x_i))$9 for MAE, local quadratic convergence for MSE when $F_T(x)=\sum_{t=1}^T f_t(x)$0, and global linear convergence for logistic loss under bounded probability assumptions (Luo et al., 2022).

A distinct line of work addresses non-convex and multimodal losses. SGLB replaces deterministic descent by a Langevin-type stochastic dynamics in prediction space, with updates that include both shrinkage and calibrated Gaussian noise. The limiting process has Gibbs invariant measure proportional to $F_T(x)=\sum_{t=1}^T f_t(x)$1, and the paper derives global empirical-risk guarantees up to an $F_T(x)=\sum_{t=1}^T f_t(x)$2 gap. Empirically, it outperforms classic gradient boosting on classification tasks with 0-1 loss, a setting the paper describes as multimodal (Ustimenko et al., 2020).

Iteration weighting can also be modified after the ensemble has been built. AGBoost assigns attention weights $F_T(x)=\sum_{t=1}^T f_t(x)$3 to boosting iterations, with a contamination-model construction

$F_T(x)=\sum_{t=1}^T f_t(x)$4

and learns $F_T(x)=\sum_{t=1}^T f_t(x)$5 by solving a quadratic program on the simplex. Across 11 regression datasets and two tree types, the paper reports higher $F_T(x)=\sum_{t=1}^T f_t(x)$6 and lower MAE than both standard GBM and its non-parametric attention variant (Konstantinov et al., 2022).

3. Tree ensembles, split finding, and systems design

The dominant practical instantiations of GB use decision trees, but they differ materially in approximation order, split search, tree topology, and regularization.

Variant	Distinguishing mechanism	Citation
GBM	First-order gradients, subsampling rate $F_T(x)=\sum_{t=1}^T f_t(x)$7, depth $F_T(x)=\sum_{t=1}^T f_t(x)$8, learning rate $F_T(x)=\sum_{t=1}^T f_t(x)$9	(Chevalier et al., 2024)
XGBoost	Second-order Taylor objective with tree-complexity and leaf-weight regularization	(Chevalier et al., 2024)
LightGBM	Leaf-wise growth with histogram-based binning	(Chevalier et al., 2024)
CatBoost	Symmetric trees and ordered target statistics for categoricals	(Chevalier et al., 2024)
DART	Randomly drops trees and renormalizes to reduce over-specialization	(Chevalier et al., 2024)
EGBM	Depth-1 stumps for main effects and depth-2 stumps for selected interactions	(Chevalier et al., 2024)

In XGBoost-style boosting, the split criterion is written explicitly in terms of gradient and Hessian aggregates. For a prospective split of a region $f^m(x)=f^{m-1}(x)+\lambda\,h_m(x)$0 into $f^m(x)=f^{m-1}(x)+\lambda\,h_m(x)$1 and $f^m(x)=f^{m-1}(x)+\lambda\,h_m(x)$2, the gain is

$f^m(x)=f^{m-1}(x)+\lambda\,h_m(x)$3

This expression clarifies why second-order methods typically converge faster than purely first-order tree boosting when curvature is informative (Chevalier et al., 2024).

More complex base learners have been used to increase local approximation power. “Gradient Boosting With Piece-Wise Linear Regression Trees” replaces constant leaf values by local linear models and fits leaf parameters in closed form via

$f^m(x)=f^{m-1}(x)+\lambda\,h_m(x)$4

The implementation combines histogram binning, incremental feature selection, half-additive fitting, continuous leafBin arrays, SIMD summation, and BMI2 PEXT-based splitting. On Higgs, the reported AUCs are 0.8602 for GBDT-PL, 0.8540 for LightGBM, 0.8541 for XGBoost, and 0.8516 for CatBoost; the optimized implementation reports ≈20× overall speedup versus its own baseline (Shi et al., 2018).

Large-scale regression has motivated still different tree surrogates. The Gradient Boosted Binary Histogram Ensemble uses randomly rotated binary histograms as weak learners, with midpoint splits and ensemble averaging inside each boosting step. Its analysis gives rates in $f^m(x)=f^{m-1}(x)+\lambda\,h_m(x)$5 and $f^m(x)=f^{m-1}(x)+\lambda\,h_m(x)$6, and in the $f^m(x)=f^{m-1}(x)+\lambda\,h_m(x)$7 setting shows that using an ensemble regressor as the base learner reduces the number of iterations needed for the fast rate. Empirically, the paper reports lowest or tied lowest MSE/MAE on most datasets and 2–10% lower error than GBRT or random forest on several large-scale datasets (Hang et al., 2021).

4. Probabilistic and distributional boosting

Standard GB predicts a point quantity, but several extensions lift the objective to full predictive distributions. PGBM does so while retaining a single ensemble of trees. Its key step is to treat each leaf weight $f^m(x)=f^{m-1}(x)+\lambda\,h_m(x)$8 as a random variable and approximate its mean and variance from within-leaf gradient and Hessian statistics, producing moment estimates

$f^m(x)=f^{m-1}(x)+\lambda\,h_m(x)$9

These are then propagated through the ensemble via stochastic update equations for the prediction mean and variance. After training, one samples from a chosen parametric distribution $\lambda$0. The method is presented as providing probabilistic estimates without compromising point performance in a single model, and the paper reports speedups of up to several orders of magnitude over existing state-of-the-art methods on large datasets, together with up to 10% improvement in point forecasting performance and up to 300% improvement in probabilistic forecasting performance in hierarchical time series (Sprangers et al., 2021).

Distributional GBM generalizes this idea into a unified likelihood-based framework. In the GBMLSS variant, each distributional parameter $\lambda$1 of a parametric family is modeled by its own boosted ensemble; in NFBoost, a conditional Normalizing Flow approximates the inverse CDF. Both are implemented as custom objectives on top of XGBoost and LightGBM, with per-parameter gradients and Hessians passed to the standard second-order tree learner. On six UCI regression datasets, GBMLSS-LightGBM attains the best average CRPS rank of 2.5 and the best runtime rank of 1.0, while NFBoost-XGBoost leads on the heavily non-Gaussian Protein set and Boston (März et al., 2022).

Another probabilistic interpretation is kernel-theoretic rather than likelihood-based. “Gradient Boosting Performs Gaussian Process Inference” shows that gradient boosting with symmetric decision trees can be reformulated as a kernel method. In the limit of small learning rate and infinitely many trees, the ensemble converges to the Kernel Ridge Regression solution

$\lambda$2

which is also the posterior mean of a Gaussian process. The same paper gives a Monte Carlo sampler that converges to exact GP posterior samples in the limit and reports better knowledge uncertainty estimates for out-of-domain detection (Ustimenko et al., 2022).

A broad actuarial comparison places these probabilistic methods in context with point-prediction boosters. Across five public datasets for claim frequency and severity, the study compares GBM, XGBoost, DART, LightGBM, CatBoost, EGBM, PGBM, XGBoostLSS, cyclic GBM, and NGBoost. It reports that LightGBM and XGBoostLSS win in terms of computational efficiency, that CatBoost sometimes improves predictive performance especially with high-cardinality categorical variables, that the fully interpretable EGBM is competitive with black-box algorithms, and that there is no trade-off between model adequacy and predictive accuracy (Chevalier et al., 2024).

5. Beyond classical tree boosting

Although decision trees remain central, several papers make explicit that the residual-fitting principle is not tree-specific. Soft Gradient Boosting Machine rewires $\lambda$3 differentiable base learners into a single DAG and augments the global loss with local residual-regression objectives,

$\lambda$4

Because all learners are differentiable, they can be jointly optimized by back-propagation rather than trained strictly one after another. The paper reports near-linear wall-clock speed-up in $\lambda$5, up to $\lambda$6 faster training for $\lambda$7 on MNIST/CIFAR, and improved accuracy relative to hard boosting with the same base learner (Feng et al., 2020).

A related direction applies boosting directly inside deep architectures. GB-CNN and GB-DNN append one dense layer at each iteration, freeze all previously learned dense layers, and fit the new head to the pseudo-residuals of cross-entropy loss. In GB-CNN, the convolutional feature extractor remains trainable; in GB-DNN, only the new dense layer is optimized. Across seven image datasets and seven tabular datasets, the paper reports consistently better classification accuracy than identically architected standard CNNs and DNNs; for Rock-Paper-Scissors with 20-unit heads, the reported accuracies are 87.37 for GB-CNN and 68.01 for CNN (Emami et al., 2023).

Wide Boosting modifies the mapping between the boosted ensemble and the loss by inserting a learnable matrix $\lambda$8, so that the boosted model outputs an intermediate vector $\lambda$9 and the final prediction is $\nu$0. This permits the boosted representation to be wider than the target dimension and to act as an embedding. On a modified multi-Bernoulli MNIST task, the paper reports average test log-loss 0.0541 for Wide Boosting and 0.0582 for standard GB, with fewer trees on average; in a transfer-style downstream task, the reported downstream log-losses are 0.180 for WB embeddings and 0.220 for GB embeddings (Horrell, 2020).

Boosting has also been adapted to scientific machine learning. GB-PINNs represent the PDE solution as a sum of subnetworks,

$\nu$1

and train each new subnetwork on the residual error left by its predecessors while freezing earlier parameters. The benchmarks include one-dimensional and two-dimensional singular perturbation problems, interior boundary layers, and nonlinear reaction–diffusion. The summary reports 10×–50× reductions in relative $\nu$2 error on challenging multi-scale PDEs, with representative results such as 0.43% versus 12.56% on a 1D singular perturbation benchmark (Fang et al., 2023).

6. Scope, recurring misconceptions, and significance

Across the cited literature, GB is applied to tabular prediction, actuarial claim frequency and severity, large-scale regression, multi-output learning, uncertainty quantification, out-of-domain detection, image classification, and PDE solving (Chevalier et al., 2024, Ustimenko et al., 2022, Emami et al., 2023, Fang et al., 2023). This breadth is already sufficient to reject the narrow identification of gradient boosting with a single tree-growing routine.

Several recurring misconceptions are addressed directly by the literature. One is that GB is inherently first-order: in fact, second-order boosting, trust-region boosting, accelerated boosting, and Langevin boosting all modify the stagewise update while keeping the additive model viewpoint (Luo et al., 2022, Lu et al., 2019, Ustimenko et al., 2020). A second is that boosting must be regularized mainly by early stopping; the $\nu$3-penalized strong-convexity analysis shows that one can study boosting “run forever” and obtain both optimization convergence and statistical consistency in that regime (Biau et al., 2017). A third is that boosting is intrinsically sequential and therefore resistant to parallelization; sGBM explicitly attacks that bottleneck by training differentiable learners jointly with a single backward pass (Feng et al., 2020).

The empirical record in these papers also complicates the view that accuracy, interpretability, and uncertainty estimation are necessarily in opposition. EGBM is described as fully interpretable yet competitive on actuarial benchmarks, PGBM and DGBM add probabilistic forecasts without requiring separate models for each quantile, and GP-style interpretations of symmetric-tree boosting connect ensemble prediction to posterior means and posterior sampling (Chevalier et al., 2024, Sprangers et al., 2021, März et al., 2022, Ustimenko et al., 2022).

A plausible implication is that GB is best understood not as a fixed algorithmic artifact, but as a general strategy for functional descent with additive models. The choice of weak learner, local approximation order, regularization scheme, and uncertainty formalism determines whether the resulting method behaves like classical tree boosting, a trust-region solver, a probabilistic forecaster, a deep residual fitter, or an ensemble method for scientific computing.