Gradient-Boosted Decision Trees

• Gradient-boosted decision tree models are additive ensembles that sequentially incorporate shallow trees via functional gradient descent to minimize a convex loss.
• They employ two main algorithmic variants that fit base learners to the negative gradient, balancing adaptive and fixed step-size strategies for convergence.
• Regularization techniques, such as L2 penalties, enable unlimited boosting rounds by controlling overfitting while ensuring theoretical stability and consistency.

Gradient-Boosted Decision Tree Models (GBDTs) are state-of-the-art predictive models that construct additive ensembles of weak learners, typically decision trees, by solving a convex optimization problem in infinite-dimensional function space. Instead of optimizing parameters of a single model, GBDTs iteratively build a function $F(x)$ by sequentially adding base learners fitted to (pseudo-)residuals, implementing a functional gradient descent scheme. This methodology underlies modern machine learning systems across domains, excelling particularly on structured/tabular data.

1. Functional Optimization Formulation and Core Algorithms

GBDTs formalize supervised learning as the minimization of a risk functional $C(F) = \mathbb{E}[\psi(F(X), Y)]$, where $\psi$ is a convex loss (\eg, squared, logistic, or exponential loss) and $F$ belongs to the linear span of a class $\mathcal{S}$ of weak learners (commonly shallow decision trees, or stumps). In the most general case, each base learner $f_j \in \mathcal{S}$ can be written, for example, as a piecewise constant function:

$f(x) = \sum_{j=1}^k \beta_j \mathbb{1}_{A_j}(x)$

for the cells (leaves) $A_j$ of the corresponding decision tree.

The boosting predictor is constructed as

$$F(x) = \sum_{j} \beta_j f_j(x), \quad f_j \in \mathcal{S}$$

The core procedure at each iteration $t$ consists of fitting a base learner to the negative gradient (or subgradient) of the current risk:

$$f_{t+1} \in \arg\max_{f \in \mathcal{S}} -\mathbb{E}[\xi(F_t(X), Y)f(X)]$$

where $\xi(\cdot, y)$ denotes the subgradient of the loss w.r.t.\ its first argument. Updates take the form $F_{t+1} = F_t + w_{t+1} f_{t+1}$ (with $w_{t+1}$ the step-size).

Two main algorithmic variants are rigorously analyzed:

• Algorithm 1: Restricts the descent direction to normalized learners and selects $f_{t+1}$ that best aligns with the negative gradient, using an adaptive step-size informed by the loss' Lipschitz constant.
• Algorithm 2: Adopts a cone of base learners, fits $f_{t+1}$ by solving a least squares problem using the empirical negative gradient, with a fixed step-size.

Both variants realize functional gradient descent in the space of models and share the principle of iteratively projecting the functional gradient into the closure of the weak learner class (Biau et al., 2017).

2. Convergence, Regularization, and Implicit Bias

With appropriate assumptions (convex risk, proper control of step-sizes, and local Lipschitz conditions on the loss), GBDT iterates $F_t$ satisfy

$$\lim_{t \to \infty} C(F_t) = \inf_{F \in \text{span}(\mathcal{S})} C(F)$$

If $\psi$ is further $\alpha$-strongly convex, then $F_t$ converges in $L^2$ to the unique minimizer $\hat{F}$. This follows from the key inequality:

$$C(F_t) - C(\hat{F}) \geq \frac{\alpha}{2} \|F_t - \hat{F}\|^2_{L^2(\mu_X)}$$

Strong convexity guarantees not just monotonic risk descent but also norm convergence, and underpins theoretical guarantees of stability and uniqueness.

When the loss is not inherently strongly convex, regularization is introduced via an $L^2$ penalty on the predictor norm:

$\psi(x, y) = \varphi(x, y) + \gamma x^2$

This modification ensures strong convexity and prevents the boosting coefficients from diverging (Biau et al., 2017). The regularization parameter $\gamma$ must be chosen to balance bias and variance, with statistical consistency achievable as $\gamma_n \to 0$ appropriately with sample size $n$.

3. Statistical Consistency and Overfitting Control

The empirical risk minimization problem, with trees of controlled complexity ($\mathcal{S}_n$ parameterized to have cell diameters shrinking with $n$), enables statistical consistency:

$$\lim_{n \to \infty} \mathbb{E}[A(\bar{F}_n)] = A(F^\star)$$

where $A(F)$ is population risk and $F^\star$ is the oracle minimizer. Provided that the number of base learners grows with $n$ at a rate such that combinatorial complexity $(\log N)/(nv_n) \to 0$ and the regularization $\gamma_n$ decays such that $$1/\big(\sqrt{n v_n \gamma_n} \zeta(\cdot)\big) \to 0$$, overfitting is avoided even in the infinite-iteration regime.

This analysis demonstrates that overfitting need not be averted by early stopping; instead, with appropriate $L^2$ penalty scaling and a carefully managed base learner class, optimization can proceed indefinitely (Biau et al., 2017). This supports and explains the empirical practice of running modern GBDT implementations (such as XGBoost) with large numbers of boosting rounds in the presence of regularization.

4. Regularization Strategies in GBDTs

Explicit regularization via an $L^2$ penalty on the ensemble norm is theoretically justified as the primary means of controlling estimator complexity and ensuring both optimization and generalization guarantees, especially when the loss lacks natural strong convexity. This approach is in contrast with relying solely on early stopping, and is formalized by augmenting the risk functional:

$$C_n(F) = \frac{1}{n}\sum_{i=1}^n \psi(F(x_i), y_i) + \gamma_n \|F\|^2_{L^2(\mu_X)}$$

Practical implications:

• Penalty $\gamma_n$ should decay at a controlled rate with $n$ to guarantee consistency.
• Penalty is "baked into" the statistical analysis, and not merely a tool to stabilize numerics in finite-sample settings.
• Regularization enables practitioners to use arbitrarily many boosting rounds, provided model complexity and penalty are correctly matched.

The above provides a theoretical foundation for regularization implementations in leading GBDT toolkits (Biau et al., 2017).

5. Relation to Tree Induction, Weak Learner Complexity, and Implementation

Each base learner in GBDT is typically a shallow, finite tree (e.g., with fixed-depth or leaf count). The implementation requires:

• At each iteration, fitting a weak learner to the current pseudo-residuals (negative gradient/subgradient vector).
• For regression/classification, the choice of loss function determines pseudo-residuals; for non-differentiable losses, subgradient approaches are employed.
• Adaptively updating step-sizes (or fixing them) per theoretical convergence conditions.
• Choosing the weak learner space to balance bias, variance, and computational load. As $n$ increases, the class of possible tree partitions must "densify" to ensure universal approximation.

Empirical strategies often involve grid-search or cross-validation to select tree depth, learning rates, and regularization penalties. In production systems, practitioners often exploit parallelized tree-growing steps, advanced histogram techniques, and optimization tricks.

6. Practical Implications and Theoretical Insights

A central insight from the convex-analytic perspective is that, given strong convexity (intrinsic or regularization-induced) and a controlled increase in weak learner complexity, GBDT can run for arbitrarily many iterations—obviating the traditional necessity of early stopping to prevent overfitting (Biau et al., 2017). This means that:

• Explicit regularization (not iteration bounds) governs generalization.
• The trade-off between empirical risk minimization and sample complexity is mediated by regularization and tree partition fineness.
• State-of-the-art toolkits such as XGBoost and LightGBM, which incorporate strong regularization (via both $L^2$ penalties and leaf-wise splitting constraints), align with the theoretical recommendations.

This theoretical framework comprehensively supports the deployment of GBDT models in high-dimensional, large-sample settings common in modern machine learning pipelines, and elucidates the convergence, statistical properties, and effective regularization for practitioners and researchers.

---

Table: Summary of Key Theoretical Elements in GBDT Optimization

Principle	Mathematical Formulation	Implication
Convex risk minimization	$C(F) = \mathbb{E}[\psi(F(X), Y)]$	Additive model as solution in function space
Update rule	$F_{t+1} = F_t + w_{t+1} f_{t+1}$	New tree aligns with negative risk gradient
Strong convexity	$\psi$ $\alpha$-strongly convex: $$C(F_t) - C(\hat{F}) \geq \frac{\alpha}{2} \|F_t - \hat{F}\|^2$$	Guarantees risk and norm convergence
Regularization	$\psi(x, y) = \varphi(x, y) + \gamma x^2$	Enables consistency, prevents divergence
Consistency conditions	$(\log N)/(n v_n) \rightarrow 0$, $1/(\sqrt{n v_n \gamma_n} \zeta) \rightarrow 0$	Avoids overfitting with growing $n$ and penalty
Infinite boosting ("no early stopping")	Run $t \to \infty$ under controlled complexity and $\gamma_n \downarrow 0$	Consistent and stable solutions

The above table restates the central mathematical elements and their practical and theoretical consequences as established in the referenced analysis (Biau et al., 2017).