Gradient boosting with vector-valued leafs

Abstract: Gradient boosting in the form of decision tree ensembles has successfully been applied to a variety of problems using simple objective functions based on log-likelihoods of a single variable. The concept extends naturally to objective functions operating on vectors - for example, multinomial logistic log-likelihood for multi-class classification, where observations have a score for each class - but popular frameworks approach these functions by either updating one value of the input vectors at a time, or by using a diagonal upper bound on the second derivative. This work extends the usual gradient boosting framework to functions of vector inputs and sketches a simple algorithm that can be used efficiently with histogram-based decision trees.

• The paper extends gradient boosting to vector objectives by leveraging true Hessians for improved multi-output prediction accuracy.
• It introduces an efficient histogram-based algorithm to compute vector-valued leaf values, reducing the number of boosting iterations drastically.
• Empirical comparisons demonstrate that the full Hessian approach achieves faster convergence and better fitting than diagonal approximations in multinomial logistic regression.

Gradient Boosting with Vector-Valued Leafs: Extending Second-Order Methods to Multi-Output Objectives

Introduction

Gradient boosting tree ensembles (GBT) are foundational in statistical modeling, with widespread implementations such as XGBoost, LightGBM, and CatBoost. The classical framework focuses on scalar objectives, typically log-likelihoods for binary or regression tasks. However, numerous practical problems require modeling objectives defined on vectors, notably multi-class classification with the multinomial logistic log-likelihood. Existing frameworks either update scores independently per vector component or utilize diagonal upper bounds of the Hessian for efficiency, sacrificing approximation quality. This paper formalizes the extension of gradient boosting to vector-valued objectives and presents an efficient algorithm exploiting full Hessians in histogram-based tree construction.

Gradient Boosting Fundamentals and Second-Order Approximation

Gradient boosting proceeds by sequentially minimizing additive convex functions using second-order Taylor expansions. Every boosting iteration generates a tree whose leaf values are computed via derivatives of the objective:

• For the scalar case, leaf values are obtained through weighted least-squares regression, exploiting first (gradient) and second (Hessian) derivatives.
• Optimal splits in trees maximize the reduction in squared error (RSS), rendered tractable via aggregation of $y$ values and their variances.
• The histogram method accelerates split computation by pre-grouping features, thus reducing computational overhead.

For general objectives without closed-form solutions, boosting fits quadratic approximations with sample-specific gradients $g_i$ and Hessians $h_i$. Solving for leaf values and split gain reduces to weighted least squares, indexed by these derivatives.

Matrix Algebra Formulation

The problem generalizes to matrix algebra: optimal leaf values are found by minimizing a quadratic form involving the Hessian matrix $\mathbf{H}$ and gradient vector $\mathbf{g}$, subject to constraints imposed by tree leaves. In scalar cases, this reduces to a simple regression intercept with observation-specific weights. The corresponding objective connects to familiar RSS computations and directly determines split criteria.

Extension to Vector-Valued Functions

When objectives operate on vector inputs, such as the multinomial log-likelihood for $k$ classes, the gradients become $k$-dimensional vectors and Hessians become $k \times k$ matrices per sample. The quadratic approximation across a leaf aggregates these, and optimal values are determined by solving:

$\mathbf{x}^{*} = \mathbf{A}^{-1} \mathbf{b}$

where $\mathbf{A} = \sum \mathbf{H}_i$, $g_i$0. The split gain is computed in terms of reduction in the quadratic criterion evaluated separately for each branch.

Efficient computation is achieved by cumulative updates of aggregates, enabling histogram-based procedures to minimize the split gain over sorted indices with incremental updates.

Multinomial Logistic Case and Hessian Singularity

Multinomial logistic loss produces a singular Hessian due to the invariance of probabilities under additive constants. Thus, its Hessian has rank $g_i$1 for $g_i$2 classes, and only $g_i$3 scores are estimable. Standard practice is to fix one class as the reference (its score set to zero), reducing the dimensionality and circumventing the singularity.

With regularization, the Hessian is modified to become positive-definite, restoring estimability of all $g_i$4 scores. Aggregation for split computations is optimized by separating diagonal and rank-one updates, reducing computational overhead.

Algorithm for Vector-Valued Splitting

The algorithm iteratively evaluates candidate split points over sorted feature values, updating leaf Hessian and gradient aggregates to compute $g_i$5, $g_i$6, and their inverses for each branch. The split maximizing gain is selected, and corresponding vector-valued leaf predictions are assigned. Histogram binning reduces the number of candidate splits considered, and practical computational limits are rarely restrictive, as $g_i$7 is typically small.

Empirical Comparison with Diagonal Approximations

XGBoost's vector boosting implementation uses a diagonal upper bound for the Hessian, simplifying leaf computations to weighted means of gradient components. The paper empirically demonstrates that using the true Hessian drastically reduces the number of boosting iterations required for convergence in multinomial logistic regression intercept estimation: four iterations with true Hessian versus over four hundred with the diagonal approach, indicating substantially more accurate and efficient optimization.

Comparisons of split thresholds show that both approaches select the same split, but differ in leaf values and resultant objective values, again favoring the true Hessian. The superior convergence and fitting accuracy of the full Hessian approach is thus numerically substantiated.

Implications and Future Directions

The extension of gradient boosting to vector-valued objectives with true second-order derivatives implies improved optimization accuracy and efficiency for multi-output tasks. The quadratic scaling with vector dimension is mitigated by typically low $g_i$8 in practical applications and the use of efficient histogram aggregation.

Theoretically, adopting true Hessians strengthens the optimization fidelity and better aligns the boosting procedure with the underlying objective structure, particularly for objectives exhibiting nontrivial covariance among components (e.g., multinomial logistic). Practically, fewer boosting iterations and more accurate split gains enhance predictive performance and resource efficiency.

Future work involves implementing the full procedure in a production-grade boosting library and benchmarking on real datasets with multiple outputs. Extensions to high-dimensional vector-valued tasks, tasks with dependent outputs, and further regularization schemes are warranted. The potential impact on deep ensemble methods, structured outputs, and new domains (e.g., multi-target regression, sequence modeling) is substantial.

Conclusion

Gradient boosting with vector-valued leaf predictions, leveraging the true Hessian in split and leaf estimation, offers a principled and efficient approach for optimizing multi-output objectives. The algorithmic extension maintains tractable computational complexity and outperforms diagonal approximation strategies used in popular libraries, especially in convergence behavior and approximation quality. Implementations and further empirical validation are necessary steps to fully realize its theoretical and practical benefits.