Soft Gradient Boosting Approaches

• Soft Gradient Boosting is a class of methods that use differentiable base learners and joint optimization to enhance traditional residual fitting.
• It leverages architectures like sGBM, GrowNet, and soft decision trees to enable parallel training and adaptive corrections in model building.
• Empirical results demonstrate improved accuracy and efficiency across image, tabular, and online learning tasks, addressing limitations of conventional boosting.

Soft gradient boosting denotes a family of gradient boosting methods in which the weak learners, their combination, or the training dynamics are made differentiable, jointly optimizable, or stage-adaptive rather than strictly sequential and fixed. In the most explicit formulation, the soft Gradient Boosting Machine (sGBM) wires multiple differentiable base learners together in a DAG and injects both local and global objectives inspired from gradient boosting so that all base learners can be jointly optimized with linear speed-up (Feng et al., 2020). Closely related usages include shallow neural networks as “soft” weak learners with fully corrective updates in GrowNet (Badirli et al., 2020), sequential neural-network construction by fitting pseudo-residuals (Emami et al., 2019), gradient-boosted CNN and DNN training through iterative dense-layer addition and freezing (Emami et al., 2023), and soft decision-tree boosting with learnable feature transforms for sequential regression (Karaca et al., 16 Sep 2025). A broader interpretation also appears in work that makes split logic a function of the global training stage and accumulated statistical context, thereby yielding a smoother and more plastic learning process (Kriuk, 17 Nov 2025).

1. Definition and conceptual scope

In standard gradient boosting, base learners are trained one at a time, each on the residuals produced by the previous ensemble. This sequential dependence prohibits parallel training and precludes the adaptation of existing learners in online or incremental settings (Feng et al., 2020). Soft gradient boosting modifies that regime by replacing non-differentiable or rigid components with differentiable base learners, joint optimization, adaptive residual objectives, or smoothly varying split logic.

One explicit definition comes from sGBM, which uses differentiable base learners such as soft decision trees, MLPs, or CNNs, wires all base learners together, and injects both local and global loss functions recreating the boosting “fit-the-residual” effect in a parallel, differentiable architecture (Feng et al., 2020). Another explicit formulation appears in GrowNet, where shallow neural networks are employed as “weak learners,” the model is additive, and a fully corrective step fine-tunes parameters of all previous weak learners and step-sizes using the true global loss (Badirli et al., 2020).

The adjective “soft” is also used more broadly. In the GB-CNN and GB-DNN framework, the “soft” aspect refers to combining the strengths of joint (end-to-end) optimization, through fine-tuning shared layers, and strict additive boosting, where each stage corrects residuals (Emami et al., 2023). In MorphBoost, “soft” refers to split logic being a function of not only the local data, but also the global training stage and accumulated statistical context, with smooth interpolation between heuristics for exploration and heuristics for refinement using continuous schedules like $\tanh$, exponential decays, and moving averages (Kriuk, 17 Nov 2025). This suggests that soft gradient boosting is not a single algorithmic template but a class of boosting procedures that soften hard stage boundaries, hard routing, or hard optimization decoupling.

2. Functional form and residual-based training

The shared mathematical core is additive modeling. In sGBM, for input $\mathbf{x}^i$, the final output is

$$O^i = \sum_{m=1}^M o_m(\mathbf{x}^i; \boldsymbol{\theta}_m),$$

where $M$ is the number of base learners and $\boldsymbol{\theta}_m$ are the parameters of base learner $m$ (Feng et al., 2020). The pseudo-residual for learner $m$ on sample $i$ is

$$r_m^i = -\frac{\partial l\Big(\sum_{j=0}^{m-1} o_j^i,\, y^i\Big)}{\partial \sum_{j=0}^{m-1} o_j^i},$$

and the local loss is

$l_m = \sum_{i=1}^{N_b} \| r_m^i - o_m^i \|_2^2,$

with global loss

$\mathbf{x}^i$0

This preserves the gradient-boosting residual logic while allowing all learners to be updated jointly (Feng et al., 2020).

Residual fitting also governs the neural-network variants. In the sequential training method for neural networks, the initial model is

$\mathbf{x}^i$1

and at step $\mathbf{x}^i$2 the pseudo-residuals are

$\mathbf{x}^i$3

A small neural network $\mathbf{x}^i$4 is trained on the pseudo-residuals, a step size is chosen by line search,

$\mathbf{x}^i$5

and the model is updated additively as

$\mathbf{x}^i$6

(Emami et al., 2019).

GB-CNN and GB-DNN follow the same residual-fitting principle. For a sample $\mathbf{x}^i$7, the residual at iteration $\mathbf{x}^i$8 is

$\mathbf{x}^i$9

and the additive update is

$$O^i = \sum_{m=1}^M o_m(\mathbf{x}^i; \boldsymbol{\theta}_m),$$0

where $$O^i = \sum_{m=1}^M o_m(\mathbf{x}^i; \boldsymbol{\theta}_m),$$1 is a shrinkage rate and $$O^i = \sum_{m=1}^M o_m(\mathbf{x}^i; \boldsymbol{\theta}_m),$$2 is a class-specific multiplier (Emami et al., 2023). For classification, the output logits are summed and a softmax is applied: $$O^i = \sum_{m=1}^M o_m(\mathbf{x}^i; \boldsymbol{\theta}_m),$$3 Across these formulations, the central mechanism remains the fit-the-residual principle, but the residual-fitting module is no longer restricted to a hard tree trained in isolation.

3. Jointly optimized differentiable ensembles

The clearest departure from classical GBM is the jointly optimized differentiable ensemble. In sGBM, all base learners are instantiated at once, each base learner receives training signal corresponding to the residuals as if it were at its place in a hypothetical sequential boosting process, and all are optimized jointly via SGD or backpropagation (Feng et al., 2020). The paper emphasizes that sGBM trains all $$O^i = \sum_{m=1}^M o_m(\mathbf{x}^i; \boldsymbol{\theta}_m),$$4 learners in a single pass per data batch, enabling linear speed-up, whereas classical GBM requires $$O^i = \sum_{m=1}^M o_m(\mathbf{x}^i; \boldsymbol{\theta}_m),$$5 training passes. It further reports that sGBM enjoys much higher time efficiency with better accuracy, given the same base learner in both on-line and off-line settings (Feng et al., 2020).

When soft decision trees are used as base learners, the resulting sGBDT is a differentiable alternative to classical GBDT. Soft decision trees replace hard thresholding at internal nodes with differentiable gating, each sample receives a probabilistic routing through the tree, and the final prediction is a weighted sum over all leaves (Feng et al., 2020). Because the ensemble is differentiable end-to-end, all learners can adapt if the data distribution shifts, which the paper highlights as an advantage for online or incremental learning.

A related but distinct jointly corrected formulation is GrowNet. GrowNet uses shallow neural networks as weak learners, with prediction

$$O^i = \sum_{m=1}^M o_m(\mathbf{x}^i; \boldsymbol{\theta}_m),$$6

and uses a 2nd order Taylor expansion to reduce the stagewise problem to a weighted least squares problem

$$O^i = \sum_{m=1}^M o_m(\mathbf{x}^i; \boldsymbol{\theta}_m),$$7

where $$O^i = \sum_{m=1}^M o_m(\mathbf{x}^i; \boldsymbol{\theta}_m),$$8 (Badirli et al., 2020). Its distinguishing feature is the fully corrective step: after adding $$O^i = \sum_{m=1}^M o_m(\mathbf{x}^i; \boldsymbol{\theta}_m),$$9, parameters of all weak learners $M$0 and the associated $M$1 are retrained jointly with backpropagation on the original task loss. The paper states that this addresses the local minima issue of greedy boosting, enforces direct progress toward the global objective, allows dynamic adaptation of the learning rate $M$2, and reduces redundancy or correlation among weak learners (Badirli et al., 2020). This is a particularly strong form of “softness,” because the additive construction is retained while the no-backward-corrections rule of classical boosting is abandoned.

4. Neural-network realizations

Several works realize soft gradient boosting through neural-network growth. The sequential training method of Gómez, et al. trains a neural network in $M$3 steps: first the bias term is initialized with a constant approximation that minimizes the average loss of the data; then, at each step, a portion of the network, composed of $M$4 neurons, is trained to approximate the pseudo-residuals on the training data computed from the previous iterations; finally, the $M$5 partial models and bias are integrated as a single NN with $M$6 neurons in the hidden layer (Emami et al., 2019). The paper reports competitive generalization performance with respect to neural networks trained with Adam, L-BFGS, SGD and deep models, and further states that the proposed method permits switching off the units that were last trained during test without a significant reduction of its generalization ability (Emami et al., 2019).

GB-CNN and GB-DNN extend the same basic logic to convolutional and deep architectures. At each iteration, the proposed method adds one dense layer to an exact copy of the previous deep NN model; the model is trained to fit the gradient of the loss function or pseudo-residuals of previous models; and the weights of the dense layers trained on previous iterations are frozen to prevent over-fitting (Emami et al., 2023). In GB-CNN, the convolutional layers are not frozen during boosting and are fine-tuned at each iteration, enabling the feature extraction part of the model to adapt to the updated dense classifier layers (Emami et al., 2023). The paper states that training continues for a fixed number of iterations, empirically 2-3 suffices, or until validation loss plateaus, and that additional boosting steps provided diminishing returns (Emami et al., 2023).

The empirical results reported for this neural formulation are concrete. GB-CNN outperforms standard CNN of the same architecture across all tested image datasets. On CIFAR-10, GB-CNN achieved 87.65% vs. 86.71% with CNN; on Rock-Paper-Scissors, 87.37% vs 68.01%; and on MNIST, 99.61% vs 99.55% (Emami et al., 2023). GB-DNN also exceeds standard DNN performance on most tabular datasets, with notable differences on Waveform and Vowel. The paper interprets these results as indicating that freezing previous dense layers reduces overfitting and increases robustness, and that the method improves generalization and final accuracy especially when conventional training plateaus (Emami et al., 2023).

A further neural boosting development appears in VPBoost, which targets separable smooth approximators of the form

$M$7

VPBoost combines second-order boosting with variable projection, yielding the closed-form linear-weight solution

$M$8

and interprets the resulting algorithm as a functional trust-region method (Chowdhary et al., 24 Mar 2026). The paper states that VPBoost converges to a stationary point under mild geometric conditions and, under stronger assumptions, achieves a superlinear convergence rate, while numerical experiments show improved evaluation metrics in comparison to gradient-descent-based boosting and competitive performance relative to an industry-standard decision tree boosting algorithm (Chowdhary et al., 24 Mar 2026). A plausible implication is that one important strand of soft gradient boosting is the replacement of hand-tuned shrinkage with analytically or trust-region controlled stage updates.

5. Soft trees, feature transforms, and adaptive split logic

Softness can also enter through the weak learner itself. In the sequential regression framework with learnable feature transforms, each boosting iteration trains a soft decision tree and learns a linear input feature transform $M$9 together (Karaca et al., 16 Sep 2025). The soft tree uses sigmoid-based internal routing,

$\boldsymbol{\theta}_m$0

leaf probabilities are products of gating probabilities along the path, and the tree output is

$\boldsymbol{\theta}_m$1

Because $\boldsymbol{\theta}_m$2 is differentiable with respect to $\boldsymbol{\theta}_m$3 and therefore with respect to $\boldsymbol{\theta}_m$4, the tree parameters and the feature transform can be jointly optimized in an end-to-end differentiable manner (Karaca et al., 16 Sep 2025). The paper states that this is particularly advantageous in high-dimensional, data-scarce scenarios and reports test MSE values such as 0.0048 for BSDT-Q, 0.0441 for BSDT, and 0.0338 for LightGBM on the Exchange example, as well as 0.0040, 0.0084, and 0.0098 respectively on ETTh2 (Karaca et al., 16 Sep 2025).

A different notion of softening appears in tree-structured boosting. TSB recursively builds a binary tree of weak learners and updates branch weights by a parameter $\boldsymbol{\theta}_m$5: $\boldsymbol{\theta}_m$6 When $\boldsymbol{\theta}_m$7, TSB is equivalent to CART; when $\boldsymbol{\theta}_m$8, it is equivalent to GBS; and intermediate $\boldsymbol{\theta}_m$9 produces a hybrid in between (Luna et al., 2017). The paper further states that for most datasets, the lowest error is achieved by TSB at an intermediate $m$0, outperforming both CART and GBS. This provides a formal spectrum between hard partitioning and fully additive boosting.

MorphBoost introduces yet another form of softness: a morphing split criterion. In early iterations,

$m$1

whereas later iterations use running gradient statistics, normalized gradients, an information-theoretic score, and the final morphing split score

$m$2

The abstract reports that MorphBoost achieves state-of-the-art performance, outperforming XGBoost by 0.84% on average, with lowest variance $m$3 and highest minimum accuracy across all models (Kriuk, 17 Nov 2025). In its own framing, the split logic smoothly interpolates between aggressive, gradient-driven splits in early stages and information-theoretic, regularized splits in late stages. This suggests that “soft” can refer not only to differentiability, but also to continuous adaptation of the learner-construction rule itself.

6. Empirical behavior, applications, and limitations

Across the cited literature, soft gradient boosting is applied to image classification, tabular classification, regression, learning to rank, online or incremental learning, sequential regression, and scientific machine learning. GrowNet is presented as a unified framework for classification, regression, and learning to rank, and reports outperforming results against state-of-the-art boosting methods in all three tasks on multiple datasets (Badirli et al., 2020). The GB-CNN and GB-DNN study evaluates image datasets including MNIST, CIFAR-10, Fashion-MNIST, Rice varieties, Rock-Paper-Scissors, MNIST-Corrupted, and Kuzushiji-MNIST, and tabular datasets including Digits, Ionosphere, Letter-26, Sonar, USPS, Vowel, and Waveform (Emami et al., 2023). VPBoost reports experiments on synthetic data, image recognition, scientific machine learning benchmarks, and HIGGS (Chowdhary et al., 24 Mar 2026). Distributional Gradient Boosting Machines model the entire conditional distribution of a univariate response variable and use XGBoost and LightGBM as computational backbones, enabling probabilistic forecasts, prediction intervals, and quantiles of interest (März et al., 2022). Although DGBM is not framed as “soft gradient boosting,” it illustrates how the boosting paradigm continues to be extended by relaxing a point-estimation view into a distributional one.

The recurring empirical themes are accuracy gains, improved adaptability, and more flexible optimization. sGBM is reported to enjoy much higher time efficiency with better accuracy in both on-line and off-line settings (Feng et al., 2020). GB-CNN and GB-DNN show superior performance in terms of classification accuracy with respect to standard CNN and Deep-NN with the same architectures (Emami et al., 2023). The sequential neural method is competitive with standard solvers and allows dynamic adaptation of the model to different classification speed requirements on the fly by switching off late-added hidden units during test (Emami et al., 2019). BSDT-Q is reported to increase performance effectively and efficiently while avoiding overfitting in high-dimensional, data-scarce settings (Karaca et al., 16 Sep 2025).

At the same time, the papers identify limitations or trade-offs. Classical sequential boosting remains less amenable to parallelization, which motivates sGBM’s joint optimization (Feng et al., 2020). Linear feature transforms may be insufficient for tasks with complex feature interdependencies, although non-linear differentiable transforms are possible if overfitting is not a problem (Karaca et al., 16 Sep 2025). VPBoost notes that for very large outputs, standard gradient-based methods become preferable unless further structure, sparsity, or iterative solvers are used (Chowdhary et al., 24 Mar 2026). MorphBoost emphasizes robustness and consistency under high-dimensionality, noisy labels, interacting features, and class imbalance, implying that adaptive split logic is partly a response to the brittleness of static criteria (Kriuk, 17 Nov 2025).

Taken together, the literature supports a technical characterization of soft gradient boosting as a move away from strictly sequential, non-adaptive, and non-differentiable boosting toward additive systems with differentiable base learners, joint or corrective optimization, probabilistic routing, feature-transform learning, or training-stage-dependent split criteria. This suggests that the defining property is not a single architecture, but a relaxation of hard boosting constraints while preserving residual-based additive learning.