Soft Bayesian Additive Regression Trees

• SBART is a hierarchical Bayesian ensemble method that replaces hard decision splits with smooth logistic gating functions, enabling flexible probabilistic modeling.
• It supports smooth function estimation and adaptive variable selection in high-dimensional regression, classification, and complex data analyses such as longitudinal and spatial clustering.
• SBART achieves near minimax-optimal rates for sparse, smooth additive functions through advanced MCMC sampling and scalable computational strategies.

Soft Bayesian Additive Regression Trees (SBART) is a hierarchical Bayesian ensemble method that generalizes the classical Bayesian Additive Regression Trees (BART) framework by introducing probabilistic, rather than deterministic, decision rules at each node of each tree. This construction supports both smooth function estimation and adaptive variable selection in high-dimensional regression and classification, as well as flexible extensions to longitudinal, survival, and spatially clustered data analysis. SBART has been theoretically validated to achieve minimax-optimal rates up to logarithmic factors for sparse, smooth, and additive functions—adapting automatically to unknown structural properties—using a MCMC sampling scheme that requires only modest adaptations to existing BART implementations (Linero et al., 2017).

1. Model Formulation and Soft Decision Trees

In SBART, the regression function $f:\mathbb{R}^p\to\mathbb{R}$ is represented as a finite ensemble of $T$ soft decision trees: $f(x) = \sum_{t=1}^T g(x; T_t, M_t)$ where each $g(x; T_t, M_t)$ is a probabilistic regression tree with structure $T_t$ and leaf parameters $M_t = \{\mu_{t,\ell}\}$. In contrast to "hard" trees, where routing at each branch node is a deterministic function of the predictor and a threshold, SBART replaces the hard indicator split with a smooth logistic (or probit) gating function. For branch $b$ indexed by splitting variable $j_b$, cutpoint $c_b$, and bandwidth (gating parameter) $\tau_b > 0$, the probability to go left is: $T$0 while the right path is taken with probability $T$1. The probability of reaching leaf $T$2 is then

$T$3

where $T$4 denotes the set of ancestor nodes for leaf $T$5 and $T$6 if the path to $T$7 from the root passes through the right branch at $T$8 (otherwise $T$9). As $f(x) = \sum_{t=1}^T g(x; T_t, M_t)$0, the gating function degenerates to the hard split $f(x) = \sum_{t=1}^T g(x; T_t, M_t)$1, recovering classical BART.

Each tree predicts a contribution

$f(x) = \sum_{t=1}^T g(x; T_t, M_t)$2

with $f(x) = \sum_{t=1}^T g(x; T_t, M_t)$3 the set of leaf nodes in $f(x) = \sum_{t=1}^T g(x; T_t, M_t)$4 (Linero et al., 2017, Linero, 2022, Basak et al., 2020).

2. Hierarchical Prior Specification

The SBART prior is fully hierarchical and regularizing, facilitating both smoothness adaptation and sparsity:

• Number of Trees ($f(x) = \sum_{t=1}^T g(x; T_t, M_t)$5): Fixed or with a light-tailed prior, e.g., $f(x) = \sum_{t=1}^T g(x; T_t, M_t)$6.
• Tree Structure: Each tree is generated by a Galton–Watson (branching process) prior: nodes at depth $f(x) = \sum_{t=1}^T g(x; T_t, M_t)$7 split with probability $f(x) = \sum_{t=1}^T g(x; T_t, M_t)$8, with typical choices $f(x) = \sum_{t=1}^T g(x; T_t, M_t)$9, $g(x; T_t, M_t)$0.
• Splitting Proportions: To encourage sparsity in high dimensions, splitting variables are chosen according to $g(x; T_t, M_t)$1, with $g(x; T_t, M_t)$2 and hyperprior on $g(x; T_t, M_t)$3.
• Gating Bandwidths: Each tree or node has a parameter $g(x; T_t, M_t)$4, with $g(x; T_t, M_t)$5 controlling the overall smoothness (smaller $g(x; T_t, M_t)$6 yields sharper splits).
• Leaf Parameters: $g(x; T_t, M_t)$7, with $g(x; T_t, M_t)$8.
• Observation Noise: $g(x; T_t, M_t)$9, where $T_t$0 is a data-driven estimate (Linero et al., 2017).

This hierarchy is readily modified for non-Gaussian likelihoods and extended for classification, as in Polya–Gamma-augmented models for binary responses (Ran et al., 2021).

3. Posterior Computation and Sampling Algorithm

SBART employs a block-wise Bayesian backfitting algorithm with Metropolis–Hastings (MH) proposals, iterating over trees:

1. Partial Residuals: For each tree $T_t$1, compute $T_t$2.
2. Update Tree Structure and Bandwidth: Propose changes to $T_t$3 (tree topology and split parameters) and bandwidth $T_t$4 via local moves (grow, prune, change), with the proposal accepted by the MH criterion incorporating the marginal likelihood of the soft tree. For bandwidth, a random walk on $T_t$5 or grid search is used.
3. Update Leaf Means: Gibbs update, using conditional Gaussian posterior.
4. Update Sparsity Weights: $T_t$6.
5. Update Hyperparameters: $T_t$7 by slice sampling or MH.

In high dimensions or with large samples, parallelization and distributed algorithms (using SPMD/MPI) allow scalable inference with near-linear speedup (Ran et al., 2021).

4. Theoretical Guarantees and Statistical Properties

SBART has rigorous posterior contraction properties:

• For regression functions in sparse Hölder classes $T_t$8 with $T_t$9, the posterior concentrates at a rate

$M_t = \{\mu_{t,\ell}\}$0

for any $M_t = \{\mu_{t,\ell}\}$1.

• SBART attains (up to logs) the minimax-optimal rate for $M_t = \{\mu_{t,\ell}\}$2-sparse, $M_t = \{\mu_{t,\ell}\}$3-smooth functions, with adaptation to unknown smoothness and sparsity.
• In additive models $M_t = \{\mu_{t,\ell}\}$4, with each $M_t = \{\mu_{t,\ell}\}$5 depending on $M_t = \{\mu_{t,\ell}\}$6 coordinates in $M_t = \{\mu_{t,\ell}\}$7, SBART contracts at the sum of minimax rates across components.
• The prior mass and approximation arguments are explicitly constructed: any Hölder $M_t = \{\mu_{t,\ell}\}$8 can be $M_t = \{\mu_{t,\ell}\}$9-norm-approximated by $b$0 soft trees, with approximation error $b$1 (Linero et al., 2017).

5. Practical Extensions and Computational Improvements

Numerous extensions of SBART leverage its foundational construction:

• Weighted SBART for Asynchronous Longitudinal Data: Each response–covariate pair is weighted via a kernel on observation lags, preserving consistency and minimax rates under independent sampling times. Empirical RMSE improves over BART and interpolation approaches (Ran et al., 2021).
• Distributed SBART: Sufficient statistics (matrix/vector products) are computed in parallel on worker nodes; master node aggregates via MPI Allreduce for global updates, yielding near-linear scaling to at least 20 cores in benchmarking (Ran et al., 2021).
• Accelerated SBART (ASBART): Grows trees from the root using the XBART strategy, then infers optimal tree softness by grid search and a single MH step. Achieves 5–100x speedup over SBART with negligible loss in RMSE and credible set coverage (Ran et al., 2023).
• Classification: Probit or logistic link with SBART sum-of-trees argument. Either direct MH on tree and leaf parameters or Polya–Gamma augmentation can be used for efficient posterior inference (Ran et al., 2021).
• SBART for Recurrent Events and Survival: The nonparametric regression tree ensemble is embedded within frailty and survival intensity models, further augmented by latent variables for interval-censored or clustered data (Chen et al., 10 Jun 2026, Basak et al., 2020, Ghosh et al., 2024).

6. Implementation, Diagnostics, and Usage

SBART is deployable with minimal changes to BART code bases:

• Key Modifications: Replace binary leaf-indicators by smooth $b$2-weights; introduce randomness and MH updates for $b$3; add Gibbs step for sparsity weights $b$4.
• Default Hyperparameters: $b$5–$b$6; $b$7; $b$8; $b$9; Cauchy or Inverse-$j_b$0 priors on variances.
• Software: Implemented as R/C++ packages (e.g., SoftBART, softbart) with support for extended models via C++ backends and R wrappers, as well as distributed (MPI) and GPU-enabled packages (Linero et al., 2017, Linero, 2022, Ran et al., 2021, Ran et al., 2023).
• Diagnostics: Trace plots of noise variance, effective sample size for leaf or prediction draws, average tree depth, and posterior fit RMSE are routine for convergence assessment (Ran et al., 2023).

7. Empirical Performance and Comparative Results

• On benchmark regression datasets (Friedman 5-d, local smooth and step functions, real tabular datasets), SBART outperforms classic BART, DART, random forests, boosting, and lasso in RMSE and uncertainty quantification.
• For variable selection in sparse high-dimensional contexts, SBART attains higher precision and recall than DART, especially as signal diminishes.
• In real-data benchmarks (ten datasets), SBART with cross-validated $j_b$1 achieves the best average relative RMSE, with default SBART tying for second, exceeding DART, hard BART, random forests, XGBoost, and Lasso (Linero et al., 2017).
• In survival, recurrent events, and longitudinal modeling, SBART attains superior predictive accuracy, improved residual calibration, and sharper credible coverage compared to BART, RecForest, and parametric models—even under model misspecification (Ran et al., 2021, Basak et al., 2020, Chen et al., 10 Jun 2026).

SBART provides a theoretically grounded, computationally practical, and empirically validated extension of machine learning tree ensembles for probabilistic modeling, smooth and high-dimensional estimation, and rich nonparametric inference across a range of domains.