Bayesian Additive Regression Trees

• BART is a fully Bayesian, nonparametric ensemble model that approximates regression functions as a sum of weak trees with rigorous uncertainty quantification via MCMC.
• It employs a regularization prior and iterative Bayesian backfitting to control each tree’s contribution, ensuring robust capture of nonlinear interactions.
• Empirical studies demonstrate BART’s competitive performance in regression, simulation, and classification tasks, along with effective model-free variable selection.

Bayesian Additive Regression Trees (BART) is a fully Bayesian, nonparametric ensemble model designed for regression and classification tasks. The method represents the unknown regression function as a sum of many weak regression trees, each constrained by a regularization prior to ensure it makes only a modest contribution to the overall fit. Model inference and uncertainty quantification are performed via an iterative, Bayesian backfitting Markov chain Monte Carlo (MCMC) algorithm, making BART both flexible in capturing nonlinearities and interactions and rigorous in terms of statistical inference (0806.3286).

1. Model Structure and Prior Specification

BART models the target variable $y$ as

$$y = f(x) + \epsilon, \qquad \epsilon \sim N(0, \sigma^2).$$

The unknown regression function $f(x)$ is approximated as a sum over $m$ weak regression trees: $f(x) \approx h(x) = \sum_{j=1}^m g(x; T_j, M_j),$ where each component $g(x; T_j, M_j)$ is a regression tree defined by its structure $T_j$ (splitting rules/decision nodes) and terminal node parameters $M_j = \{\mu_{ij}\}$.

Priors in BART are carefully designed:

• Tree Structure Prior: For a node at depth $d$, the probability it splits is

$p(\text{node splits}) = \alpha (1 + d)^{-\beta},$

with default values $\alpha = 0.95$, $\beta = 2$, favoring small/shallow trees.

• Terminal Node Parameters: For a tree $T_j$ with terminal nodes,

$\mu_{ij} \sim N(\mu_{\mu}, \sigma_\mu^2),$

where $\sigma_\mu = 0.5 / (k \sqrt{m})$ after rescaling $y$ (with $k$ controlling shrinkage).

• Error Variance: $\sigma^2 \sim \nu\lambda/\chi^2_\nu$, with hyperparameters $\nu, \lambda$ calibrated so that a chosen quantile of the prior matches a high estimate from the data.

The combination forces each tree to act as a weak learner; only the sum can fit complex structure.

2. MCMC Inference: Bayesian Backfitting Algorithm

Inference in BART is accomplished via a tailored backfitting MCMC procedure. Each iteration updates the $m$ trees and the error variance:

1. For each tree $j$, compute the "partial residual":

$R_j = y - \sum_{k \ne j} g(x; T_k, M_k)$

1. Treat $R_j$ as the outcome in a single-tree regression, updating both structure $T_j$ and parameters $M_j$ via Gibbs and Metropolis–Hastings steps.
2. Update $\sigma$ from its full conditional.
3. Repeat for all $j$ and cycle until convergence.

Terminal node parameters are normal with conjugate priors, making marginalization with respect to $M_j$ tractable during tree structure proposals.

This approach allows for draws from the joint posterior over $(T_1, M_1),\dotsc,(T_m, M_m), \sigma$ and consequently for the regression function $f(x)$.

3. Posterior Summaries and Uncertainty Quantification

Posterior samples $\{f_k^*(x)\}_{k=1}^K$ (with $f_k^*(x) = \sum_{j=1}^m g(x; T_j^*, M_j^*)$ per posterior draw) support full posterior inference:

• Point Estimates: Posterior mean or median of $\{f_k^*(x)\}$.
• Credible Intervals: Quantiles of $\{f_k^*(x)\}$.
• Partial Dependence Functions: For a predictor subset $x_s$,

$$f(x_s) \approx \frac{1}{n} \sum_{i=1}^n f^*(x_s, x_{ci}).$$

This enables assessment of marginal effects and functionals with full uncertainty quantification.

4. Variable Selection Mechanism

BART enables model-free variable selection by aggregating variable usage statistics across trees and posterior samples. Define $z_{ik}$ as the proportion of splitting rules in posterior draw $k$ that use variable $i$, and compute

$v_i = \frac{1}{K} \sum_{k=1}^K z_{ik}.$

Reducing $m$ increases competition and sharpens the distinction between relevant and irrelevant predictors, permitting empirical screening or ranking of variables based on their $v_i$.

5. Comparative Performance and Empirical Results

BART's performance was benchmarked in several contexts:

• "Bake-off" Regression: On 42 real datasets, both BART with default hyperparameters and BART with parameters chosen via cross-validation (BART-cv) yielded favorable relative RMSE compared to gradient boosting, random forests, neural nets, and lasso. BART-cv often achieved the lowest error; BART-default was competitive and easier to use.
• Friedman Simulation: Using the nontrivial synthetic function $$f(x) = 10\sin(\pi x_1 x_2) + 20(x_3-0.5)^2 + 10 x_4 + 5 x_5$$, BART recovered the true function and relevant variables with accurate credible intervals and correctly sparse variable usage profiles, even when ambient $p \gg 5$.
• Drug Discovery (Classification): With a probit extension ($P(Y=1|x) = \Phi[G(x)]$, $G(x)$ sum-of-trees), BART matched or exceeded the area under the ROC curve of random forests, boosting, neural networks, and SVMs. BART prioritized active compounds effectively when ranking predictions, outperforming the low base activity rate.

Task	BART Result	Comparison
Regression (42 sets)	Competitive-to-best RMSE; easy-to-use default	Boosting, RF, lasso, NN
Simulation	Recovered true f; identified true variables	—
Classification	High AUC; better hit rates in ranking	RF, boosting, NN, SVM

6. Theoretical and Practical Considerations

BART’s design—additive, weakly regularized trees; conjugate priors; MCMC inference—enables both flexible modeling and principled uncertainty statements. The sum-of-trees regularizes overfitting and adapts to complex interaction structures. The generative posterior facilitates:

• Estimation of both point values and functionals of $f(x)$;
• Pointwise and global uncertainty intervals;
• Empirical variable screening without fully parametric selection models.

Default prior settings and modular MCMC updates contribute to ease of use and robustness across a broad range of applications. The empirical evidence illustrates the model's practical utility in scenarios ranging from regression to high-dimensional classification.

7. Summary and Impact

BART provides a coherent, fully Bayesian sum-of-trees regression and classification model with nonparametric adaptability, robust uncertainty quantification, and model-free variable selection. Its MCMC-based inference updates trees conditionally, allowing for closed-form integration of parameters and the estimation of the full posterior for the regression function. Empirical studies demonstrate that BART is highly competitive with modern ensemble and penalized regression methods in both accuracy and uncertainty calibration, while its variable inclusion frequency mechanism offers a principled, model-agnostic approach to identifying important predictors. These properties have led to BART's adoption for a diverse array of applied statistical inference and prediction problems (0806.3286).