ANOVA-BART: Structured Bayesian Additive Trees

• The paper introduces ANOVA-BART, a method that decomposes regression functions into main effects and interactions for clearer interpretation.
• It employs a hierarchical Bayesian framework with regularized tree ensembles to capture nonlinearities and quantify uncertainty.
• Empirical and theoretical results demonstrate minimax optimal convergence rates and enhanced predictive performance over standard BART.

ANOVA Bayesian Additive Regression Trees (ANOVA-BART) is a nonparametric Bayesian modeling approach that integrates the ensemble learning principles of Bayesian Additive Regression Trees (BART) with the functional Analysis of Variance (ANOVA) decomposition. ANOVA-BART is specifically designed to enhance interpretability while maintaining the flexibility, predictive performance, and uncertainty quantification of BART. It achieves this by decomposing a regression function into additive main effects and interaction components, each represented and inferred using separate ensembles of regression trees. This structure enables direct estimation of the contributions from factors and their interactions, offering deeper insights into the underlying data-generating process and allowing for principled component selection.

1. Functional ANOVA Decomposition in BART

ANOVA-BART extends the standard BART modeling paradigm by explicitly encoding a functional ANOVA decomposition within the sum-of-trees framework. In the classical setting, BART models the unknown regression function $f(x)$ as

$f(x) = \sum_{j=1}^{m} g_j(x; T_j, M_j),$

where each $g_j$ is a weak Bayesian regression tree indexed by its structure $T_j$ and set of leaf parameters $M_j$. BART excels at capturing nonlinearities and high-order interactions without explicit feature engineering but produces models where the marginal contribution of main effects and specific interactions is not explicitly separated.

ANOVA-BART imposes a functional ANOVA structure on $f(x)$: $$f(x) = \mu + \sum_{i} f_i(x_i) + \sum_{i<j} f_{ij}(x_i, x_j) + \cdots,$$ where $f_i$ denotes a main effect, $f_{ij}$ a second-order interaction, and so on. Each component is then modeled as its own sum-of-trees. For example, for two predictors, this leads to the concrete representation: $$f(x_1, x_2) = \mu + f_1(x_1) + f_2(x_2) + f_{12}(x_1, x_2),$$ with

$$f_1(x_1) = \sum_{j=1}^{m_1} g_j^{(1)}(x_1; T_j^{(1)}, M_j^{(1)}), \quad f_2(x_2) = \sum_{k=1}^{m_2} g_k^{(2)}(x_2; T_k^{(2)}, M_k^{(2)}), \quad f_{12}(x_1, x_2) = \sum_{l=1}^{m_{12}} g_l^{(12)} ((x_1, x_2); T_l^{(12)}, M_l^{(12)} )$$

with all tree ensembles regularized via priors to remain weak learners. Each functional component (main or interaction) is thus separately identified and estimated.

2. Statistical Model and Regularization

The ANOVA-BART model operates in a Bayesian framework, specifying a likelihood and hierarchical priors for all components. For observed data $(Y_i, x_i)$, the response is modeled as

$Y_i = f(x_i) + \epsilon_i,$

with $\epsilon_i \sim N(0, \sigma^2)$. The hierarchical priors enforce the weak learner property for each tree across all components:

• The depth-dependent prior for node splitting enforces shallow, regularized trees:

$$P(\text{node splits at depth } d) = \alpha (1 + d)^{-\beta}, \quad \alpha \approx 0.95,\ \beta \approx 2$$

• Leaf (terminal node) values for each tree in each effect component group are given Gaussian priors, typically centered at 0 and variance scaled to promote shrinkage:

$$\mu_{j,\ell}^{(g)} \sim N\left(0, \frac{0.5^2}{k^2 m_g} \right),$$

where $m_g$ is the number of trees for effect group $g$, and $k$ a tuning parameter.

• For higher-order interactions, stronger shrinkage can be imposed by adjusting $m_g$ or scaling variances, paralleling hierarchical regularization in classical ANOVA.

The total model becomes: $$Y_i = \mu + \sum_{g \in \mathcal{G}} f_g(x_{i,g}) + \epsilon_i$$ where $\mathcal{G}$ indexes all main and interaction groups being modeled.

3. Posterior Inference, MCMC, and Component Selection

Model fitting is performed via a blocked Gibbs or Metropolis-within-Gibbs MCMC, iteratively cycling through each tree in each component group:

1. For each group $g$ (main or interaction), calculate partial residuals after subtracting the current sum across all other groups.
2. For each tree in $g$, update the structure ($T_j$) with a Metropolis-Hastings proposal (grow, prune, change, swap).
3. Leaf node parameters are updated from conjugate normal posteriors given the new structure and partial residuals.
4. Update error variance $\sigma^2$ from its conjugate inverse-chi-squared posterior.
5. Posterior samples for each $f_g$ yield direct estimates (mean, credible intervals) for all components.

Component selection is performed by examining posterior inclusion probabilities or effect sizes for each group. Shrinkage settings for higher-order groups can be tuned to automatically suppress unneeded interactions, enhancing model interpretability.

4. Theoretical Guarantees and Posterior Convergence

ANOVA-BART preserves the minimax optimality properties of standard BART for function estimation and extends these to interaction components. Specifically, the posterior concentration rate of the model is nearly minimax optimal, with the important property that each additive or interaction effect $f_g$ enjoys the same convergence rate as the overall function, a property not guaranteed by standard BART. This means that, for smooth $d$-dimensional functions $f_g$ of H\"older smoothness $\alpha$, the posterior contracts at the rate $n^{-\alpha/(2\alpha + d)}$, up to logarithmic factors, for each $g$.

This theoretical property ensures not only accurate recovery of the overall regression surface, but also reliable estimation and uncertainty quantification for each main effect and interaction. Thus, ANOVA-BART delivers finer-grained, structure-aware inference not available in aggregate BART posteriors.

5. Empirical Performance, Interpretability, and Comparison to BART

Experimental evidence confirms that ANOVA-BART achieves superior predictive accuracy and improved uncertainty quantification compared to standard BART, particularly in settings where the regression surface exhibits interpretable main effects or interactions. This performance improvement is attributable to:

• Enhanced identifiability arising from explicit decomposition;
• Direct posterior inference and shrinkage for each component, which prevents "masking" of effects by complex higher-order terms;
• Improved variable and component selection, allowing irrelevant effects to be pruned without sacrificing predictive ability.

Further, ANOVA-BART achieves these advances while maintaining computational scalability (since the overall model is modularized) and theoretical support for each constituent. The practical advantage is a model that is at once flexible, statistically efficient, and interpretable, facilitating discoveries such as which combinations of predictors drive response variability.

6. Extensions, Applications, and Future Directions

ANOVA-BART provides a structurally transparent scaffold for further model enrichment:

• Multilevel/Hierarchical Effects: Random or group effects can be integrated as additional additive or interaction components, either modeled parametrically or with their own tree ensembles.
• Heteroscedasticity and Non-Gaussian Outcomes: By altering the likelihood or adding BART-modeled scale functions, ANOVA-BART can be employed for heteroscedastic or exponential family response types, leveraging theory from generalized BART and heteroscedastic BART.
• Functional Data and High Dimensions: Functional extensions leverage the ability to decompose and regularize high-dimensional interactions, with relevant applications in genomics, spatial-temporal modeling, and causality where explicit effect decomposition is essential.

As a research area, ongoing directions include: data-driven selection of ANOVA component structure; hybridization with co-data or external prior information for effect groups; robust error modeling; and integration with density regression or causal inference frameworks.

---

In summary, ANOVA-BART offers a methodologically principled, empirically validated, and theoretically guaranteed framework for combining the adaptivity of BART with the interpretability and structured effect separation of ANOVA. This synthesis makes it a compelling choice for modeling and understanding complex interactions in modern regression and classification problems (Park et al., 3 Sep 2025, 0806.3286).