SBAMDT: Bayesian Additive Decision Trees with Adaptive Soft Semi-multivariate Split Rules (2501.09900v1)

Abstract: Bayesian Additive Regression Trees [BART, Chipman et al., 2010] have gained significant popularity due to their remarkable predictive performance and ability to quantify uncertainty. However, standard decision tree models rely on recursive data splits at each decision node, using deterministic decision rules based on a single univariate feature. This approach limits their ability to effectively capture complex decision boundaries, particularly in scenarios involving multiple features, such as spatial domains, or when transitions are either sharp or smoothly varying. In this paper, we introduce a novel probabilistic additive decision tree model that employs a soft split rule. This method enables highly flexible splits that leverage both univariate and multivariate features, while also respecting the geometric properties of the feature domain. Notably, the probabilistic split rule adapts dynamically across decision nodes, allowing the model to account for varying levels of smoothness in the regression function. We demonstrate the utility of the proposed model through comparisons with existing tree-based models on synthetic datasets and a New York City education dataset.

• The paper presents SBAMDT, a novel model that integrates adaptive soft split rules into Bayesian additive decision trees for enhanced multivariate interaction capture.
• It employs a latent variable framework with a logistic gating function, enabling dynamic selection between hard and soft decision rules via Bayesian updates.
• SBAMDT demonstrates superior predictive accuracy and flexibility in modeling complex spatial and geometric feature relationships compared to traditional BART methods.

Overview of SBAMDT: Bayesian Additive Decision Trees with Adaptive Soft Semi-multivariate Split Rules

The presented paper introduces a novel Bayesian additive decision tree model named SBAMDT, designed to address limitations in traditional Bayesian Additive Regression Trees (BART). BART models are acclaimed for their predictive performance and uncertainty quantification but are constrained by deterministic decision rules and axis-aligned recursive data splits, often falling short in capturing complex decision boundaries involving multivariate features. SBAMDT approaches this constraint by incorporating a probabilistic, adaptive soft split rule enabling flexibility in capturing both univariate and multivariate feature relationships with respect to geometric feature domain properties.

Model Description

SBAMDT consists of additive decision trees utilizing adaptive, soft split rules at each decision node. While traditional decision trees rely on hard binary splits using a single feature, this model allows decision rules to probabilistically assign observations across potential decision paths. It differentiates between sharp changes requiring hard decisions and domains where smooth transitions can benefit from soft decisions. This dual approach, combining hard and soft splits, allows for node-specific adaptability in representing a regression function with variable levels of smoothness across different spatial domains and feature interactions.

Methodological Innovations

SBAMDT enhances flexibility by:

1. Integrating Multivariate Splits: Employing a graph-based partitioning methodology for multivariate structured features, allowing splits to conform to data manifold geometries like road networks or cortical surfaces.
2. Adaptive Decision Rules: Introducing a latent variable determining hard or soft decision types at each node, governed by Bayesian updates, which adapt to the data characteristics.
3. Softness Control Parameter: Leveraging a logistic gating function to determine soft decision rules' probabilities based on distances to multivariate reference points, regulated by the softness control parameter to reflect various levels of smoothness.
4. Inference via Modified BART Framework: Utilizing a Metropolis-Hastings within Gibbs sampler allowing for efficient posterior inference while incorporating both decision-type adaptability and softness control flexibility.

Numerical Experiments and Results

SBAMDT's efficacy is demonstrated through simulation studies and application to a real-world NYC education dataset. In the simulation scenarios, involving both U-shaped and rectangular domains, SBAMDT significantly outperformed benchmark models, including BAMDT, BART, and SBART, particularly in scenarios where multivariate feature interactions governed the response surface. Empirically, SBAMDT displayed superior predictive accuracy and effectively managed complex domain geometries due to its adaptive decision-making framework.

Implications and Future Directions

Practically, SBAMDT's approach to probabilistic decision tree splits presents substantial improvements for spatial statistics and feature domain modeling, allowing for accurate predictions across complex spatial landscapes. The theoretical insights into SBAMDT's connections with Gaussian Processes indicate a more profound adaptability in capturing functional dependencies and smoothness across high-dimensional feature spaces.

Further research could explore:

• Computational Optimization: Implementing efficiency improvements for large-scale data applications.
• Variable Selection Mechanisms: Investigating feature importance posteriors or integrating Dirichlet priors for enhanced feature hierarchy learning.
• Extended Applications: Extending SBAMDT's utility into classifications or causal inference domains while evaluating the robustness of its probabilistic decision framework.
• Theoretical Analysis: Establishing rigorous theoretical guarantees, specifically in exploring posterior convergence rates and function approximation justification under this novel split paradigm.

The SBAMDT model showcases an advanced and effective methodological contribution to decision tree frameworks, especially relevant to complex, multivariate feature-interacting tasks, substantially raising the bar for contemporary regression models in capturing intricate real-world data relationships.