Bayesian Dynamic Trees

• Bayesian Dynamic Trees are a fully Bayesian, nonparametric framework that partitions the input space into axis-aligned regions with simple parametric models in each leaf.
• They use sequential Monte Carlo for online inference, incorporating active data retirement and a forgetting mechanism to manage memory in streaming settings.
• The approach adaptively handles non-stationarity and achieves competitive performance in regression and classification tasks with efficient, bounded computational complexity.

Bayesian Dynamic Trees (DTs) constitute a fully Bayesian, non-parametric modeling framework suitable for streaming and massive data settings. DTs maintain piecewise simple parametric models on axis-aligned partitioned subspaces while employing sequential Monte Carlo (SMC) for online inference. Key techniques include active data retirement with conjugate updating and explicit forgetting mechanisms, resulting in bounded memory and computational complexity per data point. DTs achieve adaptivity to non-stationarity and remain competitive with state-of-the-art streaming algorithms in both regression and classification tasks (Anagnostopoulos et al., 2012).

1. Model Structure

Static treed models partition the input space $X \subset \mathbb{R}^p$ into axis-aligned hyperrectangles (“leaves”) via recursive binary splits of the form $x_j \geq c$. A tree $T$ consists of internal nodes $I_T$ and leaves $\mathcal{L}_T$; for every $x \in X$, $\eta(x) \in \mathcal{L}_T$ denotes the unique leaf containing $x$.

Within each leaf $\eta$, a simple parametric model is fitted:

• Regression: $$y|x,T \sim \mathcal{N}(\beta_\eta^T x + \mu_\eta, \sigma^2_\eta)$$, with a noninformative prior $x_j \geq c$0.
• Classification: $x_j \geq c$1, with prior $x_j \geq c$2.

The Bayesian prior on tree structures is a recursive split probability: for each leaf $x_j \geq c$3 at depth $x_j \geq c$4, $x_j \geq c$5, with $x_j \geq c$6. The prior on $x_j \geq c$7 is then $x_j \geq c$8.

Dynamic operation is defined by local “grow”, “prune”, or “stay” moves triggered only at the leaf $x_j \geq c$9 where the new datum $T$0 resides. Split dimension and cut-point for a “grow” move are chosen uniformly over available dimensions and the observed range within the current leaf.

2. Bayesian Formulation

The joint Bayesian formulation is specified by independent priors across leaves, as outlined above. The full data likelihood for $T$1 samples is

$T$2

Online posterior updates within an SMC framework update the particle weight for each tree $T$3 at time $T$4 as

$T$5

3. Streaming Inference: Data Retirement and Forgetting

3.1 SMC Operation

A population of $T$6 particles $T$7 is maintained, where $T$8 holds leaf-level sufficient statistics. On receiving $T$9,

• Weight each particle by predictive density at $I_T$0.
• Resample particles in proportion to these weights.
• Propagate by performing a random local move (grow/prune/stay) at the affected leaf.
• Update sufficient statistics in the relevant leaf by including $I_T$1.

3.2 Data Retirement (Active Discarding)

Each leaf maintains at most $I_T$2 active datapoints. When the active set exceeds $I_T$3, an active point $I_T$4 is retired. The prior for the corresponding leaf is updated with the retired data point:

• Regression leaves update Normal-Inverse-Gamma sufficient statistics via $I_T$5, $I_T$6, $I_T$7, $I_T$8.
• Classification leaves update Dirichlet counts: $I_T$9. Retirement updates preserve exact marginal likelihoods and posterior predictives in each leaf.

3.3 Forgetting Mechanism

To enable temporal adaptivity, a “forgetting factor” $\mathcal{L}_T$0 is used, applying $\mathcal{L}_T$1 (and analogous updates for $\mathcal{L}_T$2). As $\mathcal{L}_T$3, full-memory is retained; as $\mathcal{L}_T$4, only the most recent observations contribute. This adaptation allows DTs to track changes in nonstationary environments.

4. High-Level Pseudo-Code Description

The online DT algorithm can be summarized as follows:

$x \in X$9

5. Computational Complexity

Memory usage is $\mathcal{L}_T$5 for active datapoints and $\mathcal{L}_T$6 for tree structures; under constant $\mathcal{L}_T$7, total memory is $\mathcal{L}_T$8. Time complexity per data point:

• Weight computation: $\mathcal{L}_T$9 ($x \in X$0 = leaf-model dimension)
• Resampling: $x \in X$1
• Propagation (local move): $x \in X$2 for split selection; $x \in X$3 for structural change
• Retirement update: $x \in X$4 amortized per active set Overall per datum cost is $x \in X$5, independent of the cumulative sample size.

6. Empirical Performance Summary

DTs were benchmarked on both synthetic and real-world datasets:

• Regression (Friedman, $x \in X$6): Keeping $x \in X$7 active points, active learning criterion (ALC) based retiring nearly matches full-data DT performance in RMSE and predictive log-density at roughly 1/10 of the memory.
• Classification (Spambase: $x \in X$8): With $w