---
title: Bayesian Dynamic Trees
url: https://www.emergentmind.com/topics/bayesian-dynamic-trees
type: topic
---

# Bayesian Dynamic Trees

Bayesian Dynamic Trees (DTs) constitute a fully Bayesian, non-parametric modeling framework suitable for streaming and massive data settings. DTs maintain piecewise simple parametric models on axis-aligned partitioned subspaces while employing sequential Monte Carlo (SMC) for online inference. Key techniques include active data retirement with conjugate updating and explicit forgetting mechanisms, resulting in bounded memory and computational complexity per data point. DTs achieve adaptivity to non-stationarity and remain competitive with state-of-the-art streaming algorithms in both regression and classification tasks [1201.5568].

## 1. Model Structure

Static treed models partition the input space $X \subset \mathbb{R}^p$ into axis-aligned hyperrectangles (“leaves”) via recursive binary splits of the form $x_j \geq c$. A tree $T$ consists of internal nodes $I_T$ and leaves $\mathcal{L}_T$; for every $x \in X$, $\eta(x) \in \mathcal{L}_T$ denotes the unique leaf containing $x$.

Within each leaf $\eta$, a simple parametric model is fitted:
- **Regression**: $y|x,T \sim \mathcal{N}(\beta_\eta^T x + \mu_\eta, \sigma^2_\eta)$, with a noninformative prior $\pi(\beta_\eta,\mu_\eta,\sigma^2_\eta) \propto 1/\sigma^2_\eta$.
- **Classification**: $y|x,T \sim \text{Categorical}\left(\{p_{\eta, \ell}\}_{\ell=1}^L\right)$, with prior $\pi(p_{\eta, \cdot}) = \text{Dirichlet}(\alpha_1, ..., \alpha_L)$.

The Bayesian prior on tree structures is a recursive split probability: for each leaf $\eta$ at depth $D_\eta$, 
$p_\text{split}(\eta) = \alpha (1 + D_\eta)^{-\beta}$, with $\alpha,\beta>0$. The prior on $T$ is then
$\pi(T) = \prod_{\eta\in I_T} p_\text{split}(\eta) \prod_{\eta\in\mathcal{L}_T} [1-p_\text{split}(\eta)]$.

Dynamic operation is defined by local “grow”, “prune”, or “stay” moves triggered only at the leaf $\eta(x_t)$ where the new datum $(x_t, y_t)$ resides. Split dimension and cut-point for a “grow” move are chosen uniformly over available dimensions and the observed range within the current leaf.

## 2. Bayesian Formulation

The joint Bayesian formulation is specified by independent priors across leaves, as outlined above. The full data likelihood for $t$ samples is
$$
p(y^t | T, x^t, \theta) = \prod_{\eta \in \mathcal{L}_T} \prod_{i : \eta(x_i) = \eta} p(y_i | x_i, \theta_\eta).
$$

Online posterior updates within an SMC framework update the particle weight for each tree $T_{t-1}^{(i)}$ at time $t$ as
$$
w^{(i)}_t \propto p(y_t | T_{t-1}^{(i)}, x_t) = \int p(y_t | \theta_{\eta(x_t)}, x_t ) \pi(\theta_{\eta(x_t)} | \text{data}_{t-1}) d\theta.
$$

## 3. Streaming Inference: Data Retirement and Forgetting

### 3.1 SMC Operation
A population of $N$ particles $\{T_{t-1}^{(i)}, S_{t-1}^{(i)}\}$ is maintained, where $S_{t-1}^{(i)}$ holds leaf-level sufficient statistics. On receiving $(x_t, y_t)$,
- **Weight** each particle by predictive density at $x_t$.
- **Resample** particles in proportion to these weights.
- **Propagate** by performing a random local move (grow/prune/stay) at the affected leaf.
- **Update** sufficient statistics in the relevant leaf by including $(x_t, y_t)$.

### 3.2 Data Retirement (Active Discarding)
Each leaf maintains at most $w$ active datapoints. When the active set exceeds $w$, an active point $r$ is retired. The prior for the corresponding leaf is updated with the retired data point:
- Regression leaves update Normal-Inverse-Gamma sufficient statistics via
  $G^{\text{new}} = G + x_r x_r^T$, $b^{\text{new}} = b + x_r y_r$, $r^{\text{new}} = r + y_r^2$, $\nu^{\text{new}} = \nu + 1$.
- Classification leaves update Dirichlet counts: $a_j^{\text{new}} = a_j + 1_{(y_r = j)}$.
Retirement updates preserve exact marginal likelihoods and posterior predictives in each leaf.

### 3.3 Forgetting Mechanism
To enable temporal adaptivity, a “forgetting factor” $\lambda \in (0,1]$ is used, applying $G \gets \lambda G + x_r x_r^T$ (and analogous updates for $b, r, \nu$). As $\lambda \rightarrow 1$, full-memory is retained; as $\lambda \rightarrow 0$, only the most recent observations contribute. This adaptation allows DTs to track changes in nonstationary environments.

## 4. High-Level Pseudo-Code Description

The online DT algorithm can be summarized as follows:

```text
Input: stream S = {(x_t, y_t)}, budget w, particles N, forgetting λ∈(0,1]
Initialize for i=1…N: T_0^{(i)} ← trivial tree, leaf-priors π_0, active-set_i=∅
For t=1,2,… do
  1. Observe (x_t, y_t)
  2. For each i: compute weight w_i = p(y_t | T_{t−1}^{(i)}, x_t ; leaf-posterior)
  3. Resample indices {i} → {j(i)} with prob ∝ w_i
  4. For new particles i=1…N:
     T̃ ← T_{t-1}^{(j(i))}
     At leaf η=η_{T̃}(x_t), choose move ∈{grow,prune,stay} uniformly
     If grow/prune: adjust T̃; redistribute retired-prior stats
     Update η: add (x_t,y_t); update S-stats
     If |active-set_η|>w:
        select r∈active-set_η by active discarding (e.g. lowest Δσ^2/regression, lowest entropy/classification)
        retire (x_r,y_r) by updating leaf-prior stats (with λ-downweighting); remove from active set
     T_t^{(i)} ← T̃
Output: predictive distribution at t via averaging p(y|x,T_t^{(i)}) over i
```

## 5. Computational Complexity

Memory usage is $O(N \cdot w)$ for active datapoints and $O(N \cdot |T|)$ for tree structures; under constant $w$, total memory is $O(N \cdot w)$. Time complexity per data point:
- Weight computation: $O(N \cdot m^2)$ ($m$ = leaf-model dimension)
- Resampling: $O(N)$
- Propagation (local move): $O(m^2)$ for split selection; $O(1)$ for structural change
- Retirement update: $O(1)$ amortized per active set
Overall per datum cost is $O(N \cdot m^2)$, independent of the cumulative sample size.

## 6. Empirical Performance Summary

DTs were benchmarked on both synthetic and real-world datasets:
- **Regression (Friedman, $p=5$)**: Keeping $w=200$ active points, active learning criterion (ALC) based retiring nearly matches full-data DT performance in RMSE and predictive log-density at roughly 1/10 of the memory.
- **Classification (Spambase: $4601 \times 57$)**: With $w

Source: https://www.emergentmind.com/topics/bayesian-dynamic-trees