Dice Question Streamline Icon: https://streamlinehq.com

Conjecture: Constancy of covariance-based degrees of freedom with ensemble size due to self-influence weights

Ascertain whether, in bagged ensembles of decision trees, the expected self-influence weight of a training point on its own prediction s^i(x_i) is invariant in the number of trees B, implying that the covariance-based degrees of freedom df(\hat{f}) remains constant as B increases, while increased randomness-induced smoothing across other training labels as B grows impacts prediction performance despite df(\hat{f}) being unchanged.

Information Square Streamline Icon: https://streamlinehq.com

Background

The authors revisit Mentch and Zhou’s degrees-of-freedom (DOF) explanation for random forest performance and observe that df(\hat{f}) fails to reflect performance differences across ensemble sizes even when performance changes. They propose a specific cause: df(\hat{f}) measures only dependence of predictions on their own training labels, not cross-label smoothing.

They conjecture that the expected self-influence si(x_i) (which determines df(\hat{f})) is constant in ensemble size B, while the ensemble’s randomness increases smoothing across other labels as B grows, thereby affecting performance without changing df(\hat{f}).

References

We conjecture that this could be because the expected dependence of a train-input prediction on its own train-time label is independent of the number of trees used (i.e. the expected value of si(x_i), which determines df(\hat{f}), is constant across B) -- this is what df(\hat{f}) captures. Yet, there can be more smoothing across all other training labels as B grows (i.e. more uniform expected sj(x_i), j\neq i) due to the randomness induced by bootstrapping -- which may impact prediction performance.

— Why do Random Forests Work? Understanding Tree Ensembles as Self-Regularizing Adaptive Smoothers (2402.01502 - Curth et al., 2 Feb 2024) in Section 3.2 (Where Mentch & Zhou’s degrees of freedom explanation falls short)