Conjecture: Spiked-smooth behavior arises with overfitting in tree ensembles

Determine whether randomized ensembles of decision trees (such as random forests) exhibit spiked-smooth behavior—i.e., they use fewer effective parameters when predicting at previously unseen test inputs than at training inputs—whenever the individual trees are overfitted to the training data. Establish this phenomenon by assessing the gap between train-time and test-time effective parameters computed from the smoother-weight vectors of the ensemble.

Background

The paper quantifies the spiked-smooth behavior conjectured by Wyner et al. by measuring effective parameters (a smoothing metric) on train versus test inputs. Experiments show that ensembles of interpolating trees use fewer effective parameters at test inputs than at training inputs, and that this train–test gap shrinks as tree depth decreases.

Extending beyond interpolation, the authors find similar spiked-smooth behavior for non-interpolating forests, with the effect appearing more pronounced when individual trees are more overfitted. This motivates their conjecture that the behavior occurs whenever there is some degree of overfitting in the individual models.