Minimax Rates and Spectral Distillation for Tree Ensembles

Abstract: Tree ensembles such as random forests (RFs) and gradient boosting machines (GBMs) are among the most widely used supervised learners, yet their theoretical properties remain incompletely understood. We adopt a spectral perspective on these algorithms, with two main contributions. First, we derive minimax-optimal convergence for RF regression, showing that, under mild regularity conditions on tree growth, the eigenvalue decay of the induced kernel operator governs the statistical rate. Second, we exploit this spectral viewpoint to develop compression schemes for tree ensembles. For RFs, leading eigenfunctions of the kernel operator capture the dominant predictive directions; for GBMs, leading singular vectors of the smoother matrix play an analogous role. Learning nonlinear maps for these spectral representations yields distilled models that are orders of magnitude smaller than the originals while maintaining competitive predictive performance. Our methods compare favorably to state of the art algorithms for forest pruning and rule extraction, with applications to resource constrained computing.

• The paper establishes minimax-optimal convergence rates for adaptive, honest random forests using spectral kernel techniques.
• The methodology leverages eigenvalue decay to derive performance guarantees and introduces SCATE for aggressive model compression.
• Empirical results show that SCATE achieves significant compression with minimal accuracy loss compared to standard pruning methods.

Minimax Rates and Spectral Distillation in Adaptive Tree Ensembles

Introduction

This paper delivers a rigorous statistical and algorithmic analysis of tree ensemble methods—specifically Random Forests (RFs) and Gradient Boosting Machines (GBMs)—from the perspective of spectral theory. It addresses two persistent open questions: (1) whether adaptively grown RFs can attain minimax-optimal rates under general nonparametric assumptions, and (2) how to leverage the spectral properties of tree ensembles for aggressive model compression, enabling resource-efficient deployment without compromising predictive fidelity. The authors present sharp minimax rate guarantees for honest, subsampled, adaptive RFs and introduce a universal spectral distillation framework—SCATE—that consistently outperforms existing model pruning and rule extraction approaches in terms of predictive performance per model size.

Theoretical Contributions: Minimax Rates for Random Forests

The minimax convergence result for RF regression (Theorem 4.1 (2605.11841)) is novel and striking, given that all prior results for nonparametric tree ensembles either required restrictive assumptions (e.g., purely random splits or oblique tessellations) or produced suboptimal rates. The analysis leverages a spectral representation of the RF ensemble as a kernel smoother. By expressing the prediction operator as a finite-sample kernel matrix and establishing eigenvalue decay rates (assumption C2), the main theorem provides:

Minimax-Optimal Convergence Rate:

If the eigenvalues of the induced RF kernel operator decay polynomially as $\lambda_i \asymp i^{-\beta}$ for $\beta > 1$ and mild smoothness holds for the regression function in the corresponding RKHS, the prediction risk attains the rate:

$$\mathbb{E}\|f^* - \widehat{f}_{\mathrm{RF}}\|^2_2 \lesssim N^{-2\alpha\beta/(2\alpha\beta+1)}$$

where $\alpha \in (0,1]$ quantifies the source condition on $f^*$. This matches the lower minimax bound under these structural conditions. The result holds for forest estimators with honest tree construction, balanced splits, and adaptive, greedy node selection. Notably, it bridges the gap between adaptive and non-adaptive ensemble theory, showing that adaptivity does not inherently preclude statistical optimality.

The proof integrates modern theories of inverse problems in Hilbert spaces and kernel methods, synthesizing bias-variance decompositions, operator norm controls, and Fano-type arguments for minimax lower bounds. Empirical assessment of the eigenvalue decay across a variety of datasets reveals that while the polynomial decay assumption is sufficient, strong practical performance of RFs persists even when this ideal is violated.

Spectral Distillation: The SCATE Framework

The statistical insights directly motivate a new, theoretically-grounded distillation protocol—SCATE (Spectral Compression of Adaptive Tree Ensembles)—which operates as follows:

1. Spectral Decomposition of Smoothers: For RFs, the kernel matrix $K$ (over $N$ training points) is PSD and admits an eigen-decomposition. For GBMs, which cannot be represented by a symmetric kernel, the smoother matrix $S$ is decomposed via SVD.
2. Identification of Informative Components: The leading $P \ll N$ eigenfunctions or singular vectors encode the dominant, predictive subspace of the ensemble.
3. Neural Map Learning: A compact MLP is trained to regress input features to these eigenfunctions, using a spectral-weighted MSE objective and an orthogonality penalty for stability.
4. Distilled Prediction Rule: Only the top-$P$ spectral directions and associated mappings are retained; predictions reduce to evaluating and linearly combining these low-dimensional functions.

This approach ensures that all compression and bottlenecking occur along maximally informative axes, in contrast to pruning, hard distillation, and rule-extraction techniques which lack a principled way to allocate model capacity or fail to give explicit control over model size-performance trade-offs.

Empirical Evaluation

Extensive benchmarking is conducted against state-of-the-art methods for ensemble pruning (e.g., ForestPrune, Level-wise Optimization and Pruning), rule extraction (e.g., FIRE, TreeExtract), and surrogate distillation (including naïve MLP-based mimics and stumps extraction). SCATE is systematically superior or competitive at every fixed resource constraint (10KB, 100KB), often providing order-of-magnitude compression with negligible loss in accuracy. Key findings include:

• Pareto Efficiency: SCATE consistently traces a Pareto-efficient rate-distortion curve relative to model size, attaining or exceeding the empirical rate of the base ensemble with substantial parameter reduction.
• Robustness to Kernel Decay: Even in regimes where eigenvalue decay is slow (i.e., spectral compressibility is low), SCATE maintains nontrivial predictive performance, outperforming existing neural and pruning-based alternatives.
• Deployment Suitability: SCATE's inference consists of evaluating shallow, parallelizable function blocks—making it ideal for deployment on microcontrollers and edge devices, in contrast to sequential tree traversal typical of rule extractors.
• Oracle Baseline: The paper introduces and closely approaches a novel oracle bound based on optimal spectral truncation, evidencing the tightness of the neuro-spectral learning approach.

Implications and Future Directions

Theoretically, the work closes a significant gap in the nonparametric theory of adaptively grown decision forests by confirming that properly regularized, honest forests can be minimax optimal, provided the kernel spectrum is suited to the function space. This result generalizes classic kernel learning theory to the highly nontrivial regime of data-adaptive, hierarchical partitioning.

Practically, SCATE establishes a new principle for ensemble distillation: compression should occur in the spectral domain, rather than via structural or symbolic pruning. The ability to set the exact size-memory footprint of the student model, while preserving predictive signal, is essential for industrial machine learning in resource-constrained contexts.

The approach's modularity admits various extensions:

• The spectral distillation mechanism is not limited to tree ensembles and could be applied to any smoother with a tractable (approximate) spectral decomposition, opening possibilities for model reduction in Gaussian Processes, kernel methods, and even attention-based architectures.
• Direct theoretical analysis of GBMs lags behind the RF case, due to challenges in non-self-adjoint operator theory; bridging this gap is a compelling direction for future work.
• Enhancing the interpretability of the spectral student remains unresolved, as the neural map is not directly human-interpretable unlike shallow trees or rule lists.

Conclusion

This work advances both the theoretical understanding and practical deployment of tree ensembles. By linking the statistical generalization of RFs to their spectral compressibility, and operationalizing this link in a scalable distillation protocol, the authors deliver a competitive, resource-aware alternative to classical ensemble reduction techniques. These methods are likely to redefine best practices for model deployment in settings where predictive accuracy and computational efficiency must be simultaneously optimized, and suggest fertile ground for future research in spectral learning and nonparametric statistical theory.

---

Reference:

"Minimax Rates and Spectral Distillation for Tree Ensembles" (2605.11841)