Forest Lasso: Sparse Regularization for Ensembles

Updated 5 April 2026

Forest Lasso is a framework that applies L1 norm penalties to tree ensembles, promoting sparsity and enhancing feature selection and model interpretability.
It incorporates methods like sparse tree weighting and selective tree pruning to achieve significant reductions in test MSE while retaining predictive accuracy.
The approach adapts classic Lasso theory to tree-based models, enabling efficient tuning, spatial fusion, and clearer insights in high-dimensional and spatial data contexts.

Forest Lasso refers to a class of methodologies that impose sparse or fusion penalties—typically based on the $\ell_1$ norm—over ensembles of trees or spatial neighborhoods to achieve improved feature selection, model parsimony, or interpretable clustering within forests or partially linear models. Although implementations and theoretical settings differ, the unifying characteristic is use of Lasso-like regularization on forests: either by assigning sparse weights directly to base trees, by regularizing feature usage, or by fusing spatially-varying regression coefficients. This framework forms the basis for algorithms such as ControlBurn, Lasso-based forest pruning, adaptive “Lassoed Forests”, and spatial Forest Lasso fusion penalties (Liu et al., 2021, Dorador, 2024, Shang et al., 10 Nov 2025, Zhang et al., 2024).

1. Lasso Penalties in Tree Ensembles: Core Formulations

Two main formulations of Forest Lasso have emerged:

Sparse Tree Weighting: Given a forest of $n$ trees with prediction vectors $A = [\alpha_1\,\ldots\,\alpha_n]\in\mathbb R^{m\times n}$ , assign nonnegative weights $w\in\mathbb R^n_+$ and solve:

$\min_{w\ge 0} \frac1m L(Aw, y) + \lambda \sum_{i=1}^n u_i w_i,$

where $u_i$ is the count of unique features used in tree $i$ , encouraging selection of trees that use overlapping subsets of features and thus drive feature sparsity (Liu et al., 2021).

Sparse Tree Selection via Lasso: Given $B$ trees with predictions $T = [\hat t_1\,\ldots\,\hat t_B]\in\mathbb R^{n_v\times B}$ , assign weights $\beta\in\mathbb R^B_+$ :

$n$ 0

encouraging only a small subset of trees to be retained. This approach is central in “Lassoed Forests” and nonnegative Lasso forest pruning (Dorador, 2024, Shang et al., 10 Nov 2025).

Spatial Fusion Forest Lasso: For spatially indexed data, the forest lasso fusion penalty enforces piecewise-constant regression coefficients across spatial graphs using tree-based incidence matrices and adaptive $n$ 1 penalties over graph edges (Zhang et al., 2024).

2. Algorithmic Steps and Practical Implementation

Forest Lasso algorithms can generally be instanced in the following summarized workflow:

Base Forest Training: Grow a diverse collection of trees, using strategies such as incremental-depth bagging, random subspace selection, or standard bagging/boosting. Diversity in feature usage per tree is essential for effective sparsification (Liu et al., 2021).
Feature/Tree Indicator Matrix Construction: For each tree, record its predictions and build a binary indicator of which features are used.
Sparse Optimization:
- Assign variables (tree weights or local regression coefficients) and pose the convex optimization problem with appropriate $n$ 2-type penalty.
- Use convex solvers (coordinate descent, glmnet, other Lasso solvers).
Tuning and Selection:
- Hyperparameters (λ for sparsity, θ for adaptive blending in “Lassoed Forests”) are tuned via cross-validation, validation set minimization, or bisection on target sparsity (Shang et al., 10 Nov 2025, Dorador, 2024).
Model Reconstruction:
- Retain only trees (or features) with nonzero weights.
- (Optional) Refit a new forest or CART model on the reduced input space for bias reduction (Liu et al., 2021, Dorador, 2024).
- In spatial settings, average fusion weights over forests, construct an adaptive spanning tree and solve a final weighted Lasso for spatial coefficient recovery (Zhang et al., 2024).

3. Theoretical Properties and Guarantees

Forest Lasso inherits several properties from classical Lasso theory:

Convexity and Global Optimality: All principal optimization problems are convex, admitting global minimizers via standard solvers (Liu et al., 2021, Dorador, 2024).
Selection Consistency: Under conditions such as restricted-eigenvalue or irrepresentable-type properties on the design (tree prediction matrix or indicator matrix), Lasso-based forest pruning and Forest Lasso achieve variable/t ree selection consistency:

$n$ 3

in the tree selection context (Dorador, 2024).

Finite-sample Generalization Bounds: For appropriate λ, high-probability excess prediction risk bounds of $n$ 4 hold, analogously to standard Lasso (Dorador, 2024).
SNR-Dependent Effects: Post-selection Lasso weighting reduces bias when signal-to-noise ratio (SNR) is high, but may increase mean squared error in low-SNR due to estimator variance. An adaptive mixture of vanilla RF and Lassoed RF always achieves MSE no greater than the better of the two (Shang et al., 10 Nov 2025).
Spatial Model Consistency: In the spatial fusion setting, under conditions on the triangulation, smoothness penalty, and fusion weights, the adaptive forest Lasso estimator recovers spatial clusters and attains minimax error rates in $n$ 5 and $n$ 6 (Zhang et al., 2024).

4. Addressing Feature Correlation, Interpretability, and Parsimony

A key motivation for Forest Lasso is the inadequacy of standard tree ensemble feature importance under high collinearity. Traditional metrics (MDI) distribute importance across correlated groups, diluting interpretability. By penalizing feature usage through weighted Lasso constraints on tree selection or feature indicators, Forest Lasso concentrates importance on a minimal subset of representatives from each correlated group, in analogy to the behavior of linear Lasso (Liu et al., 2021).

Interpretability further improves via dramatic pruning. In the Lasso-pruned forests, it is possible to collapse small sub-forests into a single decision tree by merging their partitions, providing a model that combines the accuracy of the forest with tree-level interpretability (Dorador, 2024).

5. Empirical Performance and Applications

Forest Lasso approaches have demonstrated substantial improvements across simulated and real datasets:

Standard Regression Forests: Lasso pruning achieves up to 71.5% reduction in test MSE using only a small fraction of original trees; Lasso-based selection typically achieves best pure accuracy, with best-subset and wrapper heuristics favored for maximum parsimony (Dorador, 2024).
“Lassoed Forests”: Adaptive blending of RF and PF consistently delivers lower MSE than either, with 5–20% test MSE reductions in genomics, drug sensitivity, and HIV resistance datasets (Shang et al., 10 Nov 2025).
Spatial Additive Models: Forest Lasso fusion penalty with adaptive multi-tree construction achieves 75–85% reduction in MSE of spatially varying coefficients compared to single-tree or classical graph fusion, with fast computation in contexts where all-pairs fusion would be computationally prohibitive (Zhang et al., 2024).
Feature Selection with Nonlinearity: ControlBurn’s Forest Lasso selects a minimal, interpretable feature subset while inheriting the nonlinear and interaction-capturing ability of tree ensembles, outperforming linear Lasso or standard MDI-based selection, especially under high feature correlation (Liu et al., 2021).

6. Variants: Adaptive Blending, Fusion for Spatial Clustering, and Feature Sparsification

Three main research lines can be distinguished:

Variant	Penalty Target	Application
Post-selection Forest Lasso	Trees	Regression forest pruning, model bias reduction
ControlBurn Forest Lasso	Features	Nonlinear feature selection, interpretability
Spatial Fusion Lasso	Edges of G	Clustering, spatial model heterogeneity

Post-selection Forest Lasso (Shang et al., 10 Nov 2025): Assign sparse weights to trees via Lasso or adaptive Lasso, sometimes with a hyperparameter-adaptive blend between vanilla RF and pure Lasso-pruned RF. The classifier or regressor output is adaptively combined for best error.
ControlBurn Forest Lasso (Liu et al., 2021): Assigns an $n$ 7–type penalty to total feature usage across the ensemble, forcing sparsity at the feature, not just tree, level.
Spatial Fusion Forest Lasso (Zhang et al., 2024): Imposes tree-based fusion penalties for spatial contiguity, using random spanning forests to approximate fused lasso penalties efficiently and adaptively, with theoretical support for both estimation and partition recovery.

7. Practical Guidance and Limitations

Algorithmic application of Forest Lasso methods requires careful attention to:

Training Diversity: For feature selection, ensure each tree uses only a small subset of features by strategies such as incremental-depth growth, as standard bagging can dilute the penalty’s effect (Liu et al., 2021).
Cross-Validation and Refitting: Hyperparameters (λ, θ, adaptive weights) are best chosen by validation error or cross-validation. Refit models on selected features/trees to reduce bias (Shang et al., 10 Nov 2025, Dorador, 2024).
Computational Cost: The dominant expense is forest construction; convex Lasso steps scale efficiently in the number of trees and (for spatial settings) in the number of fused edges. Forest Lasso often matches graph Lasso accuracy with order-of-magnitude lower computation in large spatial graphs (Zhang et al., 2024).
SNR Sensitivity: In low SNR, pure Lasso pruning can overfit to noise by eliminating too many trees; adaptive blending mitigates this effect (Shang et al., 10 Nov 2025).

Forest Lasso methods thus provide a principled, scalable toolkit for combining the predictive accuracy of ensemble methods with variable selection, model parsimony, and domain interpretability across classic, high-dimensional, and spatial modeling contexts.