---
title: Extremely Randomized Trees Explained
url: https://www.emergentmind.com/topics/extremely-randomized-trees
type: topic
---

# Extremely Randomized Trees Explained

Extremely Randomized Trees (Extra-Trees) are a variant of ensemble methods for regression and classification that introduce increased randomization into the induction of decision trees, both in feature selection and threshold determination. The principal motivation for Extra-Trees is to further reduce variance beyond that achievable by Random Forests, enabling more robust aggregation of piecewise constant predictors, improved computational efficiency, and, in certain regimes, better handling of sparsely sampled regions in the feature space [2006.11014][2309.01460].

## 1. Formal Definition and Algorithmic Structure

Extra-Trees, introduced by Geurts, Ernst, and Wehenkel (2006), construct a binary tree by, at each internal node:

- Drawing $K$ features (with $K \leq m$, the number of available features) uniformly at random.
- For each selected feature $j_k$, choosing a split-threshold $t_k$ uniformly at random within the range of observed values of feature $j_k$ for the current node’s subset.
- Computing the standard impurity reduction (e.g., variance for regression).
- Selecting the split $(j_{k^*}, t_{k^*})$ that maximizes this reduction.

Unlike standard decision trees or classical Random Forests, which search for the optimal split among all candidate thresholds (typically midpoints in sorted feature values), Extra-Trees randomize not just feature selection but also threshold, expanding the diversity among trees in the ensemble.

For regression, the impurity at a node with data $D = \{(x_i, y_i)\}_{i=1}^n$ is:
\[
I(D) = \mathrm{Var}(D) = \frac{1}{|D|} \sum_{(x, y) \in D} (y - \bar{y})^2,
\]
where $\bar{y}$ denotes the sample mean in $D$. At each node, Extra-Trees propose $K$ splits and branch on the split yielding the maximal decrease in variance.

The process continues recursively until a stopping criterion is reached. At the leaf level, prediction is usually the mean (for regression) or class-distribution estimate (for classification).

The following outlines the core Extra-Trees algorithm for a single regression tree:

```python
def BuildExtraTree(D, depth):
    if depth == 0 or len(D) < n_min:
        return Leaf(mean_y(D))
    for k in range(K):
        j_k = random_feature()
        t_k = random_uniform(min_xj(D, j_k), max_xj(D, j_k))
        D_left, D_right = split_data(D, j_k, t_k)
        delta_var[k] = variance_reduction(D, D_left, D_right)
    k_star = argmax(delta_var)
    left_subtree = BuildExtraTree(D_left[k_star], depth - 1)
    right_subtree = BuildExtraTree(D_right[k_star], depth - 1)
    return Node(j_k_star, t_k_star, left_subtree, right_subtree)
```
Here, $K$ is a tunable parameter controlling randomization; $K = m$ (all features) with random thresholds recovers the fully randomized Extra-Trees regime, while $K = 1$ maximizes randomization [2006.11014].

## 2. Theoretical Properties and Consistency

A key theoretical advance is the establishment of $L_2$-consistency for the Extra-Trees ensemble under probabilistic impurity-decrease conditions. Specifically [2309.01460]:

- The Extra-Trees estimator constructs trees up to depth $k_n \to \infty$, with $k_n < c \log_2 n$, $c < 1/4$, and aggregates responses at each leaf.
- Consistency is established under the “probabilistic sufficient impurity decrease” (SID) condition (C1*), which requires that for each cell, there exists a probability $\delta > 0$ such that, among $n_{\mathrm{split}}$ random uniform splits, with probability at least $\delta$, a candidate split realizing a fractionally sufficient impurity decrease is found.
- Any regression function $m$ for which sufficiently many random splits yield nontrivial impurity reduction — even if no axis-aligned split is optimal (e.g., for highly interactive targets) — can be consistently estimated.

The main consistency result states:
\[
\lim_{n \to \infty} E[(\hat{m}_T(X) - m(X))^2] = 0
\]
for tree ensembles with $n_{\mathrm{split}}$ satisfying $n_{\mathrm{split}} \geq -2\log 2 / \log(1 - \delta)$ and tree depth settings as above.

This covers target functions beyond those covered by classical SID conditions, notably functions requiring multiple coordinate splits to locally reduce error (“pure interaction” forms). This suggests Extra-Trees achieve $L_2$-consistency across a strictly broader function class than purely deterministic-split forests [2309.01460].

## 3. Comparison with Related Ensembles

Decision forests such as Random Forests (Breiman, 2001) and Extra-Trees differ in sources of randomization and search strategy:

| Method                | Feature Subsampling | Threshold Selection     | Split Score          |
|-----------------------|--------------------|------------------------|----------------------|
| CART                  | No                 | All midpoints          | Best overall         |
| Random Forest         | Random (subset)    | All midpoints          | Best per subset      |
| Extra-Trees           | Random (subset)    | Random within range    | Best random proposal |

Standard CART-based methods perform exhaustive threshold search per feature, which is computationally heavy ($O(Nm)$ per node), and, due to the deterministic alignment of split points, can introduce discontinuities when training data are sparse. Extra-Trees, by randomizing the split position within the observed range, break large inter-point gaps differently across trees, improving ensemble smoothness and reducing persistent discontinuities [2006.11014].

A partially randomized variant (termed "Partially Randomized Trees") considers all $m$ features per node, but chooses a single uniform random threshold per feature and splits on the best of $m$ random candidates. This hybrid maintains computational efficiency ($O(m)$ per node), further accelerates tree induction (yielding $N$-fold speed-ups within boosting), and injects sufficient variability in split locations to fill gaps in sparse regions [2006.11014].

## 4. Computational Aspects and Scalability

The primary computational advantage of Extra-Trees arises from the elimination of axis-aligned midpoint search. At each node, only $K$ split evaluations are performed:

- Standard CART: $O(Nm)$ per node (all midpoints).
- Extra-Trees: $O(K)$ per node (typically $K \ll N$).
- Partially Randomized Trees: $O(m)$ per node.

For a full binary tree of depth $d$, building a single Extra-Tree requires $O(K \cdot 2^d)$ operations, compared to $O(Nm \cdot 2^d)$ for classical trees. In gradient boosting or forest settings with $M$ trees, this enables considerable global speed-up. Empirically, training times are reduced by $4-5\times$ relative to standard Gradient Boosting Machines (GBM) with deterministic trees on large-scale datasets [2006.11014].

## 5. Implications for Function Approximation and Smoothness

Deterministic trees (CART) create discontinuities at leaf boundaries, which are restricted to observed midpoints, leading to persistent stepwise errors in sparsely sampled regions. Extra-Trees, because tree partitions vary randomly across the ensemble, divide large inter-point gaps at different locations, thus, the averaged ensemble achieves denser and more evenly distributed piecewise-constant approximations.

This property is especially beneficial in regression: the expected discontinuity of the ensemble’s prediction is smaller, and the approximation adapts more smoothly in regions where data is sparse — boosting algorithms with partially randomized trees demonstrate mitigated step artifacts compared to their fully deterministic counterparts [2006.11014].

## 6. Empirical Performance and Application in Boosting

Experiments on nine regression benchmarks (California housing, Boston, diabetes, HouseART, Friedman 1–3, various synthetic datasets) indicate that Gradient Boosting Machines (GBM) using partially randomized trees achieve lower mean squared error (MSE) on 8 out of 9 datasets, outperforming Random Forests, Extra-Trees regression forests, XGBoost, and CatBoost under comparable cross-validation regimes and hyperparameter search.

For boosting, the integration of partially randomized trees involves computing pseudo-residuals at each step, fitting a regression tree using randomized threshold splits for each feature, and updating the additive model. The ensemble of such partially randomized base learners yields improved accuracy and computational tractability [2006.11014].

## 7. Parameter Constraints and Practical Considerations

Successful application of Extra-Trees requires setting:

- Number of random candidate splits per node ($K$ or $n_{\mathrm{split}}$): $n_{\mathrm{split}} \gtrsim 10-20$ is typical to ensure high probability of sufficient impurity decrease.
- Tree depth: $k \to \infty$ with $k < c \log_2 n$, $c < \tfrac{1}{4}$, or equivalently, splitting until each leaf has $< n^\eta$ points, $\eta < \tfrac{1}{8}$.
- No further regularization (e.g., minimum leaf size) is required beyond bounding maximum tree depth [2309.01460].

A plausible implication is that, absent post-pruning or classical regularization, Extra-Trees ensembles can achieve consistency and low variance provided random-split parameters are properly tuned.

---

**References**

- Konstantinov, A.V., Utkin, L.V. "Gradient boosting machine with partially randomized decision trees" [2006.11014].
- Consistency of Random Forest Type Algorithms under a Probabilistic Impurity Decrease Condition [2309.01460].

Source: https://www.emergentmind.com/topics/extremely-randomized-trees