Random Forest Kernel

• Random Forest Kernel is a data-adaptive similarity measure computed as the fraction of trees where two instances fall into the same leaf.
• It leverages nonparametric adaptivity, local structure, and high-dimensional robustness to capture complex data distributions.
• The kernel integrates seamlessly with methods like kernel ridge regression and SVM, though its n² cost necessitates approximate computational strategies for large datasets.

A Random Forest Kernel is a positive semidefinite kernel implicitly constructed by an ensemble of randomized decision trees. Rather than being specified a priori, it is induced by the data-adaptive partitioning logic of the forest. The canonical construction measures similarity between instances as the fraction of trees in which they co-occur in the same leaf. This kernel inherits nonparametric adaptivity, local structure, and high-dimensional robustness from the forest, while enabling direct interfacing with the theory and practice of kernel methods.

1. Formal Definition and Construction

Given a dataset $\mathcal{D} = \{X_i, Y_i\}_{i=1}^n$, a Random Forest grows $M$ randomized trees. Each tree $m$ partitions the input space into terminal regions (leaves) $R_{m,k}$, $k=1,\ldots,T_m$. The Random Forest Kernel $$K_{\mathrm{RF}}: \mathbb{R}^d \times \mathbb{R}^d \to [0,1]$$ is defined by

$$K_{\mathrm{RF}}(x,x') = \frac{1}{M} \sum_{m=1}^M \sum_{k=1}^{T_m} \mathbf{1}\{x \in R_{m,k} \land x' \in R_{m,k}\}$$

For finite samples, this is the empirical co-occurrence frequency. For "KeRF" (Kernel based on Random Forest), and in the infinite-forest limit, this becomes the probability that $x$ and $x'$ fall into the same leaf under the randomized tree generation process (Scornet, 2015, Feng et al., 2020). Analytically, the expectation over tree-building yields

$$K_n(x, x') = \mathbb{P}_\Theta\bigl(x, x' \text{ in same leaf in tree } \Theta \bigr)$$

where $\Theta$ encodes all randomness in tree construction (feature selection, split points, subsampling, etc).

2. Mathematical and Statistical Properties

Positive Semidefiniteness and RKHS

The Random Forest Kernel is positive semidefinite: it is a convex combination (or expectation) of block-diagonal indicator kernels, each corresponding to a partition of the data (Feng et al., 2020, Davies et al., 2014, Panda et al., 2018). For any finite sample, the proximity/gran kernel matrix is symmetric and PSD, satisfying Mercer's condition. This induces a Reproducing Kernel Hilbert Space (RKHS) whose geometry and norm are determined by the data-driven partition distribution (Dagdoug et al., 29 Nov 2025, Iakovidis et al., 2023). Functions in this RKHS can be represented as

$$f(x) = \sum_{A \in \mathcal{A}} G(A) \alpha_P(A) 1_A(x)$$

where $\mathcal{A}$ is the collection of all cells/leaves, and $\alpha_P(A)$ is a suitable normalization depending on the cell's mass.

Adaptivity, Nonstationarity, and Locality

Unlike translation-invariant kernels (e.g., RBF), $K_{\mathrm{RF}}$ is highly nonstationary and data-adaptive: its behavior depends on the structure induced by the tree splits, which accommodate both the data distribution and potentially the response (for supervised forests) (Davies et al., 2014, Olson et al., 2018). The partitioning yields piecewise-constant similarity regions, locally refined where needed, and global/local smoothness is controlled by tree depth, leaf size, and split criteria.

Consistency and Limiting Behavior

As the forest size grows and split diameters decrease, the random forest kernel converges (under conditions) to a continuous limiting kernel. In the prototypical case of axis-aligned, uniformly random splits and full-depth trees, $K_{\mathrm{RF}}$ converges to a Laplace kernel,

$K_L(x,x') = \exp\bigl(-\|x-x'\|_1/\sigma\bigr)$

where $\sigma$ depends on the mean split size (Feng et al., 2020, Balog et al., 2016). With sufficient randomization, $K_{\mathrm{RF}}$ is universal and characteristic: mean-embeddings in the RF-RKHS characterize the underlying distribution (Panda et al., 2018, Dagdoug et al., 29 Nov 2025).

3. Algorithmic Realization and Variants

Proximity Kernel and Kernel Regression

In both regression and classification, the forest gives (possibly normalized) weights on training points when predicting at $x$ (Olson et al., 2018, Scornet, 2015, Qiu et al., 2022):

$$\alpha_i(x) = \frac{1}{M} \sum_{m=1}^M \frac{\mathbf{1}\{X_i \in L_m(x)\}}{N(L_m(x))}$$

where $L_m(x)$ is the leaf containing $x$ in tree $m$ and $N(L_m(x))$ its cardinality. The associated estimator is the kernel smoother:

$$\hat{f}(x) = \frac{\sum_{i=1}^n \alpha_i(x) Y_i}{\sum_{i=1}^n \alpha_i(x)}$$

or, equivalently, the Nadaraya–Watson estimator with data-driven kernel $K_{\mathrm{RF}}(x,x_i)$. Regularized regression/classification can be performed via kernel ridge regression, kernel SVM, or other approaches using $K_{\mathrm{RF}}$.

Explicit Kernels: KeRF and Mondrian Kernel

In the special case of data-independent random forests (e.g., centered or uniform splitting), explicit analytic formulas for $K_{\mathrm{RF}}$ are available (Scornet, 2015, Iakovidis et al., 2023, Isidoros et al., 4 Jul 2024). For example, the centered KeRF in $d$ dimensions and depth $k$ is

$$K_k(x,z) = \sum_{\substack{k_1 + \cdots + k_d = k \ k_j \geq 0}} \frac{k!}{k_1 ! \cdots k_d !} \left( \frac{1}{d} \right)^k \prod_{j=1}^d \mathbf{1}\{ \lceil 2^{k_j} x_j \rceil = \lceil 2^{k_j} z_j \rceil \}$$

The Mondrian kernel defines a random partition process and, in the infinite limit, gives an analytic Laplace kernel (Balog et al., 2016).

4. Theoretical Guarantees and Learning Rates

Consistency and Rates

For infinite forests grown with regularity and sufficient shrinking of partition diameters, the proximity kernel estimator is consistent for standard regression models with Lipschitz regression function (Scornet, 2015, Iakovidis et al., 2023):

$$\mathbb{E}[\widetilde m_{\infty,n}(x) - m(x)]^2 \leq C n^{-\alpha} (\log n)^\beta$$

with explicit exponents determined by the splitting mechanism, e.g., for centered KeRF, the exponent is $1/(1 + d\log 2)$ (Iakovidis et al., 2023, Isidoros et al., 4 Jul 2024).

Central Limit Theorems and Extensions

For weighted Fréchet regression (responses in metric spaces), asymptotic normality, consistency, and minimax-type rates have been established under infinite-order U-process and $M$-estimator theories (Qiu et al., 2022). In the Euclidean case, this specializes to the random forest CLT of Wager & Athey.

5. Practical Computational Issues

Component	Cost	Critical Parameters
Kernel Matrix Assembly	$O(M n^2)$	Number of trees $M$; $n$ data points
Tree Building	$O(M n \log n)$ (standard RF)	mtry, split rule, min leaf size, depth
Prediction	$O(M k)$ per test point	$k$ = tree depth; can be amortized

For large sample sizes, approximate strategies (Nyström, landmarking, cluster-based representations) can avoid the $O(n^2)$ cost of the complete kernel matrix (Davies et al., 2014, Feng et al., 2020). Storing $K_{\mathrm{RF}}$ becomes infeasible for large $n$ without such techniques.

6. Extensions and Empirical Performance

Survival, Multivariate, Manifold, and HDLSS Settings

Extensions of the Random Forest Kernel exist for survival analysis (random survival forests), multivariate and distributional regression (Fréchet, Wasserstein, SPD matrix-valued targets), and high-dimension low-sample-size (HDLSS) learning (Yang et al., 2010, Qiu et al., 2022, Cavalheiro et al., 2023). The kernel can be formed for any tree ensemble—random forest, gradient boosting, or Bayesian additive models.

Empirical evidence shows:

• RFK is often as good or better than its parent ensemble for regression, and competitive or better in classification/HDLSS contexts (Feng et al., 2020, Cavalheiro et al., 2023).
• When used in SVM (RFSVM), the kernel can significantly improve classification performance on high-dimensional, small-sample benchmarks (Cavalheiro et al., 2023).
• Manifold and metric-object responses benefit from the forest kernel's data-adaptive locality (Qiu et al., 2022).

Interpretability and Downstream Integration

The kernel aligns with variable-importance measures from the forest (mean-decrease-in-impurity, permutation importance). It is suitable for independence testing, two-sample tests, clustering, and prototype/landmark extraction. The induced RKHS admits geometric variable importance (GVI), and data geometry reflecting the structure discovered by the forest (Dagdoug et al., 29 Nov 2025, Panda et al., 2018).

7. Limitations, Open Directions, and Theoretical Developments

• The kernel is nonstationary and globally piecewise-constant, exhibiting sharper discontinuities than classical smooth kernels (e.g., Gaussian). While adaptivity is an advantage, it may require careful tuning of forest depth, mtry, and subsampling to avoid either under- or over-smoothing (Scornet, 2015, Olson et al., 2018).
• High memory cost for the full $n \times n$ kernel hinders scalability for large datasets; approximate representations are an ongoing research area (Davies et al., 2014).
• Laplace kernel convergence is not always optimal in high dimension; RFK can outperform classical Laplace/RBF but is not always minimax-optimal (Feng et al., 2020, Iakovidis et al., 2023).
• Theory of the RF-induced RKHS is developing, clarifying universality, continuity, and variational interpretations of tree ensembles (solution of penalized empirical risk in RF-RKHS, continuous-time boosting as gradient flow) (Dagdoug et al., 29 Nov 2025).
• Incorporating modern attention mechanisms and distributional splits expands the class of random-forest kernels with new properties and robustness (Utkin et al., 2022, Ćevid et al., 2020).

---

References:

(Scornet, 2015) – "Random forests and kernel methods" (Feng et al., 2020) – "(Decision and regression) tree ensemble based kernels for regression and classification" (Davies et al., 2014) – "The Random Forest Kernel and other kernels for big data from random partitions" (Qiu et al., 2022) – "Random Forest Weighted Local Fréchet Regression with Random Objects" (Iakovidis et al., 2023) – "Improved convergence rates for some kernel random forest algorithms" (Feng et al., 2020) – "Random Forest (RF) Kernel for Regression, Classification and Survival" (Panda et al., 2018) – "Learning Interpretable Characteristic Kernels via Decision Forests" (Balog et al., 2016) – "The Mondrian Kernel" (Dagdoug et al., 29 Nov 2025) – "An RKHS Perspective on Tree Ensembles" (Cavalheiro et al., 2023) – "Random Forest Kernel for High-Dimension Low Sample Size Classification" (Olson et al., 2018) – "Making Sense of Random Forest Probabilities: a Kernel Perspective" (Utkin et al., 2022) – "Attention-based Random Forest and Contamination Model"