xRFM: Scalable Kernel Learning for Tabular Data

• xRFM is a tabular-data method that integrates kernel feature learning via AGOP with a balanced binary tree to capture local, heterogeneous data structures.
• It employs recursive partitioning with median-based splits and leaf RFMs to adaptively learn distinct feature directions across subpopulations.
• Empirical evaluations demonstrate competitive regression and classification performance with O(n log n) training and O(log n) inference, while offering native interpretability.

xRFM is a tabular-data learning method that combines feature-learning kernel machines with a tree structure in order to adapt to the local structure of the data and scale to essentially unlimited amounts of training data. Introduced for inference from tabular data—collections of continuous and categorical variables organized into matrices—it is motivated by the persistence of Gradient Boosted Decision Trees (GBDTs) as the dominant baseline for tabular prediction, despite recent progress in neural networks and feature learning methods. Its defining design goals are accuracy, scalability, and interpretability: the method is reported to train roughly in $O(n\log n)$, perform inference in $O(\log n)$, and expose learned feature directions through the Average Gradient Outer Product (AGOP) (Beaglehole et al., 12 Aug 2025).

1. Problem setting and design rationale

xRFM is proposed against a specific empirical backdrop: tabular prediction has remained largely centered on GBDT variants such as XGBoost, LightGBM, and CatBoost, while kernel methods, although mathematically elegant, typically do not scale and do not adapt their features to the data (Beaglehole et al., 12 Aug 2025). The method addresses two deficiencies that the paper treats as central for tabular learning.

First, tabular datasets often exhibit local, heterogeneous structure. Different subsets of the population may depend on different variables or different combinations of variables, so a single global kernel can be too rigid. Second, direct kernel solvers become impractical at sufficiently large sample sizes, which limits the applicability of standard kernel ridge regression in the large-data regime (Beaglehole et al., 12 Aug 2025).

The xRFM construction responds to both issues simultaneously. It couples Recursive Feature Machines (RFMs), which perform kernel-based feature learning through AGOP updates, with a balanced binary tree that partitions the dataset into smaller local subproblems. Each leaf then hosts a local feature-learning kernel machine. This produces a model that can learn distinct feature representations in different regions of the input space while maintaining asymptotically efficient training and inference (Beaglehole et al., 12 Aug 2025).

2. Architecture: leaf RFMs and balanced recursive partitioning

The model has two principal components: leaf RFMs and a binary tree over the dataset (Beaglehole et al., 12 Aug 2025).

A leaf RFM is a feature-learning kernel machine trained on a small subset of the data. Rather than applying one global kernel model to the full dataset, xRFM recursively partitions the data and fits a separate leaf RFM in each terminal region. This is the mechanism by which xRFM adapts to local structure.

At each internal node, the paper specifies the following procedure:

1. Sample $m$ points from the current dataset.
2. Train a small leaf RFM for one iteration.
3. Compute the AGOP and take its top eigenvector $v$.
4. Split the current dataset by the median projection onto $v$:

$$S_1 = \{x \in S : v^\top x > \text{Median}(v^\top z \,:\, z\in S)\},$$

$$S_2 = \{x \in S : v^\top x \le \text{Median}(v^\top z \,:\, z\in S)\}.$$

This recursive partitioning continues until leaves contain at most $C$ samples, after which a final leaf RFM is trained in each leaf (Beaglehole et al., 12 Aug 2025).

The tree structure is therefore not a post hoc acceleration device. It is part of the statistical design of the method: the split directions are themselves learned from AGOP information obtained by a feature-learning kernel model. In this sense, xRFM integrates partitioning and representation learning rather than treating them as separate stages.

3. Kernel family, AGOP, and the RFM update mechanism

The mathematical core of xRFM is inherited from the RFM framework. The standard kernel predictor is written as

$$f(x)=K(x,X)\alpha, \qquad \alpha=(K(X,X)+\lambda I)^{-1}y,$$

where $X\in\mathbb{R}^{n\times d}$ denotes the inputs, $O(\log n)$0 the labels, and $O(\log n)$1 the ridge parameter (Beaglehole et al., 12 Aug 2025).

The key feature-learning object is the AGOP: $O(\log n)$2 This quantity estimates the covariance of gradients and identifies directions along which the predictor changes most (Beaglehole et al., 12 Aug 2025).

The iterative kernel-RFM update summarized in the paper is

$O(\log n)$3

$O(\log n)$4

Typical choices are $O(\log n)$5 iterations and $O(\log n)$6, and in practice xRFM uses the best validation iteration rather than always the final one (Beaglehole et al., 12 Aug 2025).

For tabular data, xRFM extends the standard rotation-invariant kernel choice and uses the family

$O(\log n)$7

The paper tunes over two notable cases: $O(\log n)$8, described as a generalized Laplace-type kernel, and $O(\log n)$9, described as a product of 1D Laplace kernels and explicitly noted to be not rotation invariant (Beaglehole et al., 12 Aug 2025).

The method also offers two AGOP variants. It can use either the full AGOP or only the diagonal of the AGOP. The diagonal-only option introduces an axis-aligned feature-selection bias, which the paper states fits tabular structure well (Beaglehole et al., 12 Aug 2025).

The appendix-level implementation details further specify the leaf RFM loop: optionally adapt bandwidth $m$0, fit kernel ridge regression on transformed inputs, compute validation error, compute AGOP,

$m$1

normalize $m$2, and return the best iteration by validation error (Beaglehole et al., 12 Aug 2025).

4. Locality, heterogeneity, and scaling behavior

The principal methodological claim of xRFM is that local partitioning changes what feature learning can represent on tabular data. A single global RFM can learn important directions over the full dataset, but it cannot separate feature relevance patterns that differ sharply across subpopulations. xRFM does so by assigning different leaves to different regions of the input space (Beaglehole et al., 12 Aug 2025).

The paper illustrates this with a synthetic example in which one part of the space depends on $m$3 while another depends on $m$4. A global kernel RFM would identify all of these features together, whereas xRFM separates them across leaves (Beaglehole et al., 12 Aug 2025). This clarifies the intended use case: hierarchical or piecewise tabular relationships in which conditioning on one variable changes which other variables matter.

The same tree also yields the scalability properties emphasized in the paper. Because the recursive splits are balanced by median projection, leaf size is fixed independently of $m$5, internal nodes use only a subsample to estimate the split direction, and each point is routed once down the tree, the resulting complexity is:

• training: $m$6,
• inference: $m$7 (Beaglehole et al., 12 Aug 2025).

The paper further states that xRFM training plus inference time scales linearly with sample size in practice, that the method is especially fast on small datasets, and that it remains practical on much larger datasets where standard kernel methods break down (Beaglehole et al., 12 Aug 2025). It also contrasts this with direct kernel solvers, which it describes as becoming intractable past roughly $m$8 samples. Taken together, these claims position xRFM as a divide-and-conquer kernel learner rather than a classical monolithic kernel machine.

5. Benchmark evaluation and comparative performance

The empirical evaluation is organized around two benchmark suites (Beaglehole et al., 12 Aug 2025).

The first is the TALENT benchmark, comprising 300 total datasets: 100 regression, 120 binary classification, and 80 multiclass classification, with training sizes from 500 to 100,000. The second is a meta-test benchmark consisting of 17 largest datasets, each with 70k to 500k samples, intended to test very-large-scale behavior where standard kernel methods become memory constrained (Beaglehole et al., 12 Aug 2025).

The comparison set includes 31 other methods, among them:

• GBDTs such as XGBoost, LightGBM, CatBoost, and RandomForest,
• neural tabular models such as RealMLP, MLP-PLR, ResNet, TabNet, and FT-Transformer variants,
• TabPFNv2,
• kernel baselines such as Laplace KRR,
• other modern tabular methods such as TabR, ModernNCA, and Excelformer (Beaglehole et al., 12 Aug 2025).

For regression, the reported metrics are RMSE and normalized RMSE (nRMSE). For classification, the metric is classification error, defined as $m$9 accuracy. Aggregate benchmark metrics include shifted geometric mean (SGM), arithmetic mean, and normalized arithmetic mean, with

$v$0

(Beaglehole et al., 12 Aug 2025)

The reported results are differentiated by task type. On the 100 TALENT regression tasks, xRFM is described as the best-performing method across all regression aggregation metrics. On TALENT classification, it is described as competitive with the best methods, around third-ranked overall, and as outperforming GBDTs (Beaglehole et al., 12 Aug 2025). On the 17 largest meta-test datasets, xRFM is reported to be best on regression and second-best on classification (Beaglehole et al., 12 Aug 2025).

These results define the empirical identity of xRFM. It is not presented merely as a scalable approximation to kernel methods; it is presented as a benchmark-competitive tabular learner whose strongest aggregate advantage appears on regression, while remaining competitive on classification against both tree ensembles and recently introduced tabular foundation models (Beaglehole et al., 12 Aug 2025).

6. Interpretability, family context, and scope

A distinctive aspect of xRFM is that interpretability is native to the learning mechanism rather than appended through a separate explainer. Each leaf RFM exposes its learned AGOP matrix, and the paper emphasizes two complementary views of that object (Beaglehole et al., 12 Aug 2025).

The first is through diagonal entries, which indicate coordinate-wise importance. A large diagonal value for feature $v$1 means that predictions are sensitive to perturbations in that coordinate. The second is through top eigenvectors, which reveal joint feature directions. If one coordinate has positive loading and another negative loading, then moving them in opposite directions changes the prediction in a structured way (Beaglehole et al., 12 Aug 2025).

Because interpretation occurs leafwise, different regions of the input space can reveal different relevance patterns. The paper illustrates this with several named examples:

• California housing: longitude is most important;
• Covertype: elevation, distance to roadways, and distance to fire points;
• NYC Taxi Tipping: different leaves emphasize different local variables, such as pickup location or fare code;
• Breast cancer: average number of concave points is highly predictive (Beaglehole et al., 12 Aug 2025).

Within the broader RFM line of work, xRFM should be situated as a specialization for tabular learning. It retains the recursive AGOP-based feature-learning mechanism of RFMs, but introduces tree-based locality and scalable partitioning as its defining innovations (Beaglehole et al., 12 Aug 2025). A later development, K-Inverse-RFM, is presented as an evolution of the broader xRFM/RFM family for corrupted mathematical tasks; it preserves the recursive AGOP-kernel structure but adds kernel-powered label projection and stabilized AGOP updates (Pasternak, 1 Jul 2026). This suggests that the RFM family is being extended along distinct axes: xRFM addresses local heterogeneity and scale in tabular data, whereas K-Inverse-RFM addresses robustness under label noise, class imbalance, and complex representations.

The scope of xRFM is therefore specific. Its central contribution is not a general-purpose replacement for all nonparametric learners, but a tabular method that combines kernel feature learning, local partitioning, and AGOP-based interpretability in a single framework. In that domain, the reported picture is precise: best aggregate performance on large regression suites, competitive classification performance, $v$2 training, $v$3 inference, and interpretable local feature structure through AGOP (Beaglehole et al., 12 Aug 2025).