---
title: Recursive Feature Machines (RFMs)
url: https://www.emergentmind.com/topics/recursive-feature-machines-rfms
type: topic
---

# Recursive Feature Machines (RFMs)

Recursive Feature Machines (RFMs) are a class of algorithms that implement feature learning by recursively updating feature representations using principled, mathematically transparent mechanisms tied to gradient information or empirical risk changes. RFMs have emerged in multiple contexts across machine learning, including kernel machines, neural networks, code decoding, conformal learning, low-rank recovery, interpretable modeling, and controlled generation. The unifying principle is the iterative adjustment or elimination of features based on their impact measured by either gradients, empirical risk, or non-conformity scores.

## 1. Core Mechanisms and Mathematical Foundations

RFMs are built upon iterative procedures that optimize feature sets or reweight feature directions to enhance model prediction or representation. There are two major classes of RFM mechanisms:

- **Recursive Feature Elimination by Empirical Risk ("Risk-RFE")**: 
  The earliest RFM approach [1304.5245] operates in the context of kernel machines. At each step, features are removed one at a time based on their marginal impact on the regularized empirical risk:
  $$
  \mathcal{R}^{\text{reg},\lambda}_{L,D,H}(f) = \lambda \|f\|_H^2 + \mathcal{R}_{L,D}(f)
  $$
  Here, $\mathcal{R}_{L,D}(f)$ is the empirical risk, $\lambda$ is a regularization parameter, and $H$ is an RKHS induced by the kernel $k$. For each feature $i$, a projected RKHS $H^{J \cup \{i\}}$ is formed by zeroing out feature $i$, and the difference in empirical risk is
  $$
  \Delta_i = \mathcal{R}^{\text{reg},\lambda}_{L,D,H^{J \cup \{i\}}}(f_{D,\lambda,H^{J \cup \{i\}}}) - \mathcal{R}^{\text{reg},\lambda}_{L,D,H^J}(f_{D,\lambda,H^J})
  $$
  The feature $i$ with the smallest $\Delta_i$ is eliminated, and the process continues until a stopping threshold $\delta_n$ is reached.

- **Recursive Feature Learning using the Average Gradient Outer Product (AGOP)**:
  In modern RFMs [2212.13881], feature adaptation is performed by recursively updating a learnable metric $M$ in the kernel via AGOP:
  $$
  M \leftarrow \frac{1}{n} \sum_{p=1}^n [\nabla f(x_p)][\nabla f(x_p)]^T
  $$
  The corresponding kernel is a Mahalanobis kernel (e.g., Laplace kernel):
  $$
  K_M(x, z) = \exp(-\gamma (x-z)^T M (x-z))
  $$
  The AGOP highlights directions in input space that most affect predictions, thus continually adapting the kernel (or metric) in feature space.

- **Algorithmic Formulation and Extensions**:
  - For matrix recovery and sparse regression [2401.04553], the RFM iterates between minimization and adaptive filtering:
    $$
    W_t = \operatorname{argmin}_W \|W\|_F^2 \ \text{subject to} \ \langle A_i M_t, W \rangle = y_i
    $$
    $$
    M_{t+1} = \phi(M_t^T W_t^T W_t M_t)
    $$
    with $\phi$ a spectral function, often a matrix power.

  - For interpretable QSPR modeling [2411.14079], feature importance per sample is calculated as $x_i^T M x_i$, with global importance as the mean over all samples.

  - In xRFM for scalable tabular learning [2508.10053], partitioning is performed by median splits along directions of maximal AGOP eigenvectors, yielding local feature learning at each leaf node.

## 2. Theoretical Guarantees and Analytical Results

RFMs provide theoretical consistency and expressivity beyond classical algorithms:

- **Consistency of Risk-RFE** [1304.5245]:
  Under convex, locally Lipschitz losses and bounded, measurable kernels, risk-RFE is uniformly consistent in identifying the correct feature subspace, provided RKHS complexity is appropriately controlled (via entropy numbers) and a minimum risk gap $\varepsilon_0>0$ exists for elimination of relevant features.

- **Duality and Function Space Extension** [2305.05642]:
  A duality framework links estimation in function spaces $\mathcal{F}_{p,\pi}$ to approximation in conjugate feature spaces, with sharp, dimension-independent approximation rates for $p>1$:
  $$
  \inf_{\{c_j\}} \left\| f - \frac{1}{m}\sum_j c_j \varphi(\cdot,v_j) \right\|_{q,\rho} \lesssim \left[ \frac{M_q^{p'-2} R_q^2 \log^3 m + M_q \log(1/\delta)}{m} \right]^{1/p'}
  $$
  For $p<2$, RFMs can approximate functions outside the RKHS, broadening learning capacity beyond kernel methods.

- **Dimensionality Reduction and Sparse Recovery** [2401.04553]:
  In linear regimes, RFM generalizes IRLS algorithms, recovering low-rank matrices and sparse vectors efficiently. Fixed points of the iterative update correspond to critical points of regularized spectral objectives, with efficient (SVD-free) scaling to matrices with millions of entries.

## 3. Practical Applications and Experimental Results

RFMs have been applied across:

- **Feature Selection and Elimination** [1304.5245, 2405.19429]:
  RFMs eliminate features by risk or non-conformity, producing concise feature sets and demonstrating higher accuracy and automatic stopping than traditional RFE or LASSO across tabular, protein, and image datasets.

- **Feature Learning for Kernel Machines and Tabular Data** [2212.13881, 2508.10053]:
  By recursive metric learning via AGOP, RFMs achieve state-of-the-art results (SOTA) on over 150 tabular datasets, outperforming both kernel and deep methods. xRFM combines these advances with tree-based partitioning for scalability and local adaptation.

- **QSPR Molecular Modeling** [2411.14079]:
  RFMs provide interpretable modeling for molecular properties using tailored fingerprints (MACCS, Morgan, hybrid). Both local and global feature importance are extracted from AGOP-weighted kernels, facilitating molecule design by identifying actionable substructures.

- **Music Generation and Controllability** [2510.19127]:
  MusicRFM adapts RFMs to steer frozen autoregressive music models by injecting concept directions (AGOP eigenvectors) into activation flows, achieving fine-grained control over musical attributes and measurable improvements in generation accuracy while preserving prompt fidelity.

- **Emergence and Generalization Phenomena** [2407.20199, 2303.15745]:
  RFMs replicate ‘grokking’ phenomena (sharp generalization phase transitions) observed in neural networks by progressively learning block-circulant features for modular arithmetic tasks, closely mirroring the learning process theorized to underlie deep learning’s emergent behavior. This aligns with the double descent MSE pattern observed under controlled noise and overparameterization [2303.15745].

## 4. Interpretability and Model Analysis

A major strength of RFMs is inherent interpretability due to gradient-based feature importance and structural transparency:

- **Feature Matrix Analysis**:
  The learned metric $M$—whether via risk-RFE, AGOP, or conformal non-conformity—can be analyzed directly for diagonal dominance (per-feature relevance), eigenstructure (feature combinations), and block structure (as in modular arithmetic with circulant blocks) [2407.20199].

- **Feature Importance Scoring**:
  RFMs support both local analysis ($x_i^T M x_i$ for individual samples) and global ranking (mean score across dataset), with scores correlating to physically or chemically meaningful descriptors, enabling actionable feedback for applications such as drug design or data screening [2411.14079].

- **Uncertainty Quantification and Consistency**:
  CRFE extends the RFM paradigm into conformal prediction, enabling recursive elimination based on non-conformity measures tied directly to sample ‘strangeness’ and introducing robust stopping and consistency indices across data splits [2405.19429].

## 5. Scalability, Efficiency, and Broader Connections

RFMs have been designed for computational efficiency and broader methodological implications:

- **Algorithmic Complexity**:
  Recursive risk-RFE exhibits linear-to-quadratic scaling, while tree-partitioned xRFM operates in $O(n \log n)$ time per training and $O(\log n)$ per inference [2508.10053]. SVD-free matrix completion provides practical scaling for linear RFM variants [2401.04553].

- **Connections to Other Recursive Paradigms**:
  The recursive structure in RM code decoding [1703.05302] offers a blueprint for layered, candidate-maintaining, and Bayesian-updating strategies—a conceptual precursor to RFM architectures.

- **Kernel Extension and Model Generalization**:
  RFMs are compatible with diverse kernels, including Laplace, Matern, Gaussian, and rational quadratic [2411.14079]. AGOP-based adaptation introduces feature learning into previously static kernel machines and is generalizable to non-neural, non-gradient-based models.

## 6. Open Questions and Future Directions

Research into RFMs suggests several future directions and open problems:

- **Theoretical Analysis**:
  Further theorization is needed to fully understand the double descent MSE pattern and emergent phase transitions in RFMs, especially parallels with neural networks and random feature models [2303.15745, 2407.20199].

- **Algorithmic Extension**:
  Investigating RFMs with richer recursive structures (multi-way splits, deeper trees, higher-order coupling), broader kernel types, and alternative domains (e.g., modular learning, sequence modeling, generative models) presents promising avenues for performance and interpretability enhancement.

- **Practical Deployment**:
  Expansion into real-time and controlled generation, large-scale data applications, and high-dimensional function approximation [2510.19127, 2411.14079, 2508.10053] is ongoing.

- **Unification with Neural Feature Learning**:
  The alignment between AGOP and neural feature matrices (NFM), as formalized in the Deep Neural Feature Ansatz [2212.13881], merits further empirical and theoretical exploration—potentially informing broader principles behind feature learning and generalization in machine learning.

## 7. Representative Mathematical Formulas and Algorithmic Components

| Mechanism              | Formula/Algorithmic Step                                             | Context         |
|------------------------|---------------------------------------------------------------------|-----------------|
| Risk-RFE               | $\mathcal{R}^{\text{reg},\lambda}_{L,D,H}(f)$, $H^J$ projection     | Feature Elimination in Kernel Machines [1304.5245] |
| AGOP Update            | $M \leftarrow \frac{1}{n}\sum_{p=1}^n \nabla f(x_p)\nabla f(x_p)^T$ | Kernel Feature Learning [2212.13881], Interpretable Modeling [2411.14079] |
| Tree Split Direction   | $v = \text{top eigenvector of AGOP}(f, S)$                          | xRFM Tabular Partitioning [2508.10053] |
| Music Generation Control | $h'_t,\ell = h_{t,\ell} + \eta_\ell(t) q_{\ell,j^*}$                | Steering MusicGen via RFMs [2510.19127] |
| Low-rank Matrix Filter | $M_{t+1} = \phi(M_t^T W_t^T W_t M_t)$, power spectral update         | Matrix Recovery [2401.04553] |
| Conformal Elimination  | Remove feature with max $\beta_j$, $\beta_j = \sum_i \phi(z_i, \mathcal{Z}, j)$ | Conformal RFMs [2405.19429] |

## Conclusion

Recursive Feature Machines comprise a systematic and theoretically robust family of algorithms for feature selection, adaptation, and learning. By leveraging empirical risk increments, gradient outer products, and recursive mechanisms, RFMs achieve strong generalization, interpretability, and scalability across a range of machine learning applications. Their mathematical transparency and adaptability make RFMs essential tools for modern machine learning, with ongoing research continuing to expand their theoretical and practical horizons.

Source: https://www.emergentmind.com/topics/recursive-feature-machines-rfms