---
title: Feature-Driven Reinforcement Learning
url: https://www.emergentmind.com/topics/feature-driven-reinforcement-learning
type: topic
---

# Feature-Driven Reinforcement Learning

Feature-driven reinforcement learning is a family of reinforcement-learning formulations in which features are not treated as incidental preprocessing artifacts, but as the primary objects of abstraction, control, search, or reward construction. In the earliest formulation, Feature Reinforcement Learning defined a map $\phi:H\to S$ from histories to a finite state space so that non-Markovian interaction histories could be reduced to an induced MDP [0906.1713]. Later work broadened the term to include wrapper-style feature selection and feature engineering, where an RL agent selects, transforms, or composes input variables for a downstream predictor [1709.07150, 2101.09460, 2203.02719, 2403.05979], as well as methods in which learned representations such as successor features, temporal tokens, or feature points directly organize exploration, planning, task inference, and control [2106.07995, 2111.09858, 2207.14723, 2509.01031]. The resulting literature is technically heterogeneous but conceptually unified by one premise: the feature space itself becomes part of the sequential decision process.

## 1. Conceptual scope

The foundational sense of feature-driven RL is representational. In Feature Reinforcement Learning, the central problem is to learn a feature map $\phi$ that compresses raw observation histories into a finite internal state representation while retaining reward-relevant structure [0906.1713]. This formulation treats “features” as learned state abstractions: the agent does not merely act in a given state space, but learns which aspects of history should count as state.

A second, later sense is operational. In wrapper-style feature selection and feature engineering, the RL agent acts over the feature space itself. The state may be a partial subset of selected variables, a transformation graph, a feature cluster, or an evolving roadmap of generated features; the actions correspond to selecting, deselecting, transforming, or combining features; and the reward is derived from downstream predictive performance [1709.07150, 2101.09460, 2505.12628, 2504.17355, 2507.03498]. In this usage, the feature space becomes the decision space.

A third sense is endogenous representation learning inside RL. Here the features are neither fixed inputs nor merely selected subsets, but learned latent structures that drive control. FPAC learns differentiable image feature points end-to-end from the SAC critic loss, with no unsupervised pretraining, no decoder, and no auxiliary feature-learning loss [2106.07995]. Successor-feature methods use learned $\phi$, reward weights $\omega$, and successor features $\psi^\pi$ to organize transfer, task inference, novelty estimation, and planning [2207.14723, 2111.09858]. TPRL-DG recasts feature extraction as an autoregressive policy over temporal tokens for cross-user activity recognition [2509.01031].

This breadth suggests that “feature-driven reinforcement learning” is not a single algorithmic school but an umbrella term spanning at least three recurring motifs: RL for discovering features, RL over features, and RL through learned feature representations. A plausible implication is that definitional ambiguity is intrinsic to the literature rather than a defect of terminology.

## 2. Formal problem formulations

The most general formulation is the $\Phi$MDP reduction. Given history $h_t$, the agent induces state $s_t=\phi(h_t)$ and evaluates the feature map by a minimum-description-length objective that trades off state-model complexity, reward predictability, and feature-map complexity [0906.1713]:
$$
Cost(\phi \mid h_n) := CL(s_{1:n}\mid a_{1:n}) + CL(r_{1:n}\mid s_{1:n},a_{1:n}) + CL(\phi).
$$
The best feature map is the minimizer of this criterion, and the induced finite-state process is then handled with standard MDP estimation and control. The later PhiMDP implementation restricted $\phi$ to context-tree classes, searched over Markov action-observation context trees, and then applied Q-learning on the induced MDP [1108.3614].

In wrapper-style feature selection, the MDP is typically defined over subsets. One explicit formulation sets the state to a selected subset $F\subseteq \psi$, the action to choosing an unselected feature $f\in \psi\setminus F$, and the transition to $F' = F \cup \{f\}$ [2101.09460]. The reward is the incremental change in classifier accuracy,
$$
Reward_f = [Accuracy]_{t+1} - [Accuracy]_t,
$$
so the policy searches directly for subsets that improve predictive utility. A related but simpler formulation treats each feature as a state and offers two actions, “select” and “do not select,” with reward tied to classifier accuracy; in that system, accuracy above $70\%$ doubles the reward, while selecting all features or no features incurs a punishment of $0.8$ [2403.05979].

Automated feature engineering generalizes the subset view to transformation search. In the transformation-graph formulation, the state is $(G_i,b_{ratio})$, where $G_i$ is the current graph and $b_{ratio}=i/B_{\max}$ is the consumed budget fraction; an action is a pair $\langle n,t\rangle$ consisting of an existing node and an applicable transformation [1709.07150]. The immediate reward is the gain in the best observed performance after adding a new node:
$$
r_i = \max_{n' \in \theta(G_{i+1})} A(n') - \max_{n \in \theta(G_i)} A(n).
$$
The objective is therefore not local transform quality but graph-level improvement under budget.

Recent feature-generation frameworks factorize the action space into staged decisions. DARL uses one agent for feature generation and a second for feature preservation, with the first deciding transformations and the second deciding whether generated features are deleted, replaced, or added [2505.12628]. TCTO uses a head-cluster agent, operation agent, and operand-cluster agent to navigate an evolving feature-transformation roadmap [2504.17355]. MAFG similarly decomposes feature generation into cluster selection, operator selection, and optional second-cluster selection, with reward equal to downstream performance improvement [2507.03498]. In all three cases, sequential decomposition is used to manage combinatorial growth.

## 3. Wrapper-style feature selection and automated feature engineering

A major branch of feature-driven RL treats feature selection and feature construction as wrapper optimization. The 2017 transformation-graph framework automated feature engineering by exploring mathematical transformations under a learned RL policy rather than handcrafted heuristics [1709.07150]. It reported that RL$_1$ was best on most of 24 datasets, that the median error reduction across the 24 datasets was $23.8\%$, that RL-based strategies were $4$–$8$ times more efficient than handcrafted breadth-first, depth-first, and global search heuristics in finding the optimal dataset in a bounded graph, and that including feature selection as a transformation improved final gain by about $51\%$.

Subsequent feature-selection papers made the wrapper loop more explicit. A TD(0)-based formulation used an SVM with Gaussian kernel to score subsets, defined feature utility through an Average of Reward statistic, and reported accuracies of $85.55 \pm 0.039\%$ on Australian, $76.29 \pm 0.007\%$ on WPBC, and $73.69 \pm 0.108\%$ on Connectionist Bench [2101.09460]. An interactive multi-agent formulation with decision tree in the loop encoded selected subsets as both an undirected feature-correlation graph and a directed decision-tree structure, personalized reward assignment through decision-tree feature importance or historical selection frequency, and achieved the best overall performance on all eight reported datasets [2010.02506]. A later synthesis paper treated these systems as instances of a broader continuum including single-agent scanning, Monte Carlo-based reinforced feature selection, dual-agent feature-instance selection, cascading feature generation, and bandit-style subset search [2503.11991].

DroidRL extended wrapper-based feature selection to high-dimensional Android malware detection by using DDQN, an RNN-family decision network, $\varepsilon$-greedy exploration, and word embeddings for feature representation [2203.02719]. The strongest result was $95.6\%$ accuracy with Random Forest using only 24 features out of 1083, together with a $97.78\%$ reduction in computational overhead when LSTM was used as the decision network. The framework thus retained wrapper feedback while avoiding exhaustive subset search.

A smaller but sharply specified medical example formulated feature selection on the Breast Cancer Coimbra dataset as a control problem solved with tabular Q-learning and SARSA [2403.05979]. Using Min-Max, $l_1$, and $l_2$ normalization, the study reported that QL@Min-Max reached $87\%$ accuracy and SARSA@$l_2$ reached $88\%$, with $l_2$ normalization producing no false negatives in the reported results and QL@Min-Max having the lowest false positives and false negatives among the displayed setups. The setup was deliberately simple: nine features, two actions per feature, reward from classifier accuracy, 1000 episodes, $\alpha = 0.03$, $\gamma = 1$, and punishment value $0.8$.

The same wrapper logic has recently been generalized into full automated feature engineering. CAFE first learns a NOTEARS-Lasso DAG, groups variables into direct, indirect, and other causal roles, and then uses cascading multi-agent DQN with causal reward shaping and adaptive exploration [2602.16435]. It reported being best on 13 of 15 datasets, improvements up to about $7\%$ over strong baselines, reaching $95\%$ of final performance in $18.3$ episodes on average versus $28.4$ episodes for GRFG, and about a $4\times$ reduction in degradation under covariate shift relative to a non-causal multi-agent baseline. DARL separated feature generation from preservation and used self-attention to encode the tabular state [2505.12628]. TCTO made the search space explicitly graph-structured and reusable through pruning, backtracking, and path reuse [2504.17355]. MAFG added a post hoc LLM-based interpretive assessment layer to RL-guided feature generation for scientific data [2507.03498]. These later systems preserve the wrapper principle but enlarge both the search space and the internal structure of the RL environment.

## 4. Learned feature representations inside reinforcement learning

A distinct branch of the literature uses learned features to organize RL itself rather than to preprocess a supervised learner. The successor-feature line is central here. In meta-RL with successor-feature-based context, rewards are decomposed as
$$
R(s,a)=\phi(s,a)^\top \omega,
$$
and the successor feature
$$
\psi^{\pi}(s)=\mathbb{E}_{\pi}\bigg[\sum_{t=0}^\infty \gamma^{t-1}\phi_{t+1}\mid s_0=s \bigg]
$$
captures policy-dependent future feature occupancy [2207.14723]. Context $z$ is then produced from $\psi$ and predicted reward weights $\omega'$ via a GRU-based encoder and used to condition the policy. The reported effect was competitive meta-RL performance with significantly fewer environmental interactions than state-of-the-art baselines.

Successor Feature Landmarks pushed the same representation into exploration and planning for long-horizon goal-conditioned RL [2111.09858]. It defined Successor Feature Similarity as the inner product of successor features, used low similarity to all landmarks as a novelty criterion, built a non-parametric landmark graph online, and derived local goal-conditioned control directly from successor-feature similarity. On ViZDoom hard settings, the reported examples included $67\%$ success on SGM-Map Hard versus $26\%$ for SGM, $48\%$ on Test-2 Hard versus $32\%$, and $60\%$ on Test-6 Hard versus $27\%$.

In reinforcement learning from images, FPAC replaced generic latent vectors with differentiable feature points extracted directly from images [2106.07995]. For an image $\mathbf{o}$, the representation is
$$
\mathbf{x} = \Phi(\mathbf{o}) = [(x_1,y_1,m_1),\ldots,(x_K,y_K,m_K)],
$$
and the actor and critic consume $[\mathbf{x}_t,\mathbf{x}_t-\mathbf{x}_{t-1}]$. The method used SAC, learned the extractor end-to-end through critic gradients, and required no unsupervised pretraining, no decoder, and no auxiliary feature-learning loss. It was reported as competitive with state of the art on DeepMind Control Suite tasks, with training on Cartpole Swingup for 500 episodes taking about 76 min 56 sec versus 70 min 41 sec for SAC from pixels on an NVIDIA V100.

Feature learning also appears in reward learning. FERL first learns features from human-provided feature traces and only then learns a linear reward over those features [2201.07082]. The representation stage uses pairwise ordering and equivalence constraints over traces to train neural feature models; the reward stage then applies standard reward-learning machinery on top of the learned feature vector. The paper reported decreased sample complexity and improved generalization of the learned reward over a deepIRL baseline in experiments with a physical 7DOF robot manipulator and a simulated user study.

More recent work generalizes feature learning to temporal domain generalization. TPRL-DG treats feature extraction itself as a sequential policy over temporal tokens, generated autoregressively by a Transformer and optimized with PPO under a multi-objective reward for class discrimination and cross-user invariance [2509.01031]. On leave-one-group-out evaluation, it reported $88.29\%$ accuracy on DSADS and $74.15\%$ on PAMAP2, surpassing the stated baselines for cross-user generalization.

## 5. Algorithms, architectural patterns, and credit assignment

Algorithmically, feature-driven RL spans nearly the full contemporary RL toolkit. Early systems used tabular or linear-function-approximation Q-learning. The transformation-graph framework employed Q-learning with function approximation in two variants, RL$_1$ and RL$_2$ [1709.07150]. Feature-selection work employed TD(0) with online value updates [2101.09460], Q-learning and SARSA with standard tabular updates [2403.05979], and Q-learning over induced MDPs after context-tree search in PhiMDP [1108.3614].

Deep value-based methods dominate larger feature spaces. DroidRL used DDQN with replay memory, target networks, an $\varepsilon$-greedy schedule, and recurrent decision networks, comparing RNN, GRU, and LSTM [2203.02719]. DARL used DQN for both the generation and discrimination agents, while its second-stage reward combined task improvement with mutual-information terms for deletion, replacement, and addition [2505.12628]. CAFE used multi-agent DQN with experience replay, hard target updates, and a reward that multiplies validation improvement by a causal bonus and supplements it with entropy and complexity terms [2602.16435].

Policy-gradient methods arise when the features themselves are continuous or sequentially generated. FPAC used SAC on differentiable feature points [2106.07995]. TPRL-DG used PPO over autoregressive token generation [2509.01031]. RLDE-AFL combined PPO with an attention-based feature extractor that learns optimization-landscape representations from evolving differential-evolution populations, using mantissa-exponent embeddings for objective values and co-training the feature learner with the dynamic algorithm-configuration policy [2503.18061]. A further application to photovoltaic intraday trading used PPO with a predominantly linear Gaussian policy whose mean is a feature-weighted action recommendation, emphasizing interpretability and real-time inference [2510.16021].

Architecturally, the literature exhibits several recurrent motifs. Recurrent networks appear when the state is a variable-length sequence of selected features or temporal tokens [2203.02719, 2509.01031]. Graph neural networks appear when feature correlations, decision hierarchies, or transformation roadmaps define the state geometry [2010.02506, 2504.17355]. Self-attention is used both for tabular state representation in DARL and for population-level landscape encoding in RLDE-AFL [2505.12628, 2503.18061]. Decision trees are often not only evaluators but feedback sources, providing importance scores, tree-structured state overlays, and trainer signals [2010.02506]. Across these systems, reward shaping typically combines performance terms with penalties for redundancy, complexity, or degenerate actions.

## 6. Domains of application and empirical profile

The empirical footprint of feature-driven RL is unusually broad. In tabular predictive modeling, the literature covers UCI, Kaggle, LibSVM, OpenML, and domain-specific scientific datasets, with downstream metrics including F-score, macro-F1, AUROC, $1-\mathrm{RAE}$, and weighted F1 [1709.07150, 2507.03498, 2602.16435]. In Android malware detection, the method is explicitly framed as RL-driven feature selection for a very large static feature dictionary [2203.02719]. In medical classification, RL is used to search compact feature subsets while tracking false positives and false negatives [2403.05979].

In robotics and control, the emphasis shifts from feature search to feature-based control. FPAC applies learned spatial feature points to continuous-control tasks from images [2106.07995]. SFL applies successor-feature abstractions to long-horizon navigation in MiniGrid and ViZDoom [2111.09858]. Meta-RL with successor-feature context targets MuJoCo task families such as Ant-Fwd-Back, Ant-Goal, HalfCheetah-Vel, and HalfCheetah-Fwd-Back [2207.14723]. FERL studies reward learning on a physical 7DOF manipulator [2201.07082].

In geometry-driven optimization, Reinforced Labels formulates point-feature label placement as a multi-agent PPO problem over local geometric observations rather than raw pixels [2303.01388]. It introduced completeness as its primary metric and reported examples of around $89\%$ completeness at 50 anchors and around $64\%$ completeness at 600 anchors, while also noting substantially higher computation time than handcrafted comparators. In human activity recognition, TPRL-DG used RL-driven feature generation for user-invariant temporal representations and reported gains over ERM, RSC, ANDMask, AdaRNN, and ACON on DSADS and PAMAP2 [2509.01031]. In photovoltaic intraday trading, feature-driven PPO achieved 311.99 kEUR profit versus 310.46 kEUR for the spot-only baseline, with mean single-step latency of 0.038 ms and full 24-step episode latency of 2.3 ms [2510.16021].

This cross-domain spread shows that feature-driven RL is not tied to a particular modality. The commonality is structural: whether the input is tabular scientific data, malware permissions, image frames, wearable-sensor windows, optimization populations, or market microstructure, the method assigns sequential decision-making authority to the representation layer.

## 7. Limitations, misconceptions, and open directions

A recurring misconception is that feature-driven RL is synonymous with feature selection. The literature is wider. PhiMDP and Feature Reinforcement Learning are concerned with state abstraction from history [0906.1713, 1108.3614]; FPAC learns feature points for control [2106.07995]; successor-feature systems use feature decompositions to support transfer, planning, and task inference [2207.14723, 2111.09858]; and FERL learns features as an intermediate target for reward learning [2201.07082]. Feature selection is therefore one branch, not the whole field.

Another misconception is that all such systems rely on hand-crafted features. Some do rely on designed descriptors, such as the ray-based mapping and self-aware vectors in label placement [2303.01388], but others learn features end-to-end through the RL objective, including FPAC’s keypoints, RLDE-AFL’s landscape encoder, and TPRL-DG’s temporal tokens [2106.07995, 2503.18061, 2509.01031]. The field therefore contains both engineered-feature and learned-feature traditions.

The limitations are equally recurrent. Search spaces are often explicitly combinatorial: the transformation graph can reach about 3.2 million nodes with 20 transformations and height 5 [1709.07150]; subset selection ranges over the power set [2101.09460]; Android malware feature selection begins from 1083 features [2203.02719]; and CAFE notes a raw action space of roughly $O(|\mathcal{O}|\cdot|\mathcal{F}|^2)$ [2602.16435]. Computational cost is therefore a first-order concern. Several papers frame their contribution as a response to inefficiency rather than only to accuracy.

Reward design is another persistent difficulty. In many wrapper systems, the reward is classifier accuracy or validation-score improvement, sometimes with auxiliary penalties for redundancy or complexity [2101.09460, 2403.05979, 2505.12628]. This means the learned feature policy is often model-dependent: different downstream evaluators can induce different feature subsets. Some papers acknowledge evaluation caveats directly; one feature-selection study states that it cannot directly compare to other methods in a fully fair way because it selects the best subset and then evaluates accuracy based on that subset [2101.09460]. This suggests that standardized benchmarking remains underdeveloped.

Interpretability is also uneven. Many feature-engineering systems emphasize explicit mathematical transformations, graph traceability, decision-tree importance, or causal grouping [2010.02506, 2504.17355, 2602.16435]. Yet feature-point methods note that learned points are not always naturally interpretable as human semantic landmarks [2106.07995], and broader surveys note that RL-based feature selection can still behave like a black box [2503.11991]. The field therefore does not present a simple interpretability hierarchy from classical to RL-based methods.

Open directions are explicit in the papers. Feature Reinforcement Learning points to structured dynamic Bayesian networks and POMDP extensions beyond unstructured finite MDPs [0906.1713]. FPAC proposes learning feature-point dynamics models and 3D feature points from monocular or multi-view images [2106.07995]. Surveys on automated feature selection call for transfer learning, continual learning, fairness-aware RL, privacy-preserving or federated settings, and better support for multi-modal heterogeneous data [2503.11991]. Causal approaches argue for soft inductive priors rather than rigid causal constraints, using adaptive exploration to remain robust when discovery is imperfect [2602.16435]. Collectively, these directions indicate that the future of feature-driven RL lies less in a single dominant formulation than in tighter integration among state abstraction, representation learning, causal structure, and computationally tractable search.

Source: https://www.emergentmind.com/topics/feature-driven-reinforcement-learning