Feature-Driven Reinforcement Learning

• Feature-driven reinforcement learning is a framework where feature maps convert raw observation histories into finite, abstract state representations to support efficient decision-making.
• It applies reinforcement learning to operate directly over features, enabling the selection, transformation, and generation of informative attributes for downstream tasks.
• The approach has proven effective in various domains, from robotics and control to automated feature engineering and classification, addressing practical challenges in complexity and computational efficiency.

Feature-driven reinforcement learning is a family of reinforcement-learning formulations in which features are not treated as incidental preprocessing artifacts, but as the primary objects of abstraction, control, search, or reward construction. In the earliest formulation, Feature Reinforcement Learning defined a map $\phi:H\to S$ from histories to a finite state space so that non-Markovian interaction histories could be reduced to an induced MDP (0906.1713). Later work broadened the term to include wrapper-style feature selection and feature engineering, where an RL agent selects, transforms, or composes input variables for a downstream predictor (Khurana et al., 2017, Rasoul et al., 2021, Wu et al., 2022, Jahed et al., 2024), as well as methods in which learned representations such as successor features, temporal tokens, or feature points directly organize exploration, planning, task inference, and control (Boney et al., 2021, Hoang et al., 2021, Han et al., 2022, Ye et al., 31 Aug 2025). The resulting literature is technically heterogeneous but conceptually unified by one premise: the feature space itself becomes part of the sequential decision process.

1. Conceptual scope

The foundational sense of feature-driven RL is representational. In Feature Reinforcement Learning, the central problem is to learn a feature map $\phi$ that compresses raw observation histories into a finite internal state representation while retaining reward-relevant structure (0906.1713). This formulation treats “features” as learned state abstractions: the agent does not merely act in a given state space, but learns which aspects of history should count as state.

A second, later sense is operational. In wrapper-style feature selection and feature engineering, the RL agent acts over the feature space itself. The state may be a partial subset of selected variables, a transformation graph, a feature cluster, or an evolving roadmap of generated features; the actions correspond to selecting, deselecting, transforming, or combining features; and the reward is derived from downstream predictive performance (Khurana et al., 2017, Rasoul et al., 2021, 2505.12628, Huang et al., 24 Apr 2025, Xiao et al., 4 Jul 2025). In this usage, the feature space becomes the decision space.

A third sense is endogenous representation learning inside RL. Here the features are neither fixed inputs nor merely selected subsets, but learned latent structures that drive control. FPAC learns differentiable image feature points end-to-end from the SAC critic loss, with no unsupervised pretraining, no decoder, and no auxiliary feature-learning loss (Boney et al., 2021). Successor-feature methods use learned $\phi$, reward weights $\omega$, and successor features $\psi^\pi$ to organize transfer, task inference, novelty estimation, and planning (Han et al., 2022, Hoang et al., 2021). TPRL-DG recasts feature extraction as an autoregressive policy over temporal tokens for cross-user activity recognition (Ye et al., 31 Aug 2025).

This breadth suggests that “feature-driven reinforcement learning” is not a single algorithmic school but an umbrella term spanning at least three recurring motifs: RL for discovering features, RL over features, and RL through learned feature representations. A plausible implication is that definitional ambiguity is intrinsic to the literature rather than a defect of terminology.

2. Formal problem formulations

The most general formulation is the $\Phi$MDP reduction. Given history $h_t$, the agent induces state $s_t=\phi(h_t)$ and evaluates the feature map by a minimum-description-length objective that trades off state-model complexity, reward predictability, and feature-map complexity (0906.1713):

$$Cost(\phi \mid h_n) := CL(s_{1:n}\mid a_{1:n}) + CL(r_{1:n}\mid s_{1:n},a_{1:n}) + CL(\phi).$$

The best feature map is the minimizer of this criterion, and the induced finite-state process is then handled with standard MDP estimation and control. The later PhiMDP implementation restricted $\phi$ to context-tree classes, searched over Markov action-observation context trees, and then applied Q-learning on the induced MDP (Nguyen et al., 2011).

In wrapper-style feature selection, the MDP is typically defined over subsets. One explicit formulation sets the state to a selected subset $\phi$0, the action to choosing an unselected feature $\phi$1, and the transition to $\phi$2 (Rasoul et al., 2021). The reward is the incremental change in classifier accuracy,

$\phi$3

so the policy searches directly for subsets that improve predictive utility. A related but simpler formulation treats each feature as a state and offers two actions, “select” and “do not select,” with reward tied to classifier accuracy; in that system, accuracy above $\phi$4 doubles the reward, while selecting all features or no features incurs a punishment of $\phi$5 (Jahed et al., 2024).

Automated feature engineering generalizes the subset view to transformation search. In the transformation-graph formulation, the state is $\phi$6, where $\phi$7 is the current graph and $\phi$8 is the consumed budget fraction; an action is a pair $\phi$9 consisting of an existing node and an applicable transformation (Khurana et al., 2017). The immediate reward is the gain in the best observed performance after adding a new node:

$\phi$0

The objective is therefore not local transform quality but graph-level improvement under budget.

Recent feature-generation frameworks factorize the action space into staged decisions. DARL uses one agent for feature generation and a second for feature preservation, with the first deciding transformations and the second deciding whether generated features are deleted, replaced, or added (2505.12628). TCTO uses a head-cluster agent, operation agent, and operand-cluster agent to navigate an evolving feature-transformation roadmap (Huang et al., 24 Apr 2025). MAFG similarly decomposes feature generation into cluster selection, operator selection, and optional second-cluster selection, with reward equal to downstream performance improvement (Xiao et al., 4 Jul 2025). In all three cases, sequential decomposition is used to manage combinatorial growth.

3. Wrapper-style feature selection and automated feature engineering

A major branch of feature-driven RL treats feature selection and feature construction as wrapper optimization. The 2017 transformation-graph framework automated feature engineering by exploring mathematical transformations under a learned RL policy rather than handcrafted heuristics (Khurana et al., 2017). It reported that RL$\phi$1 was best on most of 24 datasets, that the median error reduction across the 24 datasets was $\phi$2, that RL-based strategies were $\phi$3–$\phi$4 times more efficient than handcrafted breadth-first, depth-first, and global search heuristics in finding the optimal dataset in a bounded graph, and that including feature selection as a transformation improved final gain by about $\phi$5.

Subsequent feature-selection papers made the wrapper loop more explicit. A TD(0)-based formulation used an SVM with Gaussian kernel to score subsets, defined feature utility through an Average of Reward statistic, and reported accuracies of $\phi$6 on Australian, $\phi$7 on WPBC, and $\phi$8 on Connectionist Bench (Rasoul et al., 2021). An interactive multi-agent formulation with decision tree in the loop encoded selected subsets as both an undirected feature-correlation graph and a directed decision-tree structure, personalized reward assignment through decision-tree feature importance or historical selection frequency, and achieved the best overall performance on all eight reported datasets (Fan et al., 2020). A later synthesis paper treated these systems as instances of a broader continuum including single-agent scanning, Monte Carlo-based reinforced feature selection, dual-agent feature-instance selection, cascading feature generation, and bandit-style subset search (Nagaraju, 15 Mar 2025).

DroidRL extended wrapper-based feature selection to high-dimensional Android malware detection by using DDQN, an RNN-family decision network, $\phi$9-greedy exploration, and word embeddings for feature representation (Wu et al., 2022). The strongest result was $\omega$0 accuracy with Random Forest using only 24 features out of 1083, together with a $\omega$1 reduction in computational overhead when LSTM was used as the decision network. The framework thus retained wrapper feedback while avoiding exhaustive subset search.

A smaller but sharply specified medical example formulated feature selection on the Breast Cancer Coimbra dataset as a control problem solved with tabular Q-learning and SARSA (Jahed et al., 2024). Using Min-Max, $\omega$2, and $\omega$3 normalization, the study reported that QL@Min-Max reached $\omega$4 accuracy and SARSA@$\omega$5 reached $\omega$6, with $\omega$7 normalization producing no false negatives in the reported results and QL@Min-Max having the lowest false positives and false negatives among the displayed setups. The setup was deliberately simple: nine features, two actions per feature, reward from classifier accuracy, 1000 episodes, $\omega$8, $\omega$9, and punishment value $\psi^\pi$0.

The same wrapper logic has recently been generalized into full automated feature engineering. CAFE first learns a NOTEARS-Lasso DAG, groups variables into direct, indirect, and other causal roles, and then uses cascading multi-agent DQN with causal reward shaping and adaptive exploration (Malarkkan et al., 18 Feb 2026). It reported being best on 13 of 15 datasets, improvements up to about $\psi^\pi$1 over strong baselines, reaching $\psi^\pi$2 of final performance in $\psi^\pi$3 episodes on average versus $\psi^\pi$4 episodes for GRFG, and about a $\psi^\pi$5 reduction in degradation under covariate shift relative to a non-causal multi-agent baseline. DARL separated feature generation from preservation and used self-attention to encode the tabular state (2505.12628). TCTO made the search space explicitly graph-structured and reusable through pruning, backtracking, and path reuse (Huang et al., 24 Apr 2025). MAFG added a post hoc LLM-based interpretive assessment layer to RL-guided feature generation for scientific data (Xiao et al., 4 Jul 2025). These later systems preserve the wrapper principle but enlarge both the search space and the internal structure of the RL environment.

4. Learned feature representations inside reinforcement learning

A distinct branch of the literature uses learned features to organize RL itself rather than to preprocess a supervised learner. The successor-feature line is central here. In meta-RL with successor-feature-based context, rewards are decomposed as

$\psi^\pi$6

and the successor feature

$\psi^\pi$7

captures policy-dependent future feature occupancy (Han et al., 2022). Context $\psi^\pi$8 is then produced from $\psi^\pi$9 and predicted reward weights $\Phi$0 via a GRU-based encoder and used to condition the policy. The reported effect was competitive meta-RL performance with significantly fewer environmental interactions than state-of-the-art baselines.

Successor Feature Landmarks pushed the same representation into exploration and planning for long-horizon goal-conditioned RL (Hoang et al., 2021). It defined Successor Feature Similarity as the inner product of successor features, used low similarity to all landmarks as a novelty criterion, built a non-parametric landmark graph online, and derived local goal-conditioned control directly from successor-feature similarity. On ViZDoom hard settings, the reported examples included $\Phi$1 success on SGM-Map Hard versus $\Phi$2 for SGM, $\Phi$3 on Test-2 Hard versus $\Phi$4, and $\Phi$5 on Test-6 Hard versus $\Phi$6.

In reinforcement learning from images, FPAC replaced generic latent vectors with differentiable feature points extracted directly from images (Boney et al., 2021). For an image $\Phi$7, the representation is

$\Phi$8

and the actor and critic consume $\Phi$9. The method used SAC, learned the extractor end-to-end through critic gradients, and required no unsupervised pretraining, no decoder, and no auxiliary feature-learning loss. It was reported as competitive with state of the art on DeepMind Control Suite tasks, with training on Cartpole Swingup for 500 episodes taking about 76 min 56 sec versus 70 min 41 sec for SAC from pixels on an NVIDIA V100.

Feature learning also appears in reward learning. FERL first learns features from human-provided feature traces and only then learns a linear reward over those features (Bobu et al., 2022). The representation stage uses pairwise ordering and equivalence constraints over traces to train neural feature models; the reward stage then applies standard reward-learning machinery on top of the learned feature vector. The paper reported decreased sample complexity and improved generalization of the learned reward over a deepIRL baseline in experiments with a physical 7DOF robot manipulator and a simulated user study.

More recent work generalizes feature learning to temporal domain generalization. TPRL-DG treats feature extraction itself as a sequential policy over temporal tokens, generated autoregressively by a Transformer and optimized with PPO under a multi-objective reward for class discrimination and cross-user invariance (Ye et al., 31 Aug 2025). On leave-one-group-out evaluation, it reported $h_t$0 accuracy on DSADS and $h_t$1 on PAMAP2, surpassing the stated baselines for cross-user generalization.

5. Algorithms, architectural patterns, and credit assignment

Algorithmically, feature-driven RL spans nearly the full contemporary RL toolkit. Early systems used tabular or linear-function-approximation Q-learning. The transformation-graph framework employed Q-learning with function approximation in two variants, RL$h_t$2 and RL$h_t$3 (Khurana et al., 2017). Feature-selection work employed TD(0) with online value updates (Rasoul et al., 2021), Q-learning and SARSA with standard tabular updates (Jahed et al., 2024), and Q-learning over induced MDPs after context-tree search in PhiMDP (Nguyen et al., 2011).

Deep value-based methods dominate larger feature spaces. DroidRL used DDQN with replay memory, target networks, an $h_t$4-greedy schedule, and recurrent decision networks, comparing RNN, GRU, and LSTM (Wu et al., 2022). DARL used DQN for both the generation and discrimination agents, while its second-stage reward combined task improvement with mutual-information terms for deletion, replacement, and addition (2505.12628). CAFE used multi-agent DQN with experience replay, hard target updates, and a reward that multiplies validation improvement by a causal bonus and supplements it with entropy and complexity terms (Malarkkan et al., 18 Feb 2026).

Policy-gradient methods arise when the features themselves are continuous or sequentially generated. FPAC used SAC on differentiable feature points (Boney et al., 2021). TPRL-DG used PPO over autoregressive token generation (Ye et al., 31 Aug 2025). RLDE-AFL combined PPO with an attention-based feature extractor that learns optimization-landscape representations from evolving differential-evolution populations, using mantissa-exponent embeddings for objective values and co-training the feature learner with the dynamic algorithm-configuration policy (Guo et al., 23 Mar 2025). A further application to photovoltaic intraday trading used PPO with a predominantly linear Gaussian policy whose mean is a feature-weighted action recommendation, emphasizing interpretability and real-time inference (Abate et al., 15 Oct 2025).

Architecturally, the literature exhibits several recurrent motifs. Recurrent networks appear when the state is a variable-length sequence of selected features or temporal tokens (Wu et al., 2022, Ye et al., 31 Aug 2025). Graph neural networks appear when feature correlations, decision hierarchies, or transformation roadmaps define the state geometry (Fan et al., 2020, Huang et al., 24 Apr 2025). Self-attention is used both for tabular state representation in DARL and for population-level landscape encoding in RLDE-AFL (2505.12628, Guo et al., 23 Mar 2025). Decision trees are often not only evaluators but feedback sources, providing importance scores, tree-structured state overlays, and trainer signals (Fan et al., 2020). Across these systems, reward shaping typically combines performance terms with penalties for redundancy, complexity, or degenerate actions.

6. Domains of application and empirical profile

The empirical footprint of feature-driven RL is unusually broad. In tabular predictive modeling, the literature covers UCI, Kaggle, LibSVM, OpenML, and domain-specific scientific datasets, with downstream metrics including F-score, macro-F1, AUROC, $h_t$5, and weighted F1 (Khurana et al., 2017, Xiao et al., 4 Jul 2025, Malarkkan et al., 18 Feb 2026). In Android malware detection, the method is explicitly framed as RL-driven feature selection for a very large static feature dictionary (Wu et al., 2022). In medical classification, RL is used to search compact feature subsets while tracking false positives and false negatives (Jahed et al., 2024).

In robotics and control, the emphasis shifts from feature search to feature-based control. FPAC applies learned spatial feature points to continuous-control tasks from images (Boney et al., 2021). SFL applies successor-feature abstractions to long-horizon navigation in MiniGrid and ViZDoom (Hoang et al., 2021). Meta-RL with successor-feature context targets MuJoCo task families such as Ant-Fwd-Back, Ant-Goal, HalfCheetah-Vel, and HalfCheetah-Fwd-Back (Han et al., 2022). FERL studies reward learning on a physical 7DOF manipulator (Bobu et al., 2022).

In geometry-driven optimization, Reinforced Labels formulates point-feature label placement as a multi-agent PPO problem over local geometric observations rather than raw pixels (Bobák et al., 2023). It introduced completeness as its primary metric and reported examples of around $h_t$6 completeness at 50 anchors and around $h_t$7 completeness at 600 anchors, while also noting substantially higher computation time than handcrafted comparators. In human activity recognition, TPRL-DG used RL-driven feature generation for user-invariant temporal representations and reported gains over ERM, RSC, ANDMask, AdaRNN, and ACON on DSADS and PAMAP2 (Ye et al., 31 Aug 2025). In photovoltaic intraday trading, feature-driven PPO achieved 311.99 kEUR profit versus 310.46 kEUR for the spot-only baseline, with mean single-step latency of 0.038 ms and full 24-step episode latency of 2.3 ms (Abate et al., 15 Oct 2025).

This cross-domain spread shows that feature-driven RL is not tied to a particular modality. The commonality is structural: whether the input is tabular scientific data, malware permissions, image frames, wearable-sensor windows, optimization populations, or market microstructure, the method assigns sequential decision-making authority to the representation layer.

7. Limitations, misconceptions, and open directions

A recurring misconception is that feature-driven RL is synonymous with feature selection. The literature is wider. PhiMDP and Feature Reinforcement Learning are concerned with state abstraction from history (0906.1713, Nguyen et al., 2011); FPAC learns feature points for control (Boney et al., 2021); successor-feature systems use feature decompositions to support transfer, planning, and task inference (Han et al., 2022, Hoang et al., 2021); and FERL learns features as an intermediate target for reward learning (Bobu et al., 2022). Feature selection is therefore one branch, not the whole field.

Another misconception is that all such systems rely on hand-crafted features. Some do rely on designed descriptors, such as the ray-based mapping and self-aware vectors in label placement (Bobák et al., 2023), but others learn features end-to-end through the RL objective, including FPAC’s keypoints, RLDE-AFL’s landscape encoder, and TPRL-DG’s temporal tokens (Boney et al., 2021, Guo et al., 23 Mar 2025, Ye et al., 31 Aug 2025). The field therefore contains both engineered-feature and learned-feature traditions.

The limitations are equally recurrent. Search spaces are often explicitly combinatorial: the transformation graph can reach about 3.2 million nodes with 20 transformations and height 5 (Khurana et al., 2017); subset selection ranges over the power set (Rasoul et al., 2021); Android malware feature selection begins from 1083 features (Wu et al., 2022); and CAFE notes a raw action space of roughly $h_t$8 (Malarkkan et al., 18 Feb 2026). Computational cost is therefore a first-order concern. Several papers frame their contribution as a response to inefficiency rather than only to accuracy.

Reward design is another persistent difficulty. In many wrapper systems, the reward is classifier accuracy or validation-score improvement, sometimes with auxiliary penalties for redundancy or complexity (Rasoul et al., 2021, Jahed et al., 2024, 2505.12628). This means the learned feature policy is often model-dependent: different downstream evaluators can induce different feature subsets. Some papers acknowledge evaluation caveats directly; one feature-selection study states that it cannot directly compare to other methods in a fully fair way because it selects the best subset and then evaluates accuracy based on that subset (Rasoul et al., 2021). This suggests that standardized benchmarking remains underdeveloped.

Interpretability is also uneven. Many feature-engineering systems emphasize explicit mathematical transformations, graph traceability, decision-tree importance, or causal grouping (Fan et al., 2020, Huang et al., 24 Apr 2025, Malarkkan et al., 18 Feb 2026). Yet feature-point methods note that learned points are not always naturally interpretable as human semantic landmarks (Boney et al., 2021), and broader surveys note that RL-based feature selection can still behave like a black box (Nagaraju, 15 Mar 2025). The field therefore does not present a simple interpretability hierarchy from classical to RL-based methods.

Open directions are explicit in the papers. Feature Reinforcement Learning points to structured dynamic Bayesian networks and POMDP extensions beyond unstructured finite MDPs (0906.1713). FPAC proposes learning feature-point dynamics models and 3D feature points from monocular or multi-view images (Boney et al., 2021). Papers about automated feature selection call for transfer learning, continual learning, fairness-aware RL, privacy-preserving or federated settings, and better support for multi-modal heterogeneous data (Nagaraju, 15 Mar 2025). Causal approaches argue for soft inductive priors rather than rigid causal constraints, using adaptive exploration to remain robust when discovery is imperfect (Malarkkan et al., 18 Feb 2026). Collectively, these directions indicate that the future of feature-driven RL lies less in a single dominant formulation than in tighter integration among state abstraction, representation learning, causal structure, and computationally tractable search.