Feature MDPs: State Compression & RL

• Feature MDPs are formal frameworks that transform complex, non-Markovian observation histories into finite, Markovian state spaces via feature maps.
• They use MDL-based criteria to balance state-space complexity and reward prediction accuracy, ensuring tractable and optimal policy planning.
• Learning algorithms such as ADNN and stochastic local search efficiently discover and refine state representations in high-dimensional reinforcement learning tasks.

A Feature Markov Decision Process (Feature MDP, or $\Phi$MDP) is a formal framework that mechanizes the selection and evaluation of state representations for reinforcement learning (RL) from complex, non-Markovian observation sequences, by defining objective criteria for compressing observed histories into finite-dimensional Markovian state spaces. The central goal is to automate the discovery of feature maps $\Phi$ that transform observed history into a representation inducing a process amenable to efficient planning by standard MDP techniques, with rigorous guarantees on model selection and optimality (0812.4580, Wang et al., 2017).

1. Formal Definitions and State Compression

In sequential decision tasks, an agent interacts with its environment through discrete cycles: at each time $t$, the agent observes $o_t\in O$, takes action $a_t\in A$, and receives reward $r_t\in R$. The resulting history tuple $h_t = o_1a_1r_1\ldots a_{t-1}r_{t-1}o_t$ is an element of $H = (O\times A \times R)^*\times O$.

A Feature MDP is constructed by specifying a feature map $\Phi : H\to S$, yielding a compressed state $s_t=\Phi(h_t)\in S$. The process thus reduces the high-dimensional, possibly non-Markovian history $\Phi$0 to a finite set of states suitable for the MDP formalism. The map $\Phi$1 is to be selected such that the resulting sequence $\Phi$2 closely satisfies the Markov property, making the RL problem tractable via standard state-based algorithms (0812.4580).

A parallel, general formulation considers the original state space $\Phi$3 (potentially continuous), action space $\Phi$4, transition kernel $\Phi$5, and reward function $\Phi$6. A measurable feature map $\Phi$7 defines a reduced process $\Phi$8, where

$\Phi$9

$t$0

(Wang et al., 2017).

2. Objective Criteria for Feature Map Selection

The quality of a feature map is evaluated using compressibility criteria grounded in Minimum Description Length (MDL) principles. For a given $t$1 and a collected history $t$2, the total code-length $t$3 encodes both the state-transition sequence $t$4 and rewards $t$5: $t$6 where $t$7 denotes the MDL code-length for finite i.i.d. sequences based on observed transition and reward counts (0812.4580).

A crucial property is the trade-off between state-space complexity and reward-modeling accuracy: increasing the number of states yields improved reward predictions but increases the complexity term, while fewer states simplify the model but may degrade predictive accuracy. The optimal feature map $t$8 minimizes the overall cost: $t$9

For improved model class selection, a Bayesian (marginalized) version $o_t\in O$0 integrates out the latent state sequence, where $o_t\in O$1 is the number of nonzero parameters in the combined transition-reward matrix $o_t\in O$2 (0812.4580).

An alternative and sufficient feature selection criterion requires the conditional independence (CI): $o_t\in O$3 which, when satisfied, guarantees validity of the reduced process as an MDP and ensures that optimal policies in the reduced space are optimal in the original space (Wang et al., 2017).

3. Learning Algorithms for Feature MDPs

Given the combinatorial space of all mappings $o_t\in O$4, efficient search strategies are requisite. Stochastic local search, using split-and-merge operations, is employed: splitting a state into substates or merging sibling states, followed by accepting moves that lower (or stochastically increase under simulated annealing) the cost function.

For practical instantiation, restricting $o_t\in O$5 to the class of variable-length suffix-trees (analogous to U-Tree representations) permits tractable search by leveraging structured priors and local neighbor proposals. During learning, the agent alternates between improving $o_t\in O$6, updating counts for state transitions and rewards, and solving the finite-MDP Bellman equations to derive optimal action policies. The total computational cost per cost-evaluation is $o_t\in O$7 per time-step, but the overall search remains exponential, motivating reliance on heuristics and incremental structure refinement (0812.4580).

An architecture for high-dimensional or continuous settings employs alternating deep neural networks (ADNN), combining an encoder $o_t\in O$8 and action-conditional predictor networks $o_t\in O$9, trained with a composite loss that enforces reconstruction accuracy, Markov sufficiency (via distance-covariance residual penalty), and regularization. Alternating-gradient updates optimize the encoder and predictor parameters in mini-batch fashion (Wang et al., 2017).

4. Theoretical Guarantees and Policy Optimality

The MDL-based cost functional is asymptotically optimal when the data is truly generated by an underlying finite-state MDP. Specifically, the cost of the true model $a_t\in A$0 will, with high probability and as $a_t\in A$1, asymptotically dominate any alternative model by at most an additive constant (0812.4580). The Bayesian marginal code-length $a_t\in A$2 inherits standard consistency properties of MDL model selection.

When the conditional-independence criterion (CI) is satisfied, the main theoretical result is that the reduced process inherits the Markovian property, and an optimal policy $a_t\in A$3 for the reduced (feature) MDP pulls back to an optimal policy $a_t\in A$4 for the original process. Recursive application of this criterion allows further dimensionality reduction (Wang et al., 2017).

The use of an absorbing high-reward state as an exploration bonus ensures, under the adopted planning scheme, that the number of suboptimal actions is polynomially bounded in $a_t\in A$5 and in the sizes of $a_t\in A$6, $a_t\in A$7 (0812.4580).

5. Empirical Evaluation and Practical Impact

Empirical studies involving simulated MDPs with high-dimensional, noisy features demonstrate that feature selection via ADNN yields dramatic reductions in effective state-space dimensionality (e.g., 4–15 variables out of 64 plus noise, with compressed features $a_t\in A$8). When transitions or rewards are nonlinear, policies learned on compact ADNN representations achieve near-optimal cumulative discounted reward, outperforming principal component analysis and full-state Q-learning in both efficiency and accuracy; this robustness persists even when up to 200 noise variables are present (Wang et al., 2017).

In real-world applications such as the BASICS-Mobile study—minimizing cigarette use among students over time—ADNN distills the original 15-dimensional feature set to a 6-dimensional summary without loss of policy performance. Learned feature weights display alignment with established domain knowledge, suggesting interpretability and relevance of the compact state representations derived (Wang et al., 2017).

6. Extensions and Open Problems

The Feature MDP approach generalizes to structured settings such as Dynamic Bayesian Networks (DBNs), where the transition structure is factorized over components of $a_t\in A$9. The same MDL principles govern selection of both the network structure and the feature map—a critical step for scaling to environments with large, structured or relational state spaces (0812.4580).

Major open problems include devising efficient search algorithms over the immense space of possible feature maps $r_t\in R$0, avoiding over-penalization of large state-spaces by refined coding criteria, and establishing non-asymptotic, finite-sample guarantees for the coupled representation and policy learning. Extending the formalism to continuous observation/action spaces and handling non-ergodic environments remain critical challenges. The tractability of the feature discovery step continues to be a central bottleneck for widespread practical adoption.

---

References:

Feature Markov Decision Processes (0812.4580) Sufficient Markov Decision Processes with Alternating Deep Neural Networks (Wang et al., 2017)