---
title: 'Feature MDPs: State Compression & RL'
url: https://www.emergentmind.com/topics/feature-markov-decision-processes
type: topic
---

# Feature MDPs: State Compression & RL

A Feature Markov Decision Process (Feature MDP, or $\Phi$MDP) is a formal framework that mechanizes the selection and evaluation of state representations for reinforcement learning (RL) from complex, non-Markovian observation sequences, by defining objective criteria for compressing observed histories into finite-dimensional Markovian state spaces. The central goal is to automate the discovery of feature maps $\Phi$ that transform observed history into a representation inducing a process amenable to efficient planning by standard MDP techniques, with rigorous guarantees on model selection and optimality [0812.4580][1704.07531].

## 1. Formal Definitions and State Compression

In sequential decision tasks, an agent interacts with its environment through discrete cycles: at each time $t$, the agent observes $o_t\in O$, takes action $a_t\in A$, and receives reward $r_t\in R$. The resulting history tuple $h_t = o_1a_1r_1\ldots a_{t-1}r_{t-1}o_t$ is an element of $H = (O\times A \times R)^*\times O$.

A Feature MDP is constructed by specifying a feature map $\Phi : H\to S$, yielding a compressed state $s_t=\Phi(h_t)\in S$. The process thus reduces the high-dimensional, possibly non-Markovian history $h_t$ to a finite set of states suitable for the MDP formalism. The map $\Phi$ is to be selected such that the resulting sequence $(s_t, a_t, r_t)$ closely satisfies the Markov property, making the RL problem tractable via standard state-based algorithms [0812.4580].

A parallel, general formulation considers the original state space $S$ (potentially continuous), action space $A$, transition kernel $P(s'|s,a)$, and reward function $R(s,a,s')$. A measurable feature map $\phi : S \to \bar S \subset \mathbb{R}^q$ defines a reduced process $(\bar S, A, \bar P, \bar R, \gamma)$, where
\[
\bar P(\bar s'|\bar s,a) = P(\phi(S^{t+1}) = \bar s' \mid \phi(S^t) = \bar s, A^t = a)
\]
\[
\bar R(\bar s,a) = \mathbb{E}[R(S^t,a,S^{t+1}) \mid \phi(S^t)=\bar s, A^t = a]
\]
[1704.07531].

## 2. Objective Criteria for Feature Map Selection

The quality of a feature map is evaluated using compressibility criteria grounded in Minimum Description Length (MDL) principles. For a given $\Phi$ and a collected history $h_n$, the total code-length $\mathrm{Cost}(\Phi|h_n)$ encodes both the state-transition sequence $(s_{1:n}|a_{1:n})$ and rewards $(r_{1:n}|s_{1:n},a_{1:n})$:
\[
\mathrm{Cost}(\Phi|h_n) = CL(s_{1:n}|a_{1:n}) + CL(r_{1:n}|s_{1:n}, a_{1:n})
\]
where $CL(\cdot)$ denotes the MDL code-length for finite i.i.d. sequences based on observed transition and reward counts [0812.4580].

A crucial property is the trade-off between state-space complexity and reward-modeling accuracy: increasing the number of states yields improved reward predictions but increases the complexity term, while fewer states simplify the model but may degrade predictive accuracy. The optimal feature map $\Phi^*$ minimizes the overall cost:
\[
\Phi^* = \arg\min_\Phi \, \mathrm{Cost}(\Phi|h_n)
\]

For improved model class selection, a Bayesian (marginalized) version $\mathrm{ICost}(\Phi|h_n) = -\log P_{\hat U}(r_{1:n}|a_{1:n}) + \frac12 M \log n$ integrates out the latent state sequence, where $M$ is the number of nonzero parameters in the combined transition-reward matrix $U$ [0812.4580].

An alternative and sufficient feature selection criterion requires the conditional independence (CI):
\[
(U^t, S^{t+1}) \perp S^t \mid (\phi(S^t), A^t)
\]
which, when satisfied, guarantees validity of the reduced process as an MDP and ensures that optimal policies in the reduced space are optimal in the original space [1704.07531].

## 3. Learning Algorithms for Feature MDPs

Given the combinatorial space of all mappings $\Phi:H\to S$, efficient search strategies are requisite. Stochastic local search, using split-and-merge operations, is employed: splitting a state into substates or merging sibling states, followed by accepting moves that lower (or stochastically increase under simulated annealing) the cost function.

For practical instantiation, restricting $\Phi$ to the class of variable-length suffix-trees (analogous to U-Tree representations) permits tractable search by leveraging structured priors and local neighbor proposals. During learning, the agent alternates between improving $\Phi$, updating counts for state transitions and rewards, and solving the finite-MDP Bellman equations to derive optimal action policies. The total computational cost per cost-evaluation is $O(|S|\cdot|A|\cdot|S| + |R|)$ per time-step, but the overall search remains exponential, motivating reliance on heuristics and incremental structure refinement [0812.4580].

An architecture for high-dimensional or continuous settings employs alternating deep neural networks (ADNN), combining an encoder $\phi_{\theta_1}$ and action-conditional predictor networks $f_{\theta_2,a}$, trained with a composite loss that enforces reconstruction accuracy, Markov sufficiency (via distance-covariance residual penalty), and regularization. Alternating-gradient updates optimize the encoder and predictor parameters in mini-batch fashion [1704.07531].

## 4. Theoretical Guarantees and Policy Optimality

The MDL-based cost functional is asymptotically optimal when the data is truly generated by an underlying finite-state MDP. Specifically, the cost of the true model $\Phi_{\textrm{true}}$ will, with high probability and as $n\to\infty$, asymptotically dominate any alternative model by at most an additive constant [0812.4580]. The Bayesian marginal code-length $\mathrm{ICost}$ inherits standard consistency properties of MDL model selection.

When the conditional-independence criterion (CI) is satisfied, the main theoretical result is that the reduced process inherits the Markovian property, and an optimal policy $\bar \pi^*$ for the reduced (feature) MDP pulls back to an optimal policy $\pi^*(s) = \bar \pi^*(\phi(s))$ for the original process. Recursive application of this criterion allows further dimensionality reduction [1704.07531].

The use of an absorbing high-reward state as an exploration bonus ensures, under the adopted planning scheme, that the number of suboptimal actions is polynomially bounded in $(1-\gamma)^{-1}$ and in the sizes of $|S|$, $|A|$ [0812.4580].

## 5. Empirical Evaluation and Practical Impact

Empirical studies involving simulated MDPs with high-dimensional, noisy features demonstrate that feature selection via ADNN yields dramatic reductions in effective state-space dimensionality (e.g., 4–15 variables out of 64 plus noise, with compressed features $q\approx3-12$). When transitions or rewards are nonlinear, policies learned on compact ADNN representations achieve near-optimal cumulative discounted reward, outperforming principal component analysis and full-state Q-learning in both efficiency and accuracy; this robustness persists even when up to 200 noise variables are present [1704.07531].

In real-world applications such as the BASICS-Mobile study—minimizing cigarette use among students over time—ADNN distills the original 15-dimensional feature set to a 6-dimensional summary without loss of policy performance. Learned feature weights display alignment with established domain knowledge, suggesting interpretability and relevance of the compact state representations derived [1704.07531].

## 6. Extensions and Open Problems

The Feature MDP approach generalizes to structured settings such as Dynamic Bayesian Networks (DBNs), where the transition structure is factorized over components of $s$. The same MDL principles govern selection of both the network structure and the feature map—a critical step for scaling to environments with large, structured or relational state spaces [0812.4580].

Major open problems include devising efficient search algorithms over the immense space of possible feature maps $\Phi$, avoiding over-penalization of large state-spaces by refined coding criteria, and establishing non-asymptotic, finite-sample guarantees for the coupled representation and policy learning. Extending the formalism to continuous observation/action spaces and handling non-ergodic environments remain critical challenges. The tractability of the feature discovery step continues to be a central bottleneck for widespread practical adoption.

---

**References**:  
Feature Markov Decision Processes [0812.4580]  
Sufficient Markov Decision Processes with Alternating Deep Neural Networks [1704.07531]

Source: https://www.emergentmind.com/topics/feature-markov-decision-processes