Token-Level Markov Decision Process

• Token-Level MDP is a sequential decision model where discrete tokens serve as observations to construct state representations for predictive control.
• The approach utilizes a minimum description length principle to balance complexity and reward predictiveness, automatically selecting optimal token aggregations.
• Extensions to dynamic Bayesian networks enable the model to capture structured token dependencies in applications like NLP, robotics, and recommendation systems.

A Token-Level Markov Decision Process (MDP) is a sequential decision model where the agent's observations are discrete tokens, such as words, characters, sensor readings, or user actions, which arrive in a stream and potentially possess intrinsic structure and dependencies. The central challenge is to construct or learn a state representation derived from these tokens such that the resulting process approximates an MDP; that is, the future can be predicted from the current state and action with sufficient accuracy for reinforcement learning and optimal control. This forms the basis for a principled approach to feature extraction and state abstraction in environments with fine-grained, tokenized observations.

1. Formal Objective Criterion for State Representation

A foundational principle in deriving token-level state abstractions is the minimization of coding (or description) length, as articulated in "Feature Markov Decision Processes" (0812.4580). Each candidate state mapping $\phi: h \to s$ summarizes the history $h$ (composed of token, action, and reward sequences) into a state $s$ such that the resulting state–action–reward sequence is both succinct and predictive. The formal cost function is:

$$\operatorname{Cost}(\phi \mid h) = CL(S_{1:n} \mid a_{1:n}) + CL(r_{1:n} \mid S_{1:n}, a_{1:n})$$

where $CL(\cdot)$ denotes code length and $S_{1:n}$, $a_{1:n}$, $r_{1:n}$ are the sequences of states, actions, and rewards, respectively. The optimal mapping

$$\phi^* = \arg\min_\phi \operatorname{Cost}(\phi \mid h)$$

balances compactness against predictive fidelity: minimizing excessive state-space growth while capturing reward-relevant token dependencies.

For i.i.d. sequences over tokens, the code length is:

$$CL(x_{1:n}) = n H(\hat{\theta}) + \sum_{i=1}^{m'} \frac{1}{2} \log n$$

where $H(\hat{\theta})$ is the empirical entropy, and $m'$ is the number of observed symbols.

Applied to token-level streams, candidate $\phi$ functions extract features such as the last $k$ tokens (n-grams), recent sensor readings, or structured aggregations, with the cost function used to systematically select the optimal token grouping that most effectively “explains” the environment's reward structure.

2. Integration Into Learning Algorithms

This objective criterion is directly integrated into the learning process, not used merely for post hoc evaluation. In the $\phi$MDP-Agent algorithm, the workflow operates as follows:

• The agent collects histories of tokens, actions, rewards.
• For each candidate $\phi$, states are derived and transition/reward models estimated via empirical frequency.
• These models are used with BeLLMan backups (e.g., $Q^*(s,a) = \sum_{s'} T(s'|s,a)[R(s')+\gamma V(s')]$) to compute optimal actions.
• Stochastic local search (e.g., state splitting/merging) guided by $\operatorname{Cost}(\phi|h)$ explores the space of $\phi$ mappings.
• Exploration is balanced with exploitation so that all token-derived states are eventually sufficiently sampled.

For token-level MDPs, this framework enables automatic learning of the optimal token aggregation and state representation, leveraging the minimum description length objective to adaptively refine both the state abstraction and the control policy throughout learning.

3. Extensions to Structured Models: Dynamic Bayesian Networks

In environments where tokens have strong inter-token dependencies (such as language with syntactic or semantic structure), the basic MDP framework can be extended to Dynamic Bayesian Networks (DBNs). Rather than encoding states as unstructured labels, DBNs use collections of variables with explicit graphical dependencies. The selection of DBN structure is also guided by code-length minimization, now applied to the network. This leads to more nuanced state representations capable of capturing, for example, multi-token patterns, part-of-speech sequences, or latent syntactic constructs—enabling succinct yet expressive modeling for prediction and control.

4. Key Models and Computational Formulations

Central formulas for token-level MDP abstraction include:

Formula	Description
$CL(x_{1:n})$	Code length for i.i.d. token sequence
$CL(S_{1:n}|a_{1:n})$	State transition coding cost
$CL(r_{1:n}|S_{1:n},a_{1:n})$	Reward sequence coding cost
$\operatorname{Cost}(\phi|h)$	Overall description length objective
$Q^{*}(s,a)$ (BeLLMan Eq.)	Optimal action value in MDP

Alternate formulations, such as coding only the rewards, are also introduced:

$$\operatorname{ICost}(\phi \mid h) = - \log P_U(r_{1:n} \mid a_{1:n}) + M \log n$$

where $U$ models joint transition–reward probabilities and $M$ denotes model complexity. Transition and reward models are learned for each $\phi$ using transition counts and estimated conditional probabilities.

5. Representative Applications and Practical Scenarios

Token-level MDP abstraction methods apply broadly:

• NLP: When tokens are words, automatic selection of $\phi$ among unigram, bigram, or higher n-gram models can reveal the optimal context length for predicting downstream reward (e.g., translation accuracy, task success).
• Robotics and Control: Sensor tokens (e.g., discretized readings) are grouped into state contexts using $\phi$ that captures sufficient environmental patterning for reliable reward prediction (e.g., navigation or manipulation success).
• Online Recommendation Systems: User tokens (actions/clicks) are aggregated to states that most accurately forecast reward signals (e.g., purchases, engagement metrics).

A practical example presented in the foundational work involves binary observations and quaternary rewards; mapping $\phi$ as functions of the previous 0, 1, or 2 tokens demonstrates via cost calculations that including memory of two tokens yields superior predictive encoding for rewards.

6. Implications and Theoretical Context

The framework leverages minimum description length principles to mechanize feature extraction in reinforcement learning for sequential decision environments with fine-grained token observations. The stochastic, cost-driven search over token-to-state mapping functions delivers a systematic methodology for solving what was previously handled via domain expertise and manual feature engineering. The extension to DBNs reinforces applicability to structured environments, with the potential for further elaboration in hierarchical or relational settings.

A plausible implication is that the methodology provides a scalable route to data-driven abstraction, robust to the exponential growth of state spaces typical in token-level domains. By explicitly quantifying the tradeoff between representational cost and reward predictiveness, it drives the automatic selection of abstractions most informative for sequential decision-making.

7. Summary of Key Contributions

The token-level MDP formalism, anchored in cost-minimization for state representation, constitutes an objective paradigm for extracting, evaluating, and refining abstractions from streams of discrete tokens. The machinery—state coding costs, reward predictiveness, adaptive stochastic search, and extension to graphical models—collectively supports principled construction of MDP models in domains where observations are tokenized and potentially strongly interdependent. This unifies feature representation with optimal control theory, advancing the automation of sequential decision-making under uncertainty in high-dimensional, token-based environments.