---
title: Non-Markovian Decision Processes (nMDPs)
url: https://www.emergentmind.com/topics/non-markovian-decision-processes-nmdps
type: topic
---

# Non-Markovian Decision Processes (nMDPs)

Searching arXiv for the specified paper and closely related work on non-Markovian decision processes.
arXiv query: "2506.24026 Constructing Non-Markovian Decision Process via History Aggregator"
Non-Markovian decision processes (nMDPs) are sequential decision models in which the relevant dynamics are history-dependent: rewards, observations, and in some formulations transitions may depend on an interaction prefix rather than only on the current state and action. In the surveyed literature, this umbrella includes both general nMDPs and the narrower but influential class of non-Markovian reward decision processes (NMRDPs), where the environment dynamics remain Markovian and only the reward is history-dependent [2506.24026]. Contemporary work studies nMDPs through several complementary formalisms: temporal logic and automata encodings of history, categorical equivalence between MDPs and nMDPs, recursive internal-state policies, predictive state representations (PSRs), and robust or multi-task learning frameworks [2605.10816, 2310.13550, 2411.07514]. A recurring technical problem is to identify a sufficient summary of history and then optimize under that summary without losing the semantics of the original process.

## 1. Definitions and scope

In standard MDP formulations, the reward depends only on the current state and action. By contrast, the NMRDP literature defines rewards such as \(R:S^* \rightarrow \mathbb{R}\) or \(R_{\mathrm{NMR}}:(S \times A)^* \rightarrow \mathbb{R}\), making the reward a function of the full trajectory prefix rather than a single state or state-action pair [1109.2355, 1912.02552]. A broader nMDP formulation goes further and allows both transition and reward dependence on history; one definition uses
\[
M = \langle S, A, tr, R^{+}, \gamma \rangle,
\]
with \(tr:S^* \times A \times S \rightarrow \Pi(S)\) and \(R^{+}:(S \times A)^* \rightarrow \mathbb{R}\) [2302.14509].

This literature therefore uses the term “non-Markovian” in at least two senses. In reward-centered work, the underlying state dynamics are often still Markovian, while the reward semantics are not. In broader formulations, observations, rewards, or transitions depend on the entire interaction history, so the process is non-Markovian even before reward modeling is considered [2605.10816]. Regular decision processes (RDPs) and omega-regular decision processes (ODPs) extend the space further by guarding transitions and rewards with regular lookback or omega-regular lookahead constraints [2312.08602].

| Model family | History dependence | Standard reduction |
|---|---|---|
| MDP | Current state/action only | None required |
| NMRDP | Reward depends on \(S^*\) or \((S \times A)^*\) | Equivalent MDP via expanded state or product automaton |
| General nMDP | Transition, observation, or reward depends on history | Internal state, PSR, HAS, or history augmentation |

A common misconception is that nMDPs are synonymous with non-Markovian rewards alone. The cited works do not support that restriction. They include reward-only settings, partially observable and PSR settings, and general history-dependent processes in which the relevant dependence may enter via observations, latent state, transition structure, or external event histories [2310.13550].

## 2. Formal foundations: categories, augmentation, and equivalence

A recent formalization defines the category of MDPs \(M\) and the category of non-Markovian decision processes \(N\), with objects given by individual decision processes and morphisms given by structure-preserving maps over states, actions, and rewards [2506.24026]. In that construction, MDP objects are \(\langle \rho_0, S, A, \{T_t\}_{t=0}^\infty \rangle\), while nMDP objects replace states by histories. The paper proves that both \(M\) and \(N\) are well-defined categories and introduces a non-Markov embedding functor \(\mathbf{N}:M \to N\) and a Markov abstraction functor \(\mathbf{M}:N \to M\). The central equivalence theorem is
\[
\mathbf{M} \circ \mathbf{N} \cong 1_M, \qquad \mathbf{N} \circ \mathbf{M} \cong 1_N.
\]
The result is categorical rather than algorithmic: it shows structural equivalence up to isomorphism, not that every nMDP is computationally as easy as an MDP.

The same paper introduces the History Aggregator for State (HAS), a family of maps
\[
\mathscr{A}_{S,t}: H_t \to S',
\]
used to compress trajectory histories into aggregated states [2506.24026]. The key requirement is reversibility, expressed through an inverse-like operator \(\mathscr{A}_{S,t}^*\) that recovers the current state, or even the full state sequence, from the aggregate history. This makes HAS a constructive device for injecting controlled non-Markovianity into otherwise Markovian problems while preserving access to the original Markovian information.

The formal significance of HAS is that it turns “history dependence” into a design variable. Group-based HAS constructions are reversible if \((S,\otimes)\) forms a group, and convolution-based HAS constructions are reversible if \(w_0\) is invertible [2506.24026]. Theorems 3 and 6 in that work show that the aggregation design can specify which time steps influence the current aggregate and with what strength. This suggests a benchmark methodology in which non-Markovianity is not merely observed but explicitly constructed.

## 3. Reward-centered nMDPs: temporal logic, automata, and reward machines

A large portion of the literature addresses non-Markovian rewards. In this setting, the objective is not to abandon Markovian solution methods altogether, but to recover them by augmenting the state with enough history to make the reward Markovian again [1109.2355]. Two classical logical languages are prominent. Past Linear Temporal Logic (PLTL) specifies history-dependent reward conditions via past operators such as “previously” and “since.” Future-oriented \(\$FLTL\) adds a special reward modality \(\$\), enabling progression-based reward tracking during forward search [1212.2482, 1301.0606].

These temporal logics support automatic translations from an NMRDP to an equivalent MDP. In the expanded-state view, each e-state records both the underlying system state and the logical information needed to determine whether reward conditions have been satisfied [1212.2482]. The NMRDPP platform implements several variants of this idea—PLTLSIM, PLTLMIN, PLTLSTR, PLTLSTR(A), and FLTL/progression—and compares dynamic programming, heuristic search, and structured solution methods under a common interface [1109.2355]. The main comparative finding is not that one method dominates universally, but that performance depends strongly on reward syntax, temporal depth, and whether dynamic irrelevance can be exploited during search [1212.2482].

The progression-based line of work is especially tailored to anytime state-based methods. It constructs only the needed portion of the expanded MDP during search and achieves what the authors call “blind minimality,” meaning minimality relative to the information available without global lookahead [1301.0606]. This is significant because true minimality often requires expensive preprocessing, whereas blind-minimal constructions preserve compatibility with LAO*-style or RTDP-style incremental solvers.

A distinct but related line models non-Markovian rewards by deterministic finite automata or Mealy reward machines. In one formulation, regular non-Markovian rewards are recognized by a DFA, and the RL state is augmented to \(S' = S_M \times S_A\), where \(S_A\) is the automaton state [1912.02552]. In another, a Mealy Reward Machine
\[
\mathcal{R} = \langle U, u_0, Z, \delta_u, \delta_r, c \rangle
\]
is synchronized with an MDP to obtain a product MDP over \(S \times U\), restoring Markovian rewards on the product space [2009.12600]. These constructions support active automata learning with Angluin’s \(L^*\), offline state merging, or conformance testing when the reward machine is unknown [1912.02552, 2001.09293].

The reward-machine literature supplies explicit learning guarantees. For regular non-Markovian rewards, the \(L^*\)-based RL variant converges to an optimal policy in the limit with probability \(1\), assuming the underlying RL algorithm converges and explores sufficiently [1912.02552]. In the online Mealy-machine setting, if a domain expert provides a realistic value \(V_{\mathrm{expert}}\), the agent is guaranteed with probability \(1\) eventually to learn a hypothesis reward machine whose value is at least \(V_{\mathrm{expert}}\) [2009.12600].

## 4. Policy optimization with learned memory and latent internal state

General nMDPs require policies that act on history or on a learned summary of history. A recent policy-gradient formulation addresses this by introducing Agent State-Markov (ASM) policies, in which the agent maintains an internal state \(S_t\) updated recursively via
\[
S_t \sim \nu_t(\cdot \mid S_{t-1}, A_{t-1}, O_t),
\qquad
A_t \sim \phi_t(\cdot \mid S_t).
\]
The pair \((\nu,\phi)\) induces a history-dependent policy while keeping action selection Markovian in the internal state [2605.10816]. The same work proves policy-gradient theorems for episodic and infinite-horizon discounted nMDPs, extending classical policy-gradient results to environments where observations and rewards depend on the full interaction history.

The resulting Agent State-Markov Policy Gradient (ASMPG) algorithm jointly optimizes the agent state dynamics and the control policy in a reward-centric manner rather than learning memory through auxiliary predictive objectives [2605.10816]. The paper establishes finite-time and almost sure convergence guarantees. In the finite-horizon case, the time-averaged squared gradient norm converges at rate \(O(H^{3.5}/\sqrt{K})\), and the infinite-horizon discounted analogue scales as \(\tilde{O}((1-\gamma)^{-3.5}/\sqrt{K})\). Empirically, ASMPG outperforms AIS-KL and AIS-MMD across CheeseMaze, HallwayNavigation, HealthcareTreatment, MachineRepair, and VelocityOnlyCartPole.

A different representation-learning approach studies imitation from state-only sequences in nMDPs by treating actions as latent variables and learning a deep generative model
\[
p_\theta(\zeta)=p(s_0)\prod_{t=0}^{T-1} p_\alpha(a_t\mid s_{0:t})\,p_\beta(s_{t+1}\mid s_t,a_t),
\]
where the policy is an energy-based prior over latent actions conditioned on state history [2306.15156]. Maximum-likelihood training uses short-run MCMC for the prior and importance sampling for the posterior. The learned model supports “decision-making as inference”: model-free execution corresponds to prior sampling, while model-based planning corresponds to posterior sampling initialized from the learned prior.

Representation learning has also been used to induce policy diversity in non-Markovian environments. One such method, D2PED, learns transformer-based policy embeddings from history sequences, stacks them into a dispersion matrix, and measures diversity through the determinant of that matrix [2302.14509]. The theoretical claim is that if the dispersion matrix is positive definite, the dispersed embeddings enlarge disagreements across policies. The experiments report improved diversity and robustness in Point-v1, FrozenLake-v1, and several MuJoCo tasks.

## 5. Statistical and computational regimes: PSRs, offline robustness, and factorized model-based RL

When nMDPs are expressed through low-rank PSRs or related latent structures, statistical complexity can be quantified without reverting to raw histories. In multi-task RL over PSRs, the key quantity is the \(\eta\)-bracketing number of the joint model class [2310.13550]. The UMT-PSR algorithm is provably efficient for upstream learning when tasks share the same observation and action spaces, and its sample complexity depends logarithmically on the joint bracketing number. The central insight is that if the normalized log-bracketing number of the joint class is smaller than the single-task baseline, then multi-task PSR learning yields sample savings. The same framework extends to downstream transfer through similarity-constrained model classes with reduced bracketing number.

Robust offline nMDP learning adds model uncertainty on top of history dependence. For low-rank nominal models, one recent algorithm combines dataset distillation with a lower confidence bound design for robust values under \(T\)-type and \(P\)-type uncertainty sets [2411.07514]. The paper introduces type-I concentrability for low-rank nMDPs and type-II concentrability for general nMDPs, derives dual forms for robust values, and proves that the low-rank algorithm finds an \(\epsilon\)-optimal robust policy using \(O(1/\epsilon^2)\) offline samples. The extension to general nMDPs also has polynomial sample efficiency under all different types of the uncertainty set.

For discrete-action NMRDPs with reward machines, model-based RL can exploit a specific factorization. QR-MAX separates learning of the Markovian environment transition \(P(s' \mid s,a)\) from learning of the reward-machine transition \(P(q' \mid s',q)\), rather than relearning the same environment dynamics for every automaton state [2512.14617]. The resulting sample complexity is
\[
\widetilde{\mathcal{O}}\!\left(\frac{|S||A|+|S||Q|}{\varepsilon^{3}(1-\gamma)^{3}}\right),
\]
compared with the product-space baseline
\[
\widetilde{\mathcal{O}}\!\left(\frac{|S||Q||A|}{\varepsilon^{3}(1-\gamma)^{3}}\right).
\]
This is a concrete example in which “Markovization” is not enough; the exploitable structure of the Markovization matters.

## 6. Generalizations, applications, and recurring issues

Several works broaden nMDPs beyond reward specification alone. One formalization studies MDPs perturbed by an external temporal process whose event history changes the transition kernel, making the environment non-Markovian in the original state space [2305.16056]. The process becomes Markovian only on an augmented state containing the current state and event-history marks. Under decay assumptions expressed through summable sequences \(\{M_t\}\) and \(\{N_t\}\), there exists an optimal stationary deterministic policy for the augmented process, and finite-history truncations admit explicit value-error bounds.

RDPs and ODPs extend the expressivity of decision models through regular lookback and omega-regular lookahead. RDPs permit history-dependent rewards and transition guards via regular properties of the past, yet their expressive power coincides with finite-state MDPs [2312.08602]. ODPs add omega-regular promises about future behavior; if a promised lookahead is violated, the payoff becomes \(\bot\), overriding previously collected rewards. Optimization and learning under discounted reward are then reduced to lexicographic optimization and learning over finite MDPs.

The broader stochastic-process literature also sharpens conceptual boundaries. Positively divisible non-Markovian processes can satisfy the Chapman-Kolmogorov equation while still violating the Markov condition [2401.12715]. That result does not directly solve control problems, but it clarifies that divisibility and Markovianity are distinct notions. For finite-state processes, the paper derives a hierarchy of consistency conditions involving memory transition matrices, providing a constructive route from Markov processes to positively divisible non-Markovian ones.

Application sketches emphasize why such generalizations are pursued. In epidemic control, a non-Markovian formulation can encode time-in-state effects, heterogeneous risk, delayed intervention effects, and spatial contact structure, all of which are difficult to represent under compartmental Markov assumptions [2008.00375]. The paper presents this as a future research direction rather than a completed control framework, but it illustrates the broader motivation: many practically relevant control problems have delay distributions, promises, or temporal tasks that are more naturally described through history than through instantaneous state alone.

A final recurring issue is the relation between equivalence and tractability. Many papers show that non-Markovian problems can be translated into equivalent MDPs or product MDPs [2506.24026, 1109.2355]. This does not imply that history dependence is negligible. The expanded state may be large, logical preprocessing may be expensive, and naive product-space learning may be statistically wasteful. The major design patterns of the field—temporal-logic progression, reward machines, HAS, ASM policies, PSRs, and factorized model-based learning—can be read as different answers to the same question: which parts of history must be retained explicitly, which can be aggregated, and which can be exploited structurally once the process has been Markovized.

Source: https://www.emergentmind.com/topics/non-markovian-decision-processes-nmdps