Non-Markovian Decision Processes (nMDPs)

• Non-Markovian decision processes (nMDPs) are sequential models where rewards, transitions, or observations depend on the entire interaction history rather than just the current state or action.
• They employ formal tools such as temporal logic, automata, and history aggregators to transform history-dependent dynamics into equivalent Markovian frameworks.
• Recent advances in policy optimization and representation learning, including ASM policies and predictive state representations, offer robust methods to tackle the computational challenges of nMDPs.

Searching arXiv for the specified paper and closely related work on non-Markovian decision processes. arXiv query: "(Wang et al., 30 Jun 2025) Constructing Non-Markovian Decision Process via History Aggregator" Non-Markovian decision processes (nMDPs) are sequential decision models in which the relevant dynamics are history-dependent: rewards, observations, and in some formulations transitions may depend on an interaction prefix rather than only on the current state and action. In the surveyed literature, this umbrella includes both general nMDPs and the narrower but influential class of non-Markovian reward decision processes (NMRDPs), where the environment dynamics remain Markovian and only the reward is history-dependent (Wang et al., 30 Jun 2025). Contemporary work studies nMDPs through several complementary formalisms: temporal logic and automata encodings of history, categorical equivalence between MDPs and nMDPs, recursive internal-state policies, predictive state representations (PSRs), and robust or multi-task learning frameworks (Kar et al., 11 May 2026, Huang et al., 2023, Huang et al., 2024). A recurring technical problem is to identify a sufficient summary of history and then optimize under that summary without losing the semantics of the original process.

1. Definitions and scope

In standard MDP formulations, the reward depends only on the current state and action. By contrast, the NMRDP literature defines rewards such as $R:S^* \rightarrow \mathbb{R}$ or $$R_{\mathrm{NMR}}:(S \times A)^* \rightarrow \mathbb{R}$$, making the reward a function of the full trajectory prefix rather than a single state or state-action pair (Gretton et al., 2011, Gaon et al., 2019). A broader nMDP formulation goes further and allows both transition and reward dependence on history; one definition uses

$M = \langle S, A, tr, R^{+}, \gamma \rangle,$

with $tr:S^* \times A \times S \rightarrow \Pi(S)$ and $R^{+}:(S \times A)^* \rightarrow \mathbb{R}$ (Qu et al., 2023).

This literature therefore uses the term “non-Markovian” in at least two senses. In reward-centered work, the underlying state dynamics are often still Markovian, while the reward semantics are not. In broader formulations, observations, rewards, or transitions depend on the entire interaction history, so the process is non-Markovian even before reward modeling is considered (Kar et al., 11 May 2026). Regular decision processes (RDPs) and omega-regular decision processes (ODPs) extend the space further by guarding transitions and rewards with regular lookback or omega-regular lookahead constraints (Hahn et al., 2023).

Model family	History dependence	Standard reduction
MDP	Current state/action only	None required
NMRDP	Reward depends on $S^*$ or $(S \times A)^*$	Equivalent MDP via expanded state or product automaton
General nMDP	Transition, observation, or reward depends on history	Internal state, PSR, HAS, or history augmentation

A common misconception is that nMDPs are synonymous with non-Markovian rewards alone. The cited works do not support that restriction. They include reward-only settings, partially observable and PSR settings, and general history-dependent processes in which the relevant dependence may enter via observations, latent state, transition structure, or external event histories (Huang et al., 2023).

2. Formal foundations: categories, augmentation, and equivalence

A recent formalization defines the category of MDPs $M$ and the category of non-Markovian decision processes $N$, with objects given by individual decision processes and morphisms given by structure-preserving maps over states, actions, and rewards (Wang et al., 30 Jun 2025). In that construction, MDP objects are $\langle \rho_0, S, A, \{T_t\}_{t=0}^\infty \rangle$, while nMDP objects replace states by histories. The paper proves that both $$R_{\mathrm{NMR}}:(S \times A)^* \rightarrow \mathbb{R}$$0 and $$R_{\mathrm{NMR}}:(S \times A)^* \rightarrow \mathbb{R}$$1 are well-defined categories and introduces a non-Markov embedding functor $$R_{\mathrm{NMR}}:(S \times A)^* \rightarrow \mathbb{R}$$2 and a Markov abstraction functor $$R_{\mathrm{NMR}}:(S \times A)^* \rightarrow \mathbb{R}$$3. The central equivalence theorem is

$$R_{\mathrm{NMR}}:(S \times A)^* \rightarrow \mathbb{R}$$4

The result is categorical rather than algorithmic: it shows structural equivalence up to isomorphism, not that every nMDP is computationally as easy as an MDP.

The same paper introduces the History Aggregator for State (HAS), a family of maps

$$R_{\mathrm{NMR}}:(S \times A)^* \rightarrow \mathbb{R}$$5

used to compress trajectory histories into aggregated states (Wang et al., 30 Jun 2025). The key requirement is reversibility, expressed through an inverse-like operator $$R_{\mathrm{NMR}}:(S \times A)^* \rightarrow \mathbb{R}$$6 that recovers the current state, or even the full state sequence, from the aggregate history. This makes HAS a constructive device for injecting controlled non-Markovianity into otherwise Markovian problems while preserving access to the original Markovian information.

The formal significance of HAS is that it turns “history dependence” into a design variable. Group-based HAS constructions are reversible if $$R_{\mathrm{NMR}}:(S \times A)^* \rightarrow \mathbb{R}$$7 forms a group, and convolution-based HAS constructions are reversible if $$R_{\mathrm{NMR}}:(S \times A)^* \rightarrow \mathbb{R}$$8 is invertible (Wang et al., 30 Jun 2025). Theorems 3 and 6 in that work show that the aggregation design can specify which time steps influence the current aggregate and with what strength. This suggests a benchmark methodology in which non-Markovianity is not merely observed but explicitly constructed.

3. Reward-centered nMDPs: temporal logic, automata, and reward machines

A large portion of the literature addresses non-Markovian rewards. In this setting, the objective is not to abandon Markovian solution methods altogether, but to recover them by augmenting the state with enough history to make the reward Markovian again (Gretton et al., 2011). Two classical logical languages are prominent. Past Linear Temporal Logic (PLTL) specifies history-dependent reward conditions via past operators such as “previously” and “since.” Future-oriented $$R_{\mathrm{NMR}}:(S \times A)^* \rightarrow \mathbb{R}$$9, enabling progression-based reward tracking during forward search (Gretton et al., 2012, Thiebaux et al., 2012).

These temporal logics support automatic translations from an NMRDP to an equivalent MDP. In the expanded-state view, each e-state records both the underlying system state and the logical information needed to determine whether reward conditions have been satisfied (Gretton et al., 2012). The NMRDPP platform implements several variants of this idea—PLTLSIM, PLTLMIN, PLTLSTR, PLTLSTR(A), and FLTL/progression—and compares dynamic programming, heuristic search, and structured solution methods under a common interface (Gretton et al., 2011). The main comparative finding is not that one method dominates universally, but that performance depends strongly on reward syntax, temporal depth, and whether dynamic irrelevance can be exploited during search (Gretton et al., 2012).

The progression-based line of work is especially tailored to anytime state-based methods. It constructs only the needed portion of the expanded MDP during search and achieves what the authors call “blind minimality,” meaning minimality relative to the information available without global lookahead (Thiebaux et al., 2012). This is significant because true minimality often requires expensive preprocessing, whereas blind-minimal constructions preserve compatibility with LAO*-style or RTDP-style incremental solvers.

A distinct but related line models non-Markovian rewards by deterministic finite automata or Mealy reward machines. In one formulation, regular non-Markovian rewards are recognized by a DFA, and the RL state is augmented to $M = \langle S, A, tr, R^{+}, \gamma \rangle,$0, where $M = \langle S, A, tr, R^{+}, \gamma \rangle,$1 is the automaton state (Gaon et al., 2019). In another, a Mealy Reward Machine

$M = \langle S, A, tr, R^{+}, \gamma \rangle,$2

is synchronized with an MDP to obtain a product MDP over $M = \langle S, A, tr, R^{+}, \gamma \rangle,$3, restoring Markovian rewards on the product space (Rens et al., 2020). These constructions support active automata learning with Angluin’s $M = \langle S, A, tr, R^{+}, \gamma \rangle,$4, offline state merging, or conformance testing when the reward machine is unknown (Gaon et al., 2019, Rens et al., 2020).

The reward-machine literature supplies explicit learning guarantees. For regular non-Markovian rewards, the $M = \langle S, A, tr, R^{+}, \gamma \rangle,$5-based RL variant converges to an optimal policy in the limit with probability $M = \langle S, A, tr, R^{+}, \gamma \rangle,$6, assuming the underlying RL algorithm converges and explores sufficiently (Gaon et al., 2019). In the online Mealy-machine setting, if a domain expert provides a realistic value $M = \langle S, A, tr, R^{+}, \gamma \rangle,$7, the agent is guaranteed with probability $M = \langle S, A, tr, R^{+}, \gamma \rangle,$8 eventually to learn a hypothesis reward machine whose value is at least $M = \langle S, A, tr, R^{+}, \gamma \rangle,$9 (Rens et al., 2020).

4. Policy optimization with learned memory and latent internal state

General nMDPs require policies that act on history or on a learned summary of history. A recent policy-gradient formulation addresses this by introducing Agent State-Markov (ASM) policies, in which the agent maintains an internal state $tr:S^* \times A \times S \rightarrow \Pi(S)$0 updated recursively via

$tr:S^* \times A \times S \rightarrow \Pi(S)$1

The pair $tr:S^* \times A \times S \rightarrow \Pi(S)$2 induces a history-dependent policy while keeping action selection Markovian in the internal state (Kar et al., 11 May 2026). The same work proves policy-gradient theorems for episodic and infinite-horizon discounted nMDPs, extending classical policy-gradient results to environments where observations and rewards depend on the full interaction history.

The resulting Agent State-Markov Policy Gradient (ASMPG) algorithm jointly optimizes the agent state dynamics and the control policy in a reward-centric manner rather than learning memory through auxiliary predictive objectives (Kar et al., 11 May 2026). The paper establishes finite-time and almost sure convergence guarantees. In the finite-horizon case, the time-averaged squared gradient norm converges at rate $tr:S^* \times A \times S \rightarrow \Pi(S)$3, and the infinite-horizon discounted analogue scales as $tr:S^* \times A \times S \rightarrow \Pi(S)$4. Empirically, ASMPG outperforms AIS-KL and AIS-MMD across CheeseMaze, HallwayNavigation, HealthcareTreatment, MachineRepair, and VelocityOnlyCartPole.

A different representation-learning approach studies imitation from state-only sequences in nMDPs by treating actions as latent variables and learning a deep generative model

$tr:S^* \times A \times S \rightarrow \Pi(S)$5

where the policy is an energy-based prior over latent actions conditioned on state history (Qin et al., 2023). Maximum-likelihood training uses short-run MCMC for the prior and importance sampling for the posterior. The learned model supports “decision-making as inference”: model-free execution corresponds to prior sampling, while model-based planning corresponds to posterior sampling initialized from the learned prior.

Representation learning has also been used to induce policy diversity in non-Markovian environments. One such method, D2PED, learns transformer-based policy embeddings from history sequences, stacks them into a dispersion matrix, and measures diversity through the determinant of that matrix (Qu et al., 2023). The theoretical claim is that if the dispersion matrix is positive definite, the dispersed embeddings enlarge disagreements across policies. The experiments report improved diversity and robustness in Point-v1, FrozenLake-v1, and several MuJoCo tasks.

5. Statistical and computational regimes: PSRs, offline robustness, and factorized model-based RL

When nMDPs are expressed through low-rank PSRs or related latent structures, statistical complexity can be quantified without reverting to raw histories. In multi-task RL over PSRs, the key quantity is the $tr:S^* \times A \times S \rightarrow \Pi(S)$6-bracketing number of the joint model class (Huang et al., 2023). The UMT-PSR algorithm is provably efficient for upstream learning when tasks share the same observation and action spaces, and its sample complexity depends logarithmically on the joint bracketing number. The central insight is that if the normalized log-bracketing number of the joint class is smaller than the single-task baseline, then multi-task PSR learning yields sample savings. The same framework extends to downstream transfer through similarity-constrained model classes with reduced bracketing number.

Robust offline nMDP learning adds model uncertainty on top of history dependence. For low-rank nominal models, one recent algorithm combines dataset distillation with a lower confidence bound design for robust values under $tr:S^* \times A \times S \rightarrow \Pi(S)$7-type and $tr:S^* \times A \times S \rightarrow \Pi(S)$8-type uncertainty sets (Huang et al., 2024). The paper introduces type-I concentrability for low-rank nMDPs and type-II concentrability for general nMDPs, derives dual forms for robust values, and proves that the low-rank algorithm finds an $tr:S^* \times A \times S \rightarrow \Pi(S)$9-optimal robust policy using $R^{+}:(S \times A)^* \rightarrow \mathbb{R}$0 offline samples. The extension to general nMDPs also has polynomial sample efficiency under all different types of the uncertainty set.

For discrete-action NMRDPs with reward machines, model-based RL can exploit a specific factorization. QR-MAX separates learning of the Markovian environment transition $R^{+}:(S \times A)^* \rightarrow \mathbb{R}$1 from learning of the reward-machine transition $R^{+}:(S \times A)^* \rightarrow \mathbb{R}$2, rather than relearning the same environment dynamics for every automaton state (Trapasso et al., 16 Dec 2025). The resulting sample complexity is

$R^{+}:(S \times A)^* \rightarrow \mathbb{R}$3

compared with the product-space baseline

$R^{+}:(S \times A)^* \rightarrow \mathbb{R}$4

This is a concrete example in which “Markovization” is not enough; the exploitable structure of the Markovization matters.

6. Generalizations, applications, and recurring issues

Several works broaden nMDPs beyond reward specification alone. One formalization studies MDPs perturbed by an external temporal process whose event history changes the transition kernel, making the environment non-Markovian in the original state space (Ayyagari et al., 2023). The process becomes Markovian only on an augmented state containing the current state and event-history marks. Under decay assumptions expressed through summable sequences $R^{+}:(S \times A)^* \rightarrow \mathbb{R}$5 and $R^{+}:(S \times A)^* \rightarrow \mathbb{R}$6, there exists an optimal stationary deterministic policy for the augmented process, and finite-history truncations admit explicit value-error bounds.

RDPs and ODPs extend the expressivity of decision models through regular lookback and omega-regular lookahead. RDPs permit history-dependent rewards and transition guards via regular properties of the past, yet their expressive power coincides with finite-state MDPs (Hahn et al., 2023). ODPs add omega-regular promises about future behavior; if a promised lookahead is violated, the payoff becomes $R^{+}:(S \times A)^* \rightarrow \mathbb{R}$7, overriding previously collected rewards. Optimization and learning under discounted reward are then reduced to lexicographic optimization and learning over finite MDPs.

The broader stochastic-process literature also sharpens conceptual boundaries. Positively divisible non-Markovian processes can satisfy the Chapman-Kolmogorov equation while still violating the Markov condition (Canturk et al., 2024). That result does not directly solve control problems, but it clarifies that divisibility and Markovianity are distinct notions. For finite-state processes, the paper derives a hierarchy of consistency conditions involving memory transition matrices, providing a constructive route from Markov processes to positively divisible non-Markovian ones.

Application sketches emphasize why such generalizations are pursued. In epidemic control, a non-Markovian formulation can encode time-in-state effects, heterogeneous risk, delayed intervention effects, and spatial contact structure, all of which are difficult to represent under compartmental Markov assumptions (Eftekhari et al., 2020). The paper presents this as a future research direction rather than a completed control framework, but it illustrates the broader motivation: many practically relevant control problems have delay distributions, promises, or temporal tasks that are more naturally described through history than through instantaneous state alone.

A final recurring issue is the relation between equivalence and tractability. Many papers show that non-Markovian problems can be translated into equivalent MDPs or product MDPs (Wang et al., 30 Jun 2025, Gretton et al., 2011). This does not imply that history dependence is negligible. The expanded state may be large, logical preprocessing may be expensive, and naive product-space learning may be statistically wasteful. The major design patterns of the field—temporal-logic progression, reward machines, HAS, ASM policies, PSRs, and factorized model-based learning—can be read as different answers to the same question: which parts of history must be retained explicitly, which can be aggregated, and which can be exploited structurally once the process has been Markovized.