Non-Markovian Reward Decision Processes

• Non-Markovian Reward Decision Processes (NMRDPs) are decision models where rewards depend on the entire history rather than just the current state.
• They leverage methods like state augmentation, temporal logic specifications, and reward machines to restore Markovian structure for planning and learning.
• Recent work in NMRDPs focuses on integrating automata products with deep reinforcement learning, active learning of reward models, and PAC sample-efficiency guarantees.

Non-Markovian Reward Decision Processes (NMRDPs) are decision processes in which reward depends on history rather than only on the current state or transition. In the common formulation, the transition model remains Markovian while the reward is a functional over finite traces, for example $R:(S\times A)^*\to \mathbb{R}$, so optimal control generally depends on memory of past events. This history dependence breaks the standard Bellman decomposition on the original state space and motivates a large body of work on state augmentation, temporal-logic specification, reward machines, automata products, and learned internal memory that restore Markovian structure in an expanded state space (Miao et al., 2023, Topper et al., 2024, Kar et al., 11 May 2026).

1. Formal model and decision-theoretic foundations

A standard MDP is typically written as $M=\langle S, A, R, P, \gamma, s_0\rangle$ or $(S,A,T,R,\gamma)$, with finite state space $S$, action space $A$, transition kernel $P$ or $T$, and discount factor $\gamma$ (Miao et al., 2023, Topper et al., 2024). An NMRDP keeps the same state and transition structure but replaces the Markovian reward with a history-dependent reward. One formulation is

$$NM=\langle S, A, R, P, \gamma, s_0\rangle,\qquad R:(S\times A)^*\to \mathbb{R},$$

while another writes $R:(S\times A\times S)^*\to\mathbb{R}$ or $M=\langle S, A, R, P, \gamma, s_0\rangle$0 for a trajectory prefix $M=\langle S, A, R, P, \gamma, s_0\rangle$1 (Miao et al., 2023, Topper et al., 2024).

This change has immediate consequences for policies and value functions. In one common formalization, policies are themselves history dependent,

$M=\langle S, A, R, P, \gamma, s_0\rangle$2

and the value and action-value functions are defined on histories:

$M=\langle S, A, R, P, \gamma, s_0\rangle$3

The key distinction from an MDP is therefore not uncertainty alone, but the fact that reward-relevant information is not necessarily recoverable from the current environment state (Topper et al., 2024).

Several papers place NMRDPs inside a broader non-Markovian decision-process family. In the episodic and discounted NMDP model, both observations and rewards may depend on the whole interaction history, whereas an NMRDP is the special case in which the defining non-Markovian feature is the reward $M=\langle S, A, R, P, \gamma, s_0\rangle$4 (Kar et al., 11 May 2026). This broader view is useful because it clarifies that reward non-Markovianity and observation or transition non-Markovianity are logically distinct phenomena, even though they are often handled by similar memory-augmentation techniques.

2. Temporal logic, reward machines, and other specification formalisms

One influential line of work specifies non-Markovian rewards by temporal logic over finite traces. In the LTL$M=\langle S, A, R, P, \gamma, s_0\rangle$5 formulation,

$M=\langle S, A, R, P, \gamma, s_0\rangle$6

with derived operators such as $M=\langle S, A, R, P, \gamma, s_0\rangle$7 and $M=\langle S, A, R, P, \gamma, s_0\rangle$8. A non-Markovian task is then written as a pair $M=\langle S, A, R, P, \gamma, s_0\rangle$9, where $(S,A,T,R,\gamma)$0 is delivered when the finite trace satisfies $(S,A,T,R,\gamma)$1 (Miao et al., 2023). A representative example is the Waterworld task “touch red then green,” encoded as

$(S,A,T,R,\gamma)$2

LDL$(S,A,T,R,\gamma)$3 provides a more expressive regular-language view of reward specification. There the reward model is a finite set

$(S,A,T,R,\gamma)$4

and the reward of a prefix $(S,A,T,R,\gamma)$5 is

$(S,A,T,R,\gamma)$6

Because LDL$(S,A,T,R,\gamma)$7 captures monadic second-order logic over finite words, it strictly subsumes LTL$(S,A,T,R,\gamma)$8 while remaining compatible with automata compilation (Brafman et al., 2017).

A second major line uses explicit automata. Reward Machines are finite-state reward specifications over a label alphabet. In one formulation, a Reward Machine is

$(S,A,T,R,\gamma)$9

where $S$0 is the finite memory state set, $S$1 the initial state, $S$2 the label alphabet, $S$3 a finite reward set, $S$4 the transition function, and $S$5 the reward output (Topper et al., 2024). Closely related Mealy-style formulations write a Mealy Reward Machine as

$S$6

with deterministic next-state function $S$7 and output function $S$8 on observation symbols (Rens et al., 2020). These models make the reward-generating memory explicit rather than implicit in logic semantics.

Classical deterministic reward machines are not the only option. Probabilistic Reward Machines (PRMs) extend reward machines to stochastic reward semantics through a probabilistic transition function $S$9 and a reward map $A$0, thereby encoding reward distributions conditioned on label histories (Dohmen et al., 2021). At the opposite end of the expressivity spectrum, "Expressive Reward Synthesis with the Runtime Monitoring Language" states that RML-based Reward Machines can specify reward functions for non-regular, non-Markovian tasks by using parameterized event patterns, conditional filters, dynamic binding, and counters (Trapasso et al., 16 Dec 2025). This suggests that finite-state regular specifications are central but not exhaustive within the broader NMRDP landscape.

3. Product constructions and Markovianization

The canonical solution technique for NMRDPs is to augment the environment state with memory that summarizes the reward-relevant history. For LTL$A$1 specifications, each formula can be transformed into a deterministic finite automaton

$A$2

with label alphabet $A$3 and labeling function $A$4. The resulting product MDP has state space $A$5; in one implementation it is realized by adding a single enumerated fluent $A$6, giving

$A$7

The transition and reward are

$A$8

and

$A$9

The same paper contrasts this linear-state encoding with prior fluent-set encodings that can induce $P$0 (Miao et al., 2023).

Reward-machine products have the same structure. Given a labeled reward-free MDP

$P$1

and a reward machine

$P$2

the product MDP is

$P$3

with $P$4, $P$5,

$P$6

and

$P$7

The central intuition is that the reward machine state $P$8 is sufficient memory: rewards are no longer functions of the whole history once $P$9 is part of the Markov state (Topper et al., 2024).

The same construction extends to stochastic reward models. For a TMDP $T$0 and a PRM $T$1, the product MDP has state space $T$2 and transition kernel

$T$3

with reward emitted by $T$4. The paper proves a product-correctness theorem stating that if $T$5 encodes the stochastic reward function $T$6, then the product reproduces the original joint label-reward semantics (Dohmen et al., 2021).

A common misconception is that Markovianization is merely a heuristic engineering trick. In the cited literature it is instead the central equivalence construction: the augmented process is intended to be reward-equivalent to the original NMRDP, and optimal policies on the product induce optimal history-dependent policies on the underlying process when the memory component is sufficient (Brafman et al., 2017, Rens et al., 2020).

4. Learning and inferring non-Markovian reward structure

A major branch of the literature assumes that the environment dynamics are known but the non-Markovian reward model is not. One approach uses active automata learning. "Learning Non-Markovian Reward Models in MDPs" models the reward as a Mealy machine synchronized with a known non-rewarding MDP and learns it with Angluin’s $T$7 algorithm, using membership queries answered through interaction and equivalence queries approximated by conformance testing (Rens et al., 2020). The same paper cites a query bound of

$T$8

where $T$9 is the alphabet size, $\gamma$0 the number of states of the minimal target Mealy machine, and $\gamma$1 the maximum counterexample length.

"Online Learning of Non-Markovian Reward Models" adapts this active-learning scheme to an online control loop. It plans specifically to answer membership queries by synthesizing policies that maximize the probability of realizing a query word or minimize the expected number of steps to realize it, then alternates learning and exploitation in the synchronized product MDP. It proves that, if the expert baseline value $\gamma$2 does not exceed the true optimal value, then with probability $\gamma$3 the algorithm eventually learns a hypothesis $\gamma$4 such that $\gamma$5 (Rens et al., 2020).

A different perspective infers the reward model directly from demonstrations. In "Bayesian Inverse Reinforcement Learning for Non-Markovian Rewards," the hypothesis space is the set of reward machines with fixed size $\gamma$6 over a label alphabet $\gamma$7 and reward alphabet $\gamma$8. Demonstrations are augmented with label prefixes,

$\gamma$9

and the posterior is

$$NM=\langle S, A, R, P, \gamma, s_0\rangle,\qquad R:(S\times A)^*\to \mathbb{R},$$0

with Boltzmann likelihood

$$NM=\langle S, A, R, P, \gamma, s_0\rangle,\qquad R:(S\times A)^*\to \mathbb{R},$$1

Planning inside each candidate reward machine uses policy iteration on the product MDP, and MAP inference uses a modified simulated annealing rule that preserves prior influence even when the likelihood ratio is sharp (Topper et al., 2024).

When rewards themselves are stochastic, "Inferring Probabilistic Reward Machines from Non-Markovian Reward Processes for Reinforcement Learning" replaces deterministic reward machines with PRMs and adapts $$NM=\langle S, A, R, P, \gamma, s_0\rangle,\qquad R:(S\times A)^*\to \mathbb{R},$$2-style inference to empirical reward distributions. It introduces a sampling observation table, Hoeffding-style statistical compatibility tests, and a hypothesis PRM whose transitions are estimated from empirical counts. The paper proves almost-sure convergence to a correct PRM, or equivalence on the observable sub-language, under the stated assumptions (Dohmen et al., 2021).

Partial observability requires richer memory models. "Inferring Reward Machines and Transition Machines from Partially Observable Markov Decision Processes" introduces Transition Machines (TMs) for non-Markovian observation dynamics, Reward Machines for non-Markovian rewards, and a unified Dual-Behavior Mealy Machine (DBMM). Its passive state-merging algorithm DB-RPNI learns minimal resolvent DBMMs under structure completeness and achieves speedups of up to three orders of magnitude over the stated baselines (Wu et al., 3 Aug 2025). This broadens reward-model inference from fully observable NMRDPs to deterministic POMDP settings in which reward memory alone is insufficient.

5. Planning, reinforcement learning, and optimization algorithms

Once an NMRDP has been Markovianized, classical planning methods become applicable. Earlier translation-based work studies PLTL and future-oriented FLTL reward specifications, regression and progression operators, and explicit or symbolic translations into equivalent MDPs. "Decision-Theoretic Planning with non-Markovian Rewards" implements dynamic programming, heuristic search, and structured methods in NMRDPP, while "Anytime State-Based Solution Methods for Decision Processes with non-Markovian Rewards" develops an FLTL progression-based translation aimed at LAO*, RTDP, and LRTDP, proving that the resulting expanded MDP is blind minimal in the sense relevant to anytime forward search (Gretton et al., 2011, Thiebaux et al., 2012). The practical conclusion of this line is that the choice of temporal formalism and translation regime materially affects expansion size, reachability pruning, and suitability for explicit versus structured solvers.

More recent work integrates automata products directly with deep RL. "Using Experience Classification for Training Non-Markovian Tasks" encodes LTL$$NM=\langle S, A, R, P, \gamma, s_0\rangle,\qquad R:(S\times A)^*\to \mathbb{R},$$3 goals in a product MDP, trains SAC and TD3 in the encoded environment, and replaces TD-error prioritization with automata-guided experience classification. The shaping term is

$$NM=\langle S, A, R, P, \gamma, s_0\rangle,\qquad R:(S\times A)^*\to \mathbb{R},$$4

and replay priorities are driven by DFA-state ranks rather than by TD error. The paper reports that EC consistently accelerates learning relative to BASE, RS, and PER across tasks, except for the simplest Waterworld task with SAC where EC≈BASE, and that the single-fluent encoding yields an average training-time reduction ≈ 12.23% (Miao et al., 2023).

A distinct response dispenses with external automata and learns internal memory jointly with control. "Policy Gradient Methods for Non-Markovian Reinforcement Learning" introduces Agent State-Markov policies, in which an internal state $$NM=\langle S, A, R, P, \gamma, s_0\rangle,\qquad R:(S\times A)^*\to \mathbb{R},$$5 is recursively updated and actions are chosen from $$NM=\langle S, A, R, P, \gamma, s_0\rangle,\qquad R:(S\times A)^*\to \mathbb{R},$$6. For episodic problems, the paper proves

$$NM=\langle S, A, R, P, \gamma, s_0\rangle,\qquad R:(S\times A)^*\to \mathbb{R},$$7

and analogously for discounted infinite-horizon problems. The resulting ASMPG algorithm has finite-time and almost-sure convergence guarantees under the stated smoothness assumptions, and empirically outperforms predictive-objective baselines on the reported non-Markovian tasks (Kar et al., 11 May 2026). This provides an end-to-end memory-learning alternative to symbolic Markovianization.

Model-based RL with explicit reward machines has also acquired formal sample-efficiency guarantees. "Model-Based Reinforcement Learning in Discrete-Action Non-Markovian Reward Decision Processes" introduces QR-MAX, which factorizes learning of the Markovian environment transition model from learning of the non-Markovian reward-machine component. Its PAC-MDP sample complexity scales as

$$NM=\langle S, A, R, P, \gamma, s_0\rangle,\qquad R:(S\times A)^*\to \mathbb{R},$$8

improving over naïvely applying R-Max to the product MDP, which would scale with $$NM=\langle S, A, R, P, \gamma, s_0\rangle,\qquad R:(S\times A)^*\to \mathbb{R},$$9 (Trapasso et al., 16 Dec 2025). In the same paper, the SimHash-based Bucket-QR-MAX extends the factorized construction to continuous state spaces without manual gridding.

6. Generalizations, misconceptions, and open limitations

NMRDPs sit inside a larger family of history-dependent control models. Regular Decision Processes (RDPs) extend MDPs with regular lookback conditions on rewards and transitions; their expressive power coincides with finite-state MDPs because the relevant history can be compiled into a finite automaton state (Hahn et al., 2023). In grid-world studies of RDPs, the state is augmented with reward and transition automata, yielding a product space $R:(S\times A\times S)^*\to\mathbb{R}$0; the results show that sparse non-Markovian rewards degrade model-free learning, that automata-based reward shaping improves exploration, and that even modest non-Markovian transition structure can significantly complicate learning (Lenaers et al., 2021).

Omega-Regular Decision Processes go further by adding omega-regular lookahead constraints interpreted as promises about future behavior. Their payoff is lexicographic:

$R:(S\times A\times S)^*\to\mathbb{R}$1

so the optimization problem is no longer merely “maximize discounted reward,” but “maximize satisfaction probability first, then optimize reward conditioned on satisfaction” (Hahn et al., 2023). This clarifies that non-Markovian reward is only one axis of non-Markovianity studied in the literature.

Partial observability is another major fault line. In the DBMM framework, an RM state $R:(S\times A\times S)^*\to\mathbb{R}$2 Markovizes rewards while a TM state $R:(S\times A\times S)^*\to\mathbb{R}$3 Markovizes observation dynamics, giving an augmented product state space $R:(S\times A\times S)^*\to\mathbb{R}$4 and Bellman backup

$R:(S\times A\times S)^*\to\mathbb{R}$5

This shows that a reward-only augmentation is not, in general, sufficient under partial observability (Wu et al., 3 Aug 2025).

Several limitations recur across the literature. Automata-based approaches often assume deterministic labeling and finite-trace semantics; larger automata increase memory and training time; sparse acceptance rewards remain difficult even with shaping; and partial observability is frequently outside the base model (Miao et al., 2023). Bayesian IRL formulations commonly assume known environment dynamics, a fixed reward alphabet $R:(S\times A\times S)^*\to\mathbb{R}$6, and a fixed reward-machine size $R:(S\times A\times S)^*\to\mathbb{R}$7 (Topper et al., 2024). DBMM inference currently targets deterministic POMDPs with clean traces and meaningful labels (Wu et al., 3 Aug 2025). Policy-gradient memory-learning methods avoid symbolic specification but use high-variance likelihood-ratio estimators and do not yet resolve off-policy sample efficiency in non-Markovian settings (Kar et al., 11 May 2026).

A final misconception is that all non-Markovian reward formalisms are confined to regular languages. The RML work explicitly states that classical reward machines are typically bounded by regular languages, whereas its runtime-monitoring extension can specify reward functions for non-regular, non-Markovian tasks through counting and parameterized conditions (Trapasso et al., 16 Dec 2025). A plausible implication is that future work on NMRDPs will continue to split along two complementary directions: stronger symbolic languages for specifying temporal reward structure, and stronger learned-memory methods for discovering the relevant structure directly from interaction or demonstration.