Quasi-Markov Environments

• Quasi-Markov environments are processes where non-Markov dynamics are managed by augmenting states with finite exogenous history to regain near-Markov properties.
• They enable reinforcement learning methods to leverage classical Bellman theory and policy iteration through state augmentation and controlled truncation of past events.
• Various formulations—including external temporal processes, MEMDPs, and reactive environments—demonstrate practical techniques for balancing truncation error and computational complexity.

Searching arXiv for the provided paper and closely related work on quasi-Markov formulations, non-Markov RL with augmentation, and multiple-environment MDPs. Quasi-Markov environments are processes that are non-Markov in the agent’s native state representation yet admit a Markov, approximately Markov, or asymptotically learnable description after suitable augmentation, conditioning, or forgetfulness assumptions. In the cited literature, the term covers several technically distinct regimes: MDPs perturbed by an external temporal process; near-deterministic environments whose missing variables can be reconstructed by a deterministic factored model; Multiple-Environment MDPs with a fixed but unknown environment parameter; reactive environments satisfying value-stability or recoverability; and interacting Markov processes in a random environment where the environment coordinate is not Markov in its own filtration but the joint process is Markov (Ayyagari et al., 2023, Matuz et al., 2011, Raskin et al., 2014, 0810.5636, Das, 2016).

1. Core meaning and definitional variants

In the reinforcement-learning formulation centered on external temporal processes, an environment is quasi-Markov if the next-state and reward depend on a finite amount of history or on the state of an exogenous temporal process, rather than solely on the current state and action. Formally, a process is quasi-Markov if there exists an integer $m$ and auxiliary variables $Z_t$ such that

$$P(S_{t+1}\in\cdot, R_t\in\cdot \mid S_{0:t}, A_{0:t}) = P(S_{t+1}\in\cdot, R_t\in\cdot \mid S_t, A_t, Z_{t-m+1:t}).$$

This definition makes finite-memory sufficiency explicit: the full past matters only through the current state, current action, and a finite window of exogenous information. The same source also states that when the exogenous process is Markov of finite order $m$, or when the influence of older events decays sufficiently, a Markov model can be regained by augmenting the agent’s state with a finite history of exogenous signals or with sufficient statistics of their effect (Ayyagari et al., 2023).

Taken together, the cited works suggest that “quasi-Markov” is not one uniform formal class but a family of constructions in which Markovity is restored exactly, restored approximately, or replaced by a weaker long-run forgetfulness property.

Formulation	Mechanism	Consequence
External temporal process	Finite window of exogenous marks or sufficient statistics	Approximate or exact Markovization by augmentation
Near-deterministic world	Reconstructed state via deterministic factored FSM and look-ahead	Behavioral state becomes approximately Markov
MEMDP	Hidden static environment parameter $e$	Conditional on $e$, dynamics are Markov
Value-stable reactive environment	Sublinear catch-up and summable tail probabilities	Finite past has negligible long-run cost
Random environment process	Environment marginal is non-Markov; joint process is Markov	Markovity recovered on augmented state space

2. External temporal processes and finite-memory augmentation

The most explicit formalization models an MDP under an external temporal process with base components $S \subset \mathbb{R}^m$, $A \subset \mathbb{R}^n$, discount factor $\gamma \in (0,1)$, bounded measurable reward $r:S\times A\times S\to \mathbb{R}$ with $Z_t$0, and transition kernel $Z_t$1. At each discrete time $Z_t$2, an event indicator $Z_t$3 occurs with a mark $Z_t$4, with $Z_t$5 denoting “no event”. The exogenous history is $Z_t$6, equivalently $Z_t$7. Conditioned on $Z_t$8, the model specifies a perturbed MDP transition kernel $Z_t$9 and a distribution $$P(S_{t+1}\in\cdot, R_t\in\cdot \mid S_{0:t}, A_{0:t}) = P(S_{t+1}\in\cdot, R_t\in\cdot \mid S_t, A_t, Z_{t-m+1:t}).$$0 for the next mark, with factorization

$$P(S_{t+1}\in\cdot, R_t\in\cdot \mid S_{0:t}, A_{0:t}) = P(S_{t+1}\in\cdot, R_t\in\cdot \mid S_t, A_t, Z_{t-m+1:t}).$$1

Because $$P(S_{t+1}\in\cdot, R_t\in\cdot \mid S_{0:t}, A_{0:t}) = P(S_{t+1}\in\cdot, R_t\in\cdot \mid S_t, A_t, Z_{t-m+1:t}).$$2 and reward may depend on the entire past $$P(S_{t+1}\in\cdot, R_t\in\cdot \mid S_{0:t}, A_{0:t}) = P(S_{t+1}\in\cdot, R_t\in\cdot \mid S_t, A_t, Z_{t-m+1:t}).$$3, the process is non-Markov in $$P(S_{t+1}\in\cdot, R_t\in\cdot \mid S_{0:t}, A_{0:t}) = P(S_{t+1}\in\cdot, R_t\in\cdot \mid S_t, A_t, Z_{t-m+1:t}).$$4 alone. Tractability is recovered under regularity assumptions and decaying influence assumptions: there exist nonnegative sequences $$P(S_{t+1}\in\cdot, R_t\in\cdot \mid S_{0:t}, A_{0:t}) = P(S_{t+1}\in\cdot, R_t\in\cdot \mid S_t, A_t, Z_{t-m+1:t}).$$5 and $$P(S_{t+1}\in\cdot, R_t\in\cdot \mid S_{0:t}, A_{0:t}) = P(S_{t+1}\in\cdot, R_t\in\cdot \mid S_t, A_t, Z_{t-m+1:t}).$$6 with $$P(S_{t+1}\in\cdot, R_t\in\cdot \mid S_{0:t}, A_{0:t}) = P(S_{t+1}\in\cdot, R_t\in\cdot \mid S_t, A_t, Z_{t-m+1:t}).$$7 and $$P(S_{t+1}\in\cdot, R_t\in\cdot \mid S_{0:t}, A_{0:t}) = P(S_{t+1}\in\cdot, R_t\in\cdot \mid S_t, A_t, Z_{t-m+1:t}).$$8 such that histories differing only at an event older than $$P(S_{t+1}\in\cdot, R_t\in\cdot \mid S_{0:t}, A_{0:t}) = P(S_{t+1}\in\cdot, R_t\in\cdot \mid S_t, A_t, Z_{t-m+1:t}).$$9 induce total-variation differences bounded by $m$0 for $m$1 and by $m$2 for $m$3. These bounds quantify the decay of influence of old events (Ayyagari et al., 2023).

The exact Markovization uses the infinite augmented state space

$m$4

with augmented state $m$5. On this space, the joint transition over $m$6 is Markov:

$m$7

Practical Markovization truncates history to $m$8 recent marks, $m$9, producing an approximate MDP $e$0 on $e$1. Under the summable total-variation tails, replacing older history by non-events yields controlled error. The corresponding theorem states that there exists a deterministic stationary optimal policy on the infinite augmented MDP, and that for any $e$2 there exists a finite horizon $e$3 and a deterministic stationary policy $e$4 whose value on the true augmented process satisfies $e$5, with $e$6 controlled by the tail sums $e$7 and $e$8. When the exogenous process is finite-order Markov, choosing $e$9 restores the Markov property exactly (Ayyagari et al., 2023).

3. Bellman theory, policy iteration, and sample complexity

On the infinite augmented state space, the value of a fixed policy $e$0 satisfies

$e$1

The associated Bellman operator $e$2 is a $e$3-contraction in $e$4 and therefore has a unique fixed point $e$5. The theory thus preserves the standard Bellman structure, but only after state augmentation by exogenous history (Ayyagari et al., 2023).

The policy-iteration algorithm is explicitly adapted to finite truncation. It initializes a deterministic policy on $e$6, performs approximate policy evaluation by defining a truncated value function $e$7 that randomizes ignored older events with a distribution $e$8, and, in practice, estimates $e$9 using pathwise LSTD with linear features on $S \subset \mathbb{R}^m$0. Approximate improvement then computes an $S \subset \mathbb{R}^m$1 over actions while averaging over proposed older-event histories via $S \subset \mathbb{R}^m$2. The resulting guarantees are explicitly non-stationary. A one-step improvement bound gives

$S \subset \mathbb{R}^m$3

so degradation is bounded by the truncation tail. A stronger theorem states that at any augmented state $S \subset \mathbb{R}^m$4, either $S \subset \mathbb{R}^m$5, or the Bellman residual of $S \subset \mathbb{R}^m$6 at $S \subset \mathbb{R}^m$7 is small:

$S \subset \mathbb{R}^m$8

This localizes possible degradation to near-optimal regions where the Bellman error is already small (Ayyagari et al., 2023).

The sample-complexity analysis is organized around truncated augmentation at horizon $S \subset \mathbb{R}^m$9 and a $A \subset \mathbb{R}^n$0-dimensional linear function class $A \subset \mathbb{R}^n$1 with bounded features $A \subset \mathbb{R}^n$2, $A \subset \mathbb{R}^n$3. For a trajectory of length $A \subset \mathbb{R}^n$4, the pathwise Bellman operator $A \subset \mathbb{R}^n$5 on $A \subset \mathbb{R}^n$6 is

$A \subset \mathbb{R}^n$7

and projection $A \subset \mathbb{R}^n$8 onto sampled features gives the LSTD fixed point $A \subset \mathbb{R}^n$9. The empirical evaluation bound decomposes into approximation and estimation terms, and the expected-error bound under exponentially $\gamma \in (0,1)$0-mixing, ergodic, aperiodic augmented chains further decomposes into: truncation error from discarding events older than $\gamma \in (0,1)$1; inherent approximation error of the linear class; and stochastic estimation error depending on $\gamma \in (0,1)$2, $\gamma \in (0,1)$3, $\gamma \in (0,1)$4, the smallest eigenvalue $\gamma \in (0,1)$5 of the Gram matrix, the sample size $\gamma \in (0,1)$6, and mixing corrections. The stated intuition is that quasi-Markov exogenous dynamics add a truncation penalty that decays with $\gamma \in (0,1)$7 according to $\gamma \in (0,1)$8, whereas long memory or high-dimensional sufficient statistics increase sample complexity through $\gamma \in (0,1)$9 and $r:S\times A\times S\to \mathbb{R}$0 (Ayyagari et al., 2023).

4. Alternative formalizations in adjacent literatures

In near-deterministic reinforcement learning, non-Markovity is formulated at the level of behavioral states rather than exogenous signals. The Markov property is written as

$r:S\times A\times S\to \mathbb{R}$1

and non-Markovity is diagnosed empirically by temporal-difference error distributions: large variance, heavy tails, or multimodality indicate that the current behavioral state omits crucial variables. The proposed correction is to build and run a deterministic, factored Finite State Model of the environment and to select actions by look-ahead planning. The reconstructed state

$r:S\times A\times S\to \mathbb{R}$2

is then approximately Markov when the factored transitions are close to deterministic and include the hidden causal variables. In this formulation, quasi-Markovity is not defined by finite-memory truncation but by successful state reconstruction through factorization and look-ahead (Matuz et al., 2011).

In formal verification, Multiple-Environment Markov Decision Processes treat quasi-Markovity as uncertainty over a latent but static environment parameter $r:S\times A\times S\to \mathbb{R}$3. An MEMDP is

$r:S\times A\times S\to \mathbb{R}$4

where each fixed $r:S\times A\times S\to \mathbb{R}$5 yields an ordinary MDP $r:S\times A\times S\to \mathbb{R}$6. The controller does not know $r:S\times A\times S\to \mathbb{R}$7 a priori and must satisfy robust or multi-threshold guarantees simultaneously for all environments. Qualitative almost-sure and limit-sure reachability, safety, and parity are decidable in polynomial time; quantitative reachability and safety with thresholds are NP-hard even on acyclic MEMDPs and even for memoryless strategies; fixed-memory quantitative synthesis is solvable in PSPACE by bilinear encodings; and finite-memory strategies suffice for arbitrary $r:S\times A\times S\to \mathbb{R}$8-approximation. Here the process is quasi-Markov because it is Markov conditional on a fixed hidden parameter, while the controller’s information state is not fully Markov until that parameter is inferred or statistically distinguished (Raskin et al., 2014).

In reactive environments with arbitrary dependence, quasi-Markovity appears as a forgetfulness condition rather than an explicit state-space augmentation. The key notions are recoverability and value-stability. Recoverability requires that after any finite history with positive probability, some policy can still attain the optimal asymptotic average value almost surely. Value-stability strengthens this to a uniform catch-up condition: there exist benchmark rewards $r:S\times A\times S\to \mathbb{R}$9, a sublinear term $Z_t$00, and tail probabilities $Z_t$01 with $Z_t$02, such that after any $Z_t$03-step history the agent can catch up to the optimal-from-start benchmark up to $Z_t$04 with summable tail error. A countable class of value-stable environments admits a self-optimizing policy, and recoverability yields an upper self-optimizing policy. Finite ergodic MDPs and certain ergodic finite POMDPs satisfy these conditions, while a merely linear catch-up term $Z_t$05 is not sufficient in general. In this line of work, quasi-Markovity is an asymptotic negligibility of finite past, not a direct finite-state Markovization (0810.5636).

In interacting Markov processes in a random environment, the environment coordinate is called quasi-Markov because it is not necessarily Markov in its own filtration once its generator depends on the current basic process. The combined process $Z_t$06, possibly enlarged with transitional states $Z_t$07 or $Z_t$08, is nevertheless a continuous-time Markov chain with explicit product-form stationary measure

$Z_t$09

This construction accommodates simultaneous changes in both coordinates and allows $Z_t$10 to depend on $Z_t$11, with augmentation restoring exact Markovianity on the joint space (Das, 2016).

5. Canonical examples and empirical findings

A concrete exogenous-process example is the discrete-time marked Hawkes process. The event intensity is

$Z_t$12

and when $Z_t$13, the mark $Z_t$14 has mean $Z_t$15, while $Z_t$16 otherwise. If $Z_t$17 and $Z_t$18 decay sufficiently fast, then the total-variation bounds $Z_t$19 and $Z_t$20 hold for events older than $Z_t$21, yielding truncation error proportional to $Z_t$22. For exponential decay $Z_t$23, the tail scales as $Z_t$24 (Ayyagari et al., 2023).

The principal experiment in that framework uses a non-stationary variant of OpenAI Gym Pendulum-v1, with a Gaussian-marked discrete-time Hawkes process perturbing the torque applied to the pendulum. The setup specifies exponentially decaying $Z_t$25 with $Z_t$26, polynomially decaying $Z_t$27, a fixed neural network policy partially pre-trained by DDPG, and pathwise LSTD with linear features on the augmented state. The metric is mean squared error of the learned value function versus the number of samples $Z_t$28, the event horizon $Z_t$29, and the decay rate $Z_t$30. The reported findings are that increasing $Z_t$31 reduces MSE; increasing $Z_t$32 first decreases MSE and then may slightly increase it because feature dimension and conditioning worsen; and faster decay $Z_t$33 reduces MSE because the truncation tail is smaller. The practical guidance is to choose $Z_t$34 by balancing truncation error against feature dimensionality, to monitor Bellman residuals and improvement regions, and, under partial observability, to include available proxies such as filtered intensity estimates or recent event counts as features (Ayyagari et al., 2023).

The near-deterministic Ms. Pac-Man study provides a different empirical picture. The environment is modeled by a deterministic factored FSM with components such as dots, power dots, ghost factors, wall and corridor factors, and Pac-Man position and direction. Look-ahead search uses determinized ghost policies, no backtracking, and stopping on reset events. Shallow look-ahead with depth at most $Z_t$35 reduced performance, whereas deep look-ahead with depth $Z_t$36 improved overall score to approximately $Z_t$37, reduced average lives lost to approximately $Z_t$38, and increased score per life to approximately $Z_t$39, compared with a baseline average overall score of approximately $Z_t$40, average lives lost of approximately $Z_t$41, and score per life of approximately $Z_t$42. The paper interprets these gains as evidence that a near-deterministic, factored approximation can correct non-Markovity in behavioral macro-states when the relevant factors are identifiable and branching is low (Matuz et al., 2011).

6. Relation to adjacent notions, limitations, and open questions

A common misconception is to treat quasi-Markov environments as synonymous with generic partial observability. The cited work draws sharper distinctions. In the external-process formulation, state augmentation targets non-Markovianity created by exogenous signals and does not assume future full observability; this is explicitly contrasted with the QMDP heuristic for POMDPs. In MEMDPs, the hidden component is one static variable $Z_t$43 with full observability of $Z_t$44, which explains why several reachability, safety, and parity problems become decidable there although the corresponding POMDP problems are undecidable in general (Ayyagari et al., 2023, Raskin et al., 2014).

A second misconception is that quasi-Markovity is always exact. Some formulations are exact only after augmentation by an infinite history or by a latent environment parameter; some are approximate and controlled by decay bounds such as $Z_t$45; some are asymptotic and rely on value-stability or recoverability rather than a finite-dimensional Markov state. This suggests that the phrase should be interpreted operationally: it indicates that non-Markov dependence can be controlled well enough for planning, synthesis, or learning, but the mechanism varies across formalisms (0810.5636).

The limitations are equally formulation-specific. In the exogenous-process setting, the bounds assume bounded rewards, strong continuity, and summable total-variation tails; abrupt regime shifts, adversarial external processes, infinite-memory or heavy-tailed influence, model-estimation error in the improvement step, poor Gram-matrix conditioning, and extension to nonlinear function classes remain open issues. In near-deterministic look-ahead, incorrect determinization, large branching factor, missing factors, and brittle decision-surface construction can mislead planning, and shallow look-ahead can degrade performance. In value-stable reactive environments, the $Z_t$46 catch-up condition and countability of the environment class are essential in general. In interacting Markov processes in random environments, explicit constructions are given for finite $Z_t$47 and $Z_t$48, while extensions to diffusions, general compact metric spaces, and semi-Markov environments with heavy-tailed holding times are left open (Ayyagari et al., 2023, Matuz et al., 2011, Das, 2016).

Across these lines of work, the unifying insight is that non-Markovity need not be treated as an undifferentiated obstacle. It may arise from external temporal processes, omitted but reconstructible variables, a fixed latent environment, or arbitrary dependence whose long-run effect is negligible. The corresponding remedies—finite-memory augmentation, deterministic factor reconstruction, environment identification, and value-stability—provide distinct but mathematically precise ways to make quasi-Markov environments analyzable.