---
title: Non-Stationary Markov Process
url: https://www.emergentmind.com/topics/non-stationary-markov-process
type: topic
---

# Non-Stationary Markov Process

A non-stationary Markov process is a stochastic process in which the Markov property holds, but the transition probabilities or rates depend explicitly on time or on other evolving external variables. Unlike classical stationary Markov processes, where the law of evolution is time-homogeneous, non-stationary variants capture systems with dynamically changing environments, parameters, or underlying structures. This generalization is fundamental in accurately describing complex real-world phenomena that exhibit temporal heterogeneity, abrupt regime shifts, or gradual evolution, and underpins a large spectrum of modeling and algorithmic approaches in statistical physics, epidemiology, queueing, control, and reinforcement learning.

## 1. Mathematical Formulations

### Discrete and Continuous-Time Models

A discrete-time Markov decision process (MDP) with time-dependent transitions is formally defined as the tuple $(\mathcal{S},\,\mathcal{A},\,\{P_t\},\,\{R_t\},\,\gamma)$, where for each $t\ge0$,
\[
P_t(s' \mid s,a) = \Pr\bigl[S_{t+1}=s' \mid S_t=s,\,A_t=a\bigr]
\]
and $R_t(s,a)$ is the immediate reward function [2501.09646]. The core characteristic is that either $P_t$ or $R_t$ (or both) depend on $t$ or on an exogenous process $\theta_t$.

In continuous time, a non-stationary Markov jump process is characterized by a (possibly time-dependent) generator (rate) matrix $Q(t)$ with
\[
\frac{d}{dt}P(t_0, t) = P(t_0, t) Q(t), \qquad P(t_0, t_0) = I,
\]
where each off-diagonal $q_{ij}(t)\geq 0$ and $q_{ii}(t) = -\sum_{j\neq i}q_{ij}(t)$ [2505.21519, 2506.08282].

### Examples of Explicit Non-Stationarity

- **Time-dependent transition parameters**: $P_{ij}(t) = f_{ij}(t)$ with $\sum_j f_{ij}(t)=1$ for each $t$ [2505.21519].
- **Exogenous parameter-driven transitions**: $P_t(s'|s,a) = P(s'|s,a,\theta_t)$ with $\theta_t$ a stochastic process, e.g., a Markov chain or random walk [2501.09646].

## 2. Modeling Frameworks and Classes

### Markov Chains and Processes

Various classes arise from the specific structure of non-stationarity:

| Class                        | Defining Feature                                     | Key Reference       |
|------------------------------|-----------------------------------------------------|---------------------|
| Piecewise-stationary         | Blocks of constant $P_t$, switching at change points | [2501.09646]        |
| Smoothly time-varying        | $P_t$ drifts continuously with $t$                  | [2501.09646]        |
| Periodic (cyclostationary)   | $P_t = P_{t+T}$ for some period $T$                 | [2506.08282]        |
| Exogenous parameter-driven   | $\theta_t$ stochastic, $P_t = P(\cdot, \theta_t)$   | [2501.09646]        |
| Path-dependent Markovian     | Transition rates depend on both $n$ and $t$         | [2503.03896]        |
| Copula-based non-stationary  | Markov property encoded via time-varying copulas    | [1704.01458]        |
| Switching MDP (SNS-MDP)      | Underlying unobserved mode $\theta_t$ Markov chain  | [2503.18607]        |

Non-stationarity may be abrupt (stepwise), continuous (drift), or periodic, with modeling choices depending on the dynamics under study [2501.09646].

## 3. Analytical Results and Computational Schemes

### Chapman–Kolmogorov and Balance Systems

Non-stationary Markov processes obey time-dependent forward equations. For discrete time, balanced systems relate compartment counts via increments (e.g., $\Delta_{|S|} = \Delta_{10} - \Delta_1$, etc.) [2505.21519]. In continuous time, the Kolmogorov equation generalizes as:
\[
\frac{d\pi_i(t)}{dt} = \sum_{j\neq i} \pi_j(t)q_{ji}(t) - \pi_i(t)\sum_{j\neq i}q_{ij}(t)
\]
or, for controlled settings, with explicit policy dependence [2505.21519, 2506.08282].

### Limit Theorems and Long-run Behavior

Law of Large Numbers (LLN) and Central Limit Theorems (CLT) have been established for non-stationary Markov jump processes:
- Under mild regularity, cumulative reward $R(t)$ satisfies $R(t)/E[R(t)] \to 1$ almost surely as $t\to\infty$.
- If transitions and rewards are periodic in $t$, then the time-averaged reward converges to the periodic mean, and normalized fluctuations are asymptotically normal [2506.08282].

For certain classes, explicit limit cycles or absorbing structures can arise, as in time-inhomogeneous chains with feedback or reinforcement [2309.10974].

### Performance Approximations

For slowly varying $P_t$, rigorous first-order corrections to stationary performance measures (e.g., discounted rewards, hitting times, expected occupation times) are derived via linear systems with perturbed matrices, providing $O(\epsilon)$-accurate approximations with complexity identical to the stationary case [1805.01662].

## 4. Stochastic Diffusion, Anomalous Dynamics, and Memory

### Non-Stationary Anomalous Diffusion

Markovian replication processes (NMRP) on lattices, with time-dependent replication probability $p(t)$, yield generalized telegrapher equations:
\[
\frac{\partial \rho}{\partial t} + \mathcal{R}(t)\frac{\partial^2\rho}{\partial t^2} = \mathcal{D}(t)\frac{\partial^2\rho}{\partial x^2},
\]
with $\mathcal{D}(t)$ and $\mathcal{R}(t)$ determined by $p(t)$ [1709.02074]. Classification is governed by $p(t)$'s functional form—alternating, power-law, or marginal—producing a spectrum of diffusion behaviors (sub-, super-, or ultra-slow diffusion).

A further generalization introduces both state and time dependence in transition rates:
\[
\lambda_n(t) = \frac{\beta + \gamma n}{1+\rho t}
\]
The dynamics balance a contagion term and a time-damping, with phase diagram (sub-, superdiffusive, ballistic, hyperballistic) indexed by $H=\gamma/\rho$ [2503.03896]. Non-stationarity is necessary for all regimes but the ballistic case.

Deviations from Gaussianity and violations of the classical CLT arise generically due to non-stationarity and autocorrelation [2503.03896, 1709.02074].

## 5. Algorithmic and Control Implications

### Reinforcement Learning and Decision Processes

Non-stationarity in MDPs fundamentally impacts both policy structure and algorithm design:
- **Time-indexed value functions and Bellman recursions**: $V_t^*(s)$ and $Q_t^*(s,a)$ are recomputed for each $t$, requiring time-aware dynamic programming or Q-learning [2511.17598, 2505.21519, 2501.09646].
- **Switching environments**: SNS-MDPs with latent Markovian mode switches retain TD-learning and Q-learning convergence due to ergodicity of the joint $(\theta_t, s_t)$ process [2503.18607].
- **Delayed reinforcement**: In delayed MDPs, optimal policies must be non-stationary Markov (i.e., $a_t = d_t(s_t)$, not time-invariant), as stationary Markov policies can be strictly sub-optimal when delay $m>0$ [2101.11992].
- **Algorithmic approaches**: ASP(RL), hybridization with logical solvers, and smooth forgetting via exponential weights in value estimation support adaptation to evolving dynamics [2501.09646, 2010.12870, 1705.01399].

### Practical Benchmarks

Simulation toolkits such as NS-Gym enable systematic benchmarking of algorithms on non-stationary environments, offering a modular framework for emulating parametric (e.g., periodic, abrupt, or drifting) evolution of underlying MDP parameters [2501.09646].

## 6. Statistical, Dynamical, and Nonparametric Models

Bayesian nonparametric models construct non-stationary Markovian dynamics on real-valued data without pre-imposed functional forms or stationarity assumptions. For example, transition densities can be specified via Dirichlet process mixtures of bivariate normals, yielding time-homogeneous but marginally non-stationary Markov models suitable for capturing evolving or heteroscedastic time series [1601.04331].

Similarly, copula-based constructions facilitate both the representation and verification of $\beta$-mixing (absolute regularity) under time-varying dependence parameters, with explicit bounds on mixing rates related to the maximal-correlation coefficients of the evolving copulas [1704.01458].

## 7. Applications and Empirical Insights

Non-stationary Markov process modeling is central to a range of empirical domains:
- **Epidemiological modeling**: Time-varying compartment transition rates enable accurate simulation of disease waves, policy response, and resource allocation, outperforming stationary models which fail to capture non-equilibrium dynamics [2505.21519, 2503.03896].
- **Healthcare and system maintenance**: Feedback-driven non-stationary Markov chains predict treatment or repair cycles, optimizing resource management in complex service systems [2309.10974].
- **Queueing and service operations**: Time-of-day or week-dependent rates require LLN/CLT development for performance analysis under realistic, fluctuating workloads [2506.08282].
- **Communications and adaptive protocols**: Switching MDPs capture network channels with Markovian mode-switching (e.g., due to fading), guiding robust protocol adaptation [2503.18607].
- **Algorithmic robustness**: Benchmark environments synthesized via NS-Gym, as well as theoretical regret bounds for non-stationary linear MDPs, illustrate the necessity of temporal adaptation and model update mechanisms [2501.09646, 2010.12870].

Empirical evidence across these domains consistently demonstrates superior fidelity and policy efficacy when explicitly modeling or learning with non-stationary Markovian dynamics.

---

In summary, non-stationary Markov processes provide a canonical framework for representing, analyzing, and controlling complex systems in which time or exogenous factors drive structural shifts. Their mathematical characterization demands explicit temporal indexing or dynamic parameter evolution, and their effective deployment encompasses new algorithms, limit theorems, and empirical methodologies, all underpinned by a diverse and technically rigorous research literature [2505.21519, 2503.18607, 2501.09646, 1709.02074, 2309.10974, 2503.03896, 2506.08282, 1705.01399, 1805.01662, 2511.17598, 2101.11992, 1601.04331, 2010.12870, 1704.01458].

Source: https://www.emergentmind.com/topics/non-stationary-markov-process