---
title: 'Markov Chains: Definition, Applications, and Implications'
url: https://www.emergentmind.com/topics/markov-chains
type: topic
---

# Markov Chains: Definition, Applications, and Implications

Markov chains are stochastic processes whose next state depends on the current state and not on the complete history. For a time-homogeneous discrete-time Markov chain (DTMC) with transition matrix $P$, a row-vector distribution evolves according to $\pi(n+1)=\pi(n)P$, hence $\pi(n)=\pi(0)P^n$. A continuous-time Markov chain (CTMC) is specified by a generator $Q$, with nonnegative off-diagonal entries and vanishing row sums; its transition semigroup is $P(t)=e^{Qt}$ and its distribution evolves as $\pi(t)=\pi(0)e^{Qt}$. Markov chains provide a common formalism for random walks, population processes, queueing systems, molecular reaction networks, tensor-network optimization, graph dynamics, filtering, metastability, statistical learning, and applications in management, migration, epidemiology, and financial modeling.

## 1. Mathematical formulation and fundamental objects

Let $S$ be a finite or countable state space. A discrete-time process $(X_n)_{n\geq 0}$ is a Markov chain if

$$
\Pr(X_{n+1}=j\mid X_n=i,X_{n-1},\ldots,X_0)
=
\Pr(X_{n+1}=j\mid X_n=i).
$$

For a homogeneous chain, the transition probabilities are independent of $n$:

$$
P_{ij}=\Pr(X_{n+1}=j\mid X_n=i),
$$

with

$$
P_{ij}\geq 0,\qquad \sum_jP_{ij}=1.
$$

Under the row-vector convention, $\pi(n+1)=\pi(n)P$. The $m$-step transition probabilities are entries of $P^m$, and the Chapman–Kolmogorov equations are

$$
P^{m+n}=P^mP^n.
$$

A stationary distribution is a probability vector satisfying

$$
\pi=\pi P,\qquad \sum_i\pi_i=1,\qquad \pi_i\geq 0.
$$

In continuous time, the generator $Q$ satisfies

$$
Q_{ij}\geq 0\quad (i\neq j),\qquad Q_{ii}\leq 0,\qquad \sum_jQ_{ij}=0.
$$

The transition semigroup is

$$
P(t)=e^{Qt},
$$

and stationarity is characterized by $\pi Q=0$. A CTMC admits a jump-and-hold representation: the embedded jump chain determines the next state, while the holding time at $i$ is exponential with rate

$$
q(i)=\sum_{j\neq i}Q_{ij}.
$$

The Kendall–Gillespie construction samples an exponential holding time and then selects the next state according to normalized jump rates. For countable state spaces, the resulting minimal CTMC may be explosive if infinitely many holding times have a finite sum [2001.02183].

The Markov property is a conditional-independence statement, not temporal independence. Trajectories remain dependent over time; the current state is a sufficient summary of the past for predicting the next transition. If a process has longer memory, it can often be represented as a first-order Markov chain on an augmented state space consisting of recent histories. An order-$k$ symbolic chain, for example, uses $k$-blocks as states and transitions according to the shift rule

$$
(a_{t-k+1},\ldots,a_t)
\longrightarrow
(a_{t-k+2},\ldots,a_t,a_{t+1}).
$$

## 2. Communication structure, recurrence, and limiting behavior

A state $j$ is accessible from $i$ if $(P^m)_{ij}>0$ for some $m\geq 0$. States communicate when each is accessible from the other. Communication partitions the state space into communicating classes. A class is closed when no transition leads outside it.

A finite chain is irreducible when it has one communicating class. Irreducibility gives a unique stationary distribution with strictly positive entries. Reducible chains can contain transient states and multiple closed recurrent classes. Every stationary distribution is a convex combination of the stationary distributions supported on the positive recurrent closed classes.

The period of a state $i$ is

$$
d_i=\gcd\{m\geq 1:(P^m)_{ii}>0\}.
$$

An irreducible chain is aperiodic when its period is one. For a finite irreducible aperiodic chain,

$$
\pi(0)P^n\longrightarrow \pi
$$

for every initial distribution. If the chain is periodic, a stationary distribution can still exist, but the sequence of distributions may oscillate rather than converge. A deterministic two-state alternation has period two: its one-time distributions alternate, whereas its empirical occupation distribution converges to the uniform distribution on the two states.

An ergodic finite chain is irreducible and aperiodic. An absorbing state satisfies $P_{ii}=1$. In an absorbing chain, every state can reach an absorbing state, and long-run analysis concerns absorption probabilities, expected absorption times, and transient-state occupation rather than a nontrivial equilibrium over the entire state space.

For an absorbing chain, ordering transient states first gives

$$
P=
\begin{pmatrix}
Q&R\\
0&I
\end{pmatrix}.
$$

The fundamental matrix is

$$
N=(I-Q)^{-1}=\sum_{m=0}^{\infty}Q^m.
$$

Its entry $N_{ij}$ is the expected number of visits to transient state $j$ when starting from transient state $i$. The absorption-probability matrix is $B=NR$, and expected absorption times are given by $N\mathbf 1$. These formulas are used in models of production, inventory, delivery, graduation, epidemiological termination, and population processes [1601.01304; 2408.04902].

For countable chains, recurrence is classified through return times. A state is recurrent if it returns with probability one and transient otherwise. A recurrent state is positive recurrent if its mean return time is finite and null recurrent if the mean return time is infinite. For a positive recurrent state $x$,

$$
\pi(x)=\frac{1}{\mathbb E_x[\phi_x]},
$$

and the stationary distribution is the normalized expected occupation measure of one regeneration cycle. Regeneration at successive returns converts dependent trajectories into independent and identically distributed cycles. Foster–Lyapunov drift conditions provide practical criteria for recurrence, positive recurrence, and geometric convergence [2001.02183].

## 3. Spectral theory, reversibility, and mixing

The transition matrix always has the right eigenvector $\mathbf 1$ with eigenvalue $1$, while stationary distributions are left eigenvectors with eigenvalue $1$. If $P$ is diagonalizable, its dynamics can be represented as

$$
P^n=\sum_\omega \lambda_\omega^n r_\omega l_\omega^{T},
$$

where $r_\omega$ and $l_\omega$ are dual right and left eigenvectors. The distribution therefore decomposes into modes whose amplitudes are multiplied by $\lambda_\omega^n$.

Eigenvalues with $|\lambda|<1$ describe decaying modes. Eigenvalues near $1$ correspond to slowly relaxing or metastable structures. Negative eigenvalues produce transient oscillations, while complex eigenvalues produce rotating or cyclic modes. Eigenvalues on the unit circle encode persistent periodic behavior. For a diagonalizable ergodic chain, $1$ is the only eigenvalue of modulus one, and the nonstationary modes decay.

If $P$ is reversible with respect to $\pi$, it satisfies detailed balance:

$$
\pi_iP_{ij}=\pi_jP_{ji}.
$$

Writing $\Pi=\operatorname{diag}(\pi)$, the similarity transform

$$
K=\Pi^{1/2}P\Pi^{-1/2}
$$

is symmetric. Reversible chains consequently have real eigenvalues, a complete eigenbasis, and no defective Jordan blocks. Non-reversible chains can have complex eigenvalues, incomplete eigenbases, and Jordan blocks; in such cases, polynomial factors can multiply exponential decay. Schur decompositions and additive reversibilization provide alternative analytical tools [2207.02296].

The spectral gap controls asymptotic relaxation in many reversible settings. For non-reversible chains, spectral information alone is generally insufficient to characterize finite-time behavior. The distinction between asymptotic decay and total-variation mixing is therefore important: eigenvalues close to the unit circle indicate slow modes, but quantitative mixing bounds may require coupling, regeneration, conductance, or model-specific arguments.

The spectral structure of a chain can also be analyzed through semigroup actions. A finite Markov chain admits a random-mapping representation as a random walk generated by a finite semigroup. For chains generated by $\mathscr R$-trivial monoids, eigenvalues are explicit probability sums indexed by a lattice, while multiplicities arise by Möbius inversion. Diagonalizability follows under sufficient separation of eigenvalues. Minimal ideals yield absorption and coupling bounds, and constant maps provide strong-stationary-time interpretations [1401.4250].

For more general finite semigroup actions, Rhodes and McCammond graph expansions retain transition-history information and organize paths through unique simple routes. Weighted path sums can be converted into rational expressions using loop graphs and Kleene expressions. When the minimal ideal is left zero, the probability of not yet reaching the ideal bounds the distance to stationarity [2006.00895].

## 4. Graphs, random walks, and entropy

A weighted directed graph with weight matrix $W$ induces a DTMC through row normalization:

$$
P=D^{-1}W,\qquad D_{ii}=\sum_jW_{ij}.
$$

For an undirected graph, $W=W^T$, and the random walk is reversible with stationary distribution proportional to degree:

$$
\pi_i=\frac{d_i}{\sum_jd_j}.
$$

For a simple unweighted undirected graph with adjacency matrix $A$, the corresponding constructions are

$$
P=D^{-1}A
$$

for the discrete-time random walk and

$$
Q=A-D
$$

for the continuous-time chain. The CTMC jumps from vertex $i$ to neighboring vertex $j$ at rate one and leaves $i$ at total rate $d_i$. For connected graphs, the CTMC converges exponentially to its stationary distribution. In a general irregular undirected graph, the DTMC stationary distribution is degree-proportional rather than uniform; uniform stationarity occurs for regular graphs or other doubly stochastic normalizations [1209.3387].

For reversible graph walks, the symmetric similarity transform gives

$$
K=D^{-1/2}WD^{-1/2},
$$

and the normalized graph Laplacian is

$$
\mathcal L=I-D^{-1/2}WD^{-1/2}.
$$

The smallest Laplacian eigenvalues represent smooth, slowly varying graph modes, while transition eigenvalues near one represent persistent dynamical modes. These orderings need not coincide. The Laplacian framework underlies spectral clustering, Laplacian eigenmaps, diffusion maps, graph Fourier analysis, graph-based classification, and reinforcement-learning representations such as successor features [2207.02296].

Shannon entropy of a distribution $p$ is

$$
H(p)=-\sum_ip_i\log p_i.
$$

A doubly stochastic transition matrix makes distributions more uniform in the majorization order and therefore satisfies

$$
H(pP)\geq H(p).
$$

If the chain converges to the uniform distribution on $M$ states, then

$$
H(p_n)\longrightarrow \log M.
$$

For a graph, the normalized degree sequence

$$
p_i=\frac{d_i}{\sum_kd_k}
$$

defines a graph entropy

$$
H_{\mathrm{graph}}=-\sum_ip_i\log p_i.
$$

This entropy is maximal exactly when all vertices have the same degree. Regular graphs, including cycle and fully connected graphs, are therefore maximum-entropic in this degree-based sense [1209.3387].

Graph structure can encode Markov-chain structure in both directions. Structured transition matrices correspond to structured directed graphs, while graph Laplacians and adjacency spectra determine transient and equilibrium behavior. For a connected regular graph with degree $c$,

$$
P=\frac1cA,\qquad Q=A-cI.
$$

The eigenvalues of $P$ are adjacency eigenvalues divided by $c$, while the eigenvalues of $Q$ are adjacency eigenvalues shifted by $-c$.

## 5. Algebraic and constructive frameworks

Markov chains can be generated from algebraic operations on combinatorial objects. In a graded connected combinatorial Hopf algebra, the coproduct breaks an object into pieces and the product recombines them. A descent operator has the form

$$
m\Delta_P=\sum_D P(D)\binom{n}{d_1,\ldots,d_\ell}m\Delta_D,
$$

where $P$ is a probability distribution on weak compositions $D=(d_1,\ldots,d_\ell)$.

The resulting operator need not be stochastic. A Doob $h$-transform supplies the transition matrix:

$$
K(x,y)=\frac{\eta(y)}{\eta(x)}[y](m\Delta_P(x)),
$$

where $\eta$ is a strictly positive eigenvector of the dual operator. The probabilistic interpretation is a three-stage procedure: choose a breaking pattern, sample pieces through coproduct coefficients, and recombine them through product coefficients.

The framework includes riffle shuffles, top-to-random shuffles, restriction-then-induction chains on representations, processes on rooted trees, set partitions, rocks, graphs, and permutations. In the shuffle algebra, the Hopf-power operator produces the Gilbert–Shannon–Reeds riffle shuffle: a deck is cut into piles and the piles are uniformly interleaved. For distinct cards, the stationary distribution is uniform, and eigenvalues are powers of the number of piles [1412.8221; 1609.04312].

Under commutative or cocommutative hypotheses, Eulerian idempotents, primitive elements, Poincaré–Birkhoff–Witt theory, the Cartier–Milnor–Moore theorem, and Lyndon-word constructions provide complete eigenbases. Free-commutative bases tend to produce independent breaking, triangular transition matrices, and absorbing chains. Cofree bases tend to produce recombination-rich, irreducible chains such as riffle shuffles.

Tensor-network optimization supplies a continuous-state example. A tensor-network state is parameterized by a configuration of local unitaries, and a Metropolis chain targets the Boltzmann distribution

$$
\pi^{(\beta)}(\omega)\propto e^{-\beta h(\omega)}.
$$

Haar-random unitary perturbations together with Metropolis acceptance yield a reversible chain. Increasing $\beta$ produces simulated annealing, which concentrates probability on low-energy tensor-network configurations. The method has been applied to MPS and PEPS, with global proposals, contraction-error bounds for PEPS, and applications to ground-state, excited-state, spectral, and finite-temperature objectives [1309.4880].

## 6. Generalizations, inference, computation, and applications

### Random environments and non-homogeneous chains

A Markov chain in a stationary random environment has transition kernel $Q(Y_t,X_t,\cdot)$, where $(Y_t)$ is an environmental process. Conditionally on the environment, the state process is an inhomogeneous Markov chain; unconditionally, it is generally not homogeneous.

Under a random Lyapunov drift inequality, average contractivity over environmental blocks, and random minorization conditions, the chain admits an annealed limiting distribution and quenched convergence for almost every environmental trajectory. The same assumptions yield laws of large numbers for bounded observables. Applications include queues with dependent service times, stochastic-gradient Langevin dynamics with dependent data, and random-coefficient autoregressive systems [1911.04377].

### Metastability and finite-state reduction

Weakly noisy perturbations of deterministic maps can possess metastable neighborhoods around stable fixed points or periodic orbits. If transitions between neighborhoods have quasipotential costs $H(i,j)$, then transition probabilities scale as

$$
\exp\left\{-\frac{H(i,j)}{\sigma^2}\right\}.
$$

The corresponding Markov kernel has $N$ eigenvalues exponentially close to one, one for each metastable component, while the remainder of the spectrum is separated by a spectral gap. After local equilibration, the dynamics can be reduced uniformly in time to an $N$-state Markov chain whose transition probabilities are exponentially close to quasistationary-distribution-started first-passage probabilities [2303.12624].

### Estimation and adaptive sampling

When multiple unknown ergodic chains are available for sampling, an adaptive learner can allocate transitions among them to estimate all transition matrices. The loss

$$
L_n=\max_k\sum_x\pi_{k,n}(x)
\left\|P_k(x,\cdot)-\widehat P_{k,n}(x,\cdot)\right\|_2^2
$$

weights row-wise errors by empirical state occupancy. The oracle allocation assigns samples proportionally to

$$
G_k^\Sigma=\sum_x\sum_yP_k(x,y)(1-P_k(x,y)).
$$

The Bandit Allocation for Markov Chains policy uses optimistic upper-confidence indices to balance exploration and exploitation. Under the stated assumptions, it attains the asymptotic benchmark

$$
L_n\sim \frac{\Lambda}{n},\qquad
\Lambda=\sum_kG_k^\Sigma
$$

and provides finite-sample PAC guarantees [1905.11128].

When trajectories are observed only at random times, a gap of length $\ell$ identifies the $\ell$-step transition matrix $P^\ell(z)$ conditional on covariates $z$. The one-step matrix can be estimated through

$$
P(z)=\exp\left\{\frac1\ell\log(P^\ell(z))\right\},
$$

with kernel estimators for random-gap transition probabilities and regularization when an estimated matrix power has no stochastic matrix root [2505.12971].

### Chemical and molecular implementation

Finite CTMCs can be implemented by chemical reaction networks. Each Markov state is represented by a molecular species, and each transition $i\to j$ by a unimolecular reaction

$$
S_i\xrightarrow{k_{ij}}S_j.
$$

Under deterministic mass-action kinetics, concentrations satisfy the same forward equations as the target CTMC when $k_{ij}$ is proportional to the transition rate. Equilibrium concentrations therefore encode stationary probabilities. Bimolecular reactions provide a compact representation for second-order chains, while DNA strand displacement provides a compilation route subject to concentration, rate, leakage, and finite-sequence constraints [1802.05170].

### Population, epidemiological, and migration models

Markov binomial chains model compartment populations through simultaneous binomial transfers. For an acyclic transfer graph, a topological ordering induces a well-founded decrease in the population-state order, proving almost-sure absorption and finite expected termination time. The expected termination vector satisfies the standard absorbing-chain equations. Despite exponentially many reachable states and irrational transition probabilities, approximation belongs to $\mathbf{PSPACE}$; specialized SIR algorithms exploit triangularity to obtain an exact $O(N^4)$ algorithm in the Blum–Shub–Smale model [2408.04902].

Refugee migration can be represented by a population vector on a geographically constructed graph. Daily transition probabilities integrate several within-day movements subject to a distance budget. Time-varying location types make the process non-homogeneous. The resulting population evolution is

$$
B(t+1)=A(t)B(t).
$$

This population-level model is more computationally efficient than agent-based simulation when the number of refugees is large, but it suppresses individual heterogeneity, history, and behavioral decision-making [1903.08255].

### Filtering, importance sampling, and model comparison

For CTMC observations with target rates $\gamma_{i\to j}(t)$ and proposal rates $\overline\gamma_{i\to j}$, a jump-process likelihood ratio changes the proposal law into the target law. The same rate-change formula supports importance sampling, rejection sampling, branching particle filters, model comparison, and nonlinear filtering.

For a hidden process $X$ and observed CTMC $Y$, the unnormalized filter is

$$
\sigma_t(B)=
\mathbb E^Q\left[
A_t\mathbf 1_{\{X_t\in B\}}
\mid \mathcal F_t^Y
\right],
$$

and the normalized filter is

$$
\pi_t(f)=\frac{\sigma_t(f)}{\sigma_t(1)}.
$$

The quantity $\sigma_t(1)$ is an integrated likelihood or Bayes factor. For finite hidden state spaces, filtering reduces to linear ODE evolution between observation jumps and diagonal likelihood updates at observed transitions [2308.06192].

## 7. Limitations and conceptual scope

The Markov property is a modeling assumption. It can fail when transition probabilities depend on longer histories, seasonality, changing policies, latent variables, individual heterogeneity, or external shocks. A higher-order chain, a covariate-dependent chain, a random-environment model, or a hidden Markov model may provide a more appropriate state representation, but each enlarges the inferential or computational problem.

Stationarity and convergence are not synonymous. A stationary distribution may exist without uniqueness, and an irreducible periodic chain may possess a unique stationary distribution without converging pointwise from every initial state. Similarly, spectral gaps, asymptotic eigenvalue behavior, and finite-time mixing bounds measure related but distinct properties.

Reversibility is a special structural condition, not a generic property. Undirected random walks and symmetric generators are reversible under appropriate degree-weighted laws, whereas directed graph chains, Hopf-algebraic descent chains, non-reversible Laguerre chains, and many chains in random environments are not. Non-reversibility can introduce complex spectra, nonorthogonal eigenfunctions, transient amplification, and Jordan blocks.

Finite-state computations can conceal substantial complexity. State augmentation for higher-order models grows exponentially with the memory order; symbolic cryptocurrency models face sparse high-order contexts; epidemiological population chains can have exponentially many reachable states; and metastable systems may require exponentially long observation times. Compact representations, dynamic programming, semigroup methods, adaptive sampling, spectral reduction, and probabilistic model checking address different aspects of this complexity but do not eliminate it.

Markov chains nevertheless provide a unified language for local stochastic dynamics and global probabilistic behavior. Transition matrices and generators specify evolution; communicating classes and recurrence determine qualitative long-term structure; stationary laws describe equilibrium; eigenvalues and eigenvectors describe relaxation and metastability; graph, semigroup, Hopf-algebraic, chemical, and statistical constructions supply specialized representations; and coupling, regeneration, drift, quasistationarity, and filtering provide mechanisms for analyzing models beyond direct matrix powers.

Source: https://www.emergentmind.com/topics/markov-chains