Markov Chain Formalism

• Markov chain formalism is a rigorous probabilistic framework that models stochastic evolution where future states depend solely on the present state.
• It employs discrete-time and continuous-time methods using transition matrices and generators, enabling spectral and coupling approaches to assess system behavior.
• The formalism facilitates efficient computational techniques for stationary distributions and convergence analysis, underpinning applications like MCMC and network models.

The Markov chain formalism provides a rigorous algebraic-probabilistic framework for describing stochastic evolution on discrete or continuous state spaces, where the future trajectory of the process is conditionally independent of its past given its present state. The formalism underpins a vast array of probabilistic modeling, both theoretical and applied, and is characterized by strong structural results on asymptotic behavior, spectral theory, and explicit computations for quantities such as stationary distributions and transient probabilities.

1. Discrete-Time Markov Chains: Matrix Structure and Semigroup Properties

A (time-homogeneous) Markov chain on a finite state space $S = \{1, 2, \ldots, k\}$ is completely specified by its transition matrix

$$P = (p_{ij})_{i,j=1}^k, \quad p_{ij} \geq 0,\quad \sum_{j=1}^k p_{ij} = 1 \quad \forall\, i$$

The law of the chain at step $n$ is given by

$\mu^{(n)} = \mu P^n$

where $\mu$ is the initial distribution. The key semigroup property, encapsulated by the Chapman–Kolmogorov equations, states (for $m, n \geq 0$)

$$P^{n+m} = P^n P^m,\quad (P^{n+m})_{ij} = \sum_{\ell=1}^k (P^n)_{i\ell} (P^m)_{\ell j}$$

This algebraic structure enables the study of the system's evolution through spectral and geometric analysis of $P$ and its powers (Siems, 2019).

2. Irreducibility, Aperiodicity, and Stationary Laws

A transition matrix $P$ is called irreducible if, for every $i,j \in S$, there exists $$P = (p_{ij})_{i,j=1}^k, \quad p_{ij} \geq 0,\quad \sum_{j=1}^k p_{ij} = 1 \quad \forall\, i$$0 such that $$P = (p_{ij})_{i,j=1}^k, \quad p_{ij} \geq 0,\quad \sum_{j=1}^k p_{ij} = 1 \quad \forall\, i$$1. The period of a state $$P = (p_{ij})_{i,j=1}^k, \quad p_{ij} \geq 0,\quad \sum_{j=1}^k p_{ij} = 1 \quad \forall\, i$$2 is

$$P = (p_{ij})_{i,j=1}^k, \quad p_{ij} \geq 0,\quad \sum_{j=1}^k p_{ij} = 1 \quad \forall\, i$$3

and the chain is aperiodic if $$P = (p_{ij})_{i,j=1}^k, \quad p_{ij} \geq 0,\quad \sum_{j=1}^k p_{ij} = 1 \quad \forall\, i$$4 for all $$P = (p_{ij})_{i,j=1}^k, \quad p_{ij} \geq 0,\quad \sum_{j=1}^k p_{ij} = 1 \quad \forall\, i$$5. In the finite irreducible setting, aperiodicity of one state implies aperiodicity of all. These two properties yield strong control over the existence and uniqueness of stationary distributions and also over long-term convergence. The stationary (invariant) distribution $$P = (p_{ij})_{i,j=1}^k, \quad p_{ij} \geq 0,\quad \sum_{j=1}^k p_{ij} = 1 \quad \forall\, i$$6 satisfies

$$P = (p_{ij})_{i,j=1}^k, \quad p_{ij} \geq 0,\quad \sum_{j=1}^k p_{ij} = 1 \quad \forall\, i$$7

The Perron–Frobenius theorem applies: $$P = (p_{ij})_{i,j=1}^k, \quad p_{ij} \geq 0,\quad \sum_{j=1}^k p_{ij} = 1 \quad \forall\, i$$8 is a simple eigenvalue of $$P = (p_{ij})_{i,j=1}^k, \quad p_{ij} \geq 0,\quad \sum_{j=1}^k p_{ij} = 1 \quad \forall\, i$$9 with strictly positive left and right eigenvectors, and when normalized, this yields the unique stationary law (Siems, 2019).

3. Long-Term Behavior: Convergence and Spectral Analysis

If $n$0 is irreducible and aperiodic with unique stationary law $n$1, then for all $n$2

$n$3

$n$4

where $n$5 is the $n$6 column vector of ones. Coupling arguments (e.g., the sandwich/coupling proof) and spectral methods (projection to the eigenspace of $n$7) both formalize this convergence, which underpins the correctness of Markov Chain Monte Carlo methods on finite spaces (Siems, 2019).

4. Continuous-Time Markov Chains: Jump–Hold Construction and Generator Formalism

A continuous-time Markov chain (CTMC) on a countable state space $n$8 is determined by its generator $n$9, satisfying

$\mu^{(n)} = \mu P^n$0

The jump matrix $\mu^{(n)} = \mu P^n$1 is constructed with

$\mu^{(n)} = \mu P^n$2

The chain can be explicitly simulated by the Kendall–Gillespie algorithm, alternating exponential holding times and jumps according to $\mu^{(n)} = \mu P^n$3. The time-evolution satisfies the Chapman–Kolmogorov property and is governed by the Kolmogorov forward and backward equations: $\mu^{(n)} = \mu P^n$4

$\mu^{(n)} = \mu P^n$5

Stationary distributions $\mu^{(n)} = \mu P^n$6 solve $\mu^{(n)} = \mu P^n$7, and ergodic results provide convergence, under regularity conditions, of $\mu^{(n)} = \mu P^n$8 as $\mu^{(n)} = \mu P^n$9 (Kuntz, 2020).

5. Beyond the Standard Setting: Cycle Structure, Non-Standard Asymptotics, and Functionals

For Markov chains on finite nonrecurrent state spaces with absorbing states (as in rank–frequency analysis), the cycle structure of the transition graph determines the asymptotic behavior of the ordered path probability distribution $\mu$0. There are four main regimes:

• Acyclic (finite words): only finitely many paths exist, $\mu$1 for large $\mu$2.
• Power-law (Zipf-type): when the graph contains vertices on two distinct simple cycles, $\mu$3 where the exponent is determined by the unique $\mu$4 such that the spectral radius $\mu$5; $\mu$6.
• Intermediate (faster-than-polynomial, subexponential): if each vertex belongs to at most one simple cycle, $\mu$7 decays faster than any polynomial but slower than some exponentials.
• Exponential: if every path visits at most one cycle, $\mu$8 with explicit $\mu$9 computable from cycle weights (Bochkarev et al., 2012).

6. Markov Chains over Algebraic and Combinatorial Structures

In the setting of combinatorial Hopf algebras, descent operators induce Markov transition kernels via algebraic (co)product structure. For a graded–connected Hopf algebra $m, n \geq 0$0 with basis $m, n \geq 0$1, refined coproducts $m, n \geq 0$2 and associated descent operators $m, n \geq 0$3 (for weak-compositions $m, n \geq 0$4 of $m, n \geq 0$5) yield Markov transitions. Under mild positivity, the transition matrix is

$m, n \geq 0$6

The stationary distribution is obtained by "product–then–coproduct" enumeration, and the spectrum admits a uniform formula in terms of internal characters on symmetric functions. Explicit eigenbases can be constructed, giving exact expectations for a variety of statistics (Pang, 2016).

7. Practical Computation: Recurrence, Return Times, and Steady-State Availability

Recurrence and transience are determined by the structure of the communication classes. Regeneration (return to a state) enables ergodic theorems and long-term averages. In practical settings with CTMCs, steady-state availability (SSA) equations can often be written in closed form in networks with "decision-process" or "sequential-process" topologies:

• Decision process: for states branching from $m, n \geq 0$7 to $m, n \geq 0$8 at rates $m, n \geq 0$9, returning at rates $$P^{n+m} = P^n P^m,\quad (P^{n+m})_{ij} = \sum_{\ell=1}^k (P^n)_{i\ell} (P^m)_{\ell j}$$0,

$$P^{n+m} = P^n P^m,\quad (P^{n+m})_{ij} = \sum_{\ell=1}^k (P^n)_{i\ell} (P^m)_{\ell j}$$1

• Sequential process (closed chain): for cyclic chains, stationary probability at $$P^{n+m} = P^n P^m,\quad (P^{n+m})_{ij} = \sum_{\ell=1}^k (P^n)_{i\ell} (P^m)_{\ell j}$$2 is

$$P^{n+m} = P^n P^m,\quad (P^{n+m})_{ij} = \sum_{\ell=1}^k (P^n)_{i\ell} (P^m)_{\ell j}$$3

These formulae bypass the need to construct and invert the full generator matrix, and can be composed for networks with mixed motifs (Vasconcelos, 2017).

---

References:

• (Siems, 2019): Markov Chain Monte Carlo on Finite State Spaces
• (Bochkarev et al., 2012): Zipf and non-Zipf Laws for Homogeneous Markov Chain
• (Kuntz, 2020): Markov chains revisited
• (Pang, 2016): Markov Chains from Descent Operators on Combinatorial Hopf Algebras
• (Vasconcelos, 2017): Steady state availability general equations of decision and sequential processes in Continuous Time Markov Chain models
• (Allen-Perkins et al., 2019): Markov chain approach to anomalous diffusion on Newman-Watts networks