Discrete-Time Markov Chains

• Discrete-Time Markov Chain (DTMC) is a stochastic process defined on a countable state space where the next state depends solely on the current state.
• DTMCs are widely used in spectral analysis, model checking, queuing theory, and infrastructure reliability across various research fields.
• Key analyses involve state classification, ergodic properties, mixing time estimation, and empirical calibration for practical applications.

A discrete-time Markov chain (DTMC) is a system consisting of a countable state space and a sequence of random variables with the Markov property—future states depend only on the present state and not on the past trajectory. DTMCs are foundational in stochastic modeling, probabilistic verification, statistical physics, combinatorics, queuing theory, and infrastructure reliability. The core mathematical structure is characterized by a (sub)stochastic transition matrix $P$, with applications ranging from spectral graph theory to model checking and large-scale infrastructure monitoring.

1. Formal Structure and Definitions

A DTMC is a sequence of $S$-valued random variables $X_0, X_1, \ldots$ defined on a probability space $(\Omega, \mathcal{F}, P)$ such that for all $n$, $$P(X_{n+1}=j \mid X_n = i, X_{n-1}, ...) = P(X_{n+1}=j \mid X_n = i) =: P_{ij}$$. Here, $S$ is the state space (finite or countably infinite), and $P = (P_{ij})$ is the transition matrix—row-stochastic if $P_{ij}\geq0$ and $\sum_j P_{ij}=1$ for each $S$0; sub-stochastic if $S$1. The initial distribution is denoted by $S$2 where $S$3 (Halidias, 2017).

Paths of length $S$4 can be represented as $S$5, with probability $S$6 (Hartmanns et al., 2 Sep 2025). Absorbing and goal states may be specified for application contexts.

2. State Classification and Ergodic Properties

States in a DTMC are classified based on recurrence:

• Transient: $S$7
• Null Recurrent: Recurrence probability $S$8 but infinite expected return time
• Positive Recurrent: Recurrence probability $S$9 and finite expected return time $X_0, X_1, \ldots$0

For each pair $X_0, X_1, \ldots$1, $X_0, X_1, \ldots$2 is the hitting probability. The ergodic theorem for occupancy states that $X_0, X_1, \ldots$3 if $X_0, X_1, \ldots$4 is positive recurrent and $X_0, X_1, \ldots$5, otherwise $X_0, X_1, \ldots$6 (Halidias, 2017). For arbitrary initial law $X_0, X_1, \ldots$7, time-averaged occupancy probabilities are weighted by $X_0, X_1, \ldots$8.

3. DTMCs in Graph-Theoretic and Structural Contexts

Constructing a DTMC from a graph $X_0, X_1, \ldots$9 with adjacency (or weight) matrix $(\Omega, \mathcal{F}, P)$0 and degree matrix $(\Omega, \mathcal{F}, P)$1 is standard. For undirected graphs, $(\Omega, \mathcal{F}, P)$2 yields a symmetric or doubly stochastic transition matrix (when $(\Omega, \mathcal{F}, P)$3 is symmetric and unweighted). The stationary distribution satisfies $(\Omega, \mathcal{F}, P)$4, specializing to $(\Omega, \mathcal{F}, P)$5 for undirected weighted graphs, and $(\Omega, \mathcal{F}, P)$6 for doubly stochastic $(\Omega, \mathcal{F}, P)$7 (regular, unweighted, undirected) (Murthy, 2012). Graph structure determines chain properties, with connectedness and aperiodicity ensuring ergodicity.

Spectral decomposition applies when $(\Omega, \mathcal{F}, P)$8 is symmetric: $(\Omega, \mathcal{F}, P)$9, allowing exact analysis of convergence and mixing properties. Exponential mixing is controlled by $n$0, the second-largest eigenvalue, with mixing time $n$1 (Murthy, 2012). Shannon entropy $n$2 increases monotonically for doubly stochastic $n$3, reaching $n$4 in the limit.

4. Reachability, Path Abstraction, and Model Checking

Probabilistic model checking in DTMCs requires computation of probabilities for reaching goal states $n$5 from an initial state $n$6. The reachability probability is $n$7, summing over all non-zero probability paths from $n$8 to $n$9 (Hartmanns et al., 2 Sep 2025).

Path abstraction leverages the free monoid $$P(X_{n+1}=j \mid X_n = i, X_{n-1}, ...) = P(X_{n+1}=j \mid X_n = i) =: P_{ij}$$0: by removing subsets $$P(X_{n+1}=j \mid X_n = i, X_{n-1}, ...) = P(X_{n+1}=j \mid X_n = i) =: P_{ij}$$1 ("intermediate" states), one constructs a path-abstracted chain $$P(X_{n+1}=j \mid X_n = i, X_{n-1}, ...) = P(X_{n+1}=j \mid X_n = i) =: P_{ij}$$2. Fundamental absorption theorems establish that abstracting over nested subsets $$P(X_{n+1}=j \mid X_n = i, X_{n-1}, ...) = P(X_{n+1}=j \mid X_n = i) =: P_{ij}$$3 yields $$P(X_{n+1}=j \mid X_n = i, X_{n-1}, ...) = P(X_{n+1}=j \mid X_n = i) =: P_{ij}$$4, guaranteeing invariance under the order of abstraction over non-goal sets. The free-monoid formalism yields equational proofs of these properties and circumvents reliance on graph-theoretic SCC decompositions.

This algebraic framework enables early counterexample refinement in model checking and decouples path abstraction from combinatorial explosion in classical state elimination (Hartmanns et al., 2 Sep 2025).

5. Statistical Inference and Parametric Modeling

DTMCs provide parametric structures for empirical modeling, exemplified in infrastructure deterioration studies. For sewer pipe degradation, state space $$P(X_{n+1}=j \mid X_n = i, X_{n-1}, ...) = P(X_{n+1}=j \mid X_n = i) =: P_{ij}$$5 (discrete severity levels) is used, with homogeneous DTMC calibrated per cohort and failure type (Jimenez-Roa et al., 2023). Calibration employs weighted least-squares minimization of prediction error, under constraints enforcing Markov structure: row-stochasticity, structural zeros prohibiting "healing," and absorbing final states.

"Chain Multi" allows arbitrary non-improving transitions ($$P(X_{n+1}=j \mid X_n = i, X_{n-1}, ...) = P(X_{n+1}=j \mid X_n = i) =: P_{ij}$$6 for $$P(X_{n+1}=j \mid X_n = i, X_{n-1}, ...) = P(X_{n+1}=j \mid X_n = i) =: P_{ij}$$7), whereas "Chain Single" restricts transitions to self-loops and one-step worsening ($$P(X_{n+1}=j \mid X_n = i, X_{n-1}, ...) = P(X_{n+1}=j \mid X_n = i) =: P_{ij}$$8). Model selection favors "Chain Single" for parsimony and improved numerical stability. Expected future severity $$P(X_{n+1}=j \mid X_n = i, X_{n-1}, ...) = P(X_{n+1}=j \mid X_n = i) =: P_{ij}$$9 enables direct cohort comparison.

6. MCMC, Mixing Time, and Reversible Chains

In combinatorial optimization, reversible positive recurrent DTMCs are used for Markov Chain Monte Carlo (MCMC) sampling from complex distributions. The Metropolis chain used for Gibbs distribution sampling over perfect matchings is irreducible, aperiodic, and its stationary measure is given by $S$0 (Moothedath et al., 2017). Conductance $S$1 bounds and canonical paths yield explicit mixing time estimates, e.g., $S$2 with $S$3.

Mixing properties depend on spectral gaps (via $S$4) and detailed balance, ensuring convergence to the target distribution. This analytic machinery underpins practical rapid mixing results central to probabilistic sampling and randomized algorithms involving DTMCs.

7. Applications in Queueing, Communication, and Optimization

DTMCs underpin operational models across engineering domains. For instance, buffer-occupancy in bidirectional Fano decoders is modeled as a DTMC on occupancy states $S$5, with transition probabilities tied to the random decoding time and traffic parameters (Xu et al., 2010). Performance metrics—frame erasure, mean occupancy, throughput, and delay—are computed from the stationary distribution.

In system design, the trade-off between the number of parallel decoders and input buffer size can be optimized using DTMC-predicted performance. Minimal total area per throughput is achieved by maximizing $S$6, where $S$7 is the per-decoder input rate for buffer size $S$8.

---

DTMC theory thus provides an analytically tractable foundation for modeling, estimation, performance analysis, and verification in diverse research and application contexts, with deep interplay between structure, spectral properties, and statistical inference (Halidias, 2017, Murthy, 2012, Hartmanns et al., 2 Sep 2025, Jimenez-Roa et al., 2023, Xu et al., 2010, Moothedath et al., 2017).