Ergodic Theorems for discrete Markov chains

Abstract: Let $X_n$ be a discrete time Markov chain with state space $S$ (countably infinite, in general) and initial probability distribution $\mu^{(0)} = (P(X_0=i_1),P(X_0=i_2),\cdots,)$. What is the probability of choosing in random some $k \in \mathbb{N}$ with $k \leq n$ such that $X_k = j$ where $j \in S$? This probability is the average $\frac{1}{n} \sum_{k=1}^n \mu^{(k)}_j$ where $\mu^{(k)}_j = P(X_k = j)$. In this note we will study the limit of this average without assuming that the chain is irreducible, using elementary mathematical tools. Finally, we study the limit of the average $\frac{1}{n} \sum_{k=1}^n g(X_k)$ where $g$ is a given function for a Markov chain not necessarily irreducible.