Ergodic Theory for Controlled Markov Chains with Stationary Inputs (1604.04013v2)

Abstract: Consider a stochastic process ${X(t)}$ on a finite state space $ {\sf X}={1,\dots, d}$. It is conditionally Markov, given a real-valued `input process' ${\zeta(t)}$. This is assumed to be small, which is modeled through the scaling, [ \zeta_t = \varepsilon \zeta^1_t, \qquad 0\le \varepsilon \le 1\,, ] where ${\zeta^1(t)}$ is a bounded stationary process. The following conclusions are obtained, subject to smoothness assumptions on the controlled transition matrix and a mixing condition on ${\zeta(t)}$: (i) A stationary version of the process is constructed, that is coupled with a stationary version of the Markov chain ${X^\bullet$(t)}obtained with ${\zeta(t)}\equiv 0$. The triple $({X(t)}, {X^\bullet(t)},{\zeta(t)})$ is a jointly stationary process satisfying [ {\sf P}{X(t) \neq X^\bullet(t)} = O(\varepsilon) ] Moreover, a second-order Taylor-series approximation is obtained: [ {\sf P}{X(t) =i } ={\sf P}{X^\bullet(t) =i } + \varepsilon^2 \varrho(i) + o(\varepsilon^2),\quad 1\le i\le d, ] with an explicit formula for the vector $\varrho\in\mathbb{R}^d$. (ii) For any $m\ge 1$ and any function $f\colon {1,\dots,d}\times \mathbb{R}\to\mathbb{R}^m$, the stationary stochastic process $Y(t) = f(X(t),\zeta(t))$ has a power spectral density $\text{S}_f$ that admits a second order Taylor series expansion: A function $\text{S}^{(2)}_f\colon [-\pi,\pi] \to \mathbb{C}^{ m\times m}$ is constructed such that [ \text{S}_f(\theta) = \text{S}^\bullet_f(\theta) + \varepsilon^2 \text{S}_f^{(2)}(\theta) + o(\varepsilon^2),\quad \theta\in [-\pi,\pi] . ] An explicit formula for the function $\text{S}_f^{(2)}$ is obtained, based in part on the bounds in (i). The results are illustrated using a version of the timing channel of Anantharam and Verdu.