First-Order Markov Feedback Processes

• First-order Markov feedback processes are stochastic systems in which the future state depends only on the current augmented state, including feedback and control signals.
• Augmenting the state with controller memory or sufficient statistics preserves the Markov property, enabling tractable dynamic programming and convex optimization.
• These processes are applied in diverse fields such as signal tracking, LQG control, and statistical physics, providing rigorous frameworks for rate–distortion analysis and optimal inference.

A first-order Markov feedback process is a stochastic system in which the evolution of the state, the feedback controller, and potentially additional random inputs is governed by a first-order Markov structure. That is, at each time step, the future evolution is conditionally independent of the past given the current (possibly augmented) state and the most recent feedback or control action. This framework encompasses diverse settings in communication, control, statistical inference, and statistical physics, generalizing classical Markov chains to include feedback, memory, actuation, and inference with structural separation.

1. Formal Structure and Definitions

A process $\{Z_t\}_{t=0}^\infty$ is a first-order Markov feedback process if for all $t \ge 0$ and all histories,

$$P(Z_{t+1} \mid Z_t, Z_{t-1}, \dots, Z_0, F_t, F_{t-1}, \dots) = P(Z_{t+1} \mid Z_t, F_t)$$

where $F_t$ denotes feedback signals, controller memory, or communication acts that themselves may depend on current/past system states or measurements.

Typical realizations include:

• Gauss–Markov tracking over erasure channels, with state dynamics $x_{t+1} = \alpha x_t + w_t$ and feedback through packet arrivals/acknowledgements (Khina et al., 2017).
• Linear Additive Markov Processes (LAMP), where transitions are randomly applied from (possibly far) history locations, endowing the process with nontrivial autocorrelation and feedback properties while remaining first-order Markov on an extended state (Smart et al., 2022).
• Dynamic panel logit models with feedback, where covariates depend on the lagged outcome in a Markovian fashion (Shin, 4 Nov 2025).
• Feedback-controlled physical systems, for example, discrete-time Fokker–Planck control, where controller memory is included in the state to restore Markovianity (Wu et al., 2023, Ruiz-Pino et al., 5 Apr 2024).
• Average-reward Markov decision processes, where feedback appears via actions selected as a function of the observed state and policy (Li et al., 2022).

Maintaining the Markov property is generally achieved by augmenting the system state with all components (e.g., controller memory, feedback signals) on which either feedback or future evolution depend.

2. Key Mathematical Principles

The Markovian characterization is fundamentally based on conditional independence: the future depends only on the present, possibly augmented, state. In systems with feedback, the present state typically includes both the physical system variables and any memory or feedback variables that affect control or actuation.

A prototypical structure is

$Z_{t+1} = f(Z_t, U_t, W_t)$

where $U_t$ is selected (possibly stochastically) as a function of observation/feedback, and $W_t$ is system noise. In control problems,

• $U_t$ may be defined as $U_t = g(Z^t, C_t)$ for some memory $C_t$,
• $C_{t+1} = h(C_t, Z_t, U_t, W_t)$.

The process is first-order Markov in the pair $(Z_t, C_t)$; all future behaviour is fully determined given the current joint state.

In communication or estimation frameworks, information states (beliefs, posteriors, or occupancy measures) are often used as sufficient statistics, ensuring the same first-order property (0709.3753, Shin, 4 Nov 2025).

3. Representative Models and Rate/Distortion Properties

First-order Markov feedback processes occur in a wide range of signal, control, and inference models:

3.1 Gauss–Markov Tracking/Control over Erasure Channels

For $x_{t+1} = \alpha x_t + w_t$, with $w_t \sim {\cal N}(0,W)$, and transmission over erasure-prone links with acknowledgements, key results are:

• The optimal observer protocol is greedy in the prediction error, and for $N \to \infty$ parallel processes and instantaneous ACKs, DPCM quantization is globally optimal (Khina et al., 2017).
• The rate-distortion recursion obeys $D_t^* = (\alpha^2 D_{t-1}^* + W) B$, with $B=1-\beta(1-2^{-2R})$, under erasure rate $\beta$.
• For delayed ACKs, the expected distortion increases only slightly; the increase is quantified via Kaspi's two-decoder Gaussian result.
• In scalar LQG control, the separation principle allows propagation of estimation MSE directly into the LQG cost, both lower and upper bounds being given by $D_t^*$ and its ECDQ variant.

3.2 LAMP and Entropy Rate

A LAMP on state space $S=\{1,...,n\}$ with lag distribution $w_k$ and base Markov kernel $P$ evolves as

$$\mathbb{P}(X_n = j \mid X_0,...,X_{n-1}) = \sum_{k \geq 1} w_{k} P_{X_{n-k},j},$$

with $K_n$ (lag) i.i.d. (Smart et al., 2022). The entropy rate is

$$H_{\mathrm{LAMP}} = -\sum_{i,j} \pi_i P_{ij} \log P_{ij}$$

where $\pi$ is the stationary distribution for $P$. This rate is identical to the base chain, regardless of feedback-induced long-range dependencies.

3.3 Markovian Fokker–Planck Control

General state steering for $x_{k+1}=a_k x_k + b_k u_k + w_k$ is solvable by expressing the process in the space of its first $2n$ moments $\mathscr X_k$, and using a mixed feedback–Markov kernel law for $u_k$: $u_k = -c_k a_k x_k + \mathscr F_k$ with $\mathscr F_k$ an independent noise variable and $0 \le c_k \le 1$. Moment-matching is ensured via convex optimization under the Hankel-positivity constraint, and the kernel $\mathscr F_k$ is constructed by minimum-KL relative entropy given the moment sequence (Wu et al., 2023).

4. Feedback, Sufficient Statistics, and Identification

The presence of feedback requires redefining the system state to include any controller memory, estimation beliefs, or sufficient statistics to restore Markovianity.

• In panel data models with Markov feedback, sufficient statistics for individual heterogeneity and feedback kernels are derived via the factorization theorem. For fully general first-order Markov feedback in covariates, identification of covariate effects via conditional likelihood is generally impossible, although the lagged dependent variable is identified with $T \geq 3$ periods (Shin, 4 Nov 2025).
• In dynamic programming formulations for real-time communication with noisy feedback, optimal encoder/decoder policies depend only on the current information state (joint belief or filter value), justifying Bellman recursion structure and enabling optimal policy derivation (0709.3753).

Domain	Markov Feedback Variable	Markovization Augmentation
Estimation over channels	Packet ACK, estimator memory	$(x_t,$ ACK bits, quantizer state)
Fokker–Planck control	Controller kernel $\mathscr F_k$	$(x_k,$ controller memory)
Dynamic panel data	Lagged covariate, outcome	$(Y_{t}, X_{t})$ as joint Markov
LAMP	Lag variable $K_n$	$(X_{n}, K_{n})$ extended state
MDP (RL)	Action $a_t$ (policy output)	$(s_t, a_t)$ state–action chain

5. Practical Implications, Optimization, and Theoretical Guarantees

First-order Markov feedback processes enable concise modeling of systems with feedback, observation, and control, while retaining the computational and analytical tractability of first-order Markov systems (sometimes after suitable state augmentation).

Optimization problems reduce to finite-dimensional convex programs in several cases:

• In general discrete-time Fokker–Planck control, all required operations—moment system propagation, feedback–kernel selection, and moment-matched sampling—are convex, and uniqueness/existence are rigorously established (Wu et al., 2023).
• In stochastic control and rate–distortion problems (e.g., large-system DPCM settings), optimality is achieved by simple greedy algorithms, and the theoretical bounds are explicit and tight (Khina et al., 2017).

Feedback may create identification challenges in estimation contexts: conditioning-out unobserved heterogeneity or nuisance Markov kernels via sufficient statistics may "thin" the variation needed to identify all structural parameters (Shin, 4 Nov 2025).

In stochastic optimization (as in average-reward MDPs), first-order methods (policy mirror-descent, variance-reduced TD) are effective provided the feedback structure is properly encoded—scaling is governed by the size of the parameter space rather than state/action cardinalities (Li et al., 2022).

6. Embedding and Universality in Physical and Information Systems

The universality of the first-order Markov feedback process construction lies in the principle that any system (physical, communication, econometric) whose future evolution depends on the present augmented state (including controller/estimator memory, measurement outcomes, feedback signals) can be rendered Markovian by explicit state augmentation.

• In feedback flashing ratchets and similar systems, the joint process of system coordinate and controller memory is rigorously Markovian, admitting Chapman–Kolmogorov or Fokker–Planck master equation description (Ruiz-Pino et al., 5 Apr 2024).
• This principle ensures that powerful tools—spectral theory, stationary distributions, entropy methods, dynamic programming—apply to systems with nontrivial memory and feedback, as long as the state space is properly defined.

7. Applications and Interpretive Summary

First-order Markov feedback processes underpin diverse advances:

• Rate-distortion theory and LQG control under communication constraints (Khina et al., 2017).
• Statistical models with feedback covariates, essential for treatment inference and dynamic policy evaluation (Shin, 4 Nov 2025).
• Efficient first-order stochastic optimization for large-scale MDPs (Li et al., 2022).
• Non-equilibrium physical system modeling, enabling correct stationary and time-dependent analysis in feedback-driven dynamics (Ruiz-Pino et al., 5 Apr 2024).
• Complex dependency modeling with minimal parameter overhead, as in LAMP, allowing long-range structure without the curse of dimensionality (Smart et al., 2022).

The Markov property generally emerges only after appropriate enrichment of the state to include the feedback or memory variables—without this, naive models may misestimate entropy rates, fail to identify parameters, or admit suboptimal policy design. A plausible implication is that, in any modern inference, control, or communication architecture involving feedback applied at time scales shorter than system relaxation, it is generally necessary to design algorithms and analyses in the augmented first-order Markov feedback setting to obtain valid results and optimal performance.