Exogenous Block Controlled Markov Process (Ex-BCMP)

• Ex-BCMPs are defined by decomposing the state space into an exogenous block driven by uncontrollable factors and an endogenous block responsive to control actions.
• This framework reduces variance and improves sample efficiency in reinforcement learning by treating exogenous noise separately from controlled dynamics.
• Practical algorithms, including global optimization over the Stiefel manifold and greedy stepwise methods, enable automated discovery of the exogenous components.

An Exogenous Block Controlled Markov Process (Ex-BCMP) is a formalism used to model stochastic systems in which the state space and/or transition dynamics exhibit a block structure, where a subset of the process is driven by exogenous (uncontrolled) factors and another block is susceptible to control or endogenous intervention. Ex-BCMPs generalize classical controlled Markov processes by explicitly separating uncontrollable exogenous dynamics from controllable endogenous dynamics. This separation is crucial for efficient analysis and learning in high-dimensional, noisy, or non-stationary environments, as frequently encountered in reinforcement learning (RL), stochastic approximation (SA), queuing systems, and representation learning.

1. Mathematical Structure and Decomposition

The canonical Ex-BCMP formalization involves decomposing the state space as $S = E \times X$ where:

• $E$: endogenous subspace (states influenced by control/policy/action)
• $X$: exogenous subspace (states evolving independently of control)

The transition kernel is specified by factorization:

$$P(e', x' | e, x, a) = P(x' | x) \cdot P(e' | e, x, a)$$

where $P(x' | x)$ models the exogenous block, and $P(e' | e, x, a)$ models the conditional transitions in the endogenous block given the exogenous state and action.

The reward function is decomposable:

$r = r_{\mathrm{exo}} + r_{\mathrm{endo}}$

with $r_{\mathrm{exo}} \sim R_{\mathrm{exo}}(\cdot|x)$ (mean $m_{\mathrm{exo}}(x)$), and $$r_{\mathrm{endo}} \sim R_{\mathrm{endo}}(\cdot|e,x,a)$$ (mean $m_{\mathrm{endo}}(e,x,a)$).

This block structure allows the value function to be separated as:

$$V(e, x; h) = V_{\mathrm{exo}}(x; h) + V_{\mathrm{endo}}(e, x; h)$$

where the exogenous and endogenous components satisfy their respective (Bellman-like) recursions.

2. Stability and Convergence Properties in Stochastic Approximation

The Ex-BCMP framework is prominently utilized in the analysis of stochastic approximation algorithms driven by controlled Markov noise (Ramaswamy et al., 2015). The key stability conditions (S1 and S2) are:

• (S1) Limiting behavior: For scaled objective functions $h_c(x, y) = h(cx, y)/c$, any limit of $h_c$ as $c \rightarrow \infty$ must belong to the upper limit set $h_{\infty}(x, y)$.
• (S2) Attractor existence: The differential inclusion

$$\frac{dx}{dt} \in H(x), \quad H(x) = \overline{{\mathrm{conv}} \bigcup_{y \in S} h_{\infty}(x,y)}$$

must admit a compact attracting set $\mathcal{A}$ with the closed unit ball $B_1(0)$ as a fundamental neighborhood.

These conditions guarantee almost sure boundedness (stability) and convergence of SA iterates even over continuous state spaces, and without requiring ergodicity of the underlying controlled Markov process—a significant generalization over previous analyses requiring finiteness and ergodicity.

A representative growth bound for the set-valued map $H$ is:

$\sup \{ \|u\| : u \in H(x) \} \leq K(1+\|x\|)$

ensuring linear growth in the state parameter.

3. Practical Algorithms for Exogenous/Endogenous Discovery

Given that the separation $S = E \times X$ is often non-trivial in real systems, several algorithms have been developed to discover exogenous blocks automatically (Dietterich et al., 2018, Trimponias et al., 2023):

• Global optimization (over the Stiefel manifold): Projects high-dimensional states onto a low-dimensional candidate exogenous subspace via constrained optimization (using mutual information or conditional correlation coefficient constraints).
• Stepwise (greedy): Iteratively discovers exogenous dimensions one at a time by solving a simplified independence criterion at each step.
• Both approaches rely on minimizing conditional dependence between $X'$ and $(E, A)$, enforcing $I(X'; E, A| X) \approx 0$, and regression-based removal of exogenous reward.

These techniques enable unsupervised structure discovery even in continuous, high-dimensional MDPs where exogenous and endogenous state variables are linearly mixed.

Algorithm	Optimizes over	Independence Criterion	Comments
Global	Stiefel manifold	CCC/Mutual Information	Finds block at once
Stepwise	Unit sphere/null space	Simplified independence	One dim at a time

4. Implications for Reinforcement Learning and Sample Efficiency

A central implication of Ex-BCMP modeling is that, under additive reward decomposition and block-transition structure, the optimal policy for the endogenous MDP (focusing on $r_{\mathrm{endo}}$) is always optimal for the full MDP (Trimponias et al., 2023, Dietterich et al., 2018). This separation induces substantial variance reduction, accelerating policy evaluation and RL. Formally, the Monte Carlo sample complexity for the endogenous problem is reduced if

$$\mathrm{Var}[B_x^{\pi}] > -2\, \mathrm{Cov}(B_x^{\pi}, B_e^{\pi})$$

for $B_x^{\pi}$ (exogenous H-step return) and $B_e^{\pi}$ (endogenous H-step return).

Empirical studies reveal dramatic speedups when exogenous reward is subtracted and the endogenous block is discovered, with sample complexity and convergence time decoupled from the dimension of uncontrollable exogenous noise (Levine et al., 2024).

5. Extensions to Non-Stationary, Temporal, and Block-Structured Models

Ex-BCMPs are generalizable to models incorporating non-Markovian exogenous temporal processes (Ayyagari et al., 2023), block-structured Markov/queueing systems (Li et al., 2019), and Markov-modulated affine processes in finance (Kurt et al., 2021). In these settings:

• Augmentation by exogenous event history yields an MDP with a non-stationary transition kernel $Q_{H_t}$ perturbed by external processes.
• Optimal policies are shown to exist when the influence of old exogenous events decays sufficiently, with performance guarantees and sample complexity bounds depending on the decay rates ($M_t, N_t$) of exogenous influence.
• Matrix-geometric, RG-factorization, and differential inclusion techniques provide tractable computation and stability analysis even for infinite-dimensional or exponentially large state spaces.

6. Limitations and Directions for Further Research

While the Ex-BCMP framework enables powerful decomposition and variance reduction, certain limitations persist:

• Discovery algorithms for block structure rely on accurate conditional independence estimation; in practice, mixture of exogenous and endogenous effects may necessitate more sophisticated causal inference methods.
• Model assumptions such as deterministic controllable dynamics (Levine et al., 2024) and linear block emission mappings may not hold in all environments.
• Extensions to stochastic, continuous, or combinatorial endogenous blocks, especially in the absence of resets or in the presence of heavy noise, represent active areas of research.
• Real-world deployment requires adaptive, online updating of exogenous block estimations and robust handling of function approximation (e.g., with deep learning).

Future work may address scalable causal block discovery, more flexible block emission models, integration with temporal-difference and policy gradient architectures, and data-driven adaptive decomposition in highly non-stationary or adversarial environments.

7. Applications and Impact Across Domains

Ex-BCMP models have immediate application in:

• Reinforcement learning and temporal difference learning, enabling robust, sample-efficient algorithms with theoretical convergence guarantees under high-dimensional noise (Ramaswamy et al., 2015, Trimponias et al., 2023).
• Online supervised learning and forecasting (e.g., weather prediction by TD-updates), guaranteeing stability under delayed feedback and unknown exogenous influences (Ramaswamy et al., 2015).
• Bandit decision-making and federated learning under exogenous global processes, delivering adaptive algorithms with logarithmic regret guarantees (Gafni et al., 2022).
• Blockchain and queueing systems, allowing explicit modeling of batch/block operations subject to exogenous controls (Li et al., 2019).
• Financial risk and pricing via Markov-modulated diffusions, incorporating regime-switching and macro-economic signals (Kurt et al., 2021).

The formal separation of exogenous and endogenous blocks, together with scalable discovery and sample-efficient learning, establishes Ex-BCMPs as a foundational model class for modern stochastic control, RL, and decision-making under pervasive uncertainty.