Separately Controlled Chains

• Separately controlled chains are stochastic models where each local Markov chain is governed by its own control while global objectives couple the system.
• They decouple local transition dynamics using product-form kernels, yet introduce complexity through non-additive team variance criteria.
• Algorithmic approaches use decentralized bilevel optimization and sensitivity analysis to achieve equilibrium, as seen in smart grid and adaptive MCMC applications.

Searching arXiv for papers on “separately controlled chains” and closely related controlled Markov chain formulations. Separately controlled chains are stochastic control or game-theoretic models in which the global system comprises multiple local Markov chains, each driven exclusively by its own local control, while coupling is introduced through an aggregate objective, information structure, or performance criterion rather than through the transition law itself. In the recent stochastic-games formulation, the defining structural property is a product-form transition kernel,

$$P\bigl(\bm s_{t+1}\mid \bm s_t,\bm a_t\bigr)=\prod_{i=1}^n P^i\bigl(s'_{i}\mid s_i,a_i\bigr),$$

together with local rewards of the form $r_i(\bm s_t,\bm a_t)=r_i(s_{i,t},a_{i,t})$, so that each player’s internal state is controlled only by its own action (Xia, 30 Jul 2025). Closely related controlled-Markov-chain work studies joint processes $(\theta_n,X_n)$ where the state chain and parameter chain admit separate Lyapunov drifts that can be combined into a single recurrence argument (Andrieu et al., 2012). A different but conceptually adjacent line in partially observed control uses the term “separated” to denote the transformation of a partially observed controlled Markov chain into a fully observed control problem on the filtering state, governed by the Wonham–Zakai equations (Confortola et al., 18 Feb 2026). Taken together, these uses locate separately controlled chains within a broader family of decomposed stochastic systems in which dynamic decoupling coexists with objective- or inference-level coupling.

1. Formal model and defining structure

In the $n$-player stochastic-game formulation, player $i$ has state space $\mathcal S_i$, action space $\mathcal A_i$, local transition kernel

$$P^i(s_i' \mid s_i,a_i)=\Pr\{s_{i,t+1}=s_i' \mid s_{i,t}=s_i,a_{i,t}=a_i\},$$

and immediate reward $r_i(s_i,a_i)$ (Xia, 30 Jul 2025). The global state is $\bm s=(s_1,\dots,s_n)$, the joint action is $r_i(\bm s_t,\bm a_t)=r_i(s_{i,t},a_{i,t})$0, and the key assumption is that the controlled dynamics factorize across players: $r_i(\bm s_t,\bm a_t)=r_i(s_{i,t},a_{i,t})$1 This is the sense in which the chains are “controlled separately”: the evolution of player $r_i(\bm s_t,\bm a_t)=r_i(s_{i,t},a_{i,t})$2’s local chain is unaffected by the actions of the other players at the transition level (Xia, 30 Jul 2025).

The same paper imposes a local-information architecture. Each player observes only its own history

$r_i(\bm s_t,\bm a_t)=r_i(s_{i,t},a_{i,t})$3

and uses a stationary deterministic policy $r_i(\bm s_t,\bm a_t)=r_i(s_{i,t},a_{i,t})$4; the joint policy is $r_i(\bm s_t,\bm a_t)=r_i(s_{i,t},a_{i,t})$5 (Xia, 30 Jul 2025). Thus, dynamic independence is paired with decentralized observability.

A related controlled-Markov-chain framework considers a family of kernels $r_i(\bm s_t,\bm a_t)=r_i(s_{i,t},a_{i,t})$6 on a state space $r_i(\bm s_t,\bm a_t)=r_i(s_{i,t},a_{i,t})$7, together with parameter updates $r_i(\bm s_t,\bm a_t)=r_i(s_{i,t},a_{i,t})$8, yielding

$r_i(\bm s_t,\bm a_t)=r_i(s_{i,t},a_{i,t})$9

Although this is not an $(\theta_n,X_n)$0-player product-form model, it is a separately controlled-chain framework in the sense that the state and parameter components admit distinct control- and drift-analytic treatment before being recombined through a joint Lyapunov argument (Andrieu et al., 2012).

This suggests that “separately controlled chains” is best understood not as a single formalism but as a structural principle: one isolates subsystems whose one-step laws are locally controlled, and then studies how global behavior emerges once a common criterion or coupled analysis is imposed.

2. Objective coupling despite transition decoupling

The defining subtlety of separately controlled chains is that dynamic decoupling does not imply optimization decoupling. In the stochastic-games setting, all players share a common objective called the team variance (Xia, 30 Jul 2025). The long-run average team reward is

$(\theta_n,X_n)$1

and the team variance is

$(\theta_n,X_n)$2

Although each reward term depends only on a local state-action pair, the centering quantity $(\theta_n,X_n)$3 depends on the entire joint policy, so the objective couples the players through a global statistic (Xia, 30 Jul 2025).

The paper explicitly identifies the consequence: the variance metric is not additive or Markovian, and the dynamic programming principle fails (Xia, 30 Jul 2025). The instantaneous quantity $(\theta_n,X_n)$4 is not a standard stage cost because $(\theta_n,X_n)$5 is itself a functional of the policy. This breaks the Bellman decomposition ordinarily available for average-cost Markov decision processes.

An equivalent decomposition clarifies the structure: $(\theta_n,X_n)$6 where $(\theta_n,X_n)$7 is a pseudo team mean, and

$(\theta_n,X_n)$8

For fixed $(\theta_n,X_n)$9, the problem decomposes into $n$0 independent average-cost MDPs, one per player; the coupling reappears only through the outer minimization over $n$1 (Xia, 30 Jul 2025).

A plausible implication is that separately controlled chains are especially natural when the system designer seeks fairness, risk balancing, or coordinated variability reduction rather than purely additive throughput or reward. In such cases, independence of transitions simplifies modeling, but the non-additive criterion remains the main analytical obstacle.

3. Sensitivity analysis and bilevel reformulation

Because dynamic programming fails for team variance, the recent theory develops a sensitivity-based formulation (Xia, 30 Jul 2025). Introducing the pseudo team mean $n$2, one rewrites the global optimization problem as

$n$3

This yields a bilevel structure: an outer optimization over the coordinating scalar $n$4, and an inner level consisting of independent ergodic MDPs (Xia, 30 Jul 2025).

For a fixed $n$5 and player $n$6, the analysis uses the steady-state distribution $n$7 under a comparison policy $n$8, transition matrices $n$9, immediate-cost vectors $i$0 with entries $i$1, and a relative-value function $i$2 solving

$i$3

The performance-difference formula is then

$i$4

Aggregating across players and setting $i$5 produces the team-variance difference formula

$i$6

A derivative formula is obtained by mixing policies through $i$7: $i$8 These formulas supply the sensitivity information needed for decentralized improvement steps (Xia, 30 Jul 2025).

The algorithmic consequence is a decentralized bilevel optimization method. At iteration $i$9, each player evaluates the current team mean $\mathcal S_i$0, solves the Poisson equation for the bias vector $\mathcal S_i$1 of its own pseudo-MDP with running cost $\mathcal S_i$2, and then updates its policy independently by

$\mathcal S_i$3

Ties are broken by staying with the old action if possible, to avoid cycles (Xia, 30 Jul 2025).

The paper proves that this algorithm strictly decreases $\mathcal S_i$4 at each update, terminates in finitely many steps because the deterministic-policy space is finite and bounded below, and converges to a limit point satisfying first-order necessary conditions in the mixed-policy space; in most cases the limit is a strictly local minimum (Xia, 30 Jul 2025).

4. Equilibrium structure and recurrence analysis

For the team-variance game, the existence result is stated as follows: under the assumptions of separately controlled chains and local observations, there exists a joint policy $\mathcal S_i$5 with each $\mathcal S_i$6 stationary deterministic that minimizes $\mathcal S_i$7; equivalently, $\mathcal S_i$8 is a stationary pure Nash equilibrium of the $\mathcal S_i$9-player game (Xia, 30 Jul 2025). The proof strategy uses the fact that for fixed $\mathcal A_i$0, each inner problem is a standard ergodic MDP with average-cost $\mathcal A_i$1, so each admits an optimal stationary-deterministic policy; convexity in $\mathcal A_i$2 then yields a global minimizer (Xia, 30 Jul 2025).

A distinct but relevant stability perspective appears in the controlled-Markov-chain recurrence theory of Andrieu, Tadić, and Vihola (Andrieu et al., 2012). There, the process $\mathcal A_i$3 is governed by a kernel update for $\mathcal A_i$4 and a stepsize-indexed map for $\mathcal A_i$5. The central idea is to derive separate Lyapunov drifts for the state and parameter components and combine them into a single Foster–Lyapunov inequality for the joint chain.

The state-chain drift is

$\mathcal A_i$6

with

$\mathcal A_i$7

while the parameter-chain drift is

$\mathcal A_i$8

with

$\mathcal A_i$9

Defining

$$P^i(s_i' \mid s_i,a_i)=\Pr\{s_{i,t+1}=s_i' \mid s_{i,t}=s_i,a_{i,t}=a_i\},$$0

the paper proves that for suitable $$P^i(s_i' \mid s_i,a_i)=\Pr\{s_{i,t+1}=s_i' \mid s_{i,t}=s_i,a_{i,t}=a_i\},$$1 there exists $$P^i(s_i' \mid s_i,a_i)=\Pr\{s_{i,t+1}=s_i' \mid s_{i,t}=s_i,a_{i,t}=a_i\},$$2 such that for all large $$P^i(s_i' \mid s_i,a_i)=\Pr\{s_{i,t+1}=s_i' \mid s_{i,t}=s_i,a_{i,t}=a_i\},$$3 and all $$P^i(s_i' \mid s_i,a_i)=\Pr\{s_{i,t+1}=s_i' \mid s_{i,t}=s_i,a_{i,t}=a_i\},$$4,

$$P^i(s_i' \mid s_i,a_i)=\Pr\{s_{i,t+1}=s_i' \mid s_{i,t}=s_i,a_{i,t}=a_i\},$$5

From this compound drift, one deduces that $$P^i(s_i' \mid s_i,a_i)=\Pr\{s_{i,t+1}=s_i' \mid s_{i,t}=s_i,a_{i,t}=a_i\},$$6 returns infinitely often to $$P^i(s_i' \mid s_i,a_i)=\Pr\{s_{i,t+1}=s_i' \mid s_{i,t}=s_i,a_{i,t}=a_i\},$$7 almost surely (Andrieu et al., 2012).

This recurrence theory does not use the same product-form transition structure as the stochastic-game model, but it exemplifies an allied methodological theme: separate control or drift structure at the component level can be recombined into a global stability statement. For separately controlled chains more broadly, this suggests that equilibrium analysis and recurrence analysis are complementary rather than competing perspectives.

5. Time-scale separation and partially observed separation

Time-scale separation is explicit in the recurrence analysis of controlled Markov chains (Andrieu et al., 2012). When $$P^i(s_i' \mid s_i,a_i)=\Pr\{s_{i,t+1}=s_i' \mid s_{i,t}=s_i,a_{i,t}=a_i\},$$8, the parameter chain moves slowly relative to the state chain; the proof compares the slow $$P^i(s_i' \mid s_i,a_i)=\Pr\{s_{i,t+1}=s_i' \mid s_{i,t}=s_i,a_{i,t}=a_i\},$$9-drift, of order $r_i(s_i,a_i)$0, with the fast $r_i(s_i,a_i)$1-drift, of order $r_i(s_i,a_i)$2, by rescaling $r_i(s_i,a_i)$3 by $r_i(s_i,a_i)$4. The condition

$r_i(s_i,a_i)$5

ensures that the rescaling does not grow too quickly and that the joint process remains stable even when the components evolve on different time scales (Andrieu et al., 2012).

Another notion of separation arises in partially observed controlled Markov chains (Confortola et al., 18 Feb 2026). Here the unobserved finite-state chain $r_i(s_i,a_i)$6 has controlled transition rates, while the controller observes only a diffusion-type process

$r_i(s_i,a_i)$7

Through a Girsanov-type change of measure, one introduces the unnormalized filter

$r_i(s_i,a_i)$8

which solves the controlled Wonham–Zakai SDE

$r_i(s_i,a_i)$9

The resulting “separated optimal control problem” is fully observed in the filter state $\bm s=(s_1,\dots,s_n)$0, and its value function satisfies HJB equations in viscosity form (Confortola et al., 18 Feb 2026).

This use of “separated” is terminologically distinct from “separately controlled chains,” yet the connection is instructive. In both settings, control theory exploits a decomposition that converts an analytically difficult coupled problem into a tractable one on transformed coordinates or reduced subsystems. In the partially observed case the decomposition is epistemic, from hidden state to conditional law; in separately controlled chains it is dynamic, from joint transitions to local kernels.

A plausible implication is that future work may combine these two notions, for example by studying decentralized team objectives under local partial observations, with each player controlling a local hidden Markov chain and optimization performed on local or joint filtering states. The cited papers do not state such a model, but their technical ingredients are compatible in spirit.

6. Applications, limitations, and related interpretations

The most explicit application reported for separately controlled chains is decentralized energy management in a smart grid (Xia, 30 Jul 2025). In the numerical experiment, $\bm s=(s_1,\dots,s_n)$1 microgrids are modeled with wind-power states $\bm s=(s_1,\dots,s_n)$2, battery-level states $\bm s=(s_1,\dots,s_n)$3, actions $\bm s=(s_1,\dots,s_n)$4, and net exchanges

$\bm s=(s_1,\dots,s_n)$5

with constant demand $\bm s=(s_1,\dots,s_n)$6. The separately controlled-chains assumption holds because

$\bm s=(s_1,\dots,s_n)$7

Applying the decentralized algorithm from a random initialization, the reported team variance $\bm s=(s_1,\dots,s_n)$8 drops from about $\bm s=(s_1,\dots,s_n)$9 to $r_i(\bm s_t,\bm a_t)=r_i(s_{i,t},a_{i,t})$00 in $r_i(\bm s_t,\bm a_t)=r_i(s_{i,t},a_{i,t})$01 iterations; the team mean fluctuates mildly; and each microgrid’s pseudo variance $r_i(\bm s_t,\bm a_t)=r_i(s_{i,t},a_{i,t})$02 is monotonically reduced, while raw variance may transiently rise (Xia, 30 Jul 2025).

In adaptive MCMC, the recurrence framework applies to algorithms such as Adaptive Metropolis and coerced-acceptance-probability random-walk Metropolis (Andrieu et al., 2012). For Adaptive Metropolis,

$r_i(\bm s_t,\bm a_t)=r_i(s_{i,t},a_{i,t})$03

with updates

$r_i(\bm s_t,\bm a_t)=r_i(s_{i,t},a_{i,t})$04

$r_i(\bm s_t,\bm a_t)=r_i(s_{i,t},a_{i,t})$05

Under suitable drift and moment conditions, the compound-drift theorem yields recurrence of $r_i(\bm s_t,\bm a_t)=r_i(s_{i,t},a_{i,t})$06 to a compact $r_i(\bm s_t,\bm a_t)=r_i(s_{i,t},a_{i,t})$07 (Andrieu et al., 2012). Although this is not a decentralized game, it shows that separately analyzable controlled components arise naturally in computational statistics.

Several common misconceptions can be ruled out. First, separately controlled chains do not mean that the optimization problem is decentralized in the strong sense of being fully independent across subsystems. In the team-variance model, the objective couples all players through $r_i(\bm s_t,\bm a_t)=r_i(s_{i,t},a_{i,t})$08, and the authors emphasize that dynamic programming fails precisely because of this non-Markovian coupling (Xia, 30 Jul 2025). Second, separate control does not by itself guarantee tractable verification of optimality or global convergence; the cited algorithm converges to a first-order stationary point in the mixed-policy space and, in most cases, a strictly local minimum, not necessarily a global optimum over all mixed policies (Xia, 30 Jul 2025). Third, the term should not be conflated with unrelated uses of “chain” in robotics or blockchain systems. For example, “control chains” in hierarchical robot control refer to ordered tuples of low-level controllers coordinated for sequential manipulation (Harris et al., 2022), while “decoupled consensus between chains” concerns blockchain sidechains with independent consensus mechanisms (Garoffolo et al., 2018). These are distinct domains and definitions.

In present research usage, separately controlled chains therefore denote a class of structured stochastic systems where local transition dynamics are autonomous under local control, yet global analysis remains nontrivial because coupling re-enters through risk-sensitive criteria, equilibrium conditions, shared statistics, or stability arguments. The concept is valuable precisely because it isolates where complexity resides: not in the product-form dynamics themselves, but in the global objective or in the analytical machinery required to recombine the local pieces into a coherent control theory (Xia, 30 Jul 2025, Andrieu et al., 2012, Confortola et al., 18 Feb 2026).