Distributed Stochastic Controllers

• Distributed stochastic controllers are algorithms that employ stochastic approximations and local noisy measurements across multi-agent networks to achieve global optimization under informational constraints.
• They utilize innovative update laws such as derivative-free perturbation and scenario-based model predictive control to adapt to decentralized and uncertain environments.
• The framework offers robust performance guarantees with sublinear convergence rates and has practical applications in wireless networks, smart grids, and cooperative robotics.

Distributed stochastic controllers are algorithms and control architectures in which decisions are taken across multiple agents or nodes in a network, with each agent having access to partial system information. Agents leverage local stochastic observations—often noisy or subject to uncertainty—to achieve network-wide control or optimization objectives. These controllers operate under informational, computational, and signalling constraints inherent to distributed settings. Rather than relying on centralized gradient evaluations or global state observation, distributed stochastic controllers update their policies using stochastic approximation, decentralized information exchange, scenario-based validation, sampling, or probabilistic formulations. Applications span wireless networks, cooperative robotics, smart grids, traffic networks, and manufacturing systems.

1. Mathematical Formulation and Structural Assumptions

Distributed stochastic control problems are defined over networks of $N$ agents, each with local states, actions, and payoffs. The canonical distributed stochastic optimization problem is to maximize network utility $F(a) = \mathbb{E}_S[\sum_{i=1}^N u_i(a,S)]$ subject to individual constraints $a_i \in \mathcal{A}_i$ (Li et al., 2018). Agents possess only noisy local measurements $\tilde{u}_{i,k}$ of their payoffs (possibly without closed-form expressions), and may exchange observations to estimate global objectives. Common standing assumptions for convergence include strict concavity of $F$, uniform Lipschitz continuity, bounded martingale noise, i.i.d. perturbations, and vanishing step-sizes. Variants include linear, nonlinear, or switched-dynamics models, stochastic network-induced packet losses, multiplicative uncertainties, and additive disturbances (Wang et al., 2014, Mark et al., 2019, Muntwiler et al., 2020).

Formulations extend to receding-horizon distributed stochastic model predictive control (DSMPC) (Rostampour et al., 2017, Muntwiler et al., 2020, Pham et al., 2022), decentralized stochastic games (Mahajan et al., 2013, Wang et al., 2014), and multi-agent stochastic maximum principles (Jackson et al., 2023). The core structural distinction is between full-information protocols (each agent can aggregate all other observations) and partial-information protocols (agents have access only to subsets, typically neighbors in the network).

2. Distributed Stochastic Update Laws and Controller Architectures

Derivative-free stochastic perturbation update laws are central to distributed optimization under limited information (Li et al., 2018). Each agent applies a random perturbation $\Phi_{i,k}$ to its action $a_{i,k}$, executes the perturbed action, and observes the local payoff. The update is:

$$a_{i,k+1} = a_{i,k} + \beta_k \, \Phi_{i,k} \tilde{f}_k,$$

with $\tilde{f}_k$ an estimate of the global payoff. In the full-information variant, $\tilde{f}_k$ aggregates all local observations; in the partial-information variant, only a random subset is available. The update is a stochastic approximation for the global gradient, with bias vanishing as the perturbation size $\gamma_k$ decreases.

Distributed stochastic model predictive controllers typically employ ADMM-based consensus optimization, scenario-based chance constraint satisfaction, and tube-based uncertainty feedback (Rostampour et al., 2017, Mark et al., 2019, Muntwiler et al., 2020, Pham et al., 2022, Yoon et al., 21 Oct 2025). Agents solve local stochastic programs involving scenario samples, exchange selected state trajectories or probabilistic sets with neighbors, and enforce local and coupling constraints via consensus multipliers.

Specialized structures exist in certain domains:

• Decentralized LQ controllers with delays and losses are solved as noncooperative stochastic games with coupled Riccati recursions (Wang et al., 2014).
• Distributed Q-learning for stochastic LQ control reformulates the Riccati equation as a zero-point matrix equation, solved via consensus-plus-innovation updates (Zhang et al., 2022).
• Path-integral control yields local optimal actions as functionals of desirability gradients computed via Monte-Carlo sampling within overlapping agent subgraphs (Wan et al., 2020).
• Probability controllers for large-scale deterministic-stochastic networks use output-strictly passive memoryless maps in feedback with dissipativity-certified quadratic plants (Tsumura et al., 2020).

3. Convergence, Performance Guarantees, and Complexity

A range of stochastic approximation convergence theorems apply, guaranteeing that distributed updates converge almost surely to the global optimum under boundedness and step-size conditions (Li et al., 2018). Full and partial information variants both converge provided a sufficient rate of information exchange; rates are sublinear $O(k^{-1/2})$ under optimal step-size schedules. Strong-concavity entails explicit mean-square error bounds, with optimal parameter choices yielding the best rate.

Scenario-based DSMPC controllers admit non-asymptotic probabilistic guarantees: with $S$ scenario samples, the computed policies satisfy chance constraints at a prescribed rate with confidence $1-\beta$, per the Campi-Garatti bound (Rostampour et al., 2017). ADMM-based consensus optimization yields linear convergence in the residual, and recursive feasibility of tube-based DSMPC formulations is preserved via shift-and-append arguments (Muntwiler et al., 2020, Mark et al., 2019).

For high-dimensional networks, sharp non-asymptotic bounds have been established on the gap between full-information and distributed stochastic controls, scaling as $O(1/n)$ in mean-field and symmetric-coupling regimes (Jackson et al., 2023).

Table: Algorithmic complexity and communication

Algorithm family	Computation/agent	Communication/agent
Stochastic perturbation	$O(1)$ random draw, sum	$O(1)$ scalars per slot
DSMPC (scenario/ADMM)	$O(SN)$ scenario solve	$O(\text{neighborhood})$ per iter
Q-learning (LQ)	$O((n+m)^3)$ per iter	$O((n+m)^2)$ matrix per neighbor
Path-integral control	$O(Y K d)$ rollouts	Local neighbor state exchange

4. Applications and Numerical Validation

Distributed stochastic controllers are validated in power control for wireless networks, achieving rapid convergence and outperforming sine-perturbation references in both speed and stability even when only partial information is available (Li et al., 2018). DSMPCs have been deployed in thermal comfort control for interconnected rooms and aquifer energy networks, where scenario-based local decision-making and soft communication yield nearly centralized performance with reduced signalling (Rostampour et al., 2017), and in data-center temperature control, achieving statistically guaranteed chance constraint satisfaction despite unbounded and correlated disturbances (Muntwiler et al., 2020).

In urban traffic networks, stochastic MPC with ADMM-based distributed optimization yields significant reductions in traffic delay under high load, with performance and computational load scaling favorably compared to centralized solvers (Pham et al., 2022). For nonlinear and nonconvex multi-agent systems, distributed sampling-based MPC achieves zero collision and full completion rates in large formations (e.g., 64 Dubins-car agents) where deterministic solvers fail (Yoon et al., 21 Oct 2025). Path-integral distributed control is empirically validated in cooperative UAV formation tasks (Wan et al., 2020).

5. Design Guidelines, Extensions, and Robustness

Design of distributed stochastic controllers leverages the following principles (Li et al., 2018, Rostampour et al., 2017, Jackson et al., 2023):

• Step-size tuning prioritizes perturbation vanishing rates compatible with stochastic approximation theory; typical choices are $\nu_1=0.75$, $\nu_2=0.25$ for rates.
• Simple Bernoulli perturbation schemes suffice for derivative-free stochastic optimizers.
• Extensions to vector-valued actions, more elaborate coupling, and adaptive step-size policies are feasible in both theory and practice.
• Scenario-based soft-communication protocols trade a controlled reduction in chance constraint reliability for significant reductions in inter-agent signalling, with minimal performance loss (Rostampour et al., 2017).
• Robustness to model mismatch and unknown dynamics arises from black-box sampling methods and scenario optimization, at the cost of reduced convergence rates.

In mean-field regimes, distributed controls are provably near-optimal, with $1/n$ gap to full-information policies and scalable synthesis via McKean–Vlasov FBSDE and infinite-dimensional HJB equations (Jackson et al., 2023). Modular ADMM and consensus frameworks further enable plug-and-play compositional designs suitable for large networks.

6. Conceptual Extensions and Theoretical Insights

The foundational dichotomy in distributed stochastic control is between person-by-person rationality (Nash equilibria in non-cooperative games) and common-information coordinated policies (Mahajan et al., 2013, Wang et al., 2014). In linear-quadratic and switched systems, common-information dynamic programming yields explicit time-invariant decentralized prescriptions grounded in Riccati-type recursions and conditional mean estimation, even under unreliable packet transmission (Asghari et al., 2018). Output-strict passivity and dissipativity-based analysis enable stability certification for hybrid deterministic-stochastic actuators subject to quantization and probabilistic actuation (Tsumura et al., 2020).

Learning-theoretic extensions such as distributed Q-learning for Riccati matrix estimation accommodate unknown system statistics and uncertainty, converging peer-to-peer via consensus-plus-innovation updates (Zhang et al., 2022).

A plausible implication is the broad applicability of distributed stochastic controller frameworks across uncertain, high-dimensional, and information-constrained networks, as evidenced by robust theoretical guarantees and empirical performance in large-scale simulations. Robust, scalable distributed stochastic controllers continue to offer principled solutions for emerging decentralized control applications.