Extended CTMDP: Multi-Agent Hierarchical Models

• Extended CTMDP is a dynamic framework that generalizes traditional MDPs to continuous time, multi-agent, and hierarchical settings with game-theoretic interaction.
• It facilitates leader-follower interactions and multi-stage decision-making, driving optimal resource allocation and incentive design in complex systems.
• Advanced solution methods, including backward induction, ADMM iterations, and reinforcement learning, ensure convergence to equilibrium under strict monotonicity and convexity.

An extended continuous time Markov decision process (CTMDP) is a dynamic game-theoretic control framework generalizing traditional Markov decision processes to continuous time and to multi-agent, multi-stage, and hierarchical settings. In these extended models, agents (leaders and followers) interact sequentially and hierarchically under stochastic (often Markovian) state dynamics, continuous time evolution, and multiple stages or levels—characteristics essential to modern networked, energy, supply chain, and infrastructure systems. CTMDPs enable the analysis of time-coupled resource allocation, dynamic incentive design, and optimal control under explicit multi-level strategic interaction, rendering them indispensable for complex engineering systems, energy markets, infrastructure planning, and cyber-physical networks.

1. Model Structure and Hierarchical Dynamics

Extended CTMDPs depart from single-agent, single-stage settings and address multi-agent interactions, often under a Stackelberg (leader-follower) game structure, and frequently with additional constraints such as coupling, peak penalties, and heterogeneous time horizons. The state evolution for each agent is typically governed by either linear or nonlinear continuous-time stochastic differential equations or (in discrete-time approximations) by linear dynamics of the form: $x_i(t+1) = a_i x_i(t) + b_i u_i(t) + w_i(t)$ where $x_i$ is the state, $u_i$ the control, $w_i$ external noise, and $a_i$, $b_i$ model parameters (Li et al., 2015). The leader announces a pricing or control policy over a finite (multi-stage) horizon $t=0,1,\ldots,T-1$, while each follower reacts to the leader’s signal by selecting optimal controls to maximize their own utilities, possibly subject to state, control, and resource constraints (Li et al., 2015, Khademi et al., 2015).

Hierarchical models are extended to $N$ levels, including not only leaders and followers but also non-follower agents (for example, passive participants in congestion or infrastructure systems) (Aminikalibar et al., 4 Mar 2026). These non-follower agents adapt and reconfigure their behavior in response to congestion or external effects, influencing the entire system’s equilibrium trajectories.

2. Multi-Stage Stackelberg Game Formulation

A fundamental extension in the CTMDP setting is the explicit Stackelberg (leader-follower) hierarchy, enabling anticipation of rational reactions at each stage and providing a basis for distributed control protocols. The Stackelberg game is typically formulated as follows:

• Leader problem: Selects a sequence of control/pricing policies $\{p(t)\}_{t=0}^{T-1}$ to optimize a social welfare or profit functional, anticipating the followers’ best responses to these policies. The leader’s objective may enforce peak constraints or incorporate explicit penalties for resource violations:

$$\mathrm{SW}(\{p(t)\}) = \sum_{t=0}^{T-1} \sum_{i} U_i(u^*_i(t), p(t)) - \gamma \cdot \max_t \left[\sum_i u^*_i(t)-P_{\mathrm{peak}}\right]^+$$

(Li et al., 2015)

• Follower response: Each follower solves a convex quadratic program to maximize their utility, which combines comfort, tracking, or reward against resource/payment cost, given the leader’s signal.
• Coupled equilibrium: The joint system equilibrates when follower best responses and leader policies are mutually consistent under the system’s constraints; uniqueness and attainability follow from strict monotonicity and concavity assumptions (Li et al., 2015, Barreiro-Gomez et al., 6 May 2025).
• Extension to multiple leaders and followers: The hierarchy can be augmented to include several leaders (e.g., utility companies) solving a (possibly Nash or Stackelberg) competition, with multiple followers solving dynamic responses, as in multi-leader multi-follower network games (Alshehri et al., 2015, Chen et al., 2024).

3. Solution Methodologies and Equilibrium Computation

Extended CTMDPs with hierarchical structure require specialized solution methods beyond classic Markov chain analysis or backward dynamic programming. Prevailing approaches include:

• Backward Induction: In multi-stage games, equilibrium computation proceeds via backward induction, first solving the followers’ best-response mapping at each stage, then optimizing the leader’s objective subject to these mappings (Li et al., 2015, Khademi et al., 2015).
• Projected Subgradient/ADMM Iterations: For convex social welfare problems with coupling constraints, iterative dual-decomposition or ADMM-type updates allow decomposition of the leader’s and followers’ problems, yielding provable convergence to equilibria under monotonicity and convexity (Li et al., 2015, Kang et al., 2023).
• Best-Response Learning/Dynamic Programming: In discrete action spaces, best-response (BR) dynamics and value-iteration are used to converge to Nash or Stackelberg equilibria in each class of hierarchical game (Barreiro-Gomez et al., 6 May 2025).
• Monte Carlo and Metaheuristics: For large-scale, possibly nonconvex or black-box systems, sampling-based Monte Carlo Multilevel Optimization (MCMO) algorithms enable approximate Stackelberg equilibrium computation under limited modeling assumptions (Koirala et al., 2023).
• Reinforcement Learning Methods: Multi-agent deep reinforcement learning (e.g., MALPPO, Tiny MADRL) can be employed for distributed computation in extended CTMDPs with privacy, incomplete models, or partial information, achieving equilibrium online via experience replay and actor-critic architectures (Kang et al., 2023, Kang et al., 2024).

4. Existence, Uniqueness, and Attainability of Equilibria

Extended CTMDPs under hierarchical Stackelberg structures admit strong theoretical guarantees for equilibrium behavior, under mild assumptions:

• Strict Monotonicity and Convexity: If each agent’s best response is strictly monotonic and utility functions are strictly convex (or concave for cost), existence and uniqueness of the Stackelberg equilibrium are assured (Li et al., 2015, Barreiro-Gomez et al., 6 May 2025).
• Attainability Conditions: Under Slater-type constraint qualifications and differentiability, the social-welfare optimum achievable via centralized control can be exactly attained through price-mediated (decentralized) Stackelberg equilibria (Li et al., 2015).
• Multi-Stage and Multi-Level Extensions: The structure of existence/uniqueness generalizes to discrete and continuous time, multi-level games with multiple leaders, followers, and non-followers (Khademi et al., 2015, Aminikalibar et al., 4 Mar 2026).
• Algorithmic Convergence: Convergence to equilibrium is rigorously proven for distributed protocols leveraging projected subgradient, consensus, and barrier-function methods, under strict or strong monotonicity of the hyperspace pseudo-gradient (Chen et al., 2024).
• Reverse Stackelberg and N-Level Extensions: Existence conditions for reverse Stackelberg equilibria (where leaders announce strategies as functions of all subordinates’ decisions) require convexity or connectedness of sublevel sets and nonvanishing gradients; multiple affine equilibria may exist (Worku et al., 2022).

5. Applications in Networked Systems, Energy, and Supply Chain Coordination

Extended CTMDPs underpin resource allocation and incentive design in a range of engineering and economic settings:

• Thermostatically Controlled Loads (TCLs): Multi-stage pricing for TCLs is managed via Stackelberg frameworks, optimally balancing comfort and energy costs under peak-load constraints (Li et al., 2015).
• Demand Response in Smart Grids: Multi-leader, multi-period Stackelberg CTMDPs are used by utility companies to design temporal price signals that induce strategic consumption patterns, providing closed-form equilibrium expressions and incentive bounds (Alshehri et al., 2015).
• Green Supply Chain Management: Hierarchical Stackelberg games model investment in Corporate Social Responsibility (CSR) across supplier, manufacturer, and retailer tiers, with closed-form dynamic equilibrium and backward-induction solutions (Khademi et al., 2015).
• Networked System Co-Design: Bi-level Stackelberg games facilitate infrastructure (leader) and control (follower) co-design, accounting for both operational and investment tradeoffs within large-scale urban and water networks (Barreiro-Gomez et al., 6 May 2025).
• Congestion and Mobility Infrastructure: Multi-level Stackelberg games with non-follower agents (such as non-EV drivers) capture bidirectional interactions between infrastructure investment, pricing, and congestion, enabling robust equilibrium predictions for EV charging and urban mobility (Aminikalibar et al., 4 Mar 2026).

6. Extensions: Information Structure, Incomplete Information, and Numerical Methods

Advanced CTMDP models extend to settings with asymmetric information, explicit stochasticity, complex hierarchical agent structures, and incomplete model knowledge:

• Asymmetric and Partial Information: Three-level Stackelberg differential games with asymmetric information require the use of coupled forward-backward stochastic differential equations, iterative filtering, and hierarchical Riccati equation systems for each information sub-filtration, resulting in explicit feedback policies for all agents under nested, filtered information (Kang et al., 2022).
• Incentive Mechanism Design under Uncertainty: Stackelberg incentive games in continuous time can incorporate $H_\infty$ robust performance constraints via coupled Riccati equations and backward induction, defining affine incentive structures at every level and ensuring the team’s optimal (robust) trajectory (Xiang et al., 2024).
• Black-box and Nonconvex Settings: Monte Carlo optimization methods (MCMO) offer sample-efficient recursive optimization for multilevel nonconvex CTMDPs and have been benchmarked against evolutionary and genetic methods, outperforming these in solution accuracy and computational reliability (Koirala et al., 2023).
• Distributed Equilibrium Seeking: In networked environments with clustered or locally-observable information, distributed algorithms leveraging consensus, surrogate hyper-gradients, and barrier reformulations guarantee convergence to the Stackelberg equilibrium, even with partial leader-follower communication (Chen et al., 2024).
• Reinforcement Learning for Stackelberg Games: Deep multi-agent actor-critic and LSTM-PPO architectures implement equilibrium computation and policy discovery in large, privacy-aware, dynamic Stackelberg CTMDPs—demonstrating rapid convergence to equilibrium, robustness to incomplete information, and support for model-size-optimized neural architectures (“Tiny MADRL”) (Kang et al., 2023, Kang et al., 2024).

---

In summary, extended CTMDPs serve as the foundational formalism for analyzing and designing complex, hierarchical, time-coupled, and stochastic resource allocation problems in multi-agent networked and physical systems. They blend continuous-time Markov dynamics, multi-level Stackelberg games, convex analysis, and computational game theory, supporting both theoretical equilibrium analysis and practical algorithmic implementation across diverse application domains (Li et al., 2015, Khademi et al., 2015, Alshehri et al., 2015, Barreiro-Gomez et al., 6 May 2025, Koirala et al., 2023, Worku et al., 2022, Kang et al., 2023, Kang et al., 2024, Aminikalibar et al., 4 Mar 2026, Chen et al., 2024, Xiang et al., 2024, Kang et al., 2022).