Markov Decision Processing Networks

• Markov Decision Processing Networks are a family of models that extend MDPs by incorporating explicit network structures, such as queues, Bayesian networks, and Petri nets.
• They offer versatile formulations that combine scheduling, routing, and resource allocation decisions with stochastic system dynamics to address complex service and communication challenges.
• Key challenges include scalability, computational complexity, and the design of robust control policies in applications ranging from queueing networks to distributed multi-agent systems.

Markov Decision Processing Networks (MDPNs) are network-structured sequential decision models in which Markovian dynamics are coupled to an explicit processing, service, probabilistic, or communication network. Across the literature, the term is used in more than one way. A broad queueing-and-networks usage treats MDPNs as Markov Decision Processes formulated on stochastic processing or queueing networks—systems of buffers, servers, and routing rules driven by random arrivals and services, and controlled via scheduling, admission, routing, and resource allocation decisions. A narrower formulation introduces MDPNs as multiclass queueing networks in which service is a controlled, finite-state Markov process, so that actions both serve jobs immediately and alter future service availability (Li et al., 2019, Bhambay et al., 29 Sep 2025). Related works extend the same networked-decision idea to dynamic Bayesian networks, factored MDPs, stochastic decision Petri nets, tensor-network realizations, and distributed multi-agent control. This suggests that MDPNs are best understood as a family of network-aware Markov decision models rather than a single universally fixed formalism (Morato et al., 2020, Liu et al., 2024).

1. Terminological scope and recurring structure

A notable feature of the literature is that “MDPN” does not denote one uniquely standardized object. Instead, the term is attached to several closely related constructions that share a common design principle: the decision process is Markovian, but the state, transition, or control structure is organized by an explicit network.

Usage in the literature	Network object	Representative source
Queueing or stochastic processing MDPs	Buffers, servers, routing rules	(Li et al., 2019)
Controlled service-state queueing networks	Finite-state Markov service process	(Bhambay et al., 29 Sep 2025)
DBN-coupled POMDPs	Dynamic Bayesian network with belief-state control	(Morato et al., 2020)
Factored and robust factored MDPs	DBN structure, CSI, MILP features	(Liu et al., 2024)
Stochastic decision Petri nets	Places, transitions, controllable deactivations	(Wittbold et al., 2023)
Tensor-network and distributed realizations	MPO/MPS chains; communication graphs	(Gillman et al., 2020, Lee et al., 2023)

Despite this variation, several ingredients recur. An MDPN typically has a state description partitioned across nodes, factors, places, or agents; actions that affect both immediate reward and the future evolution of that networked state; and algorithms that exploit decomposition, conditional independence, occupancy measures, or mixing properties. In some cases the network is physical, as in queues, wireless systems, and ride-hailing. In others it is representational, as in DBNs, factored MDPs, Petri nets, or tensor networks.

At the formal level, the underlying decision model remains the MDP or one of its extensions. In the queueing survey, discrete-time formulations use a state space $S$, state-dependent action set $A(s)$, transition kernel $P(s' \mid s,a)$, reward $r(s,a)$ or cost $c(s,a)$, and discount factor $\gamma \in (0,1)$. Discounted, average-cost, and constrained formulations all appear, together with Bellman equations and linear programming over occupancy measures (Li et al., 2019). The network structure does not replace the MDP formalism; it refines it.

2. Queueing and stochastic processing foundations

In its classical form, an MDPN is an MDP defined on a stochastic processing or queueing network. Nodes $i \in \{1,\dots,I\}$ represent buffers or queues, activities $k \in \{1,\dots,K\}$ represent service modes or links, and arcs encode routing. Exogenous arrivals occur at rates $\lambda_i$, service rates $\mu_k(a)$ depend on the control action, routing is governed by probabilities $A(s)$0, and state variables may include queue lengths $A(s)$1, server availability, modes, or channel states. The control variables are scheduling, routing, admission, and service-rate decisions (Li et al., 2019).

The basic optimality equations are standard MDP objects but acquire queueing interpretation through the network state. For discounted problems,

$A(s)$2

while average-cost problems use the average-cost optimality equation

$A(s)$3

Constrained MDPs are handled through Lagrangian relaxation, and the resulting dual variables often admit an interpretation as prices for network resources (Li et al., 2019).

A large part of the mature theory concerns structural properties of optimal policies. Many admission and service-control problems exhibit monotone or threshold policies with respect to queue length or workload. In M/M/1 or M/G/1 admission control with convex holding costs, optimal policies are of threshold type: accept when $A(s)$4, reject otherwise, with hysteretic variants when switching costs are present. Index policies, including Gittins and Whittle constructions, appear in scheduling regimes with activation constraints or restless dynamics. For single-server multi-class exponential queues with holding costs $A(s)$5 and service rates $A(s)$6, the preemptive-resume $A(s)$7 rule serves the class maximizing $A(s)$8 and is optimal under standard assumptions for discounted or average holding-cost objectives (Li et al., 2019).

Network-level stabilization introduces another canonical control family: backpressure scheduling. For a directed network with link rates $A(s)$9, the controller chooses

$P(s' \mid s,a)$0

In classical communication-network settings, this stabilizes any arrival vector strictly inside the capacity region. Stability is typically certified via Lyapunov drift inequalities such as

$P(s' \mid s,a)$1

which imply positive Harris recurrence and finite expected queue lengths and delays (Li et al., 2019).

This queueing lineage is the oldest and historically dominant source of MDPN ideas. It connects Bellman optimality, stochastic stability, and network-resource allocation, and it supplies the structural motifs—thresholds, indices, pressure-based control, decomposition, and fluid or diffusion limits—that later MDPN variants reuse in other guises.

3. MDPNs as controlled service-state queueing networks

A more specific formalization appears in the 2025 work that explicitly introduces MDPNs as a multiclass queueing network model where service availability is generated by a controlled, finite-state Markov chain (Bhambay et al., 29 Sep 2025). Here customer classes $P(s' \mid s,a)$2 have infinite-capacity FIFO buffers, while a finite set $P(s' \mid s,a)$3 of service states represents endogenous resource configurations such as inventories, idle drivers, available skills, or entanglement memories. At each time slot, the controller observes the current service state $P(s' \mid s,a)$4 and possibly the queue vector $P(s' \mid s,a)$5, chooses an action $P(s' \mid s,a)$6, receives instantaneous service $P(s' \mid s,a)$7 or expected service $P(s' \mid s,a)$8, and simultaneously influences the next service state through an action-dependent kernel $P(s' \mid s,a)$9.

This model differs sharply from classical switched networks. In switched networks, the feasible service set is fixed by the current state of the queueing system; in the 2025 MDPN model, the feasible service opportunities are endogenous because actions shape the future service-state distribution. The capacity region therefore depends on long-run state-action occupancies rather than only on convex combinations of instantaneous schedules. If $r(s,a)$0 is the long-run fraction of time spent in service state $r(s,a)$1 taking action $r(s,a)$2, then the average service available to class $r(s,a)$3 is

$r(s,a)$4

and the capacity region is characterized by occupancies satisfying stationarity constraints together with $r(s,a)$5 componentwise (Bhambay et al., 29 Sep 2025).

This endogenous service process breaks a classical result: MaxWeight is no longer throughput-optimal in general. The paper gives a three-class, three-phase counterexample in which myopic service of classes 1 and 2 prevents access to the high-capacity service option for class 3. The necessary condition for MaxWeight stability becomes

$r(s,a)$6

even though a planning-aware controller can stabilize larger arrival vectors such as $r(s,a)$7 (Bhambay et al., 29 Sep 2025). The distinction is not marginal; it changes which control principle is valid.

To restore throughput optimality, the paper introduces the weighted average reward policy (WARP). For a fixed queue vector $r(s,a)$8, WARP solves an average-reward MDP over the service-state chain with one-step reward

$r(s,a)$9

and maximizes the long-run average reward

$c(s,a)$0

Operationally, it fixes the resulting optimal request-agnostic policy over epochs whose length satisfies $c(s,a)$1, so that the service-state chain mixes within an epoch while the queues evolve on a slower fluid timescale. Under bounded arrivals and service, monotone schedules, and irreducible and aperiodic service-state dynamics under stationary randomized policies, WARP is throughput-optimal: for every $c(s,a)$2 in the interior of the capacity region, the joint process $c(s,a)$3 is positive recurrent (Bhambay et al., 29 Sep 2025).

The model was designed for two-sided matching systems, and the mappings in the paper are explicit. Assemble-to-order systems interpret $c(s,a)$4 as inventory vectors; ride-hailing interprets $c(s,a)$5 as idle-driver counts by zone; cross-skilled call centers interpret $c(s,a)$6 as available skills; quantum switches interpret $c(s,a)$7 as link-level entanglement inventory. In each case, immediate service depletes or reshapes future supply (Bhambay et al., 29 Sep 2025).

4. Graphical, partially observable, and factored MDPNs

Another major branch of the literature uses MDPNs to mean a networked decision process in which the network is graphical rather than queueing. In structural reliability, a dynamic Bayesian network (DBN) is combined with a POMDP so that the DBN supplies the physical transition and observation models, and the POMDP supplies optimal control over belief states (Morato et al., 2020). Two state constructions are used: a parametric model $c(s,a)$8, where $c(s,a)$9 is discretized crack or damage size and $\gamma \in (0,1)$0 is a vector of time-invariant parameters, and a deterioration-rate model $\gamma \in (0,1)$1, where $\gamma \in (0,1)$2 indexes a time-varying deterioration regime. Actions include inspection and maintenance choices, observations come from PoD or PoI curves, and belief updates follow the Bayes filter

$\gamma \in (0,1)$3

In this construction, DBN forward filtering and POMDP belief propagation are the same operation. The DBN provides $\gamma \in (0,1)$4 and $\gamma \in (0,1)$5; the POMDP optimizes

$\gamma \in (0,1)$6

The value function is piecewise-linear and convex in belief space, with approximation

$\gamma \in (0,1)$7

The reported implementations use SARSOP, FRTDP, and Perseus. In the structural component example, the infinite-horizon rate-DBN model uses 930 states, the finite-horizon model uses 14,880 states, and the rate-DBN discretization achieves failure-probability error $\gamma \in (0,1)$8 versus Monte Carlo. In the traditional setting, finite-horizon POMDP control reduces expected cost by approximately $\gamma \in (0,1)$9–$i \in \{1,\dots,I\}$0 versus heuristics in one experiment and by approximately $i \in \{1,\dots,I\}$1–$i \in \{1,\dots,I\}$2 in another; in the detailed setting, finite-horizon POMDP costs are approximately $i \in \{1,\dots,I\}$3–$i \in \{1,\dots,I\}$4, while the best heuristics achieve approximately $i \in \{1,\dots,I\}$5–$i \in \{1,\dots,I\}$6 (Morato et al., 2020).

A closely related graphical line treats MDPNs as factored or robust factored MDPs. Here the state is decomposed as $i \in \{1,\dots,I\}$7, the action may also be factored, rewards decompose into local terms, and the DBN transition kernel factorizes as

$i \in \{1,\dots,I\}$8

Context-specific independence (CSI) further compresses the model by allowing local CPDs or reward factors to simplify in particular parent contexts. The 2024 MILP-based framework represents these contexts through binary feature maps $i \in \{1,\dots,I\}$9, solves an approximate LP with basis functions via cutting planes, and extends the same scheme to robust ambiguity sets over local marginals (Liu et al., 2024).

This graphical MDPN view is computationally attractive precisely because the full Cartesian state space is not enumerated directly. At the same time, the underlying problems remain hard. Exact optimal control of factored MDPs is EXPTIME-hard even under low-scope assumptions; identifying maximally violated constraints in the cutting-plane scheme is strongly NP-complete; and worst-case expectations under factored ambiguity sets are PPAD-hard even for two factors (Liu et al., 2024). The representation is compressed, but the control problem is not trivialized.

5. Alternative network semantics and computational realizations

MDPN ideas also appear in formalisms where the network is neither a queueing network nor a DBN. Stochastic decision Petri nets (SDPNs) equip stochastic Petri nets with controllable transitions and rewards over reached places, and they translate exactly into MDPs whose states pair the current marking with the set of places reached so far. In this semantics,

$k \in \{1,\dots,K\}$0

where $k \in \{1,\dots,K\}$1 is the set of controllable transitions. The translation preserves rewards, but concurrency can induce combinatorial state explosion. Under safe, acyclic, free-choice assumptions, the paper derives sharp complexity results: for SAFC nets, the value problem is PP-complete and policy synthesis under constant deactivation patterns is $k \in \{1,\dots,K\}$2-complete; for occurrence nets, the value problem is in P and policy synthesis is NP-complete (Wittbold et al., 2023).

A tensor-network realization replaces the usual tabular or graphical representation with a contraction architecture along the time axis. Finite episodic MDPs are encoded by a policy MPS, a dynamics MPO, and a reward MPO, yielding the return functional

$k \in \{1,\dots,K\}$3

The paper shows that DMRG-like variational sweeps can optimize policies directly in this representation. In the excursion random-walk example, a single backward sweep recovers an optimal policy achieving $k \in \{1,\dots,K\}$4, and contraction cost is dominated by the state dimension with $k \in \{1,\dots,K\}$5 per environment contraction in the reported setting (Gillman et al., 2020).

Distributed realizations reinterpret the network as a communication graph among agents. In continuous-time distributed dynamic programming for networked multi-agent MDPs, each agent observes only its own reward $k \in \{1,\dots,K\}$6 but exchanges parameters with neighbors over a connected graph with Laplacian $k \in \{1,\dots,K\}$7. One Wang–Elia-inspired algorithm evolves

$k \in \{1,\dots,K\}$8

Under the stated full-column-rank, Hurwitz, and graph-connectivity assumptions, the distributed flows are globally asymptotically stable and converge to the same value solution that a centralized algorithm would compute from the average reward (Lee et al., 2023).

A deployment-oriented interpretation appears in wireless sensor networks. There, MDPNs formalize how resource-limited autonomous sensors interact with stochastic environments through sleep, transmit, sense, route, and power-control actions under uncertainty and constraints. The literature surveyed for WSNs spans MDPs, POMDPs, Dec-MDPs, Dec-POMDPs, HMDPs, and CMDPs, with offline-computed policies often stored as lookup tables and executed online at tiny nodes. The survey also records the complexity boundary: POMDPs are PSPACE-complete in worst cases, while finite-horizon Dec-MDPs and Dec-POMDPs are NEXP-complete (Alsheikh et al., 2015).

6. Applications, misconceptions, and open directions

The application range of MDPNs is unusually broad because the networked-decision template is reusable. Queueing surveys place MDP-based network control in manufacturing, communications, service systems, call centers, healthcare operations, cloud and data centers, transportation, traffic, inventory, and pricing (Li et al., 2019). The controlled-service-state model extends this list to assemble-to-order systems, ride-hailing platforms, cross-skilled call centers, and quantum switches (Bhambay et al., 29 Sep 2025). Graphical and partially observable variants address structural inspection and maintenance planning under fatigue deterioration (Morato et al., 2020). Wireless-sensor formulations cover duty cycling, routing, MAC design, power control, tracking, security, spectrum access, and energy harvesting (Alsheikh et al., 2015).

A common misconception is that MDPNs are simply classical queueing control under a new name. The literature does not support that simplification. In some papers the term refers to queueing networks with standard MDP control; in others it denotes queueing networks with an endogenous Markov service process; in still others it denotes DBN-coupled POMDPs, factored MDPs, Petri-net semantics, tensor-network formulations, or distributed multi-agent DP. Another misconception is that the network structure automatically resolves scalability. The opposite conclusion is more accurate: network structure often makes decomposition, approximation, or partial-order methods possible, but hardness results remain central, whether as curse of dimensionality in large queues, EXPTIME-hard exact control in factored models, or PP- and $k \in \{1,\dots,K\}$9-level complexity in Petri-net variants (Liu et al., 2024, Wittbold et al., 2023).

The major open directions are consistent across the literature. Queueing surveys emphasize scalability, distributed and decentralized control, partial observability, robust and risk-sensitive control, and tighter links to learning and AI (Li et al., 2019). The controlled-service-state MDPN model adds computational cost of runtime average-reward MDP solution, non-Markov service dynamics, online learning of $\lambda_i$0, partial observability of the service state, and extensions to more general stochastic processing networks (Bhambay et al., 29 Sep 2025). Structural-reliability work points to mixed observability, multi-component systems, and POMDP-based deep reinforcement learning for very large-scale problems (Morato et al., 2020). WSN surveys stress non-stationarity, sparse feedback, coordination overhead, and the limited number of long-duration real-world testbeds (Alsheikh et al., 2015).

Representation learning is a complementary response to these difficulties. The sufficient-MDP framework with alternating deep neural networks seeks a low-dimensional representation of the original process such that the MDP model still holds and a strategy optimizing the reduced process also optimizes the original one. The core sufficient condition is

$\lambda_i$1

and the learned representation is validated by Brownian distance covariance tests on residual dependence (Wang et al., 2017). This line does not redefine MDPNs on its own, but it points toward a plausible future synthesis: network-structured control models whose state representations are themselves learned, tested for Markov sufficiency, and then optimized by dynamic programming or reinforcement learning.

Taken together, these strands show that MDPNs occupy a junction of stochastic control, queueing theory, graphical models, concurrency formalisms, and large-scale decision computation. Their unifying problem is not merely to choose an action in a Markov system, but to choose an action in a Markov system whose transition law is organized by a network and whose control objectives depend on shaping that network over time.