Diffusion-Based Policies for Dynamic Control of Stochastic Processing Networks

Abstract: We consider a processing network model with $m$ job classes or buffers, exogenous input flows into some classes, $n$ processing activities, and $p$ servers. Each activity is either a specified server processing jobs of a specified class, or a fictional input server delivering jobs of a specified class; jobs change class in Markovian fashion after completing service. A standard multiclass queueing network, with its one-to-one correspondence between job classes and activities, is a special case, but our general model allows two or more ways to process a given class, and some or all input flows may be turned away at the system manager's discretion. Costs are linear: a holding cost per time unit for each class $i$ job in the system, and a rejection penalty for each class $i$ arrival denied access $(i=1,\ldots,m)$. The system manager makes input control, job routing, and order-of-service decisions to minimize expected discounted costs over an infinite horizon. We formulate an approximating Brownian control problem (BCP) whose state space is the $m$-dimensional nonnegative orthant; control is a drift vector chosen from a bounded polyhedral set, based on dynamic state observations. Using recently developed computational methods, the BCP can be solved numerically in dimensions up to at least $m=50$, and we explain how the numerical solution is translated into an implementable control policy for the queueing system of original interest. Previous work on heavy traffic diffusion approximations suggests that this policy is nearly optimal in the heavy traffic parameter regime, and numerical examples support that conjecture. We also discuss its advantage over an alternative approach, featured in our previous work, where the BCP is replaced by a lower-dimensional "equivalent workload formulation" that is computationally efficient but difficult to interpret in the network of original interest.

• The paper develops a modified Brownian control problem with bounded drift controls and uses the value-function gradient to create decentralized, threshold-free priority and admission policies for discrete-flow networks.
• The resulting policy performs within 0.3% of exact MDP optima on two benchmark networks, improves costs by 6% to 21% over greedy methods, and reduces cost by about 9% in a three-station network.
• The method makes high-dimensional diffusion control implementable without state-space collapse, but its convergence to the original network and asymptotic optimality remain conjectures requiring theoretical proof.

Overview and contribution

This paper addresses a long-standing gap in the heavy-traffic theory of stochastic processing network (SPN) control: the systematic translation of a numerically computed Brownian control problem (BCP) solution into an implementable policy for the discrete-flow network of original interest. The authors—Ata, Guang, Harrison, and Si—consider an SPN with $m$ job classes, $n$ activities, and $p$ servers, where activities may be processing activities, input activities, or fictional "input servers" that model admission control. Costs are linear holding costs plus rejection penalties, discounted over an infinite horizon. The central methodological move is to formulate a modified BCP in which controls are bounded drift rates rather than singular processes of unbounded variation, solve it with the deep BSDE-based computational machinery of Ata–Harrison–Si (2608.14289), and then read off a continuous-review priority policy directly from the gradient $\nabla V(\cdot)$ of the value function.

The paper's positioning relative to prior work is explicit: the equivalent workload formulation (EWF) tradition initiated by Harrison reduces an $m$-dimensional BCP to a $d$-dimensional singular control problem ($d < m$, often much smaller), which is computationally attractive but offers no general recipe for interpreting its solution back in network terms. The authors state plainly that translating EWF buffer-content recommendations into routing and sequencing rules is "a problem of head-spinning complexity," and they deliberately abandon state space collapse to retain the full $m$-dimensional state descriptor, at the cost of solving a higher-dimensional control problem made tractable only by neural-network methods feasible up to roughly $m = 50$.

Model class and Brownian approximation

The general model admits alternative processing modes ($n > m$), discretionary routing, and controllable inputs; standard multiclass queueing networks are the special case $n$0. Primitives are independent integer-valued processes satisfying functional central limit theorems with first- and second-moment data $n$1 and $n$2. Under a nominal activity-rate vector $n$3 satisfying $n$4 and a heavy traffic condition requiring $n$5 to be moderate, the scaled queue-length process obeys

$n$6

with monotonicity constraints on $n$7 encoding capacity feasibility and nonbasic activity restrictions. Replacing $n$8 by Brownian motion with covariance $n$9 yields the initial BCP, whose admissible controls may have unbounded variation—hence the equivalence to a lower-dimensional EWF via state space collapse.

The modified BCP as drift-rate control

The key reformulation restricts controls to the form $p$0, where $p$1 is an adapted drift process bounded above by parameters $p$2 (argued to be of order $p$3 for fidelity to the original problem), and $p$4 is a boundary process enforcing reflection at each face $p$5 along directions given by columns of $p$6. Three constraints govern the choice of $p$7: capacity feasibility $p$8, nonnegative introduction rates for nonbasic activities, and complete-$p$9 structure of $\nabla V(\cdot)$0, which guarantees that $\nabla V(\cdot)$1 is a well-defined semimartingale reflected Brownian motion. A further design condition $\nabla V(\cdot)$2 eliminates boundary control costs whenever rejections are not part of the boundary behavior.

The interpretation of $\nabla V(\cdot)$3 is concrete and physically meaningful: when buffer $\nabla V(\cdot)$4 empties, activities serving that buffer are suspended, releasing fractions $\nabla V(\cdot)$5 of server capacity that are reallocated per the entries of $\nabla V(\cdot)$6. In all three examples the authors take $\nabla V(\cdot)$7, meaning one percent of released capacity is left idle—a device ensuring $\nabla V(\cdot)$8 is completely-$\nabla V(\cdot)$9 (in fact a $m$0-matrix, verified by exact rational arithmetic over all 255 principal minors in the eight-dimensional three-station case). Notably, $m$1 fails to be completely-$m$2 at $m$3 for two of the examples, so this slack is structural rather than cosmetic.

Solving the HJB equation for the modified BCP yields both $m$4 and $m$5, and the optimal Markov drift policy solves a linear program that decomposes by server. Translated back to the queueing network, the resulting rule is strikingly simple: each server computes indices $m$6 at the observed scaled state $m$7, then devotes full capacity to the available activity with maximal $m$8, idling only if all available indices are negative. For input servers, the same index comparison implements admission control and type-A routing. Two features distinguish this from threshold-type policies in the Bell–Williams lineage: no safety-stock parameters require tuning, and interpretation is immediate even where EWF solutions are opaque.

Numerical results

Three test problems are treated: the Harrison–Wein criss-cross network, the Pesic–Williams parallel-server system, and the Harrison BIGSTEP three-station network with dynamic routing and admission control. Where the underlying MDP is solvable (the first two, via truncated value iteration), the BCP policy is benchmarked against the exact optimum; for the three-station example, a cost-aware greedy variant of maximum pressure serves as the benchmark.

Network	MDP	Greedy	Singular control	BCP
Criss-cross	1681.7 ± 1.5	1789.4 ± 2.3	1690.3 ± 1.5	1686.6 ± 2.1
Pesic–Williams	2581.2 ± 2.2	3277.1 ± 3.9	—	2586.2 ± 3.0
Three-station	—	8844.8 ± 10.4	—	8117.4 ± 10.1

The BCP policy's cost exceeds the exact MDP optimum by roughly 0.3% on the criss-cross network and 0.2% on the Pesic–Williams system, while beating the greedy heuristic by about 6% and 21% respectively; on the three-station network it outperforms the greedy heuristic by approximately 9%. Switching-curve comparisons show the BCP policy closely tracks the optimal MDP switching boundaries in both solvable cases. These results support—but do not prove—the conjecture, grounded in the heavy-traffic diffusion literature, that the proposed policy is nearly optimal in the heavy traffic regime.

A secondary contribution worth noting: for the Pesic–Williams instance with $m$9, which Pesic and Williams explicitly could not solve ("We do not know how to solve the EWF in this case"), the authors compute the EWF solution by two independent methods (Kushner–Martins finite differences and their own singular-control method), obtaining policies and objective values agreeing within 0.15%, and confirming the predicted interior use of the nonbasic activity behind a curved free boundary. The same dual-method agreement (within 0.25%) holds for the three-station EWF.

Conjectures on convergence

Two formal conjectures frame the theoretical standing of the approach. First, letting $d$0 denote the modified BCP value with uniform bound $d$1 and $d$2 the initial BCP value, the authors argue informally that $d$3 as $d$4: with large bounds the controller can approximate instantaneous displacement in any feasible direction, effectively overpowering the pre-specified boundary policy. Second, with drift bounds scaled as order $d$5 in the $d$6-th prelimit problem, the optimal objective $d$7 should converge to $d$8 as $d$9. Neither conjecture is proved here; both are supported by heuristic argument and by the numerical evidence.

Limitations and open questions

Several limitations are conceded explicitly. Asymptotic optimality of the proposed policy is conjectural, not established; rigorous convergence results connecting the modified BCP to either the initial BCP or the prelimit network remain open. The choice of the matrix $d < m$0 and the bound parameters $d < m$1 involves analyst discretion, and although guidance is given (bounds of order $d < m$2, $d < m$3 slightly below 1), no optimization or sensitivity analysis of these tuning choices is provided. The nominal plan $d < m$4 is assumed known from a preliminary static analysis outside the scope of the method, and the approach presumes the gradient satisfies $d < m$5—an expected but unverified property enforced only heuristically during training via a penalty term. Finally, the numerical validation covers networks with $d < m$6; performance at the advertised scale of $d < m$7 is asserted on the basis of the underlying computational method rather than demonstrated for SPN control problems in this paper.

Conclusion

This paper supplies the missing implementation layer between high-dimensional Brownian control computations and deployable queueing policies. By retaining the full class-dimension state space and using bounded drift controls, it converts a neural-network solution of an HJB equation into a decentralized, threshold-free index policy whose simulated costs lie within a fraction of one percent of exact MDP optima on two benchmark networks and substantially outperform a strong greedy heuristic on a third. The open theoretical questions—proofs of the two convergence conjectures and asymptotic optimality of the induced policy—are clearly identified and constitute the natural next targets for this line of work.