Multi-Robot Multi-Queue Control via Exhaustive Assignment Actor-Critic Learning

Abstract: We study online task allocation for multi-robot, multi-queue systems with asymmetric stochastic arrivals and switching delays. We formulate the problem in discrete time: each location can host at most one robot per slot, servicing a task consumes one slot, switching between locations incurs a one-slot travel delay, and arrivals at locations are independent Bernoulli processes with heterogeneous rates. Building on our previous structural result that optimal policies are of exhaustive type, we formulate a discounted-cost Markov decision process and develop an exhaustive-assignment actor-critic policy architecture that enforces exhaustive service by construction and learns only the next-queue allocation for idle robots. Unlike the exhaustive-serve-longest (ESL) queue rule, whose optimality is known only under symmetry, the proposed policy adapts to asymmetry in arrival rates. Across different server-location ratios, loads, and asymmetric arrival profiles, the proposed policy consistently achieves lower discounted holding cost and smaller mean queue length than the ESL baseline, while remaining near-optimal on instances where an optimal benchmark is available. These results show that structure-aware actor-critic methods provide an effective approach for real-time multi-robot scheduling.

• The paper presents the EA-AC policy which integrates queueing theory with deep RL, achieving 5–12% cost reductions and 6–15% queue length improvements over the ESL baseline.
• It employs dual embedding modules for robot and queue features and a sequential masked assignment decoder, ensuring effective and scalable idle robot reassignment.
• Empirical results confirm nearly optimal performance on small systems and robust scalability with linear sample efficiency as the number of robots increases.

Exhaustive Assignment Actor-Critic for Asymmetric Multi-Robot Multi-Queue Control

Problem Setting and Motivation

The paper addresses the problem of real-time scheduling in multi-robot, multi-queue systems with stochastic, spatially distributed arrivals and explicit switching delays. Each location hosts a task queue with arrivals following heterogeneous Bernoulli processes, and robots serve tasks non-preemptively, incurring a deterministic one-step travel delay when switching locations. The objective is to minimize the expected discounted holding cost, formulated as an infinite-horizon discounted MDP. This setting generalizes classical multi-server, multi-queue scheduling by explicitly modeling non-negligible switching delays and asymmetric arrival rates, which commonly arise in modern autonomous warehousing, manufacturing, and field robotics scenarios.

Standard dynamic programming techniques are intractable for realistic system sizes due to combinatorial explosion. Prior structural results established that for homogeneous arrival rates, the exhaustive-serve-longest (ESL) policy is optimal, but this does not extend to the asymmetric case. Existing RL-based approaches for queueing and resource allocation, while effective in settings without switching delays or structural policy constraints, do not exploit system-specific policy regularities and suffer in sample efficiency and scalability.

Policy Design: Exhaustive Assignment Actor-Critic Architecture

The core contribution is a structure-aware RL architecture integrating domain knowledge from queueing control. The Exhaustive Assignment Actor-Critic (EA-AC) parameterization restricts the action space by encoding the domain-optimal “exhaustive” structure: robots continue to serve the current queue until it is depleted. Thus, learning focuses exclusively on reassignment of idle robots (robots that find their current queue depleted at a decision epoch), drastically shrinking the combinatorial control space.

The actor uses dual learned embedding modules: queue tokens encode local state features (normalized backlog, arrival intensity, occupancy), and robot tokens encode individual location context and busy/idle status. For idle robots, a sequential masked assignment decoder generates assignments to feasible locations through a compatibility function over robot and queue tokens. Strict feasibility is ensured by sequentially masking already occupied queues and reservations. The critic module is centralized, computing a value estimate via pooled embeddings and global congestion features without being tied to a particular assignment, serving primarily as a variance-reducing baseline for PPO optimization.

Figure 1: The EA-AC architecture incorporates explicit robot-queue interaction modeling with a policy network for assignment and a centralized value function for training.

Simulation Scenario Generation and Experimental Protocol

System instances are defined by $(M,N)$ (number of robots and queues), total offered load, and a vector of queue-specific arrival rates sampled using a Dirichlet model, then quantized and bounded to ensure both heterogeneity and feasible operation. The RL agent is trained per-instance, with reward signal given by negative holding cost. The baseline is the exhaustive-serve-longest (ESL) rule, known to be optimal only in the homogeneous case. For small instances, optimal policies are computed via dynamic programming for comparison.

Figure 2: Asymmetric scenario arrival rate distributions, demonstrating heterogeneity and system scale variations tailored for policy evaluation.

Empirical Results

Performance Relative to Baselines and Optimality

In all scenarios, the EA-AC policy dominates the ESL baseline in both discounted cost and mean queue length metrics. While gaps are minor for small symmetric or nearly-symmetric systems (e.g., S1: <1.5% improvement), for larger asymmetric cases, the EA-AC achieves discounted cost reductions between 5–12% and queue length reductions of 6–15%. Gains are most pronounced in system configurations featuring strong arrival-rate heterogeneity, confirming that naive heuristics such as ESL are highly suboptimal under these conditions.

On small instances where brute-force optimal solutions are computable (up to $N{=}4$), EA-AC converges to nearly optimal policies, with cost and queue metrics within the variance range of the optimal policy. On symmetric larger systems, EA-AC matches ESL, confirming that the architecture does not lose optimality when system symmetry reestablishes ESL optimality.

Scalability and Sample Efficiency

Convergence analysis shows that the number of training iterations required for stable policy recovery scales linearly with the number of robots, indicating favorable sample efficiency as system size increases. The same policy architecture generalizes across a range of robot-queue ratios and total system scales without problem-specific architectural adjustment or regularization.

Figure 3: Training convergence iterations as a function of number of robots; empirical results indicate approximately linear scaling with system size.

Further, mean queue length performance remains robust as the system grows, supporting scalability claims.

Figure 4: Comparison of mean queue length for EA-AC and ESL across diverse robot-queue parameterizations, illustrating consistent improvement by the proposed method.

Theoretical and Practical Implications

By explicitly encoding the exhaustive-service constraint at policy level and leveraging RL for learning idle assignment, the approach achieves statistically significant improvements in operational performance under realistic, asymmetric task arrival profiles. The methodology demonstrates that a hybrid of structural insights from queueing theory and powerful function approximation (deep RL) yields both sample efficiency and strong empirical results, outperforming unconstrained or fully generic RL policies where structure is ignored.

Theoretically, these results underscore the limits of heuristic prioritization rules (such as longest-queue assignment) in multi-agent stochastic control with explicit switching costs and heterogeneity. Practically, the approach offers a scalable, trainable module suitable for deployment in high-throughput, spatially distributed resource allocation domains, with direct applicability to modern robotics and logistics deployments.

Future Directions

The EA-AC approach opens several avenues for further work. First, developing analytical proofs of suboptimality gaps for ESL in generic asymmetric settings would augment the empirical validation. More generalizable actor-critic networks capable of zero-shot transfer across system scales and arrival-rate profiles are a natural next step for practical deployment. Extensions to time-varying arrival rates, weighted costs, service-time variability, or decentralized/multi-agent training regimes would further increase applicability. Finally, linking network structure to provable bounds on generalization and function approximation error remains an open challenge blending learning theory and control.

Conclusion

The EA-AC policy architecture provides an effective, scalable, and empirically near-optimal solution for online multi-robot multi-queue scheduling under realistic system constraints with switching delays and arrival-rate asymmetry. By integrating queueing-theoretic structure into deep RL policy design, it bridges the gap between practical controllability and tractability in large stochastic resource allocation networks. These results inform future research in both structural RL algorithm design and large-scale multi-agent system control.