SOAR: Real-Time Joint Optimization of Order Allocation and Robot Scheduling in Robotic Mobile Fulfillment Systems

Abstract: Robotic Mobile Fulfillment Systems (RMFS) rely on mobile robots for automated inventory transportation, coordinating order allocation and robot scheduling to enhance warehousing efficiency. However, optimizing RMFS is challenging due to strict real-time constraints and the strong coupling of multi-phase decisions. Existing methods either decompose the problem into isolated sub-tasks to guarantee responsiveness at the cost of global optimality, or rely on computationally expensive global optimization models that are unsuitable for dynamic industrial environments. To bridge this gap, we propose SOAR, a unified Deep Reinforcement Learning framework for real-time joint optimization. SOAR transforms order allocation and robot scheduling into a unified process by utilizing soft order allocations as observations. We formulate this as an Event-Driven Markov Decision Process, enabling the agent to perform simultaneous scheduling in response to asynchronous system events. Technically, we employ a Heterogeneous Graph Transformer to encode the warehouse state and integrate phased domain knowledge. Additionally, we incorporate a reward shaping strategy to address sparse feedback in long-horizon tasks. Extensive experiments on synthetic and real-world industrial datasets, in collaboration with Geekplus, demonstrate that SOAR reduces global makespan by 7.5\% and average order completion time by 15.4\% with sub-100ms latency. Furthermore, sim-to-real deployment confirms its practical viability and significant performance gains in production environments. The code is available at https://github.com/200815147/SOAR.

• The paper introduces a unified DRL framework that jointly optimizes order allocation and robot scheduling in RMFS using soft allocation and an event-driven MDP.
• It employs a heterogeneous graph transformer and p-norm reward shaping to achieve up to 18.7% makespan reduction and sub-100ms decision latency.
• Real-world deployment via a digital twin validates improved throughput and efficiency, outperforming traditional decoupled heuristics.

Real-Time Joint Optimization in Robotic Mobile Fulfillment: The SOAR Framework

Introduction and Motivation

Robotic Mobile Fulfillment Systems (RMFS) have become fundamental in large-scale, high-throughput warehouse automation. In RMFS, fleets of autonomous mobile robots transport inventory-laden shelves to stationary workstations, enabling efficient goods-to-person fulfillment. The joint optimization of order allocation (matching orders to shelves and workstations) and robot scheduling (assigning robots to pick, deliver, and return shelves) is a combinatorial problem with strict real-time requirements and strong inter-phase coupling.

Traditional approaches decompose the joint optimization into separate phases to ensure computational tractability but at the expense of global optimality. Conversely, monolithic optimization methods (e.g., MIP formulations) capture system-wide dependencies but are computationally prohibitive for dynamic, large-scale industrial use.

"SOAR: Real-Time Joint Optimization of Order Allocation and Robot Scheduling in Robotic Mobile Fulfillment Systems" (2605.03842) introduces a unified deep reinforcement learning (DRL) framework for this problem. SOAR fuses order allocation and robot scheduling in real time via a soft allocation mechanism and an event-driven Markov Decision Process (ED-MDP), and leverages heterogeneous graph neural architectures and reward shaping for robust high-dimensional reasoning and effective credit assignment.

Figure 1: Robot delivering a shelf—a key physical operation in RMFS showcasing the goods-to-person paradigm.

SOAR Architecture

The architecture of SOAR consists of a warehouse environment simulator, a Soft Order Allocation module, and an Event-Driven Robot Scheduling module. The system holistically models shelves, workstations, robots, and incoming orders as a heterogeneous entity graph evolving through asynchronous events. The primary innovations are as follows:

• Soft Order Allocation: Instead of allocating incoming orders immediately and irrevocably, SOAR computes, at each order arrival, a set of candidate shelves and workstations along with “heat” vectors reflecting their suitability for the order. This defers hard binding decisions until later, allowing the robot scheduling phase to exploit global state information and recent order flows.
• Event-Driven Markov Decision Process: Actions are triggered by discrete events (order arrival, robot idle, pick-up completion, delivery completion), yielding asynchronous, non-equidistant decision epochs—a closer reflection of the RMFS's operational reality.
• Warehouse State Encoding via Heterogeneous Graph Transformer (HGT): The warehouse is encoded as a graph with typed nodes (robots, storage locations, workstations, phase/event context), and the HGT aggregates information using role/relation-specific attention mechanisms. This captures both spatial-temporal dynamics and complex inter-entity relationships.

  Figure 2: The overall SOAR pipeline with event-driven triggers (purple arrows) and state/information flows in the event-driven MDP (orange arrows).

This joint paradigm enables the policy to propagate allocation uncertainty and exploit global context across all decision phases.

Optimization and Learning Strategies

Reward Shaping for Long-Horizon Credit Assignment

Direct makespan optimization is hindered by the sparse and delayed reward structure: makespan objective signals only arise at the end of each episode. SOAR employs $p$-norm reward shaping to create a dense surrogate reward signal. By tracking the $L_p$-norm of all robot active times at every step,

• Small $p$ encourages system-wide parallelism,
• Large $p$ approximates bottleneck minimization (makespan),
• Intermediate $p$ (empirically, $p=8$) balances training stability and optimization sharpness.

Time-aware PPO is used for policy optimization, where event delays induce variable time intervals and the discount factor is scaled accordingly. Generalized Advantage Estimation (GAE) is further adapted for event-driven transitions.

Phase-Specific Action Priors

The policy incorporates phase-specific priors directly into the action scoring logits:

• Pick-up: Prioritize shelves with higher “heat.”
• Delivery: Penalize workstations with low outstanding workload to balance queues.
• Return: Shortest-distance to available storage locations.

These priors, integrated as explicit biases, enhance exploration efficiency and initial convergence but are ultimately subsumed by the model’s deeper representations.

Empirical Validation

Experimental Setup and Metrics

SOAR's effectiveness is established through large-scale experiments on both real-world (Geekplus logs, 31 days, $40 \times 72$ warehouse, 198 robots, 861 shelves, 16 workstations) and synthetic ($100 \times 80$ warehouse) datasets at various scales (Small/Medium/Large). Performance is measured via:

• Makespan (maximum completion time),
• Average Order Completion Time,
• Computation Time per decision.

Results

SOAR demonstrates strong, consistent numerical superiority:

• Makespan reduction: up to 18.7% over best baseline in synthetic, 7.5% in real-world.
• Order completion time reduction: up to 33.9% (synthetic), 15.4% (real).
• Computation time: sub-100ms latency per decision, competitive with heuristics and orders-of-magnitude faster than MIP-based approaches.

Figure 3: Performance metrics on the Real-Large dataset, highlighting SOAR's improvement over all baselines.

Notably, traditional decoupled heuristics are fast but result in globally suboptimal utilization due to lack of inter-phase coordination. Recent joint optimization solvers based on rolling-horizon or static MIPs fail to adapt quickly to order/robot dynamics, yielding stale decisions.

Ablation studies confirm the critical contributions of soft allocation, heterogeneous relational modeling, and $p$-norm reward shaping. Setting $K$ (candidate set size for soft allocations) too low reverts to myopic local allocation; too high degrades signal with noise. Optimal $L_p$0, $L_p$1.

Sim-to-Real Deployment and Digital Twin Validation

SOAR's robustness is evidenced by sim-to-real deployment in production RMFS, using a high-fidelity digital twin as a closed-loop evaluation and A/B testing environment. The digital twin integrates all physical-level details (kinematics, collision models) and mirrors real-time order streams.

Figure 4: Digital Twin Platform with full system integration for high-fidelity sim-to-real transfer of SOAR policies.

Key industrial metrics (averaged over seven deployment days):

• Throughput: +3.85% over current best industrial heuristic.
• Shelf hit rate (item matches per shelf trip): +6%.
• Robot travel distance: −2.81% (improved item batching).
• Real-time latency: Maintained $L_p$2100ms per decision, including dynamic entity pruning for inference.

Theoretical and Practical Implications

SOAR advances the state-of-the-art in:

• Real-time, multi-stage joint optimization: It operationalizes a unified, event-driven DRL solution rather than static or decomposed strategies, achieving both responsiveness and global performance.
• Scalable DRL for high-dimensional, dynamic environments: HGT-based encodings demonstrate the importance of modeling heterogeneity in structured warehouse settings.
• Bridging sim-to-real: Domain adaptation via digital twins, robust credit assignment, and explicit latency management techniques enable practical deployment.

Theoretically, SOAR's event-driven MDP formulation generalizes conventional discrete-time MDPs and may be applicable in other high-frequency, stochastic, asynchronous logistics systems. The $L_p$3-norm reward shaping for multi-machine makespan minimization addresses a well-known challenge in DRL for long-horizon scheduling.

Future Directions

SOAR currently focuses on high-level decision-making (task allocation and scheduling). Open challenges for further research include:

• Tight integration with low-level multi-agent path-planning and collision avoidance,
• End-to-end learning frameworks that absorb both high-level scheduling and physical robot execution,
• Generalization to other combinatorial industrial applications with asynchronous, event-driven dynamics.

Conclusion

SOAR establishes a new paradigm for real-time joint order allocation and robot scheduling in RMFS by fusing soft allocation, event-driven decision processes, and scalable graph-based DRL. Its empirical superiority and demonstrated sim-to-real deployment potential underscore its utility for industrial warehouse automation, while its architectural choices provide a template for next-generation multi-agent scheduling systems in logistics and manufacturing (2605.03842).

Figure 5: Order Size Distribution in real datasets, exhibiting a long-tail characteristic—critical for soft allocation effectiveness and order batching.